Metrics of Justice Innovation

This page is under construction. We present an initial collection of resources, but there is much more to come.

There are many innovative projects happening to improve access to justice. What would indicate that a project is successful? What can we measure, to show an improvement in access to justice?

Ideally, the justice community can agree on indicators that are well-designed. Good indicators (borrowing from Liz Curran & Andrew Crockett’s article) are:

  • Relevant to the challenge at hand — improving the accessilbity & performance of the justice system for people;
  • Useful and measurable, so we can see if progress is being made at resolving the challenge;
  • Achievable, so that the expectations are realistic;
  • Practical, so that groups can use them even if they do not have extensive funding or highly-trained research groups;
  • Within the Practitioner’s Control, so that the groups we are studying could possibly have an impact on this indicator.

Measurements often are custom-designed for a project, but there can be some more general tools and principles that can be used across projects.

Metrics to measure access to justice

Substantive Justice outcomes

When trying to resolve their legal problem, does a person get a fair and just outcome according to the law? Are they able to assert their rights and make full use of the law to protect their interests?

Comparing case outcomes among litigants, to see if people of different demographics, service-types, or other factors are receiving comparable outcomes

Making of claims — are litigants who have defenses, claims, or counterclaims, able to assert these? Or are there claims that are not made, to the person’s detriment?

Legal Capability & Empowerment outcomes

To measure legal capability and empowerment, look to a person’s

  • Strategy: Ability to correctly and efficiently navigate legal procedure
  • Knowledge: Improvements to their legal knowledge, so that they know the law better and can apply it to scenarios
  • Readiness: Confidence that they can advocate for themselves and others in the legal system

Procedural Justice outcomes

While a person goes through the procedures of the legal system, do they have a dignified, transparent, efficient, and fair experience? Do they understand what is happening, are they able to navigate the process, are they able to manage the hurdles, and complete all the tasks while still feeling in control?

Measurement of procedural justice may include looking at a person’s:

  • Trust of the legal system
  • Sense that the legal procedure was fair
  • Sense that it was transparent, and a sense of ‘perceived control’ within the system
  • Feeling of dignity and respect while in the legal system
  • Sense that the legal system was usable and user-friendly

Administrative Burdens

Time and Money Costs of the Administrative Burdens: This concern, which is closely interrelated with usability, transparency, and dignity, is a growing concept in other areas of policy-making. How much burden does a given procedure (of filling in forms, collecting evidence, mailing and serving documents, attending meetings, and other tasks) put on a person? How costly are these procedural/administrative burdens to them?

Policymakers have established models for how to assess time and money costs of a given administrative/government procedure that a citizen has to go through. How burdensome is it, and how do we quantify that in terms of monetary and time cost per person?

For example, this European policymaker group has a Standard Cost Model for citizens, for use by agencies to determine the true costs of a given procedure for the people who will have to use it.

A snapshot of how this policymaking group determines the Administrative Burden of a given procedure.

Life & Social Outcomes

After a person has gone through their legal problem and/or the legal system, what is their financial, family, housing, and health situation?

Do they have negative experiences around poverty, displacement, depression, debt, sickness, or other disruption?

Anti-Poverty Indicators

Longer-term indicators might also include those focused on the prevention of poverty.

Anti-poverty effects can also be measured through SROI — the Social Return on Investment, a model developed by the Robin Hood Foundation to estimate the Anti-Poverty impact of a given intervention. See their formula here, at their Metrics page.

Open Door Legal Service’s description of the SROI model

Prevention of problems

Are justice problems prevented, so that a person doesn’t need to go through the legal system to get to resolution?

  • Prevention of Lawsuits: are potential problems and disputes resolved before it is brought to a court filing? 
  • Is there deterrence of harmful and bad behavior, because societal actors (like debt collectors or landlords) are aware that people know their rights and are able to use the legal system effectively?

One way to measure this is the Robin Hood Foundation’s model for Deterrence.

Issue-Specific Outcomes

What are the indicators that someone’s housing, family, money, or other justiciable issues are improved? Justice innovations ideally should be improving people’s life outcomes. Instruments that measure these issue-specific outcomes can show if a legal service is making someone’s life better.

Housing Stability outcomes

Housing, around landlord-tenant problems or homeownership, is a major area of civil justice. These metrics can indicate if someone’s housing situation has been improved — by a legal service or another intervention.

Stable Housing: does a person have a stable home for themselves and their family?

Is an eviction or disruptive displacement prevented, even after an unlawful detainer lawsuit or notice to quit is filed?

California State Bar Housing Outcomes

These indicators come from California, where the State Bar has defined expected outcomes that housing-oriented legal services can achieve. Instruments can measure whether this outcome occurred, as an indicator of a service’s success.

HO1.  Prevented loss of current housing. Kept client in a home; prevented eviction (eviction defense) or prevented foreclosure, including represented and/or advocated for client during the notice stage, i.e., before filing of unlawful detainer.

HO2.  Negotiated or facilitated move out to provide “soft landing”. Includes negotiated settlements, asserted defenses and procedural rights, and delayed foreclosure to help client avoid homelessness and/or transition to replacement housing. Also includes represented and/or advocated for the client during the notice stage, i.e., before filing of unlawful detainer. Results achieved may include time to move out (with or without payment of rent during that time); waiver of rent due; return of security deposit; money provided to move out; relocation payments; and kept eviction from being entered into the public record.

HO3.  Obtained or preserved access to housing. Includes obtained or maintained eligibility for all types of public, subsidized, deed restricted or other affordable housing units or housing vouchers.

HO4.  Prevented, ended or obtained relief from unfair or illegal behavior, or otherwise enforced rights or obtained remedies related to housing. Includes enforced fair housing laws, local law, regulations and rent ordinances through informal advocacy, formal administrative proceeding or filing affirmative lawsuits; includes prevented illegal rent increase, improper calculation of tenant’s income to determine rent and utility payment amounts, improper determination of household size and appropriate size unit etc. Also includes prevented unlawful discrimination; ended lockout; ensured return of security deposits and/or access to personal property; enforced DV survivor’s right to break a lease.

HO5.  Enforced rights to safe and habitable housing. Includes enforced housing, health, and safety codes, prevented utility shut offs, obtained repairs, or improved conditions.

HO6.  Obtained, preserved, enforced rights of a landlord over a tenant. Includes assisted vulnerable landlords (low-income, elderly) with evicting a tenant or enforcing a lease provision.

HO7.  Obtained relief from foreclosure or property scam. Includes delayed the impact of foreclosure to help client avoid homelessness and/or transition to replacement housing; includes obtained relief from rent skimming

HO8.  Resolved property title dispute. Includes quieted title, achieved partition, constructive trusts, or specific performance; resolved slander of title, adverse possession, boundary line or prescriptive easement issues.

HO9.  Obtained other housing benefit. Includes obtained access to transitional housing or shelter. Please use this category ONLY if your matter does not match another defined category.

Collection of Housing Stability Questionnaires

University of San Francisco California’s Social Interventions Research & Evaluation Network has a collection of survey questions about housing insecurity. This page collects questions that others can reuse to make sense of clients’ housing situations & improvements in their housing.

Some examples from the SIREN page:

Kaiser Permanente’s Your Current Life Situation Survey

Which of the following best describes your current living situation? (Select ONE only)

  • Live alone in my own home (house, apartment, condo, trailer, etc.); may have a pet
  • Live in a household with other people
  • Live in a residential facility where meals and household help are routinely provided by paid staff (or could be if requested)
  • Live in a facility such as a nursing home which provides meals and 24-hour nursing care
  • Temporarily staying with a relative or friend
  • Temporarily staying in a shelter or homeless
  • Other

Do you have any concerns about your current living situation, like housing conditions, safety, and costs?

  • Yes
    • If YES:
      • Condition of housing
      • Lack of more permanent housing
      • Ability to pay for housing or utilities
      • Feeling safe
  • No
  • Other

Accountable Health Communities Health-Related Social Needs Screening Tool

What is your living situation today?

  • I have a steady place to live
  • I have a place to live today, but I am worried about losing it in the future
  • I do not have a steady place to live (I am temporarily staying with others, in a hotel, in a shelter, living outside on the street, on a beach, in a car, abandoned building, bus or train station, or in a park)

Family Stability Outcomes

What are indicators that a person’s family situation is better off after receiving a legal service?