System Evaluation

Measuring impact of legal help websites

At the LSC ITC conference 2023, the legal help website People’s Law School in British Columbia, Canada shared their strategy to measure what works on their website.

They were motivated by knowing ‘What works?” They want to know if the website is making a difference or not. Did they help people who were seeking guidance and assistance with their problems?

They did this by asking discrete, one-off questions throughout their website.

  • Is this helpful?
  • Will this help you resolve your issue?
  • Would you recommend this to others?

These questions usually had five star rating scales. In addition, there were free text boxes. The data was then saved into a database and presented as a data dashboard.

The website administrators could then see rates around empowerment, satisfaction, net promoter score, and suggestion and comments.

The data dashboard would be a driver of regular iterations and improvements.

Current Projects

The State of Eviction Prevention Efforts

Lessons Learned from the Eviction Prevention Learning Lab cohort

As eviction rates go back up following the court shutdowns and emergency moratoria during the pandemic, communities are struggling with the question: how can we prevent evictions?

How can we help people stay in their homes, avoid lawsuits to force them out of their houses or pay extra fees, and have a scarlet ‘E’ on their credit report for years to come? And how can we help mom-and-pop landlords who are trying to avoid a costly lawsuit and turning a unit over to find a new renter?

The past several years have seen a tide of new programs, policies, and technology initiatives all aiming to prevent evictions in different ways. What are all of these initiatives, and how can we expand them to help more communities across the US?

Our team at Stanford Legal Design Lab and the National League of Cities runs the Eviction Prevention Learning Lab (EPLL), which has established a strong, local base of knowledge about how to prevent evictions, and how to start up and scale successful new programs and policies.

The Eviction Prevention Learning Lab, a peer-to-peer network of local city leaders, has just concluded our cohort. This program engaged teams from 30 municipalities in 22 different states, totaling roughly 400 individual participants. The EPLL cohort operated from early 2021 through autumn 2022 and uncovered many promising practices around how evictions can be prevented, — as well as barriers that stand in the way of local leaders looking to move the needle on housing stability.

After working closely with the city leaders over the past 18 months, our organizing team at Stanford Legal Design Lab and NLC have synthesized the cohort’s progress, insights, and next steps in this essay. These insights & patterns from the 30 cities in the EPLL can help other regions’ government agencies, courts, nonprofits, and community leaders to use in their eviction prevention work.

Why have an eviction prevention cohort?

Communities across the country are struggling with a growing eviction crisis, in which tenants are at risk of lawsuits, forced moves, and other strategies to displace them from their rental homes. Evictions lead to housing instability and homelessness, as well as collateral consequences for people’s credit reports, education, employment, health, and finances.

So what are ways to prevent evictions? In the past, local leaders have had to struggle with this question on their own. Eviction laws and housing market dynamics are often highly local. Different cities and states have different rules, protections, policies, and market forces. This fragmented landscape makes it hard to address the crisis. Local leaders often have to take the lead in navigating their local stakeholder groups, legislation, court rules, and funding relationships to find solutions.

The EPLL helped local leaders know what programs and policies are possible, what’s been tried in other similar regions, and how they can deploy these new initiatives successfully. Throughout the 18 months, EPLL cohort members learned what is happening throughout the country to deal with the eviction crisis, through presentations, interactive meetings, design workshops, and technical assistance engagements. They were trained in new data and communication skills, they heard detailed case studies about how new services or policies roll out, and they built relationships with their counterparts in other regions. All of these webinars, meetings, and share-outs have led to a strong national network that can adapt and scale the most promising practices to address evictions. The EPLL participants have used the cohort to create new innovative solutions that work for their local dynamics.

What have the past 18 months of eviction prevention been like?

At the closing meetings of the EPLL, our organizing team asked city leaders about what they have accomplished since the start of 2021. Here are some of the big accomplishments they highlighted:

Emergency rental assistance distribution to keep people in their homes

The biggest accomplishment most leaders shared from the past two years was the huge volume of rental assistance used to prevent evictions. Significant amounts of emergency rental assistance were successfully distributed to tenants and landlords at risk.

For example, $304 million dollars were distributed to people in need in Tulsa, Oklahoma. In Fort Wayne, Indiana, it was $52 million. Local leaders created completely new emergency rental assistance programs during the pandemic and successfully used ERA money to keep people in their homes and resolve landlord-tenant disputes.

Court partnerships, rule changes, and programs.

Many EPLL leaders were able to bring city agency, nonprofit, and court leadership together to create new court-based interventions. These are key to preventing evictions by getting support, mediation, and financial assistance to tenants and landlords struggling through a conflict. Some of the innovations included new diversion programs to turn lawsuits into settlement agreements, changes to courtroom setups that connect litigants with more holistic services and guidance, outreach strategies to encourage participation, and co-located services to help litigants get financial and housing navigation.

Some courts even changed their rules about how eviction cases proceed, with more judicial management, mediation sessions, integrations with ERA programs, and support for people without lawyers. These court pilots are now being formalized and spread through initiatives like the National Center for State Courts’ Eviction Diversion Initiative.

Innovative, equity-focused outreach efforts

The EPLL members, with guidance and assistance from our NLC and Stanford team, created new ways to connect tenants and landlords with eviction prevention help. This included new on-the-ground community networks and in-person assistance. City leaders and nonprofits in places like Louisville and Chattanooga have brought services to people in their neighborhoods, with door-knocking campaigns, pop-up offices, and on-site training. They built these community-based outreach efforts in order to reach people with strong word of mouth, particularly in areas where there was a high risk of eviction.

In addition to community-based outreach, the EPLL members also developed strong digital and visual outreach as well. They created (and shared) new style sheets, social media campaigns, landlord-tenant websites, text message lines, and intake partnerships with other social service groups. These outreach materials made use of many different modes and leveraged the power of the Internet to reach different audiences. These strategic outreach efforts helped increase awareness about new programs like ERA and eviction diversion programs and encouraged equitable participation in them.

The establishment of strategic coalitions and networks

EPLL cohort members, including city teams in Pittsburgh, Louisville, and Milwaukee, highlighted this theme as a key priority to deploying any new individual initiative. To really prevent evictions, city leaders discovered that it was necessary to build a local strategy group that spanned different agencies, nonprofits, courts, legal aid, and community groups.

EPLL city leaders coached each other on how to build an effective local network. One key step was aligning courts, city government, legal aid, nonprofits, and foundations around the notion of ‘eviction prevention’ as a common goal, which could help them understand how they could collaborate in new ways. Often this meant having regular meetings, having staff members at each others’ locations, finding ways to share data, and teaching each other about their own institutions. This was particularly helpful in setting up new ERA programs and then also winding them down. These ongoing strategic networks can help the leaders and communities respond to the opportunities and hurdles that emerge.

The importance of national networks

The EPLL helped city leaders to find inspiration, guidance, data, and other evidence that could support their own local work. At the EPLL sessions and in their peer-to-peer meetings, these cross-jurisdiction relationships bloomed in order to help city leaders get evidence, blueprints, strategies, and outreach templates that could help replicate and scale successful efforts. These national networks also provide an undercurrent of support to professionals who often are stressed and over-committed. Because they could find peers dealing with similar challenges in other locations, the EPLL relationships ended up being a place for leaders to recharge.

What’s next for eviction prevention?

Now regional leaders are turning to the next phase of eviction prevention work. When we asked them what their current goals and challenges are, here is the agenda they laid out for the next several years:

Moving to longer-term planning for sustainable innovation

Beyond local programs and short-term mandates, city leaders want to use what they learned during the short-term planning of the emergency to design permanent eviction programs, that also fit into the bigger picture at the state and national levels.

Shift to mediation and problem resolution

Now that there is less money to support ERA programs, local coalitions are exploring how to revise their prevention programs — especially court diversion programs. Many are focusing on community mediators, preventative outreach in high-risk neighborhoods, and mandatory pre-filing diversion programs to help landlords and tenants resolve problems without going through an eviction.

State government involvement

City leaders have also discovered the limits of what they can do at the local level to stop evictions. More are now exploring what they might be able to do in partnership with governors, state legislatures, and other statewide agencies. In some cases, this might be about statewide programs.

In other places, it’s legislative changes or funding. For example, some cities are trying to deal with statewide laws that prohibit eviction record sealing. Other leaders are cautious about proposing reforms to statewide landlord-tenant laws, for fear that once the law is open for changes, there might end up being changes that inhibit eviction prevention.

Strengthening local coalitions

Now that some agencies are moving away from eviction prevention as a core priority, leaders are working on how to keep their pandemic-era coalitions intact and stronger. Many cities are exploring more service co-location, referral networks, and funding plans. Some coalitions are considering breaking into different themes or seeking out common funding in order to keep the coalition strong.

Special attention to court involvement & judge leadership

A particular concern is court and judge involvement in these coalitions. Many cities have had difficulty getting local judges and court executives to collaborate on eviction diversion.

In response, they are creating data to show court leaders the impact of current court practices on the community’s housing and homelessness situation. They are also spotlighting how courts can better administer the justice system when they partner with other agencies on eviction diversion programs, resource networks, record sealing, and other efforts.

Judges can play a role in setting better court rules, scheduling, and courtroom setup that allow more participation in court, and more services to get a better outcome for both tenants and landlords.

Infrastructure around common models and data efforts

Groups are also thinking about regional and national coordination efforts that can spread best practices more efficiently and establish more knowledge about the eviction system. Groups are working on national data standardization and sharing efforts, like those led by New America. Some groups, including a coalition in Philadelphia and states in the Justice For All program, are considering how to make standardized, well-designed legal documents like court forms, eviction summons, settlement agreements, and leases, that improve prevention efforts. Some city teams are considering investing in broader public awareness of housing rights and equity, so that more people see how these interconnected programs around rental assistance, court diversion, mediation, affordable housing, and homelessness prevention all work together.

Working Towards Sustainable Eviction Prevention

The past two years have been a flurry of eviction prevention innovation, with new initiatives like emergency rental assistance, court diversion efforts, and community justice navigators starting up in regions across the US. But will these fledgling new initiatives take root, to become permanent, growing eviction prevention efforts?

Much of that will depend on what happens in the next year. Local leaders, as well as our NLC and Stanford Legal Design Lab, will continue to work on innovative, coordinated new efforts to help prevent evictions and all of the harm they cause to families and communities.

More Resources

Explore pilots and data about eviction prevention at our site Eviction Innovation.

Find practical guides in the Emergency Rental Assistance Toolkit for more effective outreach, program design, partnerships, and more.

Read more about the Eviction Prevention Learning Lab here.


How the justice system can learn from unemployment insurance

The federal government is newly focused on Customer Experience (CX). That has meant that their teams are creating better websites, tools, and forms that can help people get their business with agencies done more easily.

The Department of Labor has a team working on modernizing Unemployment Insurance. See their examples & guidance about how to use better design and tech to improve people’s experience with the government.

These strategies and examples can be used by courts, legal aid groups, and other justice system actors to improve their processes, as well. These practices include:

  • Plain Language
  • Mobile-Friendliness
  • Automated Quality Testing Tools
  • Keeping the Backend and improving the frontend
  • Redesigning Notices and Application Forms

The website has specific guidance on improving the usability of each of these initiatives.

For example, here are quality feedback tools to run:
Current Projects

Why doesn’t every new justice reform policy come with a community navigator program?

Margaret Hagan, Dec 1, 2022

Reflecting back on all of our legal design work over the past 10 years, I’ve spotted a common pattern happening:

  1. Momentum builds around a justice reform policy that intends to address a fundamental problem in people’s ability to access the justice system. This could be around giving people facing traffic tickets the ability to apply for a financial waiver, so it doesn’t set off a financial crisis for them. Or it could be a new program to get tenants & landlords emergency rental assistance, to stop an eviction. Or it could be a policy to allow for people to expunge criminal records for marijuana-related offenses.
  2. Hard, detailed legal work goes into creating & passing this new policy. State, local, or federal bodies make this proposal into an official ‘thing’. The policy is announced! Change is happening! There are press releases, announcements, and maybe a website to tell the public about this new policy that should help them.
  3. Groups are put in charge of making this policy into programs, that the public can find and use. This is usually where our Legal Design Lab has been invited in to help. How do we make the public aware that this policy exists and can help them? How do we translate the policy from a legal text into real-life programs, processes, and impact?
  4. Programs struggle to get effective public outreach, uptake & usage. It turns out to be quite hard to reach people who might benefit from a new traffic ticket, eviction, or expungement policy. It’s also hard to design a new process that’s user-friendly, low-burden, and engaging enough for a person to stick with it until the end. In some cases, programs also need to adjust when there’s fraud activity, too. We often work with these nascent, struggling programs to redesign how they’re doing outreach, services, and evaluations.

Having observed many of these policy → program journeys — and how hard it is to get public awareness, uptake, and engagement with new policies — there is one solution that should be baked into the earlier phases, as groups are proposing policies:

Community Justice Navigators are an essential part to making justice reforms impactful.

If every new ‘justice reform’ policy came with a mandate & funding for community justice navigators, this could be a big step towards making the policy have the impact it is intended to.

Community justice navigators can be in the neighborhoods and community groups, to alert the public about this new policy & programming. They can help people understand what the opportunity is, and why it’s trustworthy and valuable. Navigators can:

  • Awareness: Get more people to know about the policy & programs: They can hold ‘official’ public outreach events, and also do informal, peer-to-peer outreach. Ideally, community justice navigators will have ties to local groups, and can have a much stronger reach to build awareness.
  • Spot-and-Refer: Help people know what’s available to them. Navigators can talk to people who are trying to figure out what their life problem is in legal terms — and what services might help them. They can help legal (or financial, or social) issue-spot, and refer people to expert groups or programs that can help them.
  • Procedural Coach: Be there with a person throughout their justice journey. Navigators can play a companion role, helping a person stay engaged, attending hearings, dealing with legal notices, addressing hiccups, and keeping a person motivated and supported. A justice journey can be draining and intimidating — and a navigator can play a support role that lawyers and others often can’t.
  • Task Assistance: Help the person get their forms and papers right. People often struggle with filling in forms, gathering evidence, or prepping for their day in court. A navigator can help them do these tasks, to know what’s expected & how to get it done.
  • In-Court Navigation: Guide the person at hearings, meetings, and trials. When a person comes to a court or government agency on their own, they may not know how to behave, what to say, how to get their story across, or how to interact with officials. A navigator can help a person know what to do & support them through the stress of a court day.

A Community Justice Navigator doesn’t have to play all of these roles — but even playing a few of them can help increase access to justice. A navigator can increase a person’s legal capability — improving their knowledge, strategy, and confidence.

A Navigator can also help policy-makers and service providers, by getting their new rules and programs crucial public exposure, doing the ‘customer service’ work of helping people navigate this new system, and encouraging the person to participate in a timely, well-prepared way.

Any new justice reform policy or program is likely going to struggle with public awareness, uptake, and navigation. So why doesn’t every new policy effort come baked in with a Community Justice Navigator program?

For all the effort that goes into getting new legislation or programs launched, the real goal is to impact people’s lives for the better. We want people to know about this new policy, be able to use it, and have improved life outcomes because of it. To make an impact, we need to invest in & train more Community Justice Navigators, to reach the public & help them use new programs that are meant to help them.

Read more about Justice Navigator programs for examples:


Benchmark principles for A2J Tech

The State Courts in Washington established a set of guiding technology principles for the development and release of new technologies in the justice system. They are benchmarks that teams can use to evaluate their new idea, pilot, or even existing program with.

  1. Equitable access to the system, with technology enhancing (and not diminishing) opportunities to participate in the justice system. This tech-enabled participation should be usable, accountable, efficient, and transparent.
  2. Openness, Privacy and Safety. Tech should protect confidential information, and regulate access to it. It should ensure that information is controlled, and not introduced into the public domain without appropriate controls.
  3. Accountability and fairness: so that groups are monitoring how the technology is performing, and that it is not exacerbating harms or inequities.
  4. Maximizing public awareness and use: so that people know it exists, can discover and use it, and this is equitable among demographic groups.
  5. Usability: so that people can easily navigate it, go through its steps, and complete the user journey with support, low-cost and low-burden.
  6. Accessibility: so that people with different skills and physical facilities are able to use the tool equally
  7. Plain Language: so that it’s understandable to people who are not legal professionals, or who have limited reading skills.
  8. Cultural Responsiveness: so that it treats people, even with different value systems and cultural norms, with dignity and respect — and doesn’t inadvertently harm or exclude them.
  9. Enhancing Human Interaction: so that the system is still humane, responsive, and empathetic — and the ‘human touch’ is not lost even with more technology involvement
  10. Language Access: so that people with limited English Proficiency are still able to access and use it.

See the full list of principles here
Current Projects

An International R&D Community for Better Justice Innovations

Margaret Hagan, Aug 16, 2022

Building a network that’s researching, designing, and evaluating what works to increase access to justice

Earlier this summer I was lucky to spend a Saturday in conversation with Professor Monica Palmirani & her research group CIRSFID at the University of Bologna.

It was a pleasant afternoon talking about projects & giving feedback on early-stage initiatives. Especially because of COVID, it has been hard to build up international (or even inter-university) connections about what our different legal labs and research groups are working on.

It turns out that there was a surprising amount of overlap between our work at Stanford Legal Design Lab and the CIRSFID group in Bologna.

Both of our groups are focused on an overarching theme:

How do we empower more people to participate in the government systems meant to benefit and protect them? In particular, how do we increase people’s ability to understand, use, and participate in the legal system to resolve housing & finance problems?

For us at Stanford Legal Design Lab, we have been very focused on people’s participation in the United States’ state court and legal aid systems. For people facing life problems around housing, debt, employment, and other situations — can we increase their participation in this legal help system to resolve the problem and move forward with their life?

Parallels in A2J innovation from California to Italy

Professor Palmirani’s group just launched a project with very similar themes, in another jurisdiction’s context. They have launched a 5-year research project on digital participation in government. This focus aligns with ours at the Legal Design Lab — and it turns out our past projects have as well.

Building data standards for justice systems

The Bologna group has already established infrastructure around legislation and NLP, through their data standards of Akoma Ntoso & Legal Document ML to capture legislation in standardized notation.

This parallels our Stanford’s group focus on data standards:

It turns out that in our different regional contexts, both of our research groups have made data standards infrastructure projects a central part of our work. These standards project lay the ground for more groups to develop new solutions & intelligent applications.

That said, there’s another — less technical, more design — theme that we also have both aligned on in Bologna & California.

Legal Design in the Courts for User Empowerment

Professor Palmirani’s group in Bologna has turned to legal design as another theme for innovation and research. Their group has partnered with judges in Sicily, to do user research, create new designs, and evaluate them. The challenge they’re tackling through legal design is this:

  • There is new legislation regionally, that creates opportunities for people struggling with debt.
  • Many people do not necessarily know about this new policy and related programs, that could help them pay their debts down & get relief.
  • How can the courts increase awareness of this new set of rights and services? And how can they empower more people to navigate this system strategically, to find relief?
  • Can this intervention, if it helps people Engage -> Motivate -> Act on the law, ultimately get people to better outcomes? The hypothetical chain goes as follows: People use the law to relieve their debt, because the guide has helped them find and use the legal system→ They are cleared of significant financial stress → They get stabilized with their house, car, family, and job → They have better physical health, mental health, and quality of life — as does their family.
The judges’ challenge to the R&D group: can you make the law clear for citizens?

This project in Bologna/Siciliy parallels our Stanford projects in which we’ve been working with courts and city governments on rental relief, eviction diversion initiatives, and moratoria. Like in the Italian example, there also have been recent roll-outs of new laws, assistance programs, and court policies. We have been working on the outreach and service design, with similar goals to our European counterparts.

The Bologna group has created several design interventions that they are now going to evaluate in the Sicily context. They have created a visual explainer book, along with online videos, that onboard people to their options to deal with debt. The goals of the explainers are as follows:

  • Can they engage more members of the public to know about this law, and motivate them to take action if it’s applicable to them?
  • Can they help a person understand the 3 legal pathways they have to deal with a debt — and to make a strategic choice among the 3?
  • Can they make the most commonly applicable pathway very approachable and clear to the person? With the use of process maps, human stories, and staged information — can they keep a person engaged, with a sense of control, and a sense of dignity?

The Design Patterns & Strategies for Legal Empowerment

Their group of lawyers and designers have created visual guides that they will test.

My notes on the design patterns in use by the Bologna group, with their debt guides

The guides were created with many visual and messaging strategies, meant to increase engagement, motivation, and action:

  • Process Map and a Choice Matrix. These 2 visual patterns are commonly used in legal design, to show how a person can navigate a system, and how to weigh strategic choices against each other. The Bologna group made a color-coded procedural map that shows the steps as a timeline with simplified steps. Then a person can go from this high-level overview into detail. They can also see in tables about exactly when the law applies and when it doesn’t.
  • Representations of many kinds of local people, to show the potential users of the law that it is for them. The designers showed distinct ‘Personas’ as visual characters in the guide. This includes a variety of gender, ages, abilities, family situations, and backgrounds.
  • Messages around dignity, normalcy, and support. The language and visuals reinforce the message that debt and money problems are normal. That a person shouldn’t be ashamed or feel guilty about them.
  • Framing the legal system as a service rather than a punishment. There is also deliberate messaging that debt is not criminal. This is not a trial, and it doesn’t have to be adversarial or punitive. Rather, the focus is on collaborative service and problem-solving, with the legal system as a service rather than a punishment.
  • Visual metaphors around weight & uplift. At the same time as emphasizing ‘this is normal’, the guide also acknowledges how burdensome the situation is for a person. It literally shows people being weighed down with their debt problems. And it uses the hot-air balloon as a visual metaphor throughout the guide to communicate uplift. As a person navigates the path towards relief, the balloon lifts up.
  • Emphasizing & clarifying choices. The guide shows the 3 legal procedures a person could use for debt relief as 3 different hot air balloons. They’re colored differently, based on difficulty & accessibility. It can let a person visualize which one would be best for them to use, to take a justice journey. At the same time as the guide presents the 3 pathways, it doesn’t present them as equal. They use indicators to guide people to the most commonly relevant and the least burdensome path.
  • Remixing Cultural Symbols. The legal design team discovered that symbols from a popular card game in Sicily could be adapted for the guide. They could use these familiar, local symbols — swords, vases, spears, coins — to represent different system parts and people. They chose these symbols specifically because playing cards is a familiar experience for making strategic, careful choices. Just like in the legal system, you have to be thinking about your goals and steps. Also, these symbols help frame the lawyer as the ‘protector’ rather than the enemy. This can help let the person see the lawyer as someone who can help them, rather than someone to fear.
  • Building Familiarity with the Different Professionals. One problem to deal with is that all of the different ‘system people’ can all seem the same to a new user of a system. The guide makes a point to differentiate who all these different support and decision-maker experts are: the lawyer, the accountant, the trustee, the judge. These should lessen people’s fear of these ‘experts’, and also help them ask the right questions to the right people.
  • Messages that set expectations. Though much of the guide is emphasizing that the legal system is approachable and engaging — the designers have also realized that too much of this language might make people think that navigating the system will be easy, or they will have complete support from the professionals to do it. This might then lead to frustration & burnout when they find that they will still have to do many tasks, make hard choices, and come to high-stakes meetings, negotiations, and hearings. Even if there are people to help them, the process will still be hard. It will take effort and dedication. The guide emphasizes this along with the other, more encouraging messages above.

This Justice Innovation Work Can Be Networked

This effort in Sicily has been amazing to watch. The Bologna group will have more to report about the effectiveness of their Engage -> Motivate -> Act theory of change.

In the meantime, there is much to learn for other folks (myself included) who are working on the research & design of new justice system interventions, meant for legal empowerment. Like the Bologna group, my Lab has also been busy working on new guides, websites, outreach fliers, and maps to help institutions better engage & empower the public. When I talked with Professor Palmirani and her team, I realized that we had been working in a parallel way but had much to learn from each other.

In particular, there is an opportunity for the law school labs, court innovation groups, legal tech entrepreneurs, and others focused on ‘justice innovation’ to be creating some standard ways of working. These would not be hard-and-fast data standards. Rather it would be a collection of best practices, strategies, and instruments that we can share.

In particular, we could be spreading & scaling our ways of working, including:

  • The visual design patterns, messages, and strategies that work. Each cultural context may need different symbols, content, and messages. But as we learn what engages people to take on hard tasks around their money, housing, family, and employment problems — it’s worth it to share these strategies explicitly with each other. That may mean sharing the actual visuals being used, or the strategies underlying them. We might replicate these successful models in different jurisdictions.
  • The design process to use to create effective interventions. This would include how to get the right mix of stakeholders, involve them in the creation and testing of the interventions, and ensure that there is equitable, lively participation. How do we check for blind spots, missing participants,
  • The metrics & evaluation instruments we use to measure outcomes. How do we know whether a new legal help design works? And what are the protocols we can use to measure this performance — in the lab or in the field? We need more discussion and collaboration across our R&D groups about evaluation. Most groups don’t have the luxury to run gold-standard randomized control trials because of how burdensome they are to establish, and partners who are unwilling to randomize who gets the new intervention. So we must create other experimental and quasi-experimental research designs that can still help us understand if a justice intervention is worthwhile — -or if it should be changed or discontinued.

I’m excited to build more of these ‘design standards’ for our group in Stanford and to spread and edit them while we collaborate with groups like Professor Palmirani’s group in Italy, and other labs and research teams around the world.

Please feel free to share your own best practices on developing & evaluating ‘justice interventions’ in the comments!

Current Projects

How do you design a user-friendly court form?

Margaret Hagan, Jun 29, 2022

(Even if we should be moving away from forms altogether…)

I am thinking a lot about forms these days! At the Stanford Filing Fairness Project, our team is working on a near future in which PDF forms no longer are key to people’s access to the court system. In that vision, it’s all about moving from populating PDFs to gathering data fields and sending them straight from a user-friendly website to the court’s case management system. No PDF forms needed!

But we still live in a justice system full of PDF forms. Even when software exists to help people fill in forms, through logic trees and document assembly programs, still it’s all going through PDFs.

So… if we still have to live in a world of PDF forms, let’s make sure that this world is as accessible & empowering to as many people as possible. We still need to care about the design of PDF forms.

The Basics of Good Court Form Design

This morning I had the privilege to kick off the National Center for State Courts summertime webinar series: Forms Camp. Over 750 attendees showed up to learn how to design better court forms. Usually, I think of court form design as a super-specialized, design geek kind of topic. But it turns out that a lot of people are unhappy with their current form design, and want to make them better.

So what should you to design better forms? I had 3 main areas for work:

  1. Shift your form team’s point of view, from the institution-centered perspective to a people-centered perspective.
  2. Follow key visual design principles — as well as good court form design principles that I proposed, based on the Legal Design Lab’s work.
  3. Get started on a design process — in which multi-stakeholder groups, including community members, visual designers, lawyers, court staff, and others are reviewing current forms, sketching out new drafts, testing these prototypes, and finalizing the most successful ones.

1. Shifting from Institution-Centered Forms to People-Centered Forms

To really design a better court form, we need to shift from an institutional perspective to a user’s perspective. In particular, we really need to think of the end-user, the litigant who has a PDF or a digital pdf in front of them, is trying to deal with some kind of civil justice problem, and now has to use this form document to get their stories and choices into the court.

This user point of view is different than the institutional one — which is usually a ‘forms committee’ point of view. There the POV is about trying to put as many of the legal requirements in a PDF document, covering all possible use cases, and getting this document passed through a series of judicial committees and approval structures as quickly as possible.

Once we start looking at the court form from the user’s POV, we can see how important the humble form is to big lofty issues of access to justice, procedural fairness, and social policy. We also can set some better metrics & indicators by which to measure whether a form is successful or not.

Moving to People-Centered Measures of Forms

At the highest level, a successful form will increase a person’s access to justice. It will increase the litigant’s procedural justice & substantive justice.

It will make a person feel that the court process is transparent, fair, and trustworthy. (That’s procedural justice).

It will let them share their info, perspective, & experiences so that the judge can make a fair decision, applying the law to the situation in a just way. (That’s the substantive justice).

So how do we get to a form that increases access to justice?

At the intermediate level, a successful form will be useful, usable, and engaging to litigants. A litigant should be able to find & use iteasily. They would find it useful,to get their key info across & prepare for court. And it would engagethem, so they want to spend time & resources filling it in. These are all core principles of user-centered design in all realms — applied here to the specific court form scenario.

If we are going to define a court form’s key performance indicators (or, KPIs for acronym fans), we should make a user-centered KPI list. We put litigants first — since the justice system is meant to serve the public. We also include court staff and leadership as secondary users as well, since we know that they are users of forms, too. To make a good form, we need to track indicators for both litigants & staff.

Key Form Success Indicators

Here is how courts can track a form’s performance & aim for improvements:

People-Centered Indicators

  • Increased Engagement of possible litigants — so more people are actually finding the form, filling it out, and submitting it successfully
  • Higher sense of confidence & strategic ability in how they fill it out — that they feel they can ‘get it right’
  • Lower Administrative Burden with less cost, time, and resources needed to fill it out
  • Increased Procedural Justice, with people feeling that the court is respectful & fair
  • Improved Substantive Justice, with people able to achieve resolution with the conflict, and the law correctly applied to their scenario.

Staff-Centered Indicators

  • Reduce the number of calls and visits to court staff
  • Reduce the amount of mistakes clerks must deal with
  • Get correct & comprehensive info, in a clear & accessible presentation, to clerks and judges

Setting up these metrics and indicators is key to getting the first part of form design right. We need to know what we’re aiming to achieve. Once we have user-centered metrics and indicators in place, we make sure all our design and legal work are oriented in the right direction.

2. Follow Visual & Court Form Design Principles

So, then how do we make it more likely that litigants will engage with these forms, use them easily & correctly, and find value in filling them out? That’s where design principles come in.

I presented two sets of guiding principles for court form designers to follow. The first principles are general graphic or visual design principles, that have been honed over the years in various (non-legal) fields. The second principles are court-form-specific principles that I distilled from the past decade of working on legal design at Stanford.

General Visual Design Rules

How do you lay out a paper-based (or digital PDF) form in the most user-friendly way? Here is a short run down of useful visual design principles:

  1. Support the User Journey: by making it easy for a person to navigate the document, and guiding them through a ‘story’.
  2. Have a Clear, Strategic Hierarchy: prioritize the info & tasks, with clear navigation. Define a clear strategy that has a hierarchy — not all info is the same.
  3. Provide a Standard, Clear Layout: all content should be aligned, with a single, coherent visual language. Use a grid and possibly column design, to create distinct zones for the person to explore.
  4. Give Generous White Space: let the eye breathe, make people calm, and give space to the most important info.
  5. Deploy Selective Pops: use limited amounts of special fonts or colors to draw attention to high-priority info.

Key design principles that court form designers can use

If you want to get more into very detailed communication design choices (like font, color, and accessibility) there are more resources on general visual design rules here:

Court Form Design: the key components of a form

These general principles are great, but they can be hard to translate to the specific world of court forms. We need to dive into the specifics of the court form’s components — and people’s experiences of a form.

Court forms usually contain this shortlist of components. These are the ‘materials’ we can use in our design. We can also track — does the form have all these key things, does it get the hierarchy right, and does it treat them consistently?

  • Credentials that signal the form is official for a jurisdiction
  • Title and purpose, what the form is & what it’s about
  • Instruction info for the user, so they know what to do overall, and then in each section
  • Questions and tasks, asking the user for key information and posing choices to them
  • Entry fields for the user to put information into
  • Insider fields for the court staff to mark notes, enter info
  • Links to more help and associated documents
  • Next Steps & Deadlines of how to get this form into the court correctly, and what to expect after

Court Form Design: How litigants experience a form

People don’t magically see a form, get excited, and spend the next two hours completing it before they walk it over to the court clerk’s office. Their experience is usually much more complicated and stressful.

We need to see the court form not as a static document, but as an experience. Understanding people’s experience of a form helps us figure out specific principles that can help increase engagement, usability, and usefulness.

From our user testing and observation sessions at the Legal Design Lab, we’ve found trends in how litigants use forms — especially when they are self-represented.

  • They scan them over quickly. What should I expect? How long is this going to take? Can I even do this? Am I up for this challenge?
  • They do work in bursts. They may have 1 burst when they first engage. But they’ll likely pause & disengage after they get tired. Hopefully, they’ll re-engage with later bursts!
  • They might get distracted or discouraged when they can’t understand or feel overwhelmed. Can you help make sure this doesn’t lead to disengagement? Instead, support them there.
  • They want to be ‘normal’ and strategic. The form should amplify their sense of legal capability — not make them feel lonely or dumb.

This is where we come back to the big goal of access to justice. Court forms could be an essential gateway to participating in the justice system. But they are hard! They ask for complex, high-stakes information. And people are often in a high-stress situation, afraid of getting things wrong, and with a lot of other things to do in their life. So if a court form is badly designed, it can disengage users, and shut down their access to justice.

Key Court Form Design Principles

So with that background, I propose a handful of key principles for good court form design.

  1. Have a clear navigation scheme & glance-able structure. Can a person ‘get’ the key zones of info & tasks within a 1-minute glance-over?
  2. Be calm & readable. Don’t overcrowd with info and tasks. Does it make the person feel more capable or less? Does it have distinct zones of work?
  3. Support stressed-out users. Does it have off-ramps to info, examples, & assistance — especially near the hardest tasks?
  4. Be easy to fill in. Have consistent, ample space to fill info in. Make it clear through spacing, boxes, and lines about what is ‘right’ and ‘normal’ to put in.
  5. Don’t prioritize ‘insider’ tasks & info over the users’ tasks. Are user tasks in high-priority places? Are insider tasks put in discrete, low-importance places?

3. Running a Design Process on Your Court Forms

Great! Now we have user-centered metrics to aim for, we’re aiming at more engagement, usability, and usefulness, and we know some principles about how to do this. Let’s think even more concretely. How do we get courts, judicial councils, and forms committees to start improving the design of their court forms? How do we get more people-centered shifts, and more use of key design principles?

Kick off a Redesign by doing a User-Centered Design Review

If you’re inside the court, you can start a design process. I often like to do it in a series of workshops — beginning with design review sessions, involving many different stakeholders and especially court users.

You can lay out existing court forms on a board or a digital whiteboard, and then start going through what works or doesn’t. You can mark it up with post-its, or pen, or even cut it up and reorganize it.

We did this using a Miro board at today’s Form Camp session. We start a design review by beginning with a specific user’s POV. Today, we began with this user scenario:

A tenant in California has just been sued for eviction.

They’ve searched online & found a pdf of this form at the California courts’ webpage.

How can we make it usable, useful, and engaging to this litigant?

So we situated our forms discussion in the user’s journey. They’ve come from a Google search, then to a court website, then to a digital pdf. Let’s imagine we are this tenant — and look at the pdf to see what we could improve.

Laying out the user’s journey on a whiteboard, for a forms design review

I screenshot the 5 pages of the California unlawful detainer answer response. This is the document that a tenant would have to fill out, in order to defend themselves against an eviction lawsuit. I laid out the 5 pages on the digital whiteboard, and then asked my team to answer a series of questions about how well the form lived up to the key design principles. I took notes with post-its as they gave feedback.

Doing the design review, with post-its capturing stakeholders’ comments about what could be improved

I prepared a series of specific questions, to get the stakeholders to critique the current design based on the court form and visual design principles.

User Journey Review of a court form

If a person sat down with this form, could they navigate it?

  • At the start, does the document establish a clear, trustworthy relationship between the court & the user?
  • Do the tasks follow a logical, clear order? Are they grouped into clear ‘zones’ that make sense to a user? Are the zones labeled with clear Section Headings? Are there instructions/guidance about sections?
  • Are there ‘Offramp’ links for info & more help in the right place — in a context where the person might be looking for them?
  • At the end of the form, does it make the person confident about the next steps to take?

Area 1: Hierarchy Review of a court form

Do you have a clear hierarchy of information & tasks?

Your strategic ranking: Have you reviewed everything you want to convey & get from the user? What is most important? What is the middle? What is the least important for the user?

Giving the right treatment: For the most important things, have you put them:

  • In the prime locations
  • With bigger fonts
  • With ‘pop’ of color, font, or bold

Strong headings: Have you put strong, clear headings for the distinct ‘zones’?

Area 2: Standardized Layout Review of your court form

Do you have standard ways you’re laying out groups of info?

Are things consistently in the same place on the page, in the same font/sizing/alignment:

  • Instructions
  • Questions
  • Entry boxes
  • Offramps to More info & links
  • Court/clerk ‘insider task boxes’

Are they grouped in clear & distinct zones for a user to navigate?

  • Different tasks/topics are clearly delineated from each other
  • So a person can ‘take a break’ in-between zones

Area 3: Legibility & Capability-building review

Are the zones, text, and layout all accessible & enhancing legal capability — instead of overwhelming the person?

  • Is there plain language or legal jargon, code references, etc?
  • Is the text presented in a large enough font, with enough line spacing, for it to be easy to read?
  • Does the text go all the way across the page (too long)?
  • Are the different zones of tasks cluttered together on page? Or is there breathing, white space at margins and between zones?

These different sets of questions can make sure the stakeholders are doing a critique based on the user’s POV and design principles derived from past best practices. If you don’t use these questions, it’s easy to start shifting back to an institutional-centered POV.

After design review workshops, the team can then move onto other kinds of workshops to generate prototypes, test them, and decide on final versions of forms.

What kinds of form design workshops can you run in your court?

Better Court Form Design will be a continual process

It would be great if I could publish a template for all court forms to follow. It might have hard rules about maximum page length, the ideal font size, and the perfect grid layout. Perhaps these patterns will emerge in the coming years, as we do more testing of form designs with a wide range of users.

But for now, the best way to make user-friendly court forms is to have a continual process of reviewing current forms with community members, drawing on established design principles, testing any new prototype with litigants and court staff, and using our people-centered indicators as the metrics to determine if a court form should be released to the public.

In any court form design, there will be lots of trade-offs. How many details, claims, defenses, and rights can we let a person know about — before we exhaust them to the point of disengagement? How much generous white space can we give a person, until the form becomes so lengthy that it turns them off?

That’s why a multi-stakeholder process, still rooted in the litigant’s (and staff members’) points of view is necessary to decide what works best in making these trade-offs. Ideally, more courts will be engaging in this design work, gathering even more principles and best practices, and leading towards more standardization of the best form designs.

That said — perhaps this whole conversation will be moot soon, if we move towards more interactive form-filling websites, with no more PDFs at all. We’ll then move on to the best design of software interfaces. There will be slightly different principles in play — but our metrics and goals should still be the same.

We want more people to be able to protect their rights in court and do so with a sense of confidence, capability, and dignity. Good court forms are fundamental to good access to justice. As more courts embrace user-centered design, we can start moving to this better future.

Current Projects

Making Good Legal Design the Law

Margaret Hagan, Jan 13, 2022

We have been talking and working on the importance of the justice system’s user experience — as have many others in other public interest sectors.

We have been talking and working on the importance of the justice system’s user experience — as have many others in other public interest sectors.

These civic services should be usable, useful, and engaging. It should be easy for all different types of people to find their options for courts, benefits, taxes, government documents, and other key government tasks. The processes should be as simple as possible, with the fewest time and money burdens, and with an emphasis on a supportive, dignified experience.

Now we are seeing these goals become legal mandates.

The White House issued an Executive Order in December 2021 setting out explicit rules about good design of federal government services. It sets out human-centered design as an official obligation for federal agencies.

Here’s a quote of their policy intent with this order:

“ The Federal Government must design and deliver services in a manner that people of all abilities can navigate. We must use technology to modernize Government and implement services that are simple to use, accessible, equitable, protective, transparent, and responsive for all people of the United States.”

It also uses the UX-oriented concept of administrative burdens to set out metrics. The order requires that these agencies be measuring the quality of the design and user experience.

“It is the policy of the United States that, in a Government of the people, by the people, and for the people, improving service delivery and customer experience should be fundamental priorities. The Government’s performance must be measured empirically and by on-the-ground results for the people of the United States, especially for their experiences with services delivered.”

The Order sets out staffing, agenda-setting, and other required actions to make these policy goals real. In some in cases, the Order calls out specific websites, services, and forms/applications that need good civic design overhauls. It mandates that government staff centralize and simplify processes, and then create more user-friendly platforms and tools so that people can find and use them.

One other possible legal design policy is happening in New York City. (Hat tip to a colleague at the Racial Justice office of the city for alerting me to this).

The NYC Racial Justice Commission has some Ballot Proposals for 2022 — one of which (Proposal 2) would establish a Racial Equity Office in the City. Part of this office would include a mandate for good civic design, language access design, and overall human-centered service design.

Proposed policies to add to the NYC Charter, that would make good civic design law

These proposals are linking racial equity with good design. It’s baking in key design principles of empathetic, non-paternalistic respect for community members — and a commitment to inclusion through better government design. If passed, this Citywide Access Design project would be a leader in ensuring that diverse community members can access the government services they need.

Do you have other examples of government agencies recognizing the importance of good legal and civic design? And how it was established as a law, rule, or policy? Please share!

Class Blog Uncategorized

Human-Centered Computable Contracts

Margaret Hagan, Dec 16, 2021

In Winter Quarter, our Lab Team is working with the Stanford Law CodeX team, to co-teach a new class at Stanford Law School. It is a hands-on, project-based class, about how to make insurance contracts more accessible, intelligent, and human centered. It builds on our past classes on user-friendly privacy policies, and contract design labs.

The class is 808L Human Centered Computable Contracts (

How do we make insurance contracts that consumers can understand — and than harness all the potential of tech & choice engines?

We will be working with regulators of the insurance industry & consumer protection advocates. The focus is ‘How can we develop more user-accessible models of insurance contracts, so that more people can be strategic & capable in this complex system?’

Students will be:

  • Interviewing consumers about how they typically interact with insurance contracts (like in healthcare, housing, and otherwise). Also understand what consequences have happened for people, depending on how they’ve shopped for and used insurance.
  • Map out key user breakdowns, incentives and interests, and behavioral heuristics that could support people’s strategic decision making
  • Research what new tech models are possible, to make complex choices & documents more user-friendly
  • Test prototypes for computable insurance contracts, to see if people can use them — what works & doesn’t
  • Propose key principles, interfaces, and metrics for what makes a human-centered insurance contract

The class work will feed into the large initiatives in the law school, including CodeX’s Insurance Initiative. It will also be part of global research on proactive law, computable contracts, and new kinds of disclosure design.

Current Projects

Can a legal aid group send proactive texts to people who have been sued?

More legal aid and court groups are excited to use text message strategies to reach people facing lawsuits. Texting may help encourage participation, increase uptake of free services, and empower people to avoid defaults & other bad legal outcomes.

But the problem many of them face is how to legally & ethically reach out to members of the public through text. If they don’t already have a relationship with a person, can they initiate a texting relationship with them?

For example, if a legal aid group sees that a person has been sued for eviction in a local county court, can they proactively send them a text message (especially if they have the person’s phone number from the court docket info or from a phone-lookup service)?

The Legal Services Corporation recently published an Advisory Opinion that walks through guidance to this question, to show legal aid groups how they can do this proactive text message outreach on a topic like eviction services — and still be in compliance with the law. They also have a Program Letter that explains how legal aid groups can do proactive texting without violating federal anti-spam laws.

Advisory Opinion on legal aid proactive texting

See this 2020 Advisory opinion on proactive text message outreach from the LSC to legal aid groups.

Program Letter on legal aid proactive texting

Also see this 2022 Program Letter from the LSC that walks through explicitly how proactive text message out reach can be done, in compliance with federal laws. Our Lab had written to the LSC to ask this question, because we had many legal aid colleagues who were thinking about adopting a text message reminder strategy & we wanted to make sure that the community knew how to do this in compliance with laws against spam or other violations.