The Access to Justice team at the National Center for State Courts has a new User Testing Toolkit out. It can help courts and their partners get user feedback on key papers, services, and tools, like:
Court Forms: are they understandable and actionable?
Self-Help Materials: can litigants find and engage with them effectively?
Court Websites: are they discoverable, accessible, and useful?
Efiling Systems: are they easy to use, and to get right the first time?
Signage and Wayfinding: can people easily find their way around in-person and digital court spaces, with dignity?
Accessibility: are the courts physical and digital platforms sufficiently easy to use for all different kinds of people?
The toolkit has background guidance on user testing, strategies for planning testing sessions, and example materials to use in planning, recruitment, facilitation, and analysis.
For our upcoming AI+Access to Justice Summit and our AI for Legal Help class, our team has made a new design workbook to guide people through scoping a new AI pilot.
We encourage others to use and explore this AI Design Workbook to help think through:
Use Cases and Workflows
Specific Legal Tasks that AI could do (or should not do)
User Personas, and how they might need or worry about AI — or how they might be affected by it
Data plans for training AI and for deploying it
Risks, laws, ethics brainstorming about what could go wrong or what regulators might require, and mitigation/prevention plans to proactively deal with these concerns
Quality and Efficiency Benchmarks to aim for with a new intervention (and how to compare the tech with the human service)
Support needed to go into the next phases, of tech prototyping and pilot deployment
Responsible AI development should be going through these 3 careful stages — design and policy research, tech prototyping and benchmark evaluation, and piloting in a controlled, careful way. We hope this workbook can be useful to groups who want to get started on this journey!
Our team is excited to announce the new, 2024-25 version of our ongoing class, AI for Legal Help. This school year, we’re moving from background user and expert research towards AI R&D and pilot development.
Can AI increase access to justice, by helping people resolve their legal problems in more accessible, equitable, and effective ways? What are the risks that AI poses for people seeking legal guidance, that technical and policy guardrails should mitigate?
In this course, students will design and develop new demonstration AI projects and pilot plans, combining human-centered design, tech & data work, and law & policy knowledge.
Students will work on interdisciplinary teams, each partnered with frontline legal aid and court groups interested in using AI to improve their public services. Student teams will help their partners scope specific AI projects, spot and mitigate risks, train a model, test its performance, and think through a plan to safely pilot the AI.
By the end of the class, students and their partners will co-design new tech pilots to help people dealing with legal problems like evictions, reentry from the criminal justice system, debt collection, and more.
Students will get experience in human-centered AI development, and critical thinking about if and how technology projects can be used in helping the public with a high-stakes legal problem. Along with their AI pilot, teams will establish important guidelines to ensure that new AI projects are centered on the needs of people, and developed with a careful eye towards ethical and legal principles.
Is your team working on a legal innovation project, using a human-centered design approach? Then you are likely focused on different kinds of ‘users’, ‘stakeholders’, or ‘audience members’ as you plan out your innovation.
Our Legal Design Lab team has a free Canva template to make your own user personas easily.
This Canva template gives blank & example user persona templates that your team can fill in with your interviewees’ and stakeholders’ details. We recommend making multiple personas, and then possibly also making a schematic comparing different kinds of users.
The template document includes example images drawn by Margaret Hagan, that you can use in making your persona.
Have fun & let us know how your user persona creation goes!
Last week, Margaret Hagan traveled to Houston Texas for the National Center for State Court convening of Eviction Diversion Initiative facilitators. She ran a half day workshop on how to use human centered design to improve the program design, paperwork, and Service delivery of eviction diversion help at housing courts around the country.
This Design Workshop built on the several years of design work that the Legal Design Lab has done with courts and legal aid groups across the country, to improve how people are helped when facing eviction and their life outcomes.
Workshop participants, including lawyers, social workers, and court staff who work on running new eviction diversion programs in courts across the country, were able to go through the following sequence:
learning the basics of design mindsets, including focused on your users point of view and creating new experiments to see what can work better
choosing an example user to focus on, creating a Persona to summarize the person’s situation, needs, goals, and preferences
detailing that person’s current user journey through the housing and eviction system, and if they get too good or bad outcomes around housing, money, credit report, and other factors
zooming in on a particular touchpoint on this user journey, where a new intervention could improve the person’s experiences and outcomes
brainstorming many different ways that this problem/opportunity touchpoint could be improved, including with new paperwork, new Service delivery models, new space designs, new cultural or rule shifts, or new technology tools. Participants were shown an array of possible Innovation projects, which they could build on top of
Choosing a handful of the brainstormed ideas, to then bring home to share with colleagues and to try out in short pilots
It was wonderful to work with leaders from across the country, especially those who are so creative, empathetic, and ready to try out new ideas to make the court system work better for normal people.
Downtown HoustonIdentifying User PersonasMapping out User JourneysSpotting OpportunitiesBrainstorming Ideas
Some of the ideas included:
new paperwork that’s more supportive, clear & encouraging
space redesigns in hallways and courtrooms, to make it more human, breathable, polite, and dignified
technology tools that offer coaching and check-ins
data connections to improve efficiencies, and more!
See the presentation slides for the eviction diversion design workshop.
This past week, I had the privilege of attending the State of Privacy event in Rome, with policy, technical, and research leaders from Italy and Europe.
I was at a table focused on the intersection of Legal Design, AI platforms, and privacy protections.
Our multidisciplinary group spent several hours getting concrete: what are the scenarios and user needs around privacy & AI platforms? What are the main concerns and design challenges?
We then moved towards an initial brainstorm. What ideas for interventions, infrastructure, or processes could help move AI platforms towards greater privacy protections — and avoid privacy problems that have arisen in similar technology platform advancements in the recent past? What could we learn from privacy challenges, solutions, and failures that came with the rise of websites on the open Internet, the advancement of search engines, and the popular use of social media platforms?
Our group circled around some promising, exciting ideas for cross-Atlantic collaboration. Here is a short recap of them.
Learning from the User Burdens of Privacy Pop-ups & Cookie Banners
Can we avoid putting so many burdens on the user, like with cookie banners and privacy pop-ups on every website? We can learn from the current crop of privacy protections, which warn European visitors when they open any new website and require them to read, choose, and click through pop-up menus about cookies, privacy, and more. What are ways that we can lower these user burdens and privacy burn-out interfaces?
Smart AI privacy warnings, woven into interactions
Can the AI be smart enough to respond with warnings when people are crossing into a high-risk area? Perhaps instead of generalized warnings about privacy implications — a conversational AI agent can let a person know when they are sharing data/asking for information that has a higher risk of harm. This might be when a person asks a question about their health, finances, personal security, divorce/custody, domestic violence, or another topic that could have damaging consequences to them if others (their family members, financial institutions, law enforcement, insurance companies, or other third parties) found out. The AI could be programmed to be privacy-protective, to easily let a person choose at the moment about whether to take the risk of sharing this sensitive data, to help a person understand the risks in this specific domain, and to help the person delete or manage their privacy for this particular interaction.
Choosing the Right Moment for Privacy Warnings & Choices
Can warnings and choices around privacy come during the ‘right moment’? Perhaps it’s not best to warn people before they sign up for a service, or even right when they are logging on. This is typically when people are most hungry for AI interaction & information. They don’t want to be distracted. Rather, can the warning, choices, and settings come during the interactions — or after it? A user is likely to have ‘buyer’s remorse’ with AI platforms: did I overshare? Who can see what I just shared? Could someone find out what I talked about with the AI? How can privacy terms & controls be easily accessible right when people need it, usually during these “clean up” moments?
Conducting More Varied User Research about AI & Protections
We need more user research in different cultures and demographics about how people use AI, relate to it, and critique it (or do not). To figure out how to develop privacy protections, warning/disclosure designs, and other techno-policy solutions, first we need a deeper understanding of various AI users, their needs and preferences, and their willingness to engage with different kinds of protections.
Building an International Network Working on AI & Privacy Protections
Could we have anchor universities, with strong computer science, policy, and law departments, that host workshops and training on the ethical development of AI platforms? These could help bring future technology leaders into cross-disciplinary contact with people from policy and law, to learn about social good matters like privacy. These cross-disciplinary groups could also help policy & law experts learn how to integrate their principles and research into more technical form, like by developing labeled datasets and model benchmarks.—Are you interested in ensuring there is privacy built into AI platforms? Are you working on user, technical, or policy research on what the privacy scenarios, needs, risks, and solutions might be on AI platforms? Please be in touch!Thank you to Dr. Monica Palmirani for leading the Legal Design group at the State of Privacy event, at the lovely Museo Nazionale Etrusco di Villa Giulia in Rome.
The federal government is newly focused on Customer Experience (CX). That has meant that their teams are creating better websites, tools, and forms that can help people get their business with agencies done more easily.
The Department of Labor has a team working on modernizing Unemployment Insurance. See their examples & guidance about how to use better design and tech to improve people’s experience with the government.
These strategies and examples can be used by courts, legal aid groups, and other justice system actors to improve their processes, as well. These practices include:
Entering home stretch of the Autumn quarter, we spent today’s class first synthesizing our findings and working on our final pitch to the California Judicial Council and then selecting one of our prototypes for further development.
To start the the synthesis process, we grabbed a whiteboard and divided it into two halves — with one side dedicated to answering “What we heard or saw?” and the other dedicated to answering “What do we do in response?” Starting with the former question, we started jotting down quotes and experiences we had catalogued over the past few weeks from our interviews and observations, before clustering them around common topics. This exercise yielded two incredibly salient themes that we hope to address with our revised prototype:
Time: People fear the courthouse, because it takes an inordinate amount of time and as a result deprives of economic and educational opportunity that they would be accumulating, had they not spent hours upon hours and days upon days within the courthouse. One woman we interviewed exclaimed that, “divorce right now is almost a full-time job”; while another lamented that the amount of time she had to spend in court affected her kids’ academics, as they had to accompany her so that she could have the proper assistance necessary to fill out the English-language forms.
The process of getting proper help seems to take too much time because the self-help desk is understaffed and because court users produce a large number of errors while filling out their paperwork. Many of the people have interviewed over the past several weeks have mentioned that they often spend several hours waiting to be helped, only to be told that they made a mistake in their documents and are then sent to back the of line to seek guidance.
As a result, we witness a vicious cycle. The self-help desk is constantly creating its own backlog of requests, ultimately increasing stress and time allotted per case — for both the clerks and the court users. As a result of this feedback, one of our primary goals is to reduce the amount of time that a user has to spend in order to fill and submit their proper paperwork. This will help users have a more pleasurable and accessible court experience, while reducing the stress upon the self-help clerks.
Language Barriers Are Multifaceted: One thing we did not realize until we began user testing was how multifaceted of a problem language barriers actually are. When presenting our “Redesigned-Form” prototype to non-native speakers last week, we established a situation where we asked our interviewees to file for divorce. On the second page of the prototype, we asked our court users to declare whether they wanted a “Divorce”, “Legal Separation”, or “Nullity” from a “Marriage” or “Domestic Partnership”. While it was clear to the user that they were on a page associated with divorce, they were unsure as to what the differences were between a “divorce”, “legal separation”, and a “nullity”. As a native English speaker these terms seem foreign, as they are rooted in precise legal terminology; so one apparent aspect of “language access” is to provide court users with simple language that unpacks these precise terms. But the problem with language access extends far behind legal terminology and words in different languages. There are often significant cultural barriers as well. When interviewing a technologically-savvy Uyghur woman, we saw her even tried opening Google Translate on her phone, writing the phrase, and having the service produce a Mandarin version of the text. The problem was, however, that the concept did not exist in her culture; so even though she had the translated phrase, the concept did not register. This highlights the fact that language access does not simply include English-barriers, but also cultural ones. We must overcome both in order to provide true access to court systems.
With this in mind, we shifted to the other half of the board, answering “What do we do in response?” Here are a few of the ideas from that brainstorm:
Split the current forms into manageable chunks so that we do not overwhelm court users and narrow context of any page down to a singular topic so that it become easier for a non-native speaker to identify the goal of the page, even if they struggle to understand the bulk of it.
Provide native-language instructions and definitions that unpack legal ease in laymen’s terms and pay attention to cultural differences, in their explanations of legal terms.
Add legal advice forums like r/legal-advice into the court website; and provide a platform for non-native speakers to voice their experiences to others within their communities. We heard from many younger court users, that they looked online to blogs in order to understand the experience they were about to undertake, as a user of the court. These blogs reassured them and provided guidance, when they were most confused. It would be cool to provide this type of support on the court website and extend it to non-native speakers.
Moving forward, we are going to further pursue our “Redesigned-Form” prototype, diving deeper into the Divorce Experience to provide a more nuanced prototype experience.
Last Friday was the final class in the Stanford Law School/d.school class Prototyping Access to Justice. Kursat Ozenc and I were teaching the course as a practical, service design effort.
The big question guiding the work: if hundreds of thousands of Californians go to the courts to deal with their divorce, child custody, debt, and housing problems — how can we make the courts work for them, on their own terms? We know that growing numbers of people are trying to use the courts without a lawyer, but that the courts have been designed for lawyers — with complex procedure and intimidating jargon is so complicated that only lawyers can really figure it out.
Students were given initial design briefs that we had crafted from our earlier research into California Courts’ Self Help Centers last year. In the first version of this class, we followed litigants through their court journeys and interviewed professionals to identify key opportunities and breakdown points.
This quarter’s classes aimed to use this groundwork to jump more quickly into prototyping and testing. Each of the design teams worked on site at San Mateo County or Santa Clara County courthouses, and at the Stanford d.school labs — going through 3 cycles of scoping out a concept design, making a prototype of it, and testing it with many different stakeholders.
We ended up with seven proposals for the courts to pilot. Two concerned how to remake the court building and design of physical space. Two were new modes of guides, to present better ways to guide litigants through complicated tasks. One was about better form completion. One was about new modes for court feedback. And one was about better preparing court users before they come to court for the first time.
The teams made videos, maps, and presentations to capture their proposals, and we present them here for you to review. We ask you for your feedback now — because we are vetting these seven proposals to decide which to continue working on and possibly pilot with the courts.
1. Redesign of the Court Building: Visual Lines + Signs for Empowered Wayfinding
Team Chuka Ryori were tasked with helping people just as they arrived at the court the first time. How could we make people feel more supported, less confused and intimidated, and more capable of getting through the process efficiently?
Their proposal is to launch a coordinated, color-coded, pictogram-based wayfinding system in the court building. There should be color paths on the floor for the most common user destinations, with pictograms and a palette that supports finding the right place.
They did guerrilla-design work, by “decorating” the actual court with new lines, signs and pictograms to test how users reacted. The results were overwhelmingly positive. Our next steps are to refine the color palette and pictograms, and then work with the court to implement the new lines and signs.
2. Redesign of the Court Building: Respectful, Transparent Line Waiting
Team Golden Design Warrior was focused on the next moment in the user’s journey, when a person found the Self Help Center, but now must deal with the long and confusing wait to get services. After several different ideas to change the layout of the space, the team moved to focus on how to set up lines that gave users greater transparency and more comfort while waiting to be served.
The team identified that people were rushing to wait in a confusing line. They were stressed out, and in turn stressing out the staff who felt as if they had to barricade themselves in against a huge amount of people who wanted things from them. The goal of the system is to give people a clear ticket that would give them an explicit place in line, and would let them relax, sit down, and see when they could expect to be served.
The first pilot is just with laminated cards and a person distributing them near the entrance. Then it can be scaled to an automated ticketing service.
This prototype has tested remarkably well with both litigants and professionals, reducing both stakeholders’ stress and giving them more of a sense of control. With the simple intake during the sign-up, the professionals can better prep for the clients’ cases. They also get insulated from the pressure of a huge group of people hovering around their doors.
The joy of this design is how a simple service intervention can have a huge experiential payoff — making the experience of visiting court or working there be less anxious, confusing, and stressful.
3. Visual Book Guide to Following a Legal Process
Instead of worksheets and forms, or instructions told out loud before a person leaves the Center, how do we convey instructions and guidance to them? How do we make it easier for them to follow the procedure, so they stay on track and get it all completed correctly and on time?
Team Jiffy Justice proposes a visual booklet, that gives people a step-by-step map of what the process will look like, what to do, and how exactly to finish the steps. It’s about envisioning, modeling, and taking legal actions out of abstract text language, and into clear, grounded situations.
The team made a map that can be printed as a poster, a handout, or part of the book. It gives the systems-level view of the case. People liked this as an orientation material, but still wanted more detail about exactly what each of these steps entails. A high-level view helps give a person the mental model of the system, but they want to dig into more specific instructions and strategies.
The team made the booklet to enhance the guide, to go from the map to the detailed instructions.
They built it specifically so it could be easily printed on common paper sizes by the Self Help Center. It incorporates the map, but then with details of the forms, the filing info, and common flags and warnings.
The next, scaled-up version of this would be a digital version (most likely on mobile) that has the step-by-step guidance and the map for the person to follow along as they go through the process.
4. Text-based coaching through complicated process
Team Exit took this same challenge — how to help people through complicated procedures that they often fail at? Their proposal is more tech-centered, harnessing the power of the mobile phone. They created a prototype of the RemindMe Text system, in which litigants would get coaching reminders, customized due dates, and clear blasts of instructions about what to do to serve process (a particularly thorny part of a process, that people often screw up).
The team embraced the principle of staging information and providing it at the right moment and context. Rather than give huge worksheets with general information all at once, segment it into specific messages and customize it with the user’s own information.
This program could later incorporate other kinds of messages, beyond reminders — including the maps and visuals that Team Jiffy Justice had in their booklets, or the wayfinding and prep materials that other teams proposed.
The great part of this proposal is that the text message channel, opened up between the courts and the litigants, can allow for a diversity of services to be provided in the future. As more technology is developed for court services, they can be integrated into this same channel.
5. Prep People Before Court with Warnings and Key Info
Even before people come to court, how do we make sure they come prepared to make the most of the day — and not waste it? Especially if it takes several hours to even get to speak to someone at court, how do we make sure people come with the paper, translators, and knowledge enough to get their tasks accomplished?
PrepMe is an idea to do better outreach around this Prep information, via websites, mobile apps, and other court materials. It should be in multiple languages, and show very prominently the most common prep information people don’t know: about translators, child care, and timings.
This information can be presented also in court correspondence, posters, fliers, and any other ‘touchpoint’ where people are thinking about using the court system and planning for how to do it.
It prioritizes language access as a fundamental principle of design of court information, rather than as an add-on afterthought.
6. Help People Fill in Forms Better
One of the big failpoints in the legal process is the correct completion of forms. Team Remind proposed two prototypes — one paper-based, the other tech-enabled for improving litigants’ ability to complete Service of Process forms.
The paper-based system involves tagging up and creating a model completed form, that would guide a person through exactly how to follow this model.
The tech-based guide uses a Google Doc form to let people enter in the key data points, and then uses Python to fill in the form with this data. The litigant (or the process-server) never needs to see the Judicial Council form except when they print and file it. The Python script does the completion for them.
The vision of this prototype is to have a 2-pronged tech/paper strategy, so that resources are allocated to different types of users in the system. It is also to come up with cheap hacks to use the power of technology. Rather than contract with an expensive, proprietary vendor to provide for form-filling, the goal here is to mash together existing, modern, mobile-friendly services (like Google Docs) to get a very cheap and quick working system of filling in forms.
The other big insight here was in the power of having an interdisciplinary team, with lawyers and computer scientists working together to find the most strategic uses of technology that would serve the legal system. Lawyers should know the power of Python — a major takeaway for our partners.
7. Gather user input and experiences to feed back to the Courts
Team Law4U drafted a prototype of a kiosk in the Self Help Center’s office, that would ask simple questions from people as they’re waiting to get service. They’d be able to rate the court’s quality of service and give ideas for improvements.
In the future, this program could also recruit litigants to join a Standing User Testing panel, in which they’d be compensated for reviewing new court efforts or giving more feedback to the courts. This would feed into a broader culture of testing and experimentation in the system.
—
These seven prototypes are the result of 9 weeks of hard, creative work by our Prototyping Access to Justice class. Many thanks to the wonderful students and coaches!
We are soliciting feedback now on these prototypes, so that we can then proceed to pilot implementations of some of them in the courts. Let us know what you think!
In our Prototyping Access to Justice class, Kursat Ozenc and I are leading student teams to get quickly from speculating about how the courts could be improved to implementing new concepts.
In our class today, in week 3 of the course, we had the students make some more progress along the Journey of Prototypes.
The journey metaphor is useful to hep anyone who has an idea for how a system (like courts or self-help centers) could be better — but doesn’t know how to make forward progress on it. We are creating a staged set of prototypes to develop, to go from the spark of an idea to a vetted, feasible concept.
Last week, the students had digested their research into how the Self-Help centers currently function. From this, they made journey maps and spotted opportunity points for intervention. Then, they brainstormed 30 new concepts for each intervention-point.
For example, what are 30 new ways to make sure a person can find their way from the entrance of the court to the right office? Or, what are 30 new ways we can help people get service of process right, and not fail here?
This was the first stage of their Prototype Journey — one line ideas on a post-it.
This is the most basic capture of the idea — hard to even call it a prototype. But it’s the seed that starts to lay out what the ‘it’ is, that will guide the rest of the design cycle.
From the post-it, the next stop is the Concept Poster — a slightly more robust prototype. We had the students sort out all of their post-it ideas into a hierarchy, and choose the top 6 ideas they thought had promise to be breakthrough and feasible.
For each of these top 6, they drafted a Concept Poster version of the post-it’s concept, that added more details on functions, target users, high-level value, and implementation requirements. The Concept Posters were then compiled into a catalogue, of the Ideabook.
The value of the Idea Poster as a prototype is that it can let your team quickly, visually present a range of ideas to experts and users. They can almost go ‘shopping’ through the ideas and figure out which ones are most exciting, do-able, and worth pursuing. It can help the team whittle down further to the best one to pursue first, or to reorient based on this initial input.
The Concept Poster/Ideabook prototype can also be a check-in, with the testers letting the team know if similar ideas have been tried before, could be replicated from elsewhere, or if the team should be talking to certain experts.
Once the team gets Ideabook feedback, then they can focus in even further to the first idea they want to pursue. For this single idea, they will flesh it out to the next stage of prototype: the Storyboard.
They go from one sketch of the idea to a visual flow of the entire, ideal journey the user has with the new concept. It lays out the key steps of how the concept will work, what the user will have to do, what the system/service-provider will have to do, and what the resolution looks like. All of this is the ‘happy path’ version of an implementation, where everything goes right and works out.
Once the team documents this ideal flow, they then use the prototype to identify the key assumptions they’re making about the user, the service-provider, and the system. By laying out the step-by-step of the ideal, they can see what needs to happen or exist in order for this ideal to be realized. The prototype helps the team spot what they need to test.
From there, the team quickly moves to the next prototype mode: The Wizard of Oz version. For each of those assumptions they’ve discovered in their storyboard flow, they have to make a way to behaviorally test whether it will really work the way they expect it to.
A Wizard of Oz prototype is an interactive realization of the idea. The user is not just reacting to a presentation of an idea — she is playing around with the ‘app’, or moving around in the new ‘space’, or considering whether to visit a ‘website’. The design team does not actually build a functional new app, space or website — but they use paper, sketches, enactment, and other hacks to simulate the concept.
The value of this prototype is getting more realistic feedback, about whether people will really engage with the thing you’re creating, if they’ll be understand it, and if they’ll get value from it. You can test the many assumptions baked into it.
Some common Wizard of Oz versions of prototypes are making paper-sketches of apps and then assembling them into a click-through using a tool like Marvel. Or putting a sign-up sheet for a new service or organization, and seeing how many people sign up or take the ad. Or, manually acting out a new tech-based service.
Each of these prototypes will let the team count behavior: how many people sign up? How many people are able to use the app you’re envisioning? How many people click on an advertisement for a new service? How many people stop to look and use a new electronic sign you want to install?
It also is valuable for more formal organizations like the court, to do something temporary and non-permanent — to demonstrate what is possible without asking for a large commitment immediately. Rapid prototypes of new signs, worksheets, space layout, or tech can show value, get feedback, and build momentum without any budget requests or committee approvals.
After the Wizard of Oz prototype, the teams will continue on their journey to more refined, content-rich, and tech-enabled versions of their ideas. But these first four steps of the journey help them get quick input, vetting, and refinements to make sure they’re on a promising track and doing something that will resonate with their various stakeholders.
