AI + Access to Justice Current Projects

User Research Workshop on AI & A2J

In December 2023, our lab hosted a half-day workshop on AI for Legal Help.

Our policy lab class of law students, master students, and undergraduates presented their user research findings from their September through December research.

Our guests, including those from technology companies, universities, state bars, legal aid groups, community-based organizations, and advocacy/think takes, all worked together in break-out sessions to tackle some of the big policy and legal opportunities around AI in the space.

We thank our main class partners, the Technology Initiative Grant team from the Legal Services Corporation, for assisting us with the direction and main feedback to our class user research work.

Current Projects

Filing Fairness Toolkit

The Stanford Legal Design Lab & the Rhode Center on the Legal Profession have just released the Filing Fairness Toolkit.

The toolkit covers 4 areas, with diagnostics, maturity models, and actionable guidance for:

  1. improving Filing Technology Infrastructure
  2. building a healthy Filing Partner Ecosystem
  3. establishing good Technology Governance
  4. refining Forms & Filing Processes

This Toolkit is the product of several years of work, design sessions, collaborations with courts and vendors across the country, and stakeholder interviews. It is for court leaders, legal tech companies, legal aid groups, and government officials who are looking for practical guidance on how to make sure that people can find, complete, and file court forms.

Check out our diagnostic tool to see how your local court system measures up to national best practices in forms, efiling, and services.

We know efiling and court technology can be confusing (if not intimidating). We’ve worked hard to make these technical terms & processes more accessible to people beyond IT staff. Getting better efiling systems in place can unlock new opportunities for access to justice.

Please let us know if you have questions, ideas, and stories about making forms, efiling, and other court tech infrastructure more accessible, user-friendly, and impactful.

Read more at the Toolkit announcement page.

AI + Access to Justice Current Projects

Schedule for AI & A2J Jurix workshop

Our organizing committee was pleased to receive many excellent submissions for the AI & A2J Workshop at Jurix on December 18, 2023. We were able to select half of the submissions for acceptance, and we extended the half-day workshop to be a full-day workshop to accommodate the number of submissions.

We are pleased to announce our final schedule for the workshop:

Schedule for the AI & A2J Workshop

Morning Sessions

Welcome Kickoff, 09:00-09:15

Conference organizers welcome everyone, lead introductions, and review the day’s plan.

1: AI-A2J in Practice, 09:15-10:30 AM 

09:15-09:30: Juan David Gutierrez: AI technologies in the judiciary: Critical appraisal of LLMs in judicial decision making

09:30-09:45: Ransom Wydner, Sateesh Nori, Eliza Hong, Sam Flynn, and Ali Cook: AI in Access to Justice: Coalition-Building as Key to Practical and Sustainable Applications

09:45-10:00: Mariana Raquel Mendoza Benza: Insufficient transparency in the use of AI in the judiciary of Peru and Colombia: A challenge to digital transformation

10:00-10:15: Vanja Skoric, Giovanni Sileno, and Sennay Ghebreab: Leveraging public procurement for LLMs in the public sector: Enhancing access to justice responsibly

10:15-10:30: Soumya Kandukuri: Building the AI Flywheel in the American Judiciary

Break: 10:30-11:00 

2: AI for A2J Advice, Issue-Spotting, and Engagement Tasks, 11:00-12:30 

11:00: Opening remarks to the session

11:05-11:20: Sam Harden: Rating the Responses to Legal Questions by Generative AI Models

11:20-11:35: Margaret Hagan: Good AI Legal Help, Bad AI Legal Help: Establishing quality standards for responses to people’s legal problem stories

11:35-11:50: Nick Goodson and Rongfei Lui: Intention and Context Elicitation with Large Language Models in the Legal Aid Intake Process

11:50-12:05: Nina Toivonen, Marika Salo-Lahti, Mikko Ranta, and Helena Haapio, Beyond Debt: The Intersection of Justice, Financial Wellbeing and AI

12:05-12:15: Amit Haim: Large Language Models and Legal Advice12:15-12:30: General Discussions, Takeaways, and Next Steps on AI for Advice

Break: 12:30-13:30

Afternoon Sessions

3: AI for Forms, Contracts &  Dispute Resolution, 13:30-15:00 

13:30: Opening remarks to this session13:35-13:50: Quinten Steenhuis, David Colarusso, and Bryce Wiley: Weaving Pathways for Justice with GPT: LLM-driven automated drafting of interactive legal applications

13:50-14:05: Katie Atkinson, David Bareham, Trevor Bench-Capon, Jon Collenette, and Jack Mumford: Tackling the Backlog: Support for Completing and Validating Forms

14:05-14:20: Anne Ketola, Helena Haapio, and Robert de Rooy: Chattable Contracts: AI Driven Access to Justice

14:20-14:30: Nishat Hyder-Rahman and Marco Giacalone: The role of generative AI in increasing access to justice in family (patrimonial) law

14:30-15:00: General Discussions, Takeaways, and Next Steps on AI for Forms & Dispute Resolution

Break: 15:00-15:30

4:  AI-A2J Technical Developments, 15:30-16:30

15:30: welcome to session
15:35-15:50: Marco Billi, Alessandro Parenti, Giuseppe Pisano, and Marco Sanchi: A hybrid approach of accessible legal reasoning through large language models
15:50-16:05: Bartosz Krupa – Polish BERT legal language model
16:05-16:20: Jakub Dråpal – Understanding Criminal Courts
16:20-16:30: General Discussion on Technical Developments in AI & A2J

Closing Discussion: 16:30-17:00

What are the connections between the sessions? What next steps do participants think will be useful? What new research questions and efforts might emerge from today?


Report on litigants’ outcomes on Zoom court hearings

The Indiana University team, led by professor Victor Quintanilla, has released the report Accessing Justice with Zoom: Experiences and Outcomes in Online Civil Courts.

The team had set up a novel system to recruit court users to give feedback about their experience attending court in-person or remotely, combining that with administrative data and observational data about how the hearings proceeded. This allows them to examine the various effects, preferences, and outcomes that are at play now that online/remote/Zoom court proceedings are now available.

Explore some of the findings that the team found with Indiana court users, including

  • the technological capability and usage of litigants
  • comparison of preferences for remote hearings vs in person
  • how people participated in remote hearings
  • how frequently litigants experienced technical issues
  • what kinds of concerns and dynamics were interrelated

See more at the Indiana Equity Accelerator.

Current Projects

How do we measure mistakes and harms of legal services?

As new services and tech projects launch to serve the public, there’s a regular question being asked:

  • How do we measure if these new justice innovations do better than the status quo?
  • How can we compare the risk of harm to the consumers by these new services & technologies, as compared to a human lawyer — or compared to no services at all?

This entails diving into the discussion of legal services mistakes, risks, harms, errors, complaints, and problems. Past discussions of these legal service problems tend to be fairly abstract. Many regulators & industry groups focus on consumer protection at the high level: how can we protect people from low-quality, fraudulent, or problematic legal services?

This high-level discussion of legal service problems doesn’t lend itself well to specific measurements. It’s hard to assess whether a given lawyer, justice worker, app, or other service-tech tool is more or less protective of a consumer’s interest.

I’ve been thinking a lot about how we can more systematically and clearly measure the quality level (and risk of harm) of a given legal service. As I’ve been exploring & auditing AI platforms for legal problem-solving, this systematic evaluation is needed to be able to assess the quality issues on these AI platforms.

Measuring Errors vs Measuring Consequences

As I’ve been reading through work in other areas (particularly health information and medical systems), I’ve found the work of medical & information researchers to be very instructive. See one such article here.

One of the big things I have learned from medical safety analysis has been the importance of separating the Mistake/Error from the Harm/Consequence. Medical domain experts have built 2 sets of resources:

This is somewhat of a revelation: to separate the provider error from the user harm. Not all errors result in harm — and not all harms have the same severity & level of concern.

As I am studying AI provision of legal services is that AI might make an error, but this does not always result in harm. For example, the AI might tell a person the wrong timeline around eviction lawsuits. The person might screenshot this incorrect AI response and send it their landlord – “I actually have 60 days to pay back rent before you can sue me – see what ChatGPT says!”. The landlord might cave, and give that person 60 days to pay back rent. The user hasn’t experienced harm, even though there was an error. That’s why it’s worthwhile to separate these problems into the Mistake and the Harm.

Planning out a protocol to measure legal services errors & harms

Here is how I have been developing mistake-harm protocol, to assess legal services (including AI platforms answering people’s questions). Here is a first draft, that I invite feedback to:

Step 1: Categorize what Legal Service Interaction you’re assessing. Does the legal service interaction fit into one of these common categories?

  • Provision of info and advice in response to a client’s description of their problem, including statement of law, listing of options, providing plan of steps to take (common in brief services, hotlines, chats, AI)
  • Filling in a document or paperwork that will be given to court or other party, including selection of claims/defenses
  • Intake, screening about whether the service can help you
  • Prep and advocacy in a meeting, hearing, mediation, or similar
  • Negotiation, Assessment of options, and Decision advice on key choices
  • (Meta) Case Management of the person’s problem, journey through the system
  • (Meta) Pricing, billing, and management of charges/payments for the service

Step 2: Categorize what Problem or Mistake has happened in this interaction (with the thought that we’ll have different common problems that happen in these different service interactions above)Preliminary list of problems/mistakes

  • Provider supplies incorrect (hallucinated, incorrect jdx, out of date, etc) info about the law, procedure, etc
  • Provider supplies correct info, but in a way that user does not understand enough to make wise choice
  • User misinterprets the provider’s response
  • Provider provides biased information or advice
  • User experiences provider as offensive, lack of dignity/respect, hurtful to their identity
  • Provider incorrectly shares private data from user
  • Provider is unreasonably slow
  • Provider charges unreasonable amount for service

Step 3: Identify if any Harm or Consequence occurred because of the problem. Acknowledging that not all of the situations above result in any harm at all – or that there are different degrees of harm.Possible harms that user or broader community might experience if the problems above occur.

  • User does not raise a claim or defense that they are entitled to, and might have gotten them a better legal judgment/outcome.
  • User raises an inapplicable claim, cites an incorrect law, brings in inadmissible evidence – makes a substantive or procedural mistake that might delay their case, increase their costs, or  lead to a bad legal judgment.
  • User spends $ unnecessarily on a legal service.
  • User’s legal process is longer and costlier than needed.
  • User brings claim with low likelihood of success, and goes through an unnecessary legal process.
  • User’s conflict with other party worsens, and the legal process becomes lengthier, more expensive, more acrimonious, and less likely to improve their (or their family’s) social/financial outcomes.
  • User feels legal system is inaccessible. They are less likely to use legal services, court system, or government agency services in future problems.
AI + Access to Justice Current Projects

Presentation to Indiana Coalition for Court Access

On October 20th, Legal Design Lab executive director presented on “AI and Legal Help” to the Indiana Coalition for Court Access.

This presentation was part of a larger discussion about research projects, a learning community of judges, and evidence-based court policy and rules changes. What can courts, legal aid, groups, and statewide justice agencies be doing to best serve people with legal problems in their communities?

Margaret’s presentation covered the initial user research that the lab has been conducting, about how different members of the public think about AI platforms in regards to legal problem-solving and how they use these platforms to deal with problems like evictions. The presentation also spotlit the concerning trends, mistakes, and harms around public use of AI for legal problem-solving, which justice institutions and technology companies should focus on in order to prevent consumer harms while harnessing the opportunity of AI to help people understand the law and take action to resolve their legal problems.

The discussion after the presentation covered topics like:

  • Is there a way for justice actors to build a more authoritative legal info AI model, especially with key legal information about local laws and rights, court procedures and timelines, court forms, and service organizations contact details? This might help the AI platforms, avoid mistaken, information or hallucinated details.
  • How could researchers measure the benefits and harms of AI provided legal answers, compared to legal expert-provided legal answers, compared to no services at all? Aside from anecdotes and small samples, is there a more deliberate way to analyze the performance of AI platforms, when it comes to answering peoples questions about the law, procedures, forms, and services? This might include systematically measuring how often these platforms make mistakes, categorizing exactly what the mistakes are, and estimating, or measuring how much harm emerges from these mistakes. A similar deliberate protocol might be done for the benefits that these platforms provide.
AI + Access to Justice Current Projects

Interest Form signup for AI & Access to Justice

Are you a legal aid lawyer, court staff member, judge, academic, tech developer, computer science researcher, or community advocate interested in how AI might increase Access to Justice — and also what limits and accountability we must establish so that it is equitable, responsible, and human-centered?

Sign up at this interest form to stay in the loop with our work at Stanford Legal Design Lab on AI & Access to Justice.

We will be sending those on this list updates on:

  • Events that we will be running online and in person
  • Publications, research articles, and toolkits
  • Opportunities for partnerships, funding, and more
  • Requests for data-sharing, pilot initiatives, and other efforts

Please be in touch through the form — we look forward to connecting with you!

AI + Access to Justice Current Projects

Report a problem you’ve found with AI & legal help

The Legal Design Lab is compiling a database of “AI & Legal Help problem incidents”. Please contribute to this database by entering in information on this form, that feeds into the database.

We will be making this database available in the near-future, as we collect more records & review them.

For this database, we’re looking for specific examples of where AI platforms (like ChatGPT, Bard, Bing Chat, etc) provide problematic responses, like:

  • incorrect information about legal rights, rules, jurisdiction, forms, or organizations;
  • hallucinations of cases, statutes, organizations, hotlines, or other important legal information;
  • irrelevant, distracting, or off-topic information;
  • misrepresentation of the law;
  • overly simplified information, that loses key nuance or cautions;
  • otherwise doing something that might be harmful to a person trying to get legal help.

You can send in any incidents you’ve experienced here at this form: 

We will be reviewing submissions & making this incident database available in the future, for those interested.

Fill in the form to report an AI-Justice problem incident

AI + Access to Justice Current Projects

Call for papers to the JURIX workshop on AI & Access to Justice

At the December 2023 JURIX conference on Legal Knowledge and Information Systems, there is an academic workshop on AI and Access to Justice.

There is an open call for submissions to the workshop. There is an extension to the deadline, which is now November 20, 2023. We encourage academics, practitioners, and others interested in the field to submit a paper for the workshop or consider attending.

The workshop will be on December 18, 2023 in Maastricht, Netherlands (with possible hybrid participation available).

See more about the conference at the main JURIX 23 website.

About the AI & A2J workshop

This workshop will bring together lawyers, computer scientists, and social science researchers to discuss their findings and proposals around how AI might be used to improve access to justice, as well as how to hold AI models accountable for the public good.

Why this workshop? As more of the public learns about AI, there is the potential that more people will use AI tools to understand their legal problems, seek assistance, and navigate the justice system. There is also more interest (and suspicion) by justice professionals about how large language models might affect services, efficiency, and outreach around legal help. The workshop will be an opportunity for an interdisciplinary group of researchers to shape a research agenda, establish partnerships, and share early findings about what opportunities and risks exist in the AI/Access to Justice domain — and how new efforts and research might contribute to improving the justice system through technology.

What is Access to Justice? Access to justice (A2J) goals center around making the civil justice system more equitable, accessible, empowering, and responsive for people who are struggling with issues around housing, family, workplace, money, and personal security. Specific A2J goals may include increasing people’s legal capability and understanding; their ability to navigate formal and informal justice processes; their ability to do legal tasks around paperwork, prediction, decision-making, and argumentation; and justice professionals’ ability to understand and reform the system to be more equitable, accessible, and responsive. How might AI contribute to these goals? And what are the risks when AI is more involved in the civil justice system?

At the JURIX AI & Access to Justice Workshop, we will explore new ideas, research efforts, frameworks, and proposals on these topics. By the end of the workshop, participants will be able to:

  • Identify the key challenges and opportunities for using AI to improve access to justice.
  • Identify the key challenges and opportunities of building new data sets, benchmarks, and research infrastructure for AI for access to justice.
  • Discuss the ethical and legal implications of using AI in the legal system, particularly for tasks related to people who cannot afford full legal representation.
  • Develop proposals for how to hold AI models accountable for the public good.

Format of the Workshop: The workshop will be conducted in a hybrid form and will consist of a mix of presentations, panel discussions, and breakout sessions. It will be a half-day session. Participants will have the opportunity to share their own work and learn from the expertise of others.

Organizers of the Workshop: Margaret Hagan (Stanford Legal Design Lab), Nora al-Haider (Stanford Legal Design Lab), Hannes Westermann (University of Montreal), Jaromir Savelka (Carnegie Mellon University), Quinten Steenhuis (Suffolk LIT Lab).

Are you generally interested in AI & Access to Justice? Sign up for our Stanford Legal Design Lab AI-A2J interest list to stay in touch.

Submit a paper to the AI & A2J Workshop

We welcome submissions of 4-12 pages (using the IOS formatting guidelines). A selection will be made on the basis of workshop-level reviewing focusing on overall quality, relevance, and diversity.

Workshop submissions may be about the topics described above, including:

  • findings of research about how AI is affecting access to justice,
  • evaluation of AI models and tools intended to benefit access to justice,
  • outcomes of new interventions intended to deploy AI for access to justice,
  • proposals of future work to use AI or hold AI initiatives accountable,
  • principles & frameworks to guide work in this area, or
  • other topics related to AI & access to justice

Deadline extended to November 20, 2023

Submission Link: Submit your 4-12 page paper here:

Notification: November 28, 2023

Workshop: December 18, 2023 (with the possibility of hybrid participation) in Maastricht, Netherlands

More about the JURIX Conference

The Foundation for Legal Knowledge Based Systems (JURIX) is an organization of researchers in the field of Law and Computer Science in the Netherlands and Flanders. Since 1988, JURIX has held annual international conferences on Legal Knowledge and Information Systems.

This year, JURIX conference on Legal Knowledge and Information Systems will be hosted in Maastricht, the Netherlands. It will take place on December 18-20, 2023.

The proceedings of the conferences will be published in the Frontiers of Artificial Intelligence and Applications series of IOS Press. JURIX follows the Golden Standard and provides one of the best dissemination platforms in AI & law.

Current Projects

Paths Toward Access to Justice at Scale presentation

In October 2023, Margaret Hagan presented at the International Access to Justice Forum, on “Paths toward Access to Justice at Scale”. The presentation covered the preliminary results of stakeholder interviews she is conducting with justice professionals across the US about how best to scale one-off innovations and new ideas for improvements, to become more sustainable and impactful system changes.

The abstract

Pilots to increase access to justice are happening in local courts, legal aid groups, government agencies, and community groups around the globe. These innovative new local services, technologies, and policies aim to build people’s capability, reduce barriers to access, and improve the quality of justice people receive. They are often built with an initial short-term investment, to design the pilot and run it for a period. Most of them lack a clear plan to scale up to a more robust iteration, or spread to other jurisdictions, or sustain the program past the initial investment. This presentation presents a framework of theories of change for the justice system, and stakeholders’ feedback on how to use them for impact.

The research on Access to Justice long-term strategies

The presentation covered the results of the qualitative, in-depth interviews with 11 legal aid lawyers, court staff members, legal technologists, funders, and statewide justice advocates about their work, impact, and long-term change.

The research interviews asked these professionals about their long-term, systematic theories of change — and to rate other theories of change that others have mentioned. They were asked about past projects they’ve run, how they have made an impact (or not), and what they have learned from their colleagues about what makes a particular initiative more impactful, sustainable, and successful.

The goal of the research interviews was to gather the informal knowledge that various professionals have gathered over years of work in reforming the justice system and improving people’s outcomes when they experience legal problems.

This knowledge often circulates casually at meetings, dinners, and over email, but is not often laid out explicitly or systematically. It was also to encourage reflection among practitioners, to move from a focus just on day-to-day work to long-term impact.

Stay tuned for more publications about this research, as the interviews & synthesis continue.