Reading List on Justice Innovation

The Legal Design Lab offers this library of studies, reports, and articles that focus on access to justice research, innovation, and impact.

Please use these readings in your coursework, research efforts, and strategic plans.

Our Reading List covers access to justice, user experience of courts and legal aid, evaluation of innovation pilots, and more.

Let us know if there are more pieces to add. This Reading List on Justice Innovation is a growing resource.

Design Methods for Justice

How can groups create & validate better justice innovations?

These readings describe design methodologies that can better identify promising interventions, and that can validate these proposed initiatives before they are deployed in pilots or broader implementations. These aim to direct scarce resources to more promising and impactful solutions, that have a higher chance of increasing access to justice.

Redesigning Justice Innovation: A Standardized Methodology

Bernal, Daniel W, and Margaret D Hagan. “Redesigning Justice Innovation: A Standardized Methodology.” Stanford Journal of Civil Rights and Civil Liberties XVI, no. 2 (June 2020): 335–84.

This paper proposes the use of a human-centered legal design process to create better possible interventions for civil justice reform. It identifies how this process can be carried out so that the strongest possible intervention is then piloted and evaluated during randomized control trials or other evaluations.

Human-Centered Civil Justice Design

Victor Quintanilla, 121 Penn State Law Review 745 (2017)

Quintanilla’s 2017 paper lays out how design can be used by stakeholders in the civil justice system to reform the services, technology, and system — particularly to improve accessibility to members of the public.

It connects design with the outcome of procedural justice — and how using empathetic, participatory, user-centered methods can lead to new reforms and interventions that are more likely to be used by litigants, make them feel more dignity, and increase their justice outcomes.

The paper also points to this design process as a way to move courts to a more empathetic, human-centered culture.

Charles L Owen, Ronald W. Staudt & Edward B. Pedwell, Access to Justice: Meeting the Needs of Self-Represented Litigants (2001), https://www.kentlaw.iit.edu/Documents/Institutesand Centers/CAJT/access-to-justice-meeting-the-needs.pdf

Margaret Hagan, Participatory Design for Innovation in Access to Justice, Winter Daedalus (2019), https://www.amacad.org/publication/participatory-design-innovation-access-justice (last visited Jan 24, 2019).

Margaret Hagan, Legal Design as a Thing: A Theory of Change and a Set of Methods to Craft a Human-Centered Legal System, 36 Des. Issues 3–15 (2020), https://doi.org/10.1162/desi_a_00600 (last visited Jul 10, 2020).

Margaret D Hagan, A Human-Centered Design Approach to Access to Justice Generating New Prototypes and Hypotheses for Intervention to Make Courts User-Friendly, 6 Indiana J. Law Soc. Equal. 199–239 (2018), https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3186101

Shannon Salter & Darin Thompson, Public-centered Civil Justice Redesign, 3 McGill J. Dispute. Resolut. 113–136 (2016), https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2955796

Lorne Sossin, Designing Administrative Justice, 25 SSRN (2017).

Dan Jackson, Human-Centered legal tech: Integrating design in legal education, 50 Law Teach. 82–97 (2016).

Emily Houh & Kristin (Brandser) Kalsem, It’s Critical: Legal Participatory Action Research, University of Cincinnati College of Law Scholarship and Publications (2013), http://scholarship.law.uc.edu/fac_pubs (last visited Jun 29, 2017).


What works to increase access to justice?

What should an Access to Justice agenda be?

Civil Justice For All: A Report and Recommendations from the Making Justice Accessible Initiatives

American Academy of Arts & Sciences, Civil Justice for All: A Report and Recommendations from the Making Justice Accessible Initiative (2020), https://www.amacad.org/sites/default/files/publication/downloads/2020-Civil-Justice-for-All_0.pdf

This report recommends that civil justice be an urgent policy and funding priority. It outlines 7 key themes for improved access to justice.

  1. Increased Investment in this work
  2. Legal Services lawyers to help people with their problems
  3. Pro Bono Assistance to supplement dedicated legal services lawyers
  4. New Advocates that can supplement lawyers with other capacities and roles
  5. Greater Collaboration among legal professionals and other human service providers
  6. Simplification and Innovation of legal offerings, including the use of technology
  7. National Coordination among providers and others working in this space

Rhode, Deborah L. “Access to Justice: An Agenda for Legal Education and Research.” https://jle.aals.org/home/vol62/iss4/2/.

Rickard, Erika J. “The Role of Law Schools in the 100% Access to Justice Movement.” Indiana Journal of Law and Social Equality 6, no. 2 (2018): 240–66. https://www.repository.law.indiana.edu/cgi/viewcontent.cgi?article=1084&context=ijlse

Celebrating the “Null” Finding: Evidence-Based
Strategies for Improving Access to Legal Services

Jeanne Charn, Celebrating the “Null” finding: Evidence-based strategies for improving access to legal services, 122 Yale Law J. 2206–2234 (2013). https://www.yalelawjournal.org/pdf/1180_gojype4a.pdf

Jeanne Charn critically examines whether a right-to-counsel movement is the answer to the access to justice crisis. She emphasizes that there needs to be lawyer-lite, self-help innovations along with investment in expanded access to lawyers.

She advocates a consumer-centered approach, that follows what people prefer in responding to their legal problems (which is not always using a full-representation lawyer).

These articles identify specific interventions, strategies, principles, and other guidance for effective access to justice interventions. In particular, these articles propose interventions that can increase engagement with the legal system, build legal capability, and increase justice outcomes.

Uptake of Legal Self-Help Resources: What Works, For Whom, and For What?

by Hugh M. McDonald, Suzie Forell & Zhigang Wei
 30 Justice Issues 1–36 (2019)

This paper used results from the Legal Australia-Wide (LAW) Survey to use a multilevel binary logistic regression model to test the independent influence of legal problems and demographic characteristics on the helpfulness of those resources. From this, the analysis reports findings on how self-help resources influenced individuals’ use of resources to prepare for legal issues, and disaggregates the use of self help resources by legal issues and demographic characteristics. The paper, overall, finds that respondents were just as likely to use self-help resources as they were to take no action at all.

The Justice is in the Details: Evaluating Different Self-Help Designs for Legal Capability in Traffic Court

by Margaret Hagan
Journal of Open Access to the Law: Vol. 7, No.1, October 17 (2019)

This paper presents different visual & technology prototypes that aim to help build legal capability among people who are navigating traffic court. It explains the design process that created these new proposed ways to deliver legal help, as well as the lab-based evaluation of these different interventions. Which should move forward — which are less valuable based on this lightweight testing? It finds that litigants want visual designs to guide them through the court process, and then supplement them with digital tools like text messaging and web guides to get ongoing engagement and follow-through.

A Design Space for Legal and Systems Capability: Interfaces for Self-Help in Complex Systems

by Margaret Hagan and F. Kursat Ozenc
36 Design Issues 61–81 (2020), https://doi.org/10.1162/desi_a_00604

What are the best practices for guiding a person through a complex system, like the legal system?

This paper proposes a common framework that designers, technologists, and policymakers can use to create effective help.

This includes:

  • Understanding the user’s experience as a Justice Journey with a common arc of phases. You should find which part of the justice journey you are trying to affect, and choose the right pattern and messaging for that phase.
  • There are standard patterns of interfaces, tools, and visuals that people and experts agree are good fits for helping people in these different phases

J.J. Prescott, Assessing Access-to-Justice Outreach Strategies, 174 J. Institutional Theor. Econ. 34 (2018), https://repository.law.umich.edu/articles/1965 (last visited Oct 8, 2020).

Logan Cornett et al., Redesigning Divorce: User-Driven Design for a Better Process (2019), http://iaals.du.edu (last visited Jul 7, 2020).

Richard Zorza, The Access to Justice Sorting Hat: Towards a System of Triage and Intake that Maximizes Access and Outcomes, 89 Denver U 859–886 (2012).

Margaret D Hagan, A Human-Centered Design Approach to Access to Justice Generating New Prototypes and Hypotheses for Intervention to Make Courts User-Friendly, 6 Indiana J. Law Soc. Equal. 199–239 (2018), https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3186101

Margaret Hagan & Miso Kim, Design for dignity and procedural justice, 585 in AFHE 2017: Advances in Affective and Pleasurable Design 135–145 (W. Chung ed., 2018), https://ssrn.com/abstract=2994354 (last visited Sep 23, 2018).

Charles L Owen, Ronald W. Staudt & Edward B. Pedwell, Access to Justice: Meeting the Needs of Self-Represented Litigants (2001), https://www.kentlaw.iit.edu/Documents/InstitutesandCenters/CAJT/access-to-justice-meeting-the-needs.pdf

John M. Greacen, Resources to Assist Self-Represented Litigations: A Fifty-State Review of the “State of the Art” 1–54 (2011).

Richard Zorza, Some First Thoughts on Court Simplification: The Key to Civil Access and Justice Transformation, 61 Drake Law Rev. 845–881 (2013).

How can legal websites best help people?

Seeking Legal Help Online: Understanding the Missing Majority

By Jo Szczepanska and Emma Blomkamp, Justice Connect, November 2020, Australia

This report focuses on finding new models of providing cost-efficient and effective free legal assistance at scale for the ‘missing majority’, a redubbing of an Australian term to refer to the growing cohort of people in the ‘justice gap’ that is having difficulty accessing a lawyer, free public or community assistance. The report presents a series of recommendations and design principles tailored for five main areas:

  • Invest in information design and user experience
  • Involve people with lived experience in making online resources
  • Break down silos between sectors, organizations, communities, and self-help 
  • Establish communities of practice to support makers of online self-help resources
  • Invest in consumer outreach, search engine optimization, communications, and marketing

Margaret Hagan, The User Experience of the Internet as a Legal Help Service: Defining standards for the next generation of user-friendly online legal services, 20 Va. JL Tech. 395–465 (2016), https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2942478.

Michigan Advocacy Program & Graphic Advocacy Project, Build a better _______: Strategies for user-informed legal design https://www.srln.org/system/files/attachments/MAP%20x%20GAP%20Report%20-%20Strategies%20for%20User-Informed%20Legal%20Design_0.pdf (2021).

Statewide Website Assessment

Legal Services Corporation, Statewide Website Assessment public report (2017), https://lsc-live.app.box.com/s/zjv5cjuos1kbp43yhicc6pnhccjtboew

This report from Legal Services Corporation lays out benchmark standards for good legal help websites. This includes sites that have improved:

  • Content, with plain language and language access and good presentation
  • Accessibility, user support, mobile-friendliness, and community engagement
  • User-friendly navigation and visual presentation

Catrina Denvir, Nigel J. Balmer & Pascoe Pleasence, Surfing the web – Recreation or resource? Exploring how young people in the UK use the Internet as an advice portal for problems with a legal dimension, 23 Interact. Comput. 96–104 (2011), http://iwc.oxfordjournals.org/cgi/doi/10.1016/j.intcom.2010.10.004 (last visited Mar 22, 2014).

Catrina Denvir, Online and in the know? Public legal education, young people and the Internet, 92–93 Comput. Educ. 204–220 (2016), http://ssrn.com/abstract=2711489 (last visited Jun 25, 2019).

Catrina Denvir, Nigel J. Balmer & Pascoe Pleasence, Portal or pot hole? Exploring how older people use the ‘information superhighway’ for advice relating to problems with a legal dimension, 34 Ageing Soc. 670–699 (2012), http://www.journals.cambridge.org/abstract_S0144686X12001213 (last visited Jul 21, 2014).

Crowdsourcing legal help & online access to legal tools

CB Robertson, The Facebook Disruption: How Social Media May Transform Civil Litigation and Facilitate Access to Justice, 65 Ark. L. Rev. 75 (2012), https://heinonline.org/HOL/Page?handle=hein.journals/arklr65&id=79&div=7&collection=journals

This article points to the effect that social media can have on access to justice. As more people go online with their life & legal problem-solving, there can be new justice journeys & access to solutions. This includes solutions built on:

  • crowdsourced legal advice on social media platforms like Facebook
  • informal discovery, gathering evidence and data using online platforms to track communications & status
  • connecting people with automated document assembly & other software products for doing legal tasks
  • connecting people with global outsourcing & ghostwriting for their legal tasks

(And, what are ways that technology can go wrong — that groups should avoid)

Rebecca L Sandefur, Legal Tech for Non-Lawyers: Report of the Survey of US Legal Technologies (2019), http://www.americanbarfoundation.org/uploads/cms/documents/report_us_digital_legal_tech_for_nonlawyers.pdf (last visited Jan 10, 2020).

James E Cabral et al., Using Technology to Enhance Access to Justice, 26 Harv. J. Law Technol. (2012).

John M. Greacen, Eighteen Ways Courts Should use Technology to Better Serve Their Customers, 57 Fam. Court Rev. 515–538 (2019), http://iaals.du.edu/sites/default/files/documents/publications/eighteen_ways_courts_should_use_technology.pdf (last visited Dec 19, 2018).

Ronald W. Staudt, All the Wild Possibilities: Technology That Attacks Barriers to Access to Justice, 42 Loy. LAL Rev. (2008), http://heinonlinebackup.com/hol-cgi-bin/get_pdf.cgihandle=hein.journals/lla42&section=43

Phil Malone et al., Preliminary Report Best Practices in the Use of Technology to Facilitate Access to Justice Initiatives (2010), https://cyber.harvard.edu/sites/cyber.harvard.edu/files/A2J_Report_Final_073010.pdf

The Engine Room, Technology for Legal Empowerment: A Global Review (2019), https://www.theengineroom.org/wp-content/uploads/2019/01/Tech-for-Legal-Empowerment-The-Engine-Room.pdf . (last visited Jul 3, 2019).

Jason Tashea, Justice-as-a-Platform, MIT Computational Law Report (2021), https://law.mit.edu/pub/justiceasaplatform/release/3 (last visited Jun 15, 2022).

Vincent Morris, Navigating Justice: Self-Help Resources, Access To Justice, and Whose Job Is It Anyway ?, 82 Supra 161–181 (2013).

Court Compass: Existing Programs, (2017), https://online.divorce-education.com/

Chris Johnson, Leveraging Technology to Deliver Legal Services, 23 Harv. JL Tech. (2009), http://heinonlinebackup.com/hol-cgi-bin/get_pdf.cgi?handle=hein.journals/hjlt23&section=9

Ronald W Staudt, Technology for Justice Customers : Bridging the Digital Divide Facing Self-Represented Litigants, 5 Univ. Maryl. Law J. Race, Relig. Gend. Cl. 71–94 (2005).

WIlliam E. Hornsby, Jr., Gaming the System: Approaching 100% Access to Legal Services through Online Games, 1 (2014).

Self-represented efiling: surveying the accessible implementations National Center for State Courts, (2022), www.ncsc.org/a2j. (last visited Jun 14, 2022).

How to promote participation, and reduce default rates, in the justice system?

Daniel Bernal, Taking the Court to the People: Real-World Solutions for Nonappearance, 59 Ariz. Law Rev. 547–571 (2017), http://arizonalawreview.org/pdf/59-2/59arizlrev547.pdf (last visited Sep 18, 2017).

Alissa Fishbane, Aurelie Ouss & Anuj K. Shah, Behavioral nudges reduce failure to appear for court, 370 Science (80). (2020), https://doi.org/10.1126/science.abb6591 (last visited Nov 24, 2020).

Brice Cooke et al., Using Behavioral Science to Improve Criminal Justice Outcomes: Preventing Failures to Appear in Court, IDEAS 42 Policy Pap. 1–20 (2018), https://urbanlabs.uchicago.edu/attachments/store/9c86b123e3b00a5da58318f438a6e787dd01d66d0efad54d66aa232a6473/I42-954_NYCSummonsPaper_Final_Mar2018.pdf (last visited Aug 10, 2021).

Direct Mail Messaging Strategies for Improving Debt Collection Hearing Court Appearance Rates

Harvard Law School Access to Justice Lab, Center on the Legal Profession

This memo outlines effective ways to use paper mailing to encourage litigants’ participation in their court case. This includes having improved content, cartoons, bold headings, action plans, and letter sequences that improve timely engagement.

D James Greiner, Dalie Jimenez & Lois R Lupica, Self-Help, Reimagined, 92 Indiana Law J. 1119–1173 (2017), https://www.repository.law.indiana.edu/ilj/vol92/iss3/6/

Timothy R Schnacke, Michael R Jones & Dorian M Wilderman, Increasing Court-Appearance Rates and Other Benefits of Live-Caller Telephone Court-Date Reminders : The Jefferson County, Colorado, FTA Pilot Project and Resulting Court Date Notification Program, 48 86–95 (2005), http://digitalcommons.unl.edu/ajacourtreviewhttp://digitalcommons.unl.edu/ajacourtreview/393 (last visited Sep 8, 2019).

Court & Judge Reforms to increase access to justice

Shanahan, Colleen F, Jessica Steinberg, Alyx Mark, and Anna E Carpenter. “The Institutional Mismatch of State Civil Courts.” Columbia Law Review, 2022. https://doi.org/10.2139/ssrn.4143712.

Carpenter, Anna E, Jessica K Steinberg, Colleen F Shanahan, and Alyx Mark. “Studying the ‘New’ Civil Judges.Wisconsin Law Review, 2018. https://ssrn.com/abstract=3174977.

Shanahan, Colleen F. “The Keys to the Kingdom: Judges, Pre-Hearing Procedure, and Access to Justice.Wisconsin Law Review, 2018, 215–48.


Regulatory Reforms to increase access to justice

Hadfield, Gillian K., and Deborah L. Rhode. “How to Regulate Legal Services to Promote Access, Innovation, and the Quality of Lawyering.” In Hastings Law Journal, 67:1191–1223, 2016. https://repository.uchastings.edu/hastings_law_journal/vol67/iss5/2.

Engstrom, David Freeman, Lucy Ricca, Graham Ambrose, and Maddie Walsh September. “Legal Innovation After Reform: Evidence From Regulatory Change.” Stanford, CA, September 2022. https://law.stanford.edu/wp-content/uploads/2022/09/SLS-CLP-Regulatory-Reform-REPORTExecSum-9.26.pdf

The Utah Work Group on Regulatory Reform. “Narrowing the Access-to-Justice Gap by Reimagining Regulation Report and Recommendations.” Salt Lake City, UT, 2019. https://www.utahbar.org/wp-content/uploads/2019/08/FINAL-Task-Force-Report.pdf.

Steinberg, Jessica K, Anna E Carpenter, Colleen F Shanahan, and Alyx Mark. “Judges and the Deregulation of the Legal Monopoly.” Fordham Law Review 89, no. 4 (2021): 1315–49. https://perma.cc/D948-NGMH].

Hadfield, Gillian K. “Innovating to Improve Access: Changing the Way Courts Regulate Legal Markets,” no. 14 (2014). https://doi.org/10.1162/daed_a_00290.

Campbell, RW. “Rethinking Regulation and Innovation in the U.S. Legal Services Market.NYUJL & Bus. 9, no. 1 (2012): 1–70.

Clarke, Thomas, and Rebecca L. Sandefur. “Preliminary Evaluation of the Washington State Limited License Legal Technician Program.” SSRN Electronic Journal, 2018. https://doi.org/10.2139/ssrn.2949042.

AI & Access to Justice

How could AI help (or harm) access to justice?

Opportunities & Risks for AI, Legal Help, and Access to Justice, report by Margaret Hagan from Spring 2023 multi-stakeholder workshop co-hosted by Stanford Legal Design Lab & Self-Represented Litigation Network

Justice & AI Crossover development work report, by Margaret Hagan after July 2023 technical-legal crossover event to discuss possible projects

Guzman, H. (2023). AI’s “Hallucinations” Add to Risks of Widespread Adoption. Retrieved June 19, 2023, from https://www.law.com/corpcounsel/2023/03/23/ais-hallucinations-add-to-risks-of-widespread-adoption/?slreturn=20230519164801 

Granat, R. (2023). ChatGPT, Access to Justice, and UPL. Retrieved June 19, 2023, from https://www.lawproductmakers.com/2023/03/chatgtp-access-to-justice-and-upl/ 

Janatian, Samyar et al, “From Text to Structure: Using Large Language Models to Support the Development of Legal Expert Systems” (2023), online: <http://arxiv.org/abs/2311.04911>.

Hagan, M. D. Towards Human-Centered Standards for Legal Help AI. Philosophical Transactions of the Royal Society A. (Forthcoming)

Holt, A. T. (2023). Legal AI-d to Your Service: Making Access to Justice a Reality. Vanderbilt Journal of Entertainment and Technology Law. Retrieved from https://www.vanderbilt.edu/jetlaw/2023/02/04/legal-ai-d-to-your-service-making-access-to-justice-a-reality/

Kanu, H. (2023, April). Artificial intelligence poised to hinder, not help, access to justice. Reuters. Retrieved from https://www.reuters.com/legal/transactional/artificial-intelligence-poised-hinder-not-help-access-justice-2023-04-25/ 

Perlman, A. (2023). The Implications of ChatGPT for Legal Services and Society. The Practice. Cambridge, MA. Retrieved from https://clp.law.harvard.edu/knowledge-hub/magazine/issues/generative-ai-in-the-legal-profession/the-implications-of-chatgpt-for-legal-services-and-society/ 

Poppe, E. T. (2019). The Future Is ̶B̶r̶i̶g̶h̶t̶ Complicated: AI, Apps & Access to Justice. Oklahoma Law Review, 72(1). Retrieved from https://digitalcommons.law.ou.edu/olr/vol72/iss1/8 

Simshaw, D. (2022). Access to A.I. Justice: Avoiding an Inequitable Two-Tiered System of Legal Services. Yale Journal of Law & Technology, 24, 150–226. https://yjolt.org/access-ai-justice-avoiding-inequitable-two-tiered-system-legal-services 

Stepka, M. (2022, February). Law Bots: How AI Is Reshaping the Legal Profession. ABA Business Law Today. Retrieved from https://businesslawtoday.org/2022/02/how-ai-is-reshaping-legal-profession/ 

Telang, A. (2023). The Promise and Peril of AI Legal Services to Equalize Justice. Harvard Journal of Law & Technology. Retrieved from https://jolt.law.harvard.edu/digest/the-promise-and-peril-of-ai-legal-services-to-equalize-justice 

Tripp, A., Chavan, A., & Pyle, J. (2018). Case Studies for Legal Services Community Principles and Guidelines for Due Process and Ethics in the Age of AI. Retrieved from https://docs.google.com/document/d/1rEvg5xuOs_o1njPHHpF9jtuaGi0ren6DYUElBu0Fkfk/edit 

Westermann, Hannes and Karim Benyekhlef. JusticeBot: A Methodology for Building Augmented Intelligence Tools for Laypeople to Increase Access to Justice. ICAIL 2023. https://arxiv.org/pdf/2308.02032.pdf , https://www.cyberjustice.ca/en/logiciels-cyberjustice/nos-solutions-logicielles/justicebot/ 

Wilkins, S. (2023, February). DoNotPay’s Downfall Put a Harsh Spotlight on AI and Justice Tech. Now What? Legaltech News. Retrieved from https://www.law.com/legaltechnews/2023/02/10/donotpays-downfall-put-a-harsh-spotlight-on-ai-and-justice-tech-now-what/ 

JusticeBot project at Cyberjustice Laboratory in Montreal

See the research paper: JusticeBot: A Methodology for Building Augmented Intelligence Tools for Laypeople to Increase Access to Justice by Hannes Westermann, Karim Benyekhlef

Laypeople (i.e. individuals without legal training) may often have trouble resolving their legal problems. In this work, we present the JusticeBot methodology. This methodology can be used to build legal decision support tools, that support laypeople in exploring their legal rights in certain situations, using a hybrid case-based and rule-based reasoning approach. The system ask the user questions regarding their situation and provides them with legal information, references to previous similar cases and possible next steps. This information could potentially help the user resolve their issue, e.g. by settling their case or enforcing their rights in court. We present the methodology for building such tools, which consists of discovering typically applied legal rules from legislation and case law, and encoding previous cases to support the user. We also present an interface to build tools using this methodology and a case study of the first deployed JusticeBot version, focused on landlord-tenant disputes, which has been used by thousands of individuals. 

Spot Classifier from Suffolk LIT Lab, a classifier that can analyze people’s descriptions of their life/legal problems, and identify specific legal issue codes.

Read more about the project: “Online Tool Will Help ‘Spot’ Legal Issues That People Face

Learned Hands machine learning labelling project, to build a labeled dataset of people’s legal problem stories.

See the LawNext write-up: Stanford and Suffolk Create Game to Help Drive Access to Justice

Karl Branting, The Justice Access Game: Crowd-Sourced Evaluation of Systems for Pro Se Litigants, 3435 in CEUR Workshop Proceedings (2023), https://ceur-ws.org/Vol-3435/short5.pdf

Karl Branting & Sarah McLeod, Narrative-Driven Case Elicitation, 3435 in CEUR Workshop Proceedings (2023), https://ceur-ws.org/Vol-3435/short1.pdf

ChatGPT as an Artificial Lawyer

Jinzhe Tan, Hannes Westermann & Karim Benyekhlef, ChatGPT as an Artificial Lawyer?, 3435 in CEUR Workshop Proceedings (2023), https://ceur-ws.org/Vol-3435/short2.pdf

Abstract: Lawyers can analyze and understand specific situations of their clients to provide them with relevant legal information and advice. We qualitatively investigate to which extent ChatGPT (a large language model developed by OpenAI) may be able to carry out some of these tasks, to provide legal information to laypeople. This paper proposes a framework for evaluating the provision of legal information as a process, evaluating not only its accuracy in providing legal information, but also its ability to understand and reason about users’ needs. We perform an initial investigation of ChatGPT’s ability to provide legal information using several simulated cases. We also compare the performance to that of JusticeBot, a legal information tool based on expert systems. While ChatGPT does not always provide accurate and reliable information, it acts as a powerful and intuitive way to interact with laypeople. This research opens the door to combining the two approaches for flexible and accurate legal information tools.

Access to AI Justice: Avoiding an Inequitable Two-Tiered System of Legal Services by Drew Simshaw, Yale Journal of Law & Technology, 2022

Artificial intelligence (AI) has been heralded for its potential to help close the access to justice gap. It can increase efficiencies, democratize access to legal information, and help consumers solve their own legal problems or connect them with licensed professionals who can. But some fear that increased reliance on AI will lead to one or more two-tiered systems: the poor might be stuck with inferior AI-driven assistance; only expensive law firms might be able to effectively harness legal AI; or, AI’s impact might not disrupt the status quo where only some can afford any type of legal assistance. The realization of any of these two-tiered systems would risk widening the justice gap. But the current regulation of legal services fails to account for the practical barriers preventing effective design of legal AI across the landscape, which make each of these two-tiered systems more likely.

Therefore, this Article argues that jurisdictions should embrace certain emerging regulatory reforms because they would facilitate equitable and meaningful access to legal AI across the legal problem-solving landscape, including by increasing competition and opportunities for collaboration across the legal services and technology industries. The Article provides a framework that demonstrates how this collaboration of legal and technical expertise will help stakeholders design and deploy AI-driven tools and services that are carefully calibrated to account for the specific consumers, legal issues, and underlying processes in each case. The framework also demonstrates how collaboration is critical for many stakeholders who face barriers to accessing and designing legal-AI due to insufficient resources, resilience, and relationships. The Article then advocates for regulatory priorities, reforms, and mechanisms to help stakeholders overcome these barriers and help foster legal AI access across the landscape. 

AI, Pursuit of Justice & Questions Lawyers Should Ask in Bloomberg Law, by Julia Brickell, Columbia University, Jeanna Matthews, Clarkson University, Denia Psarrou, University of Athens & Shelley Podolny, Columbia University

Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021). On the dangers of stochastic parrots: Can language models be too big? FAccT 2021 – Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, 610–623. https://doi.org/10.1145/3442188.3445922 

Bickmore, T. W., Trinh, H., Olafsson, S., O’Leary, T. K., Asadi, R., Rickles, N. M., & Cruz, R. (2018). Patient and consumer safety risks when using conversational assistants for medical information: An observational study of siri, alexa, and google assistant. Journal of Medical Internet Research, 20(9). https://doi.org/10.2196/11510

Neel Guha et al., LegalBench: Prototyping a Collaborative Benchmark for Legal Reasonging, (2022), https://arxiv.org/abs/2209.06120 .

Weidinger, L., Uesato, J., Rauh, M., Griffin, C., Huang, P. Sen, Mellor, J., … Gabriel, I. (2022). Taxonomy of Risks posed by Language Models. In ACM International Conference Proceeding Series (Vol. 22, pp. 214–229). ACM. https://doi.org/10.1145/3531146.3533088

How do people use AI tools for problem-solving?

The User Experience of ChatGPT: Findings from a Questionnaire Study of Early Users, July 2023 12 page study by Marita Skjuve, Asbjorn Folstad, and Petter Bae Brandtzaeg

Should My Agent Lie for Me? A Study on Attitudes of US-based Participants Towards Deceptive AI in Selected Future-of-work

On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? by Emily M. Bender, Timni Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell, March 2021

Abstract: The past 3 years of work in NLP have been characterized by the development and deployment of ever larger language models, especially for English. BERT, its variants, GPT-2/3, and others, most recently Switch-C, have pushed the boundaries of the possible both through architectural innovations and through sheer size. Using these pretrained models and the methodology of fine-tuning them for specific tasks, researchers have extended the state of the art on a wide array of tasks as measured by leaderboards on specific benchmarks for English. In this paper, we take a step back and ask: How big is too big? What are the possible risks associated with this technology and what paths are available for mitigating those risks? We provide recommendations including weighing the environmental and financial costs first, investing resources into curating and carefully documenting datasets rather than ingesting everything on the web, carrying out pre-development exercises evaluating how the planned approach fits into research and development goals and supports stakeholder values, and encouraging research directions beyond ever larger language models.

Deceptive AI Ecosystems: The Case of ChatGPT, July 2023, 6-page Extended Abstract from Xiao Zhan, Yifan Xu, Stefan Sarkadi

Patient and Consumer Safety Risks When Using Conversational Assistants for Medical Information: An Observational Study of Siri, Alexa, and Google Assistant 2018 Journal of Medical Internet Research.

Abstract: Conversational assistants, such as Siri, Alexa, and Google Assistant, are ubiquitous and are beginning to be used as portals for medical services. However, the potential safety issues of using conversational assistants for medical information by patients and consumers are not understood.

Objective:To determine the prevalence and nature of the harm that could result from patients or consumers using conversational assistants for medical information.

Methods:Participants were given medical problems to pose to Siri, Alexa, or Google Assistant, and asked to determine an action to take based on information from the system. Assignment of tasks and systems were randomized across participants, and participants queried the conversational assistants in their own words, making as many attempts as needed until they either reported an action to take or gave up. Participant-reported actions for each medical task were rated for patient harm using an Agency for Healthcare Research and Quality harm scale.

Results: Fifty-four subjects completed the study with a mean age of 42 years (SD 18). Twenty-nine (54%) were female, 31 (57%) Caucasian, and 26 (50%) were college educated. Only 8 (15%) reported using a conversational assistant regularly, while 22 (41%) had never used one, and 24 (44%) had tried one “a few times.“ Forty-four (82%) used computers regularly. Subjects were only able to complete 168 (43%) of their 394 tasks. Of these, 49 (29%) reported actions that could have resulted in some degree of patient harm, including 27 (16%) that could have resulted in death.

Conclusions: Reliance on conversational assistants for actionable medical information represents a safety risk for patients and consumers. Patients should be cautioned to not use these technologies for answers to medical questions they intend to act on without further consultation from a health care provider.

Doctor GPT-3: Hype or Reality? report from medical tech company Nabla about their attempt to build an AI-powered application to help people with medical issues.

Read a write-up at The Register, “Researchers made an OpenAI GPT-3 medical chatbot as an experiment. It told a mock patient to kill themselves”

Evaluating Outcomes

How do we evaluate which justice innovations are successful?

These articles lay out instruments and metrics by which groups can evaluate the impact of justice innovations. Do they work as intended? What are the best methods to use to determine if they are successful?

Prescott, J.J.. “The Challenges of Calculating the Benefits of Providing Access to Legal Services.” Fordham Urban Law Journal 37, no. 1 (2010): 303–48. http://ssrn.com/abstract=1582229.

D James Greiner, The new legal empiricism & its application to access-to-justice inquiries, 148 Daedalus 64–74 (2019), https://www-mitpressjournals-org.stanford.idm.oclc.org/doi/pdf/10.1162/daed_a_00536 (last visited May 3, 2019).

Hugh McDonald, Assessing access to justice: How much “legal” do people need and how can we know?, 11 UC Irvine Law Rev. 693–752 (2021), https://scholarship.law.uci.edu/ucilrhttps://scholarship.law.uci.edu/ucilr/vol11/iss3/6 (last visited Sep 13, 2021).

Dalié Jiménez et al., Improving the Lives of Individuals in Financial Distress Using a Randomized Control Trial: A Research and Clinical Approach, XX Georg. J. Poverty Law {&} Policy 449–477 (2013), http://ssrn.com/abstract=2213000 (last visited Jun 13, 2022).

D. James Greiner & Andrea Matthews, Randomized Control Trials in the United States Legal Profession, 12 Annu. Rev. Law Soc. Sci. 295–312 (2016), http://www.ssrn.com/abstract=2726614 (last visited Nov 22, 2018).

Richard Moorhead, RCTs in civil justice: Greiner and Pattanyak, Lawyer Watch (2012), https://lawyerwatch.blog/2012/07/03/rcts-in-civil-justice-greiner-and-pattanyak/ (last visited Nov 22, 2018).

OECD and Open Society Foundations, Understanding Effective Access to Justice, Underst. Eff. Access to Justice 30 (2016), http://www.oecd.org/gov/Understanding-effective-access-justice-workshop-paper-final.pdf (last visited Nov 22, 2018).

J.J. Prescott, Assessing Access-to-Justice Outreach Strategies, 174 J. Institutional Theor. Econ. 34 (2018), https://repository.law.umich.edu/articles/1965 (last visited Oct 8, 2020).

Tom R. Tyler, Procedural Justice and the Courts, 44 Court Rev. J. Am. Judges Assoc. 26–31 (2007), http://digitalcommons.unl.edu/ajacourtreview%5Cn

Martin Gramatikov, Methodological Challenges in Measuring Cost and Quality of Access to Justice: TISCO Working Paper Series on Civil Law and Conflict Resolution Systems (2007), http://ssrn.com/abstract=1099392 (last visited May 5, 2018).

National Center for State Courts & Conference of Chief Justices, Civil Justice Initiative: Performance Measures for Civil Justice (2017), https://www.ncsc.org/~/media/Microsites/Files/Civil-Justice/CJIPerformanceMeasures.ashx (last visited Apr 14, 2019).

Semple, Noel, Tending the Flame: Technological Innovation and the Legal Services Act Regime (August 6, 2019). A paper prepared for the Legal Services Board (2019), Available at SSRN: https://ssrn.com/abstract=3616343 or http://dx.doi.org/10.2139/ssrn.3616343

Abstract: This paper considers the different types of challenges that technological innovation poses to the Legal Services Act 2007’s regulatory framework, and whether the framework is capable of supporting technological innovation that benefits consumers while also addressing the risks it poses to them. The paper makes some recommendations about how the current regime could be adapted to better address these challenges, but its overall conclusion is that the LSA regime remains capable, for the time being, of responding to them.

Saks, Michael and Alice R. Benedict. “Evaluation and quality assurance of legal services.” Law and Human Behavior 1 (1977): 373-384, https://www.semanticscholar.org/paper/Evaluation-and-quality-assurance-of-legal-services-Saks-Benedict/c2a7d16bd1814f5bbd2780964d5e095cff2198cf

Abstract: The issue of evaluating and assuring quality in legal services delivery systems is addressed by considering findings and concepts from research on health-care systems and from the social and behavioral sciences. A framework for conceptualizing the components of quality assurance and for classifying research on quality is offered. The concepts of outcome, process, and structure are the major elements of the framework. Also discussed is the linking of the various components to provide the knowledge necessary to create effective quality-assurance systems. 

Linna, Daniel W. “Evaluating legal services: The need for a quality movement and standard measures of quality and value.” (2021), https://www.semanticscholar.org/paper/Evaluating-legal-services%3A-The-need-for-a-quality-Linna/9d76ec53ca5e1de7ee1458c5fc814ae468a95471

Abstract: The legal industry has not undergone a quality movement and lacks standard measures of legal services quality and value. Whereas medicine long ago embraced evidence-based practice and empiricism, law muddles along, deferring to lawyers’ untested, loosely conceived, normative standards for practice. As a result, existing data about legal-services delivery is of questionable quality, and we lack standard metrics to evaluate this data and any applications developed with it. The lack of empirical rigor in legal services threatens progress in data analytics and artificial intelligence, which require high-quality input and output data. Additionally, the failure to undertake a quality movement in law contributes to numerous legal industry problems, including inadequate access to legal services and justice, a lack of diversity, and work-life imbalances. This article discusses the need for a quality movement (focused on standard work, error detection, peer review, performance measurement, and continuous improvement) and standards for legal-services quality and value. The article discusses output, process, and input data and metrics for measuring quality and value. The article includes summaries of multiple holistic models for measuring legal-services value, including from Noel Semple, Rebecca Sandefur and Thomas Clarke for “roles beyond lawyers,” and Paul Lippe for contracts. The article also identifies several initiatives contributing to the development of quality and value metrics. Finally, the article briefly summarizes the stakeholder benefits of a quality movement and standard metrics for legal-services quality and value.

Douglas, Alex, Lindsey Muir and Karon Meehan. “E‐quality in the e‐services provision of legal practices.” Managing Service Quality 13 (2003): 483-491.

Abstract: This paper examines the quality issues facing private legal practices as they try to offer services on the Internet. The threats and opportunities from offering high quality e‐services are examined as well as potential problems. E‐service operations are divided into “hard”, that is those operations concerned with ensuring the customer receives what was ordered at the right time, place, cost and condition, and “soft”, that is those concerned with Web site design, data information readiness and transactions. Quality measures for both types are proposed and the implications for the legal profession examined. The second half of the paper reports the findings of a survey of Merseyside legal practices in order to see the extent to which they are facing the challenges of the Internet and the issues of e‐service provision. The results show that the legal profession is being slow to introduce Internet technology and only a few firms are offering limited e‐services to their clients. The old attitudes of an aged and learned profession are proving difficult barriers to overcome as the profession battles to bring itself into the twenty‐first century.

Understanding Effective Access to Justice working paper

The OECD and Open Society Justice Initiative published a working paper in November 2016 in the run-up to an expert workshop in Paris.

The paper lays out three key areas of knowledge and research questions:

  1. What is the unmet legal and justice needs? How can we measure this justice gap, and understand the legal needs?
  2. What impact do unmet legal needs have on people, the community, and the government?
  3. What models of legal help can be effective in meeting these legal needs?

Understanding People’s Justice Needs and Experiences

What are their main needs? And what does ‘access to justice mean?

Justice Needs and Satisfaction in the United States of America 2021: Legal problems in daily life

The Hague Institute for Innovation of Law & The Institute for the Advancement of the American Legal System, Justice Needs and Satisfaction in the United States of America (2021), https://iaals.du.edu/sites/default/files/documents/publications/justice-needs-and-satisfaction-us.pdf

This report reports on a wide-ranging survey of Americans about what justice problems they have, what help they have used, what sources of information and advice they sought out, and what the procedural and substantive outcomes were.

Katherine S Wallat, Reconceptualizing Access to Justice, 103 Marquette Law Rev. (2019), https://scholarship.law.marquette.edu/mulr/vol103/iss2/10 (last visited Feb 24, 2021).

Tanina Rostain, Techno-optimism & access to the legal system, 148 Daedalus 93–97 (2019), https://www.mitpressjournals.org/doi/pdf/10.1162/daed_a_00540 (last visited Apr 24, 2019).

Why the “Haves” Come out Ahead: Speculations on the Limits of Legal Change

Marc Galanter, Law & Society ReviewVol. 9, No. 1, Litigation and Dispute Processing: Part One (Autumn, 1974), pp. 95-160 (66 pages)

Galanter’s article explores the game-like setup of court.

Pascoe Pleasence, Nigel J Balmer & Rebecca L Sandefur, Paths to Justice: A past, present and future roadmap (2013).

Rebecca L Sandefur, Access to what?, 148 Daedalus 49–55 (2019).

Lisa Wintersteiger, Legal Needs, Legal Capability and the Role of Public Legal Education, Law for Life: the Foundation for Public Legal Education (2015).

Rebecca L Sandefur, What We Know and Need to Know About the Legal Needs of the Public, 67 S. C. Law Rev. 443–459 (2016).

Strengthening Access to Civil Justice with Legal Needs Surveys, https://www.opensocietyfoundations.org/sites/default/files/strengthening-access-to-civil-justice-legal-needs-surveys-20180409.pdf (last visited Nov 3, 2018).

Pascoe Pleasence, Nigel J Balmer & Rebecca L Sandefur, Paths to Justice: A past, present and future roadmap (2013).

Legal Services Corporation, The Justice Gap: Measuring the Unmet Civil Legal Needs of Low-income Americans (2017), https://www.lsc.gov/justicegap2017 .

Legal Services Corporation, Documenting the Justice Gap in America: The Current Unmet Civil Legal Needs of Low-Income Americans.An Updated Report of the Legal Services Corporation (2009).

The State Bar of California, 2019 California Justice Gap Study (2019), https://www.lsc.gov/media-center/publications/2017-justice-gap-report (last visited Apr 27, 2021).

Victor Quintanilla & Rachel Thein, Indiana Civil Legal Needs Study and Legal Aid System Scan (2019), https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3376257 (last visited May 10, 2019).

Christine Coumarelos et al., Legal Australia-Wide Survey: Legal Need in Australia: Access to justice and legal needs (2012).

How people currently go through justice journeys, and what problems they experience

Barbara Bezdek, Silence in the court: participation and subordination of poor tenants’ voices in legal process, 20 20 Hofstra L. Rev. 533 533–608 (1992), http://scholarlycommons.law.hofstra.edu/hlr (last visited Mar 12, 2020).

Daniel W. Bernal, Ashamed, Judged, and Unsafe: A Qualitative Study of Tenant Justice Perceptions to Inform the Redesign of Housing Court, 52 N.M. L. Rev. 70 (2022).
Available at: https://digitalrepository.unm.edu/nmlr/vol52/iss1/5

Scholars have long suspected that tenants were skeptical of housing court, but prior studies—relying principally on surveys— have not borne that out. This qualitative empirical study draws from in-depth interviews and finds, in contrast to these previous studies, that tenants find the housing court process anything but fair, and describe a startling disconnect between their reasons for court attendance and their experiences of the hearings. Such negative justice perceptions may affect participation in housing court, compliance with judgments, and overall confidence in the judicial process. This Article suggests several legal and policy reforms to better align the housing court experience and tenant attendance goals, including more readable and empathetic court documents, amendments to rules of procedure for housing courts, and structural changes to the eviction hearing.

Utah Bar Foundation, Utah Bar Foundation Report on Debt Collection and Utah’s Courts (2022), https://www.utahbarfoundation.org/static/media/UBF2022.912d30c10e5681bf5f8c.pdf (last visited Jun 9, 2022).

Ellen Degnan et al., Trapped in Marriage, SSRN Electron. J. (2018), (last visited Jun 14, 2022).

NCSC, Civil Justice Initiative: The Landscape of Civil Litigation in State Courts (2015), http://www.ncsc.org/~/media/Files/PDF/Research/CivilJusticeReport-2015.ashx

Emily La Gratta, Court Voices Project: Using Court User Feedback to Guide Courts’ Pandemic Responses, August 2022, https://static1.squarespace.com/static/5d260fabfdbd0000011668cb/t/630d145e090f8e58d8380900/1661801576870/court+voices+project+user+feedback

Equitable Access to Justice

What are the best ways to get access for people from different demographic groups?

U.S. Department of Justice Civil Rights Division, Language Access in State Courts, SSRN (2016), https://www.justice.gov/crt/file/892036/download (last visited Sep 27, 2018).

State Justice Institute & National Center for State Courts, A National Call to Action Access to Justice for Limited English Proficient Litigants: Creating Solutions to Language Barriers in State Courts (2013), www.ncsc.org (last visited Sep 23, 2018).

Natasha N. Jones & Miriam F. Williams, The Social Justice Impact of Plain Language : A Critical Approach to Plain-Language Analysis, 60 IEEEE Trans. Prof. Commun. 1–18 (2017).

The Working Party & Amanda Finlay, Preventing Digital Exclusion from Online Justice (2018), www.justice.org.uk.

Sara Sternberg Greene, Race, Class, and Access to Civil Justice, 101 Iowa Law Rev. 1263 (2016), http://www.americanbar.org/content/dam/aba/migrated/legalservices/downloads/sclaid/ (last visited Feb 24, 2021).

Sheila O’Hare & Sanda Erdelez, Legal information acquisition by the public: The role of personal and environmental factors, 54 Proc. Assoc. Inf. Sci. Technol. 298–307 (2017).

New Court Models, or Alternatives to Courts

Interventions to make courts more accessible

Richard Zorza, The Self-Help Friendly Court: Designed from the Ground Up to Work for People Without Lawyers (2002).

Colleen F. Shanahan, The Keys to the Kingdom: Judges, Pre-Hearing Procedure, and Access to Justice, Wis. L. Rev. 215–248 (2018).

Jessica K Steinberg, A Theory of Civil Problem Solving Courts, 93 NYU Law Rev. (2018), https://ssrn.com/abstract=3219306Electroniccopyavailableat:https://ssrn.com/abstract=3219306 (last visited Nov 7, 2019).

Jessica K Steinberg, Informal, Inquisitorial, and Accurate: An Empirical Look at a Problem-Solving Housing Court, 42 Law Soc. Inq. 1058–1090 (2017). https://scholarship.law.gwu.edu/cgi/viewcontent.cgi?article=2427&context=faculty_publications

Jessica Steinberg, Demand Side Reform in the Poor People’s Court, 47 Conn. Law Rev. 741 (2015), http://scholarship.law.gwu.edu/faculty_publications

New models for dispute resolution systems

Ayelet Sela, Streamlining Justice: How Online Courts Can Resolve the Challenges of Pro Se Litigation, Cornell J. Law Public Policy (2016), https://ww3.lawschool.cornell.edu/research/JLPP/upload/Sela-final.pdf (last visited May 2, 2019).

John Graecen, Serving Self-Represented Litigants Remotely: A Resource Guide (2016).

James J Prescott, Improving Access to Justice in State Courts with Platform Technology, 70 Vand. L. Rev (2017), https://repository.law.umich.edu/articles

Analysis of State Courts’ role in problem-solving

The Institutional Mismatch of State Civil Courts

Colleen F Shanahan, Jessica Steinberg, Alyx Mark, and Anna E. Carpenter, The Institutional Mismatch of State Civil Courts, Columbia Law Rev. (2022), https://perma.cc/WUV8-YJPT] (last visited Jul 5, 2022).

This article looks at case data and court observations to understand how courts deal with the social needs that people come to the civil justice system to deal with. The article posits that courts are making social policy, becoming social service providers, and rule-makers.

Other topics for exploration

  • How do people talk about their problems?
  • How do we measure different kinds of interventions?
  • What goes wrong with online searches and information brokers to legal help?
  • What do people want vs. what do experts think they need when it comes to legal help?
  • What are the barriers to more innovative offerings in legal help?
  • How can courts become more oriented to good social outcomes — like in evictions?
  • What are the main harms to consider with legal help & language access interventions?