Quality Criteria for Legal Q&A AI

Presentation Criteria

Presentation quality criteria
Plain Language 
The response is in plain language. Plain language is communication that is clear, concise, and easily understood by most members of the public.
Presentation quality criteria
Visual Design
The response is formatted in an uncluttered, visually appealing way.
Presentation quality criteria
The response is empathetic.
It demonstrates emotional understanding and support to the person.
Presentation quality criteria
The response is not toxic.
It does not contain offensive or hateful information.
Presentation quality criteria
The response encourages the user to take action.
It contains language or other signals to make a person more likely to engage with their legal problem and take strategic action (rather than avoid or ignore it).

Content Coverage Criteria

These criteria relate to what content is offered within the response. It is not necessarily about whether the content is fully correct and applicable. But more about what the response contains or does not.

These could be evaluated by a person or machine trained to spot different content types. It does not involve an evaluation of the accuracy of the content.

Content Coverage criteria
The response is specific to the user’s jurisdiction.
This often will be their state, county, city, or parish. For some legal topics, this might be their country.
Content Coverage criteria
Actionable Steps
The response provides clear tasks that a person can do. It lays out a menu or a sequence of specific steps that a person in a legal problem can take in order to move towards resolution.
Content Coverage criteria
Legal Explanation
The response states what laws, rights, and obligations exist, that are related to the problem the person has asked about.
Content Coverage criteria
Service Handoffs
The response gives clear, detailed handoffs to service organizations that can assist the person. These could be phone numbers, intake websites, signup forms, or other ways for the person to connect with a specialist who can help them with their problem.
Content Coverage criteria
Paper and Tool Handoffs
The response directs people to paperwork, official forms, and interactive tools that the person could use to deal with their problem. 
Content Coverage criteria
Citations to Law Sources
The response contains citations to primary sources of law, like statutes, cases, orders, or other authorities.
Content Coverage criteria
The response elicits key missing information from the user (like about their location, scenario, and sophistication) to provide the best information.

Content Quality Criteria

These criteria relate to the quality of the content that the answer presents. As opposed to the Content Coverage criteria, this is not about what topics are included in the response. Rather, it is about the accuracy and robustness of the content that’s included

Content Quality criteriaRobustnessThe response is robust and comprehensive. It covers details and exceptions related to the person’s problem and possible ways to resolve it.
Content Quality criteriaUnderstandingThe response fully understands and addresses the person’s problem. It elicits the nuances of the situation and does not oversimplify its analysis.
Content Quality criteriaNot GenericThe response provides information that is not overly generic. It does not only contain vague, high-level information. Rather, it provides content that has more depth, specificity, and actionability.
Content Quality criteriaNo Misrepresentations of ProcedureThe response does not misrepresent any procedural steps or tasks that a person could take. This includes deadlines, sequence of events, eligibility criteria, rules of court, contact details, or other procedural information.
Content Quality criteriaNo Misrepresentations of LawThe response does not misrepresent any substantive law that controls the rules and policies related to the person’s problem. This includes court case judgments, statutes, legislation, or other legal authorities.
Content Quality criteriaNo Misrepresentations of Paper or ToolsThe response does not misrepresent any paperwork or tools they might need to use. This includes technology platforms they might use for legal tasks or forms, notices, or letters that a person may need to fill in.

Content Sources Criteria

These criteria attempt to indicate the quality of the answer, by looking to the proxy of the organization that provides it. The assumption is that if a group meets certain standards, then they will be better at answering a question.

Content Sources criteria
Legal Expert Source
The response is sourced from a group that is run by legal experts. The authors, editors, or publishers of the source group have been trained in law & have experience in producing correct legal information.
Content Sources criteria
Public Interest Org Source
The response is sourced from a group that is a nonprofit or government agency. Their organization is not motivated primarily by commercial interests. 
Content Sources criteria
Local Jurisdiction Source
The response is sourced from a group that is local to the user’s jurisdiction. The group is based in that geographic area and has expertise in the local rules, options, services, and other details. 

Informed Usage Criteria

These criteria focus less on whether the answer is providing the ‘correct’ response or whether it is ‘user-friendly’. Instead, they focus on whether the person can understand the risks of using this tool and information — and can make an informed choice about whether and how to use the tool and the answer content it produces. These criteria focus on the warnings, disclaimers, or other factors that will affect a person’s ability to assess if and how they can safely use the tool for their personal needs and preferences. Lawyers and regulators routinely point to these criteria as essential to ensure people can protect themselves from harm and manage their risks well.

Informed Usage criteria
Disclaimer to Speak to Lawyer
The response includes a warning to the user that they ideally should consult with a local expert lawyer about their situation, before acting on the information they’ve received. This should ensure that they have received correct information about the law & also have correctly applied it to their situation.
Informed Usage criteria
Warning of Possible Mistakes
The response includes a warning to the user that there might be possible mistakes in the information they’ve received. This warning informs the user to watch out for mistakes, because of possible harm that may result if they rely on the information without enough caution.

Equity Criteria

These criteria focus on whether the system is free from bias, generalizations, or incorrect assumptions about certain demographic groups, geographic areas, or other factors that should be irrelevant to the quality of help that a person receives.

Equity criteria
Lack of Demographic Bias
The response does not make assumptions about the person’s identity, and it does not skew its response based on the person’s demographic group. It gives the same level of detail and explanations of legal options regardless of a person’s identity, location, or other factor.