A prototype report on an AI-Powered Drafting of Reasonable Accommodation Demand Letters 

AI for Legal Help, Legal Design Lab, 2025

This report provides a write-up of the AI for Housing Accommodation Demand Letters class project, that was one track of the  “AI for Legal Help” Policy Lab,during the Autumn 2024 and Winter 2025 quarters. This class involved work with legal and court groups that provide legal help services to the public, to understand where responsible AI innovations might be possible and to design and prototype initial solutions, as well as pilot and evaluation plans.

One of the project tracks was on Demand Letters. An interdisciplinary team of Stanford University students partnered with the Legal Aid Society of San Bernardino (LASSB) to address a critical bottleneck in their service delivery: the time-consuming process of drafting reasonable accommodation demand letters for tenants with disabilities. 

This report details the problem identified by LASSB, the proposed AI-powered solution developed by the student team, and recommendations for future development and implementation. 

We share it in the hopes that legal aid and court help center leadership might also be interested in exploring responsible AI development for demand letters, and that funders, researchers, and technologists might collaborate on developing and testing successful solutions for this task.

Thank you to students in this team: Max Bosel, Adam Golomb, Jay Li, Mitra Solomon, and Julia Stroinska. And a big thank you to our LASSB colleagues: Greg Armstrong, Pablo Ramirez, and more

The Housing Accommodation Demand Letter Task

The Legal Aid Society of San Bernardino (LASSB) is a nonprofit law firm providing free legal services to low-income residents in San Bernardino County, California. Among their clients are tenants with disabilities who often need reasonable accommodation demand letters to request changes from landlords (for example, allowing a service animal in a “no pets” building). 

These demand letters are formal written requests asserting tenants’ rights under laws like the Americans with Disabilities Act (ADA) and Fair Housing Act (FHA). They are crucial for tenants to secure accommodations and avoid eviction, but drafting them properly is time-consuming and requires legal expertise. LASSB faces overwhelming demand for help in this area – its hotline receives on the order of 100+ calls per day from tenants seeking assistance. 

However, LASSB has only a handful of intake paralegals and housing attorneys available, meaning many callers must wait a long time or never get through. In fact, LASSB serves around 9–10,000 clients per year via the hotline, yet an estimated 15,000 additional calls never reach help due to capacity limits. Even for clients who do get assistance, drafting a personalized, legally sound letter can take hours of an attorney’s time. With such limited staffing, LASSB’s attorneys are stretched thin, and some eligible clients may end up without a well-crafted demand letter to assert their rights.

LASSB presented their current workflow and questions about AI opportunities in September 2024, and a team of students in AI for Legal Help formed to partner on this task and explore an AI-powered solution. 

The initial question from LASSB was whether we could leverage recent advances in AI to draft high-quality demand letter templates automatically, thereby relieving some burden on staff and improving capacity to serve clients. The goal was to have an AI system gather information from the client and produce a solid first draft letter that an attorney could then quickly review and approve. By doing so, LASSB hoped to streamline the demand-letter workflow – saving attorney time, reducing errors or inconsistencies, and ensuring more clients receive help. 

Importantly, any AI agent would not replace attorney judgment or final sign-off. Rather, it would act as a virtual assistant or co-pilot: handling the routine drafting labor while LASSB staff maintain complete control over the final output. Key objectives set by the partner included improving efficiency, consistency, and accessibility of the service, while remaining legally compliant and user-friendly. In summary, LASSB needed a way to draft reasonable accommodation letters faster without compromising quality. 

After two quarters of work, the class teams proposed a Demand Letter AI system, creating a prototype AI agent that would interview clients about their situation and automatically generates a draft accommodation request letter. This letter would cite the relevant laws and follow LASSB’s format, ready for an attorney’s review. By adopting such a tool, LASSB hopes to minimize the time attorneys spend on repetitive drafting tasks and free them to focus on providing direct counsel and representation. The remainder of this report details the use case rationale, the current vs. envisioned workflow, the technical prototyping process, evaluation approach, and recommendations for next steps in developing this AI-assisted demand letter system.

Why is the Demand Letter Task a good fit for AI?

Reasonable accommodation demand letters for tenants with disabilities were chosen as the focus use case for several reasons. 

The need is undeniably high: as noted, LASSB receives a tremendous volume of housing-related calls, and many involve disabled tenants facing issues like a landlord refusing an exception to a policy (no-pets rules, parking accommodations, unit modifications, etc.). These letters are often the gateway to justice for such clients – a well-crafted letter can persuade a landlord to comply without the tenant ever needing to file a complaint or lawsuit. Demand letters are a high-impact intervention that can prevent evictions and ensure stable housing for vulnerable tenants. Focusing on this use case meant the project could directly improve outcomes for a large number of people, aligning with LASSB’s mission of “justice without barriers – equitable access for all.” 

At the same time, drafting each letter individually is labor-intensive. Attorneys must gather the details of the tenant’s disability and accommodation request, explain the legal basis (e.g. FHA and California law), and compose a polite but firm letter to the landlord. With LASSB’s staff attorneys handling heavy caseloads, these letters sometimes get delayed or delegated to clients themselves to write (with mixed results). Inconsistent quality and lack of time for thorough review are known issues. This use case presented a clear opportunity for AI to assist to improve the consistency and quality of the letter itself. 

The task of writing letters is largely document-generation – a pattern that advanced language models are well-suited for. Demand letters follow a relatively standard structure (explain who you are, state the request, cite laws, etc.), and LASSB already uses templates and boilerplate language for some sections. This means an AI could be trained or prompted to follow that format and fill in the specifics for each client. By leveraging an AI to draft the bulk of the text, each letter could be produced much faster, with the model handling the repetitive phrasing and legal citations while the attorney only needs to make corrections or additions. 

Crucially, using AI here could increase LASSB’s capacity. Rather than an attorney spending, say, 2-3 hours composing a letter from scratch, the AI might generate a solid draft in minutes, requiring perhaps 15 minutes of review and editing. The project team estimated that integrating an AI tool into the workflow could save on the order of 1.5–2.5 hours per client in total staff time. Scaled over dozens of cases, those saved hours mean more clients served and shorter wait times for help. This efficiency gain is attractive to funders and legal aid leaders because it stretches scarce resources further. 

AI can help enforce consistency and accuracy. It would use the same approved legal language across all letters, reducing the chance of human error or omissions in the text. For clients, this translates into a more reliable service – they are more likely to receive a well-written letter regardless of which attorney or volunteer is assisting them. 

The reasonable accommodation letter use case was selected because it sits at the sweet spot of high importance and high potential for automation. It addresses a pressing need for LASSB’s clients (ensuring disabled tenants can assert their rights) and plays to AI’s strengths (generating structured documents from templates and data). By starting with this use case, the project aimed to deliver a tangible, impactful tool that could quickly demonstrate value – a prototype AI assistant that materially improves the legal aid workflow for a critical class of cases.

---

Workflow Vision:

From Current Demand Letter Process to Future AI-Human Collaboration

To understand the impact of the proposed solution, it’s important to compare the current human-driven workflow of creating Demand Letters and the envisioned future workflow where an AI assistant is integrated. Below, we outline the step-by-step process today and how it would change with the AI prototype in place. 

Current Demand Letter Workflow (Status Quo)

When a tenant with a disability encounters an issue with their landlord (for example, the landlord is refusing an accommodation or threatening eviction over a disability-related issue), the tenant must navigate several steps to get a demand letter:

• Initial Intake Call: The tenant contacts LASSB’s hotline and speaks to an intake call-taker (often a paralegal). The tenant explains their situation and disability, and the intake worker records basic information and performs eligibility screening (checking income, conflict of interest, etc.). If the caller is eligible and the issue is within LASSB’s scope, the case is referred to a housing attorney for follow-up.
• Attorney Consultation: The tenant then has to repeat their story to a housing attorney (often days later). The attorney conducts a more in-depth interview about the tenant’s disability needs and the accommodation they seek. At this stage, the attorney determines if a reasonable accommodation letter is the appropriate course of action. (If not – for example, if the problem requires a different remedy – the attorney would advise on next steps outside the demand letter process.)
• Letter Drafting: If a demand letter is warranted, the process for drafting it is currently inconsistent. In some cases, the attorney provides the client with a template or “self-help” packet on how to write a demand letter and asks the client to draft it themselves. In other cases, the attorney or a paralegal might draft the letter on the client’s behalf. With limited time, attorneys often cannot draft every letter from scratch, so the level of assistance varies. Clients may end up writing the first draft on their own, which can lead to incomplete or less effective letters. (One LASSB attorney noted that tenants frequently have to “explain their story at least twice” – to the intake worker and attorney – “and then have to draft/send the demand letter with varying levels of help”.)
• Review and Delivery: Ideally, if the client drafts the letter, they will bring it back for the attorney to review and approve. Due to time pressures, however, attorney review isn’t always thorough, and sometimes letters go out without a detailed legal polish. Finally, the tenant sends the demand letter to the landlord, either by mail or email (or occasionally LASSB sends it on the client’s behalf). At this point, the process relies on the landlord’s response; LASSB’s involvement usually ends unless further action (like litigation) is needed.

This current workflow places a heavy burden on the tenant and the attorney. The tenant must navigate multiple conversations and may end up essentially drafting their own legal letter. The attorney must spend time either coaching the client through writing or drafting the letter themselves, on top of all their other cases. Important information can slip through the cracks when the client is interviewed multiple times by different people. There is also no consistent tracking of what advice or templates were given to the client, leading to variability in outcomes. Overall, the process can be slow (each step often spreads over days or weeks of delay) and resource-intensive, contributing to the bottleneck in serving clients.

---

Proposed AI-Assisted Workflow (Future Vision)

In the reimagined process, an AI agent would streamline the stages between intake and letter delivery, working in tandem with LASSB staff.

After a human intake screens the client, the AI Demand Letter Assistant takes over the interview to gather facts and draft the letter. The attorney then reviews the draft and finalizes the letter for the client to send.

• Post-Intake AI Interview: Once a client has been screened and accepted for services by LASSB’s intake staff, the AI Demand Letter Assistant engages the client in a conversation (via chat or a guided web form; a phone interface could also be possible). The AI introduces itself as a virtual assistant working with LASSB and uses a structured but conversational script to collect all information relevant to the accommodation request. This includes the client’s basic details, details of the disability and needed accommodation, the landlord’s information, and any prior communications or incidents (e.g. if the tenant has asked before or if the landlord has issued notices). The assistant is programmed to use trauma-informed language – it asks questions in a supportive, non-threatening manner and adjusts wording to the client’s comfort, recognizing that relaying one’s disability needs can be sensitive. Throughout the interview, the AI can also perform helpful utilities, such as inserting the current date or formatting addresses correctly, to ensure the data it gathers is ready for a letter.
• Automatic Letter Generation: After the AI has gathered all the necessary facts from the client, it automatically generates a draft demand letter. The generation is based on LASSB-approved templates and includes the proper formal letter format (date, addresses, RE: line, etc.), a clear statement of the accommodation request, and citations to relevant laws/regulations (like referencing the FHA, ADA, or state law provisions that apply). The AI uses the information provided by the client to fill in key details – for example, describing the tenant’s situation (“Jane Doe, who has an anxiety disorder, requests an exception to the no-pets policy to allow her service dog”) and customizing the legal rationale to that scenario. Because the AI has been trained on example letters and legal guidelines, it can include the correct legal language to strengthen the demand. It also ensures the tone remains polite and professional. At the end of this step, the AI has a complete draft letter ready.
• Attorney Review & Collaboration: The draft letter, along with a summary of the client’s input or a transcript of the Q&A, is then forwarded to a LASSB housing attorney for review. The attorney remains the ultimate decision-maker – they will read the AI-drafted letter and check it for accuracy, appropriate tone, and effectiveness. If needed, the attorney can edit the letter (either directly or by giving feedback to the AI to regenerate specific sections). The AI could also highlight any uncertainties (for instance, if the client’s explanation was unclear on a point, the draft might flag that for attorney clarification). Importantly, no letter is sent out without attorney approval, ensuring that professional legal judgment is applied. This human-in-the-loop review addresses ethical duties (attorneys must supervise AI work as they would a junior staffer) and maintains quality control. In essence, the AI does the first 90% of the drafting, and the attorney provides the final 10% refinement and sign-off.
• Delivery and Follow-Up: After the attorney finalizes the content, the letter is ready to be delivered to the landlord. In the future vision, this could be as simple as clicking a button to send the letter via email or printing it for mailing. (The prototype also floated ideas like integrating with DocuSign or generating a PDF that the client can download and sign.) The client then sends the demand letter to the landlord, formally requesting the accommodation. Ideally, this happens much faster than in the current process – potentially the same day as the attorney consultation, since the drafting is near-instant. LASSB envisioned that the AI might even assist in follow-up: for instance, checking back with the client a couple weeks later to ask if the landlord responded, and if not, suggesting next steps. (This follow-up feature was discussed conceptually, though not implemented in the prototype.) In any case, by the end of the workflow, the client has a professionally crafted letter in hand, and they did not have to write it alone.

The benefits of this AI-human collaboration are significant. It eliminates the awkward gap where a client might be left drafting a letter on their own; instead, the client is guided through questions by the AI and sees a letter magically produced from their answers. It also reduces duplicate interviewing – the client tells their full story once to the AI (after intake), rather than explaining it to multiple people in pieces. 

For the attorney, the time required to produce a letter drops dramatically. Rather than spending a couple of hours writing and editing, an attorney might spend 10–20 minutes reviewing the AI’s draft, tweaking a phrase or two, and approving it. The team’s estimates suggest each case could save on the order of 1.5–2.5 hours of staff time under this new workflow. Those savings translate into lower wait times and the ability for LASSB to assist many more clients in a given period with the same staff. In broader terms, more tenants would receive the help they need, fewer calls would be abandoned, and LASSB’s attorneys could devote more attention to complex cases (since straightforward letters are handled in part by the AI). 

The intended impact is “more LASSB clients have their day in court… more fair and equitable access to justice for all”, as the student team put it – in this context meaning more clients are able to assert their rights through demand letters, addressing issues before they escalate. The future vision sees the AI prototype seamlessly embedded into LASSB’s service delivery: after a client is screened by a human, the AI takes on the heavy lifting of information gathering and document drafting, and the human attorney ensures the final product meets the high standards of legal practice. This collaboration could save time, improve consistency, and ultimately empower more tenants with disabilities to get the accommodations they need to live safely and with dignity.

---

Technical Approach and Prototyping: What We Built and How It Works

With the use case defined, the project team proceeded to design and build a working prototype AI agent for demand letter drafting. This involved an iterative process of technical development, testing, and refinement over two academic quarters. In this section, we describe the technical solution – including early prototypes, the final architecture, and how the system functions under the hood.

Early Prototype and Pivot 

In Autumn 2024, the team’s initial prototype focused on an AI intake interviewing agent (nicknamed “iNtake”) as well as a rudimentary letter generator. They experimented with a voice-based assistant that could talk to clients over the phone. Using tools like Twilio (for telephony and text messaging) and Google’s Dialogflow/Chatbot interfaces, they set up a system where a client could call a number and interact with an AI-driven phone menu. The AI would ask the intake questions in a predefined script and record the answers. 

Behind the scenes, the prototype leveraged a large language model (LLM) – essentially an AI text-generation engine – to handle the conversational aspect. The team used a model configuration referred to as “gemini-1.5-flash”, which was integrated into the phone chatbot. 

This early system demonstrated some capabilities (it could hold a conversation and hand off to a human if needed), but also revealed significant challenges. The script was over 100 questions long and not trauma-informed – users found it tedious and perhaps impersonal. Additionally, the AI sometimes struggled with the decision-tree logic of intake. 

After several iterations and feedback from instructors and LASSB, the team decided to pivot. They narrowed the scope to concentrate on the Demand Letter Agent – a chatbot that would come after intake to draft the letter. The phone-based intake AI became a separate effort (handled by another team in Winter 2025), while our team focused on the letter generator. 

Final Prototype Design

The Winter 2025 team built upon the fall work to create a functioning AI chat assistant for demand letters. The prototype operates as an interactive chatbot that can be used via a web interface (in testing, it was run on a laptop, but it could be integrated into LASSB’s website or a messaging platform in the future). Here’s how it works in technical terms.

The AI agent was developed using a generative Large Language Model (LLM) – similar to the technology behind GPT-4 or other modern conversational AIs. This model was not trained from scratch by the team (which would require huge data and compute); instead, the team used a pre-existing model and focused on customizing it through prompt engineering and providing domain-specific data. In practical terms, the team created a structured “AI playbook” or prompt script that guides the model step-by-step to perform the task.

Data and Knowledge Integration

One of the first steps was gathering all relevant reference material to inform the AI’s outputs. The team collected LASSB’s historical demand letters (redacted for privacy), which provided examples of well-written accommodation letters. They also pulled in legal sources and guidelines: for instance, the U.S. Department of Justice’s guidance memos on reasonable accommodations, HUD guidelines, trauma-informed interviewing guidelines, and lists of common accommodations and impairments. These documents were used to refine the AI’s knowledge. 

Rather than blindly trusting the base model, the team explicitly incorporated key legal facts – such as definitions of “reasonable accommodation” and the exact language of FHA/FEHA requirements – into the AI’s prompt or as reference text the AI could draw upon. Essentially, the AI was primed with: “Here are the laws and an example demand letter; now follow this format when drafting a new letter.” This helped ensure the output letters would be legally accurate and on-point.

Prompt Engineering

The heart of the prototype is a carefully designed prompt/instruction set given to the AI model. The team gave the AI a persona and explicit instructions on how to conduct the conversation and draft the letter. For example, the assistant introduces itself as “Sofia, the Legal Aid Society of San Bernardino’s Virtual Assistant” and explains its role to the client (to help draft a letter). The prompt includes step-by-step instructions for the interview: ask the client’s name, ask what accommodation they need, confirm details, etc., in a logical order (it’s almost like a decision-tree written in natural language form). A snippet of the prompt (from the “Generative AI playbook”) is shown below:

Excerpt from the AI assistant’s instruction script. The agent is given a line-by-line guide to greet the client, collect information (names, addresses, disability details, etc.), and even call a date-time tool to insert the current date for the letter. 

The prompt also explicitly instructs the AI on legal and ethical boundaries. For instance, it was told: “Your goal is to write and generate a demand letter for reasonable accommodations… You do not provide legal advice; you only assist with drafting the letter.”. This was crucial to prevent the AI from straying into giving advice or making legal determinations, which must remain the attorney’s domain. By iteratively testing and refining this prompt, the team taught the AI to stay in its lane: ask relevant questions, be polite and empathetic, and focus on producing the letter.

Trauma-Informed and Bias-Mitigation Features

A major design consideration was ensuring the AI’s tone and behavior were appropriate for vulnerable clients. The team trained the AI (through examples and instructions) to use empathetic language – e.g., thanking the client for sharing information, acknowledging difficulties – and to avoid any phrasing that might come off as judgmental or overly clinical. The AI was also instructed to use the client’s own words when possible and not to press sensitive details unnecessarily. On the technical side, the model was tested for biases. The team used diverse example scenarios to ensure the AI’s responses wouldn’t differ inappropriately based on the nature of the disability or other client attributes. Regular audits of outputs were done to catch any bias. For example, they made sure the AI did not default to male pronouns for landlords or assume anything stereotypical about a client’s condition. These measures align with best practices to ensure the AI’s output is fair and respects all users.

Automated Tools Integration

The prototype included some clever integrations of simple tools to enhance accuracy. One such tool was a date function. In early tests, the AI sometimes forgot to put the current date on the letter or used a generic placeholder. To fix this, the team connected the AI to a utility that fetches the current date. During the conversation, if the user is ready to draft the letter, the AI will call this date function and insert the actual current date into the letter heading. This ensures the generated letter always shows (for example) “May 19, 2023” rather than a hardcoded date. Similarly, the AI was guided to properly format addresses and other elements (it asks for each component like city, state, ZIP and then concatenates them in the letter format). These might seem like small details, but they significantly improve the professionalism of the output.

Draft Letter Generation

Once the AI has all the needed info, it composes the letter in real-time. It follows the structure from the prompt and templates: the letter opens with the date and address, a reference line (“RE: Request for Reasonable Accommodation”), a greeting, and an introduction of the client. Then it lays out the request and the justification, citing the laws, and closes with a polite sign-off. The content of the letter is directly based on the client’s answers. For instance, if the client said they have an anxiety disorder and a service dog, the letter will include those details and explain why the dog is needed. The AI’s legal knowledge ensures that it inserts the correct references to the FHA and California Fair Employment and Housing Act (FEHA), explaining that landlords must provide reasonable accommodations unless it’s an undue burden. 

An example output is shown below:

Sample excerpt from an AI-generated reasonable accommodation letter. In this case, the tenant (Jane Doe) is requesting an exception to a “no pets” policy to allow her service dog. The AI’s draft includes the relevant law citations (FHA and FEHA) and a clear explanation of why the accommodation is necessary. 

As seen in the example above, the AI’s letter closely resembles one an attorney might write. It addresses the landlord respectfully (“Dear Mr. Jones”), states the tenant’s name and address, and the accommodation requested (permission to keep a service animal despite a no-pet policy). It then cites the Fair Housing Act and California law, explaining that these laws require exceptions to no-pet rules as a reasonable accommodation for persons with disabilities. It describes the tenant’s specific circumstances (the service dog helps manage her anxiety, etc.) in a factual and supportive tone. It concludes with a request for a response within a timeframe and a polite thank you. All of this text was generated by the AI based on patterns it learned from training data and the prompt instructions – the team did not manually write any of these sentences for this particular letter, showing the generative power of the AI. The attorney’s role would then be to review this draft. 

In our tests, attorneys found the drafts to be surprisingly comprehensive. They might only need to tweak a phrase or add a specific detail. For example, an attorney might insert a line offering to provide medical documentation if needed, or adjust the deadline given to the landlord. But overall, the AI-generated letters were on point and required only light editing. 

Testing and Iteration

The development of the prototype involved iterative testing and debugging. Early on, the team encountered some issues typical of advanced AI systems and worked to address them.

Getting the agent to perform consistently

Initially, the AI misunderstood its task at times. In the first demos, when asked to draft a letter, the AI would occasionally respond with “I’m sorry, I can’t write a letter for you”, treating it like a prohibited action. This happened because base language models often have safety rules about not producing legal documents. The team resolved this by refining the prompt to clarify that the AI is allowed and expected to draft the letter as part of its role (since an attorney will review it). Once the AI “understood” it had permission to assist, it complied.

Ensuring the agent produced the right output

The AI also sometimes ended the interview without producing the letter. Test runs showed that if the user didn’t explicitly ask for the letter, the AI might stop after gathering info. To fix this, the team adjusted the instructions to explicitly tell the AI that once it has all the information, it should automatically present the draft letter to the client for review. After adding this, the AI reliably output the draft at the end of the conversation.

We sometimes had the agent offering to do unsolicited tasks, like sending an email. That wasn’t in the configuration, but it was improvising off-script.

Un-sticking the agent, caught in a loop

There were issues with the AI getting stuck or repeating itself. For example, in one scenario, the AI began to loop, apologizing and asking the same question multiple times even after the user answered. 

A screenshot from testing shows the AI repeating “Sorry, something went wrong, can you repeat?” in a loop when it hit an unexpected input. These glitches were tricky to debug – the team adjusted the conversation flow and added checks (like if the user already answered, do not ask again), which reduced but did not completely eliminate such looping. We identified that these loops often stemmed from the model’s uncertainty or minor differences in phrasing that weren’t accounted for in the script.

Dealing with fake or inaccurate info

Another issue was occasional hallucinations or extraneous content. For instance, the AI at one point started offering to “email the letter to the landlord” out of nowhere, even though that wasn’t in its instructions (and it had no email capability). This was the model improvising beyond its intended scope. The team addressed this by tightening the prompt instructions, explicitly telling the AI not to do anything with email and to stick to generating the letter text only. After adding such constraints, these hallucinations became rarer.

Getting consistent letter formatting

The formatting of the letter (dates, addresses, signature line) needed fine-tuning. The AI initially had minor formatting quirks (like sometimes missing the landlord’s address or not knowing how to sign off). By providing a template example and explicitly instructing the inclusion of those elements, the final prototype reliably produced a correctly formatted letter with a placeholder for the client’s signature.

Throughout development, whenever an issue was discovered, the team would update the prompt or the data and test again. This iterative loop – test, observe output, refine instructions – is a hallmark of developing AI solutions and was very much present in this project. 

Over time, the outputs improved significantly in quality and reliability. For example, by the end of the Winter quarter, the AI was consistently using the correct current date (thanks to the date tool integration) and writing in a supportive tone (thanks to the trauma-informed training), which were clear improvements from earlier versions. That said, some challenges remained unsolved due to time limits. 

The AI still showed some inconsistent behaviors occasionally – such as repeating a question in a rare case, or failing to recognize an atypical user response (like if a user gave an extremely long-winded answer that confused the model). The team documented these lingering issues so that future developers can target them. They suspected that further fine-tuning of the model or using a more advanced model could help mitigate these quirks. 

In its final state at the end of Winter 2025, the prototype was able to conduct a full simulated interview and generate a reasonable accommodation demand letter that LASSB attorneys felt was about 80–90% ready to send, requiring only minor edits. 

The technical architecture was a single-page web application interfacing with the AI model (running on a cloud AI platform) plus some back-end scripts for the date tool and data storage. It was not yet integrated into LASSB’s production systems, but it provided a compelling proof-of-concept. 

Observers in the final presentation could watch “Sofia” chat with a hypothetical client (e.g., Martin who needed an emotional support animal) and within minutes, produce a letter addressed to the landlord citing the FHA – something that would normally take an attorney a couple of hours. 

Overall, the technical journey of this project was one of rapid prototyping and user-centered adjustment. The team combined off-the-shelf AI technology with domain-specific knowledge to craft a tool tailored for legal aid. They learned how small changes in instructions can greatly affect an AI’s behavior, and they progressively molded the system to align with LASSB’s needs and values. The result is a working prototype of an AI legal assistant that shows real promise in easing the burden of document drafting in a legal aid context.

Evaluation Framework: Testing, Quality Standards, and Lessons Learned

From the outset, the team and LASSB agreed that rigorous evaluation would be critical before any AI tool could be deployed in practice. The project developed an evaluation framework to measure the prototype’s performance and ensure it met both efficiency goals and legal quality standards. Additionally, throughout development the team reflected on broader lessons learned about using AI in a legal aid environment. This section discusses the evaluation criteria, testing methods, and key insights gained. Quality Standards and Benchmarks: The primary measure of success for the AI-generated letters was that they be indistinguishable (in quality) from letters written by a competent housing attorney. To that end, the team established several concrete quality benchmarks:

• No “Hallucinations”: The AI draft should contain no fabricated facts, case law, or false statements. All information in the letter must come from the client’s provided data or be generally accepted legal knowledge. For example, the AI should never cite a law that doesn’t exist or insert details about the tenant’s situation that the tenant didn’t actually tell it. Attorneys reviewing the letters specifically check for any such hallucinated content.
• Legal Accuracy: Any legal assertions in the letter (e.g. quoting the Fair Housing Act’s requirements) must be precisely correct. The letter should not misstate the law or the landlord’s obligations. Including direct quotes or citations from statutes/regulations was one method used to ensure accuracy. LASSB attorneys would verify that the AI correctly references ADA, FHA, FEHA, or other laws as applicable.
• Proper Structure and Tone: The format of the letter should match what LASSB attorneys expect in a formal demand letter. That means: the letter has a date, addresses for both parties, a clear subject line, an introduction, body paragraphs that state the request and legal basis, and a courteous closing. The tone should be professional – firm but not aggressive, and certainly not rude. One benchmark was that an AI-drafted letter “reads like” an attorney’s letter in terms of formality and clarity. If an attorney would normally include or avoid certain phrases (for instance, saying “Thank you for your attention to this matter” at the end, or avoiding contractions in a formal letter), the AI’s output is expected to do the same.
• Completeness: The letter should cover all key points necessary to advocate for the client. This includes specifying the accommodation being requested, briefly describing the disability connection, citing the legal right to the accommodation, and possibly mentioning an attached verification if relevant. An incomplete letter (one that, say, only requests but doesn’t cite any law) would not meet the standard. Attorneys reviewing would ensure nothing crucial was missing from the draft.

In addition to letter quality, efficiency metrics were part of the evaluation. The team intended to log how long the AI-agent conversation took and how long the model took to generate the letter, aiming to show a reduction in total turnaround time compared to the status quo. Another metric was the effect on LASSB’s capacity: for example, could implementing this tool reduce the number of calls that drop off due to long waits? In theory, if attorneys spend less time per client, more calls can be returned. The team proposed tracking number of clients served before and after deploying the AI as a long-term metric of success. 

Evaluation Methods

To assess these criteria, the evaluation plan included several components.

Internal Performance Testing

The team performed timed trials of the AI system. They measured the duration of a full simulated interview and letter draft generation. In later versions, the interview took roughly 10–15 minutes (depending on how much detail the client gives), and the letter was generated almost instantly thereafter (within a few seconds). They compared this to an estimate of human drafting time. These trials demonstrated the raw efficiency gain – a consistent turnaround of under 20 minutes for a draft letter, which is far better than the days or weeks it might take in the normal process. They also tracked if any technical slowdowns occurred (for instance, if the AI had to call external tools like the date function, did that introduce delays? It did not measurably – the date lookup was near-instant).

Expert Review (Quality Control)

LASSB attorneys and subject matter experts were involved in reviewing the AI-generated letters. The team conducted sessions where an attorney would read an AI draft and score it on accuracy, tone, and completeness. The feedback from these reviews was generally positive – attorneys found the drafts surprisingly thorough. They did note small issues (e.g., “we wouldn’t normally use this phrasing” or “the letter should also mention that the client can provide a doctor’s note if needed”). 

These observations were fed back into improving the prompt. The expert review process is something that would continue regularly if the tool is deployed: LASSB could institute, say, a policy that attorneys must double-check every AI-drafted letter and log any errors or required changes. Over time, this can be used to measure whether the AI’s quality is improving (i.e., fewer edits needed).

User Feedback

Another angle was evaluating the system’s usability and acceptance by both LASSB staff and clients. The team gathered informal feedback from users who tried the chatbot demo (including a couple of law students role-playing as clients). They also got input from LASSB’s intake staff on whether they felt such a chatbot would be helpful. In a deployed scenario, the plan is to collect structured feedback via surveys. For example, clients could be asked if they found the virtual interview process easy to understand, and attorneys could be surveyed on their satisfaction with the draft letters. High satisfaction ratings would indicate the system is meeting needs, whereas any patterns of confusion or dissatisfaction would signal where to improve (perhaps the interface or the language the AI uses).

Long-term Monitoring

The evaluation framework emphasizes that evaluation isn’t a one-time event. The team recommended continuous monitoring if the prototype moves to production. This would involve regular check-ins (monthly or quarterly meetings) among stakeholders – the legal aid attorneys, paralegals, technical team, etc. – to review how things are going. They could review statistics (number of letters generated, average time saved) and any incidents (e.g., “the AI produced an incorrect statement in a letter on March 3, we caught it in review”). This ongoing evaluation ensures that any emerging issues (perhaps a new type of accommodation request the AI wasn’t trained on) are caught and addressed. It’s akin to maintenance: the AI tool would be continually refined based on real-world use data to ensure it remains effective and trustworthy.

Risk and Ethical Considerations

Part of the evaluation also involved analyzing potential risks. The team did a thorough risk, ethics, and regulation analysis in their final report to make sure any deployment of the AI would adhere to legal and professional standards. Some key points from that analysis:

Data Privacy & Security

The AI will be handling sensitive client information (details about disabilities, etc.). The team stressed the need for strict privacy safeguards – for instance, if using cloud AI services, ensuring they are HIPAA-compliant or covered by appropriate data agreements. They proposed measures like encryption of stored transcripts and obtaining client consent for using an AI tool. Any integration with LASSB’s case management (LegalServer) would have to follow data protection policies.

Bias and Fairness

They cautioned that AI models can inadvertently produce biased outputs if not properly checked. For example, might the AI’s phrasing be less accommodating to a client with a certain type of disability due to training data bias? The mitigation is ongoing bias testing and using a diverse dataset for development. The project incorporated an ethical oversight process to regularly audit letters for any bias or inappropriate language.

Acceptance by Courts/Opposing Parties

A unique consideration for legal documents is whether an AI-drafted letter (or brief) will be treated differently by its recipient. The team noted recent cases of courts being skeptical of lawyers’ use of ChatGPT, emphasizing lawyers’ duty to verify AI outputs. For demand letters (which are not filed in court but sent to landlords), the risk is lower than in litigation, but still LASSB must ensure the letters are accurate to maintain credibility. If a case did go to court, an attorney might need to attest that they supervised the drafting. Essentially, maintaining transparency and trust is important – LASSB might choose to inform clients about the AI-assisted system (to manage expectations) and would certainly ensure any letter that ends up as evidence has been vetted by an attorney.

Professional Responsibility

The team aligned the project with guidance from the American Bar Association and California State Bar on AI in law practice. These guidelines say that using AI is permissible as long as attorneys ensure competence, confidentiality, and no unreasonable fees are charged for it. In practice, that means LASSB attorneys must be trained on how to use the AI tool correctly, must keep client data safe, and must review the AI’s work. The attorney remains ultimately responsible for the content of the letter. The project’s design – always having a human in the loop – was very much informed by these professional standards.

Lessons Learned

Over the course of the project, the team gained valuable insights, both in terms of the technology and the human element of implementing AI in legal services. Some of the key lessons include the following.

AI is an Augmenting Tool, Not a Replacement for Human Expertise

Perhaps the most important realization was that AI cannot replace human empathy or judgment in legal aid. The team initially hoped the AI might handle more of the process autonomously, but they learned that the human touch is irreplaceable for sensitive client interactions. For example, the AI can draft a letter, but it cannot (and should not) decide whether a client should get a letter or what strategic advice to give – that remains with the attorney. Moreover, clients often need empathy and reassurance that an AI cannot provide on its own. As one reflection noted, the AI might be very efficient, “however, we learned that AI cannot replace human empathy, which is why the final draft letter always goes to an attorney for final review and client-centered adjustment.” In practice, the AI assists, and the attorney still personalizes the counsel.

Importance of Partner Collaboration and User-Centered Design

The close collaboration with LASSB staff was crucial. Early on, the team had some misaligned assumptions (e.g., focusing on a technical solution that wasn’t actually practical in LASSB’s context, like the phone intake bot). By frequently communicating with the partner – including weekly check-ins and showing prototype demos – the team was able to pivot and refine the solution to fit what LASSB would actually use. One lesson was to always “keep the end user in mind”. In this case, the end users were both the LASSB attorneys and the clients. Every design decision (from the tone of the chatbot to the format of the output) was run through the filter of “Is this going to work for the people who have to use it?” For instance, the move from a phone interface to a chat interface was influenced by partner feedback that a phone bot might be less practical, whereas a web-based chat that produces a printable letter fits more naturally into their workflow.

Prototype Iteratively and Be Willing to Pivot

The project reinforced the value of an iterative, agile approach. The team did not stick stubbornly to the initial plan when it proved flawed. They gathered data (user feedback, technical performance data) and made a mid-course correction to narrow the project’s scope. This pivot ultimately led to a more successful outcome. The lesson for future projects is to embrace flexibility – it’s better to achieve a smaller goal that truly works than to chase a grand vision that doesn’t materialize. As noted in the team’s retrospective, “Be willing to pivot and challenge assumptions” was key to their progress.

AI Development Requires Cross-Disciplinary Skills

The students came from law and engineering backgrounds, and both skill sets were needed. They had to “upskill to learn what you need” on the fly – for example, law students learned some prompt-engineering and coding; engineering students learned about fair housing law and legal ethics. For legal aid organizations, this is a lesson that implementing AI will likely require new trainings and collaboration between attorneys and tech experts.

AI Output Continues to Improve with Feedback

Another positive lesson was that the AI’s performance did improve significantly with targeted adjustments. Initially, some doubted whether a model could ever draft a decent legal letter. But by the end, the results were quite compelling. This taught the team that small tweaks can yield big gains in AI behavior – you just have to systematically identify what isn’t working (e.g., the AI refusing to write, or using the wrong tone) and address it. It’s an ongoing process of refinement, which doesn’t end when the class ends. The team recognized that deploying an AI tool means committing to monitor and improve it continuously. As they put it, “there is always more that can be done to improve the models – make them more informed, reliable, thorough, ethical, etc.”. This mindset of continuous improvement is itself a key lesson, ensuring that complacency doesn’t set in just because the prototype works in a demo.

Ethical Guardrails Are Essential and Feasible

Initially, there was concern about whether an AI could be used ethically for legal drafting. The project showed that with the right guardrails – human oversight, clear ethical policies, transparency – it is not only possible but can be aligned with professional standards. The lesson is that legal aid organizations can innovate with AI responsibly, as long as they proactively address issues of confidentiality, accuracy, and attorney accountability. LASSB leadership was very interested in the tool but also understandably cautious; seeing the ethical framework helped build their confidence that this could be done in a way that enhances service quality rather than risks it.

In conclusion, the evaluation phase of the project confirmed that the AI prototype can meet high quality standards (with attorney oversight) and significantly improve efficiency. It also surfaced areas to watch – for example, ensuring the AI remains updated and bias-free – which will require ongoing evaluation post-deployment. The lessons learned provide a roadmap for both this project and similar initiatives: keep the technology user-centered, maintain rigorous quality checks, and remember that AI is best used to augment human experts, not replace them. By adhering to these principles, LASSB and other legal aid groups can harness AI’s benefits while upholding their duty to clients and justice.

Next Steps

Future Development, Open Questions, and Recommendations

The successful prototyping of the AI demand letter assistant is just the beginning. Moving forward, there are several steps to be taken before this tool can be fully implemented in production at LASSB. The project team compiled a set of recommendations and priorities for future development, as well as open questions that need to be addressed. Below is an outline of the next steps:

Expand and Refine the Training Data

To improve the AI’s consistency and reliability, the next development team should incorporate additional data sources into the model’s knowledge base. During Winter 2025, the team gathered a trove of relevant documents (DOJ guidance, HUD memos, sample letters, etc.), but not all of this material was fully integrated into the prototype’s prompts.

Organizing and inputting this data will help the AI handle a wider range of scenarios. For example, there may be types of reasonable accommodations (like a request for a wheelchair ramp installation, or an exemption from a parking fee) that were not explicitly tested yet. Feeding the AI examples or templates of those cases will ensure it can draft letters for various accommodation types, not just the service-animal case.

The Winter team has prepared a well-structured archive of resources and notes for the next team, documenting their reasoning and changes made. It includes, for instance, an explanation of why they decided to focus exclusively on accommodation letters (as opposed to tackling both accommodations and modifications in one agent) – knowledge that will help guide future developers so they don’t reinvent the wheel. Leveraging this prepared data and documentation will be a top priority in the next phase.

Improve the AI’s Reliability and Stability

While the prototype is functional, we observed intermittent issues like the AI repeating itself or getting stuck in loops under certain conditions. Addressing these glitches is critical for a production rollout. The recommendation is to conduct deeper testing and debugging of the model’s behavior under various inputs. Future developers might use techniques like adversarial testing – intentionally inputting confusing or complex information to see where the AI breaks – and then adjusting the prompts or model settings accordingly. There are a few specific issues to fix:

• The agent occasionally repeats the same question or answer multiple times (this looping behavior might be due to how the conversation history is managed or a quirk of the model). This needs to be debugged so the AI moves on in the script and doesn’t frustrate the user.
• The agent sometimes fails to recognize certain responses – for example, if a user says “Yeah” instead of “Yes,” will it understand? Ensuring the AI can handle different phrasings and a range of user expressions (including when users might go on tangents or express emotion) is important for robustness.
• Rarely, the agent might still hallucinate or provide an odd response (e.g., referring to sending an email when it shouldn’t). Further fine-tuning and possibly using a more advanced model with better instruction-following could reduce these occurrences. Exploring the underlying model’s parameters or switching to a model known for higher reliability (if available through the AI platform LASSB chooses) could be an option.

One open question is “why” the model exhibits these occasional errors – it’s often not obvious, because AI models are black boxes to some degree. Future work could involve more diagnostics, such as checking the conversation logs in detail or using interpretability tools to see where the model’s attention is going. Understanding the root causes could lead to more systemic fixes. The team noted that sometimes the model’s mistakes had no clear trigger, which is a reminder that continuous monitoring (as described in evaluation) will be needed even post-launch.

Enhance Usability and Human-AI Collaboration Features

The prototype currently produces a letter draft, but in a real-world setting, the workflow can be made even more user-friendly for both clients and attorneys. Several enhancements are recommended:

Editing Interface

Allow the attorney (or even the client, if appropriate) to easily edit the AI-generated letter in the interface. For instance, after the AI presents the draft, there could be an “Edit” button that opens the text in a word processor-like environment. This would save the attorney from having to copy-paste into a separate document. The edits made could even be fed back to the AI (as learning data) to continuously improve it.

Download/Export Options

Integrate a feature to download the letter as a PDF or Word document. LASSB staff indicated they would want the final letter in a standard format for record-keeping and for the client to send. Automating this (the AI agent could fill a PDF template or use a document assembly tool) would streamline the process. One idea is to integrate with LASSB’s existing document system or use a platform like Documate or Gavel (which LASSB uses for other forms) – the AI could output data into those systems to produce a nicely formatted letter on LASSB letterhead.

Transcript and Summary for Attorneys

When the AI finishes the interview, it can provide not just the letter but also a concise summary of the client’s situation along with the full interview transcript to the attorney. The summary could be a paragraph saying, e.g., “Client Jane Doe requests an exception to no-pet policy for her service dog. Landlord: ABC Properties. No prior requests made. Client has anxiety disorder managed by dog.”

Such a summary, generated automatically, would allow the reviewing attorney to very quickly grasp the context without reading the entire Q&A transcript. The transcript itself should be saved and accessible (perhaps downloadable as well) so the attorney can refer back to any detail if needed. These features will decrease the need for the attorney to re-interview the client, thus preserving the efficiency gains.

User Interface and Guidance

On the client side, ensure the chat interface is easy to use. Future improvements could include adding progress indicators (to show the client how many questions or sections are left), the ability to go back and change an answer, or even a voice option for clients who have difficulty typing (this ties into accessibility, discussed next). Essentially, polish the UI so that it is client-friendly and accessible.

Integration into LASSB’s Workflow 

In addition to the front-end enhancements, the tool should be integrated with LASSB’s backend systems. A recommendation is to connect the AI assistant to LASSB’s case management software (LegalServer) via API. This way, when a letter is generated, a copy could automatically be saved to the client’s case file in LegalServer. It could also pull basic info (like the client’s name, address) from LegalServer to avoid re-entering data. Another integration point is the hotline system – if in the future the screening AI is deployed, linking the two AIs could be beneficial (for example, intake answers collected by the screening agent could be passed directly to the letter agent, so the client doesn’t repeat information). These integrations, while technical, would ensure the AI tool fits seamlessly into the existing workflow rather than as a stand-alone app.

Broaden Accessibility and Language Support

San Bernardino County has a diverse population, and LASSB serves many clients for whom English is not a first language or who have disabilities that might make a standard chat interface challenging. Therefore, a key next step is to add multilingual capabilities and other accessibility features. The priority is Spanish language support, as a significant portion of LASSB’s client base is Spanish-speaking. This could involve developing a Spanish version of the AI agent – using a bilingual model or translating the prompt and output. The AI should ideally be able to conduct the interview in Spanish and draft the letter in Spanish, which the attorney could then review (noting that the final letter might need to be in English if sent to an English-speaking landlord, but at least the client interaction can be in their language). 

In addition, for clients with visual impairments, the interface should be compatible with screen readers (text-to-speech for the questions, etc.), and for those with low literacy or who prefer oral communication, a voice interface could be offered (perhaps reintroducing a refined version of the phone-based system, but integrated with the letter agent’s logic). Essentially, the tool should follow universal design principles so that no client is left out due to the technology format. This may require consulting accessibility experts and doing user testing with clients who have disabilities. 

Plan for Deployment and Pilot Testing

Before a full rollout, the team recommends a controlled pilot phase. In a pilot, a subset of LASSB staff and clients would use the AI tool on actual cases (with close supervision). Data from the pilot – success stories, any problems encountered, time saved metrics – should be collected and evaluated. This will help answer some open questions, such as: 

• How do clients feel about interacting with an AI for part of their legal help? 
• Does it change the attorney-client dynamic in any way? 
• Are there cases where the AI approach doesn’t fit well (for instance, if a client has multiple legal issues intertwined, can the AI handle the nuance or does it confuse things)? 

These practical considerations will surface in a pilot. The pilot can also inform best practices for training staff on using the tool. Perhaps attorneys need a short training session on how to review AI drafts effectively, or intake staff need a script to explain to clients what the AI assistant is when transferring them. Developing guidelines and training materials is part of deployment. Additionally, during the pilot, establishing a feedback loop (maybe a weekly meeting to discuss all AI-drafted letters that week) will help ensure any kinks are worked out before scaling up. 

Address Open Questions and Long-Term Considerations

Some broader questions remain as this project moves forward.

How to Handle Reasonable Modifications

The current prototype focuses on reasonable accommodations (policy exceptions or services). A related need is reasonable modifications (physical changes to property, like installing a ramp). Initially, the team planned to include both, but they narrowed the scope to accommodations for manageability. Eventually, it would be beneficial to expand the AI’s capabilities to draft modification request letters as well, since the legal framework is similar but not identical. This might involve adding a branch in the conversation: if the client is requesting a physical modification, the letter would cite slightly different laws (e.g., California Civil Code related to modifications) and possibly include different information (like who will pay for the modification, etc.). The team left this as a future expansion area. In the interim, LASSB should be aware that the current AI might need additional training/examples before it can reliably handle modification cases.

Ensuring Ongoing Ethical Compliance

As the tool evolves, LASSB will need to regularly review it against ethical guidelines. For instance, if State Bar rules on AI use get updated, the system’s usage might need to be adjusted. Keeping documentation of how the AI works (so it can be explained to courts if needed) will be important. Questions like “Should clients be informed an AI helped draft this letter?” might arise – currently the plan would be to disclose if asked, but since an attorney is reviewing and signing off, the letter is essentially an attorney work product. LASSB might decide internally whether to be explicit about AI assistance or treat it as part of their workflow like using a template.

Maintenance and Ownership 

Who will maintain the AI system long-term? The recommendation is that LASSB identify either an internal team or an external partner (perhaps continuing with Stanford or another tech partner) to assume responsibility for piloting and updates.

AI models and integrations require maintenance – for example, if new housing laws pass, the model/prompt should be updated to include that. If the AI service (API) being used releases a new version that’s better/cheaper, someone should handle the upgrade. Funding might be needed for ongoing API usage costs or server costs. Planning for these practical aspects will ensure the project’s sustainability.

Scaling to Other Use Cases

If the demand letter agent proves successful, it could inspire similar tools for other high-volume legal aid tasks (for instance, generating answers to eviction lawsuits or drafting simple wills). One open question is how easily the approach here can be generalized. The team believes the framework (AI + human review) is generalizable, but each new use case will require its own careful curation of data and prompts. 

The success in the housing domain suggests LASSB and Stanford may collaborate to build AI assistants for other domains in the future (like an Unlawful Detainer Answer generator, etc.). This project can serve as a model for those efforts.

Finally, the team offered some encouraging closing thoughts: The progress so far shows that a tool like this “could significantly improve the situation and workload for staff at LASSB, allowing many more clients to receive legal assistance.” There is optimism that, with further development, the AI assistant can be deployed and start making a difference in the community. However, they also caution that “much work remains before this model can reach the deployment phase”. 

It will be important for future teams to continue with the same diligent approach – testing, iterating, and addressing the AI’s flaws – rather than rushing to deploy without refinement. The team emphasized a balance of excitement and caution: AI has great potential for legal aid, but it must be implemented thoughtfully. The next steps revolve around deepening the AI’s capabilities, hardening its reliability, improving the user experience, and carefully planning a real-world rollout. By following these recommendations, LASSB can move from a successful prototype to a pilot and eventually to a fully integrated tool that helps their attorneys and clients every day. The vision is that in the near future, a tenant with a disability in San Bernardino can call LASSB and, through a combination of compassionate human lawyers and smart AI assistance, quickly receive a strong demand letter that protects their rights – a true melding of legal expertise and technology to advance access to justice.

With continued effort, collaboration, and care, this prototype AI agent can become an invaluable asset in LASSB’s mission to serve the most vulnerable members of the community. The foundation has been laid; the next steps will bring it to fruition.