This month, our team commenced interviews with landlord-tenant subject matter experts, including court help staff, legal aid attorneys, and hotline operators. These experts are comparing and rating various AI responses to commonly asked landlord-tenant questions that individuals may get when they go online to find help.
Our team has developed a new ‘Battle Mode’ of our rating/classification platform Learned Hands. In a Battle Mode game on Learned Hands, experts compare two distinct AI answers to the same user’s query and determine which one is superior. Additionally, we have the experts speak aloud as they are playing, asking that they articulate their reasoning. This allows us to gain insights into why a particular response is deemed good or bad, helpful or harmful.
Our group will be publishing a report that evaluates the performance of various AI models in answering everyday landlord-tenant questions. Our goal is to establish a standardized approach for auditing and benchmarking AI’s evolving ability to address people’s legal inquiries. This standardized approach will be applicable to major AI platforms, as well as local chatbots and tools developed by individual groups and startups. By doing so, we hope to refine our methods for conducting audits and benchmarks, ensuring that we can accurately assess AI’s capabilities in answering people’s legal questions.
Instead of speculating about potential pitfalls, we aim to hear directly from on-the-ground experts about how these AI answers might help or harm a tenant who has gone onto the Internet to problem-solve. This means regular, qualitative sessions with housing attorneys and service providers, to have them closely review what AI is telling people when asked for information on a landlord-tenant problem. These experts have real-world experience in how people use (or don’t) the information they get online, from friends, or from other experts — and how it plays out for their benefit or their detriment.
We also believe that regular review by experts can help us spot concerning trends as early as possible. AI answers might change in the coming months & years. We want to keep an eye on the evolving trends in how large tech companies’ AI platforms respond to people’s legal help problem queries, and have front-line experts flag where there might be a big harm or benefit that has policy consequences.
Stay tuned for the results of our expert-led rating games and feedback sessions!
If you are a legal expert in landlord-tenant law, please sign up to be one of our expert interviewees below.
From a team in the Justice By Design: Eviction Class, 2022.
I: Overview of Activities
Our policy lab interviewed sixteen tenants, navigators, and landlords across the country, learning from their experiences and hearing their ideas. We asked general questions about their experiences with eviction, their experiences with seeking out help, and their ideas for change.
We synthesized interviews by creating personas, user journeys, and visual representations of salient moments gleaned from the interviews.
Finally, we shared common findings to capture pervasive issues and suggest potential reforms.
II: Problems identified based off interviews with tenants
Informal evictions
Many tenants described falling behind on rent and feeling that they had to move out, even before they had been served with any formal eviction documents. Landlords often don’t follow proper notice procedures for eviction, telling their tenants to pay what they owe or start planning to move out. Considering a pervasive fear of the legal system, as discussed below, it is difficult to imagine tenants being empowered to hold their landlords accountable for breaking the law.
Especially for tenants behind on rent, many lack a feeling of agency to look for resources. They assume that because they are behind on rent, they will not have any recourse to resist displacement.
The fact that many evictions occur informally presents unique challenges for policy implementation. Eviction reforms centered around courts are common, but legal and court reforms will not affect the experiences of those evicted extralegally. These experiences highlight the need for empowering interventions that occur before the eviction experience; tenants need to know of their rights and resources before a housing scare occurs. Any intervention that does not reach clients pre-eviction may be too late.
Tenant Story: John John was informally evicted from his home in San Francisco. Due to local tenant protections, John very likely could have received legal aid—if he knew where to look. But John was evicted informally; he was told to vacate by his landlord, without being provided any proper legal notice. John was recovering from injuries he sustained during an accident, so he did not feel that he had the ability to look for any financial or legal resources. Unable to make up the rent he owed, John and his family had to move out. They were able to live temporarily with friends and family until they found a new place to live. John’s story is a prime example of how even when robust legal or financial resources exist, these resources provide no recourse to informally evicted tenants who lack awareness of their options. Ensuring that tenants are informed of their rights and resources before crisis occurs is critical.
Complex eviction notices
Receiving a Notice to Quit or an eviction summons couldbe a potential point of intervention; these notices ideally would tell tenants: (1) why they are receiving the notice; (2) how they can respond to the notice; and (3) resources they can seek if they need assistance.
Formal eviction notices are far from this ideal. To most tenants, they appear to be warnings that they need to leave, rather than indicators that they have options as part of an ongoing process.
Notices tend to be written in confusing English, and are often not served in foreign languages. Some states have attempted to simplify eviction notices. In Massachusetts, for example, an eviction summons gives the tenant a court date. Getting to court can be difficult, but being given a date and location seems easier to comply with than the requirement of making an official legal filing. Greater Boston Legal Services has a free online service that prompts tenants with questions to answer in plain English, then creates a form that tenants can use in Housing Court to help them defend themselves. Instead of forcing people to file an official Answer, giving tenants the option to fill out an online form where they can explain their situation could be much more tenant-friendly.
We also learned that the landlord-tenant relationship is becoming increasingly bureaucratized. Many tenants live not under mom-and-pop landlords, but rather under large, impersonal property management companies. These companies can churn out Notices to Quit summarily after tenants fall behind on rent—even if they fall behind for just a few days. Tenants feel slighted by this impersonal process; they are asked to vacate without anyone checking in on them or trying to work things out informally.
Property management companies provide an interesting wrinkle in how we think about policy implementation. Because their systems are bureaucratized (and may be less personally antagonistic toward non-paying tenants), it may be simpler for them to implement positive changes—like attaching an NAACP Navigator flier whenever they serve a Notice to Quit.
Tenant Story: Linda Linda works as a case manager for people affected by COVID, and her work includes assisting people through eviction scares. She is completely knowledgeable of all the resources available to tenants in her home state of Colorado. Because she lives under an impersonal property management company, she received a Notice to Quit after falling behind on rent for three days. Having lived in her home for some time without any issues, Linda was shocked and offended that the company would try to kick her out after being behind for just three days. And even though she knows the law, she reported that her ability to comprehend her rights was compromised when she received her notice—she started to second-guess her own knowledge. Linda acknowledges that if she did not have her specialized background knowledge, the notice would likely have prompted her to leave.
Fear of court and court inaccessibility
Most tenants we interviewed never really pictured their eviction scare as a legal issue. For most who sought recourse, their emphasis was on finding enough money to pay. Some tenants expressed uncertainty about what, if any, legal resources were available to them. Certain tenants expressed that they did not qualify for legal aid, yet they could not independently afford legal assistance.
Beyond the problem of access to legal advice, many tenants expressed broad skepticism about court. There is a shared understanding that court is a protracted, exhausting endeavor. Having to balance that experience with a family, a job, and other obligations is challenging, and sometimes impossible. For some, going to court does not feel worth the risk of losing time for their other commitments, potentially having the black mark of a formal eviction on their record, exposing their children to a courthouse, or going against their landlord—who they identify as having more power within the system.
Any interventions that focus on the legal process of eviction must consider the fact that many tenants are evicted informally, and that even tenants with the opportunity to go to court choose to avoid the process of legal resistance. If interventions are designed to make court more tenant-friendly and more feasible to navigate, these changes need to be communicated to tenants to change a widespread negative perception of the legal system.
Tenant Story: Linda As discussed above, Linda works with people being evicted, so she is very aware of tenant resources and legal rights. When she faced her own eviction scare, however, she did not see the court as a viable option, and she instead opted for finding financial assistance. Certainly, going to court could yield a positive result, but the prospect of being formally evicted and having that on her permanent record was too risky. The fact that even someone as knowledgeable as Linda was scared of the courts is highly telling.
Fear of “fighting,” desire for help
Related to the fear of court, tenants generally had overall apprehension at the thought of “fighting for their rights” or resisting. Due to the high stress of eviction, as well as the numerous obligations many tenants have to balance, the notion of resisting doesn’t always seem feasible or attractive. Most tenants focused not on resisting, but rather on getting some assistance and moving on with their lives.
Many eviction prevention policies place a heavy emphasis on lawyering, and encouraging tenants to resist through the various legal defenses they can raise. But to better meet tenants’ needs and desires, non-legal help (like the Navigators) may be a preferable intervention. Several tenants sought out rental assistance, but not legal assistance, suggesting that tenants may disfavor interventions that are seen as overly combative. There was also a widespread consensus that rental assistance was more accessible than legal services. Because legal interventions seem to be disfavored, policy that focuses on strengthening the legal backbone of eviction defense may fail to affect tenants who are simply seeking to move on as soon as possible and reach a place of stability. A good area for further inquiry would be asking tenants how they feel about lawyers generally as a resource. Would they be comfortable reaching out to a lawyer, or do they feel more comfortable reaching out to non-lawyer advocates?
One organization that focuses on prevention, rather than resistance, is HomeStart in Boston. HomeStart’s first line of defense in eviction prevention is a rental assistance payment program that seeks to help tenants halt the eviction process and pay back rent. HomeStart also has non-lawyer advocates who accompany clients to Housing Court, where they assist in negotiating feasible payment plans with landlords. HomeStart’s focus on holistic services and stability, rather than legal defense, may feel more accessible and comforting to tenants.
Tenant Story: Ken Ken fell behind on rent and was served with an eviction notice after failing to resolve the issue informally with his landlord. Ken decided not to seek out legal aid or resist the eviction. He figured that the legal process would be too expensive. Plus, because he was behind on rent, he believed that he had no chance of asserting a legal defense. Ken was more comfortable reaching out to Southwest Behavioral and Health Services, where he was placed with a caseworker. Ken had a great experience seeking out holistic services. He was able to secure financial assistance to find a new home, and his caseworker also assisted him in filling out housing assistance applications. Ken now has Section 8 housing.
High stress
Several tenants communicated that they might have the ability to search for resources if the housing problems were happening to someone else, but that their ability to problem-solve was significantly clouded by their high levels of stress. Tenants have to balance family obligations, work, health, etc., and the emotional turmoil of housing insecurity means that it is often not feasible to seek out proper channels of assistance under these circumstances.
The reality of eviction is that even the most resourceful of tenants are often unable to figure out where to go to get help. Even if tenants know their rights, it may be asking too much for tenants undergoing this traumatizing process to resist. Perhaps interventions should therefore be centered around providing tenants the assistance of a third party, like a Navigator, who can take on the burden of finding resources. In other words, interventions that focus solely on empowerment and self-advocacy may fall short in these situations of heightened vulnerability.
General difficulty in securing resources
Many tenants had frustrations with the process of attempting to secure resources. One tenant, Darlene, actually sought legal aid, but the offices she contacted were unresponsive due to overwhelming demand. Darlene became frustrated, and ultimately stopped trying to seek out legal aid when the stress of her impending eviction became overwhelming. Another tenant, Linda, was frustrated by the ERAP process. Her ERAP payment would take months to process, but she had very little time to pay the rent she owed. Linda ended up having to borrow from friends and family to stay in her home. Multiple tenants expressed a desire for an easy-to-access, uniform service for rental assistance.
A desire—but no outlet—to help
One of the most unfortunate ironies of eviction is that it is such a widely shared experience in some communities, yet the experience of being evicted is completely isolating. Many tenants who have experienced an eviction scare gain practical knowledge about best practices, but that knowledge is lost if not shared with others.
Several tenants expressed gratitude that they were able to share their eviction stories, and were hopeful that the information they relayed would help others in similar situations. A surprising number of tenants showed an interest in becoming more formally involved in eviction prevention and attending events to share their experiences. Being evicted is a disempowering experience, and we heard tenants express that talking about their experiences was helpful. People seemed to appreciate having their voices heard, even if just for a brief interview. Eviction is a community problem, not an individual problem, so interventions should seek to integrate larger communities.
Tenant Story: Jen Jen experienced manipulation and invasions of privacy when she had unofficial housing contracts. After being in two situations in which she was taken advantage of by landlords, she now feels empowered to speak up for others in the Vietnamese community. She knows many people are facing the same issues, and she wants to use her voice to stand up for her community.
III: Experience-Centric Solutions
Key Takeaways
Based on the conversations we had with tenants across the country, we found three key takeaways from the eviction process that are integral to any user-centered, experientially-motivated solutions:
Communication is key. For each tenant that we spoke to, communication, primarily between tenants and landlords, though also with families, employers, court employees, judges, government officials, and more, seemed to fail. The tight timelines of evictions can jam already busy communications lines, and even a day of unresponsiveness or a misunderstood court order can be the difference between a family staying in their home with their back rent paid, or living in temporary housing while struggling to find a new home. Facilitating clear communication throughout the eviction process will be key to ensuring fair, mutually beneficial outcomes.
Isolation is disastrous. Almost each conversation that we conducted with evicted tenants revealed the overwhelming sense of isolation that endured throughout their eviction processes. With no one to turn to, tenants were consistently forced to adopt short-term, fight-or-flight thinking to best cope with the situation at hand. This often meant accepting unlawful evictions, or not knowing who to call to access the legal aid they were eligible for. When tenants have no support through the eviction process, they must consistently make decisions out of necessity. Supported, connected tenants, on the other hand, are much more likely to fight for their rights and reach mutually beneficial solutions.
Awareness is lacking. Tenants are nearly universally lost when they receive an eviction notice or are made aware of an informal eviction process. Up to the point of eviction, they have received no education on how to manage an eviction process or their rights as a tenant. Generally, once the eviction process has begun, eviction education is almost useless—dealing with a current landlord, in addition to working to find a suitable new home, is stressful enough. Even in cities with robust tenant services and resources, like San Francisco, tenants still do not know who to reach out to when they are served with an eviction notice, and are thus not able to make use of the available services. Tenants must be informed enough to know where to turn, even if this is just knowing an urgent, non-emergency number, like 311.
Key Opportunities
Inspired by current policy solutions and pilots across the US, we used these key takeaways from tenant interviews to determine three potential opportunities for intervention in the current eviction landscape:
Mandatory Mediation
Currently, almost all jurisdictions see eviction cases go straight to the courtroom. Tenants often choose to forgo their right to a trial out of intimidation. With mandated mediation, tenants have the opportunity to meet the landlord on a more even playing field, where mutual benefit is incentivized for both parties, in addition to offering a better opportunity to maintain the tenant-landlord relationship. Courts benefit, too, from reduced caseloads. This program has worked well during the pandemic in Philadelphia, where the city’s Eviction Diversion program has mandated that landlords go to mediation with their tenants before they are able to evict them. Philadelphia is unique, though, and many municipal and state jurisdictions face political opposition to any measures perceived to be biased toward renters or more costly than conventional courts. The program also fails to address informal evictions. While not a cure-all, and while an eviction notice mandating mediation remains frightening for many, we believe this could be an important step toward empowering both landlords and tenants to achieve an agreeable, workable solution that cuts costs and effort for all involved.
Navigator Programs
Given the discouraging prevalence of isolation during the eviction process, the potential to empower tenants to find their best solution through support and companionship is very important to experience-centric innovation in the eviction landscape. With housing navigator programs, like the NAACP pilot program in Richland County, SC, tenants at any stage of the eviction process can be connected with a community member who has been trained to understand the local eviction landscape and can educate tenants on their options and the available resources. This engages the local community on the issue of eviction, and provides both support and a know-your-rights knowledge base for tenants. Still, this comes with challenges: navigator recruitment and training, maintaining the boundary between advice and UPL, and the organizational overhead. Even when those are addressed, if tenants in need don’t know about the program, it can also be yet another helpful resource that goes unused. Nonetheless, when executed correctly, navigator programs have the potential to guide isolated and uninformed tenants to their best interest outcomes.
Renter Education and Simplified Notices
Most importantly, in our conversations with tenants, we found that eviction is nearly always an emergency. Even when renters expect recourse for nonpayment of rent, or were threatened by their landlord in the past, an eviction is always a moment of stress that no one feels prepared for. The opportunity here is obvious: What if eviction were something that every renter was prepared for? Or, what if every tenant at least knew one website to visit or number to call in case of urgent eviction needs? This is the case in Milwaukee, where the Rent For Success Program has worked hard to ensure that every tenant in the city has access to basic information and education to enable successful renting, beneficial to both tenants and landlords. While this solution may meet the most needs, and serves a clear function to better enable the earlier two, it too has challenges. How does one implement such a program? Is it mandatory for all municipal renters? Despite these questions, education is an exciting opportunity for individual municipalities to develop unique, local programs that can iterate, evolve, and grow to have tangible impacts on both landlords and tenants.
Initial Observations at the Santa Clara Family Justice Center (Week 2)
By Sahil Chopra
During our second week of the course, we paid our first visit to the Santa Clara Family Justice Center in order to observe, explore, and immerse ourselves in the court experience. Our day at court was structured around exploring the self-help facilities before branching out into smaller, more intimate portions of the courthouse in smaller groups. My team drove down to the court and arrived at around 8:30 am, just as the self-help waiting room started to fill up. We jotted down a few stray observations before convening with the rest of our class in the lobby at 9:00 am, where our instructors Margaret and Jonty handed out a few Design Review pamphlets for our day at court, wherein we continued to write down our observations and thoughts.
Here are the highlights from our first trip to court. Next week, we shall pool our individual observations and insights, as we brainstorm what potential problems and solutions might be.
Self-Help Desk
Definition:
Many users do not have access to a lawyer, so the court provide a self-help desk, where individuals wait in a queue until court staff call up their ticket number and can help them address their problem — whether that be information about the filing process or guidance as to which forms must be filled out in order to proceed with their case. While the self-help desk provides an invaluable service, it is often understaffed. As a result, court users often lineup outside the Family Court around 7:00 am, though the center does not open till 8:30 am and does not start processing tickets until about 9:00 am. When it comes to language access, there is not much the self-help desk can provide on its limited budget. If one does not speak English, he/she/they must bring along a translator, a legal adult in the state of California, i.e. 18 years or older, who is preferably a relative. If they come without a translator, they will ultimately be turned away.
Highlights:
The self-help waiting room feels like a hybrid of the DMV and a doctor’s office. Everyone sits side-by-side, but in their own little-world. Entering the room, there are black chairs lining the perimeter of the room, except for the left-hand-wall, where there is a wall full of assorted forms. While it seemed very well organized, i.e. color-coded, accessible, etc., there were very few people who approached the wall to pick up flyers. Perhaps, the singular placement of all essential forms seemed overwhelming?
Sitting in the crowd, it was easy to spot parents who had brought their teenagers to help them with their paperwork. In hushed voices, I saw a sixteen year boy reading over an assortment of forms, quickly translating them to their mom. Translation services would help decongest the overflowing waiting room, by limiting the number of family members that would need to be brought along. Additionally, it would be beneficial for both the kids and the parents, if the children did not have to take time off school.
Workshop
Definition:
Throughout the week, there are several workshops that the self-help desk hosts, wherein the process for filing a specific motion is discussed and then assistance is provided with form-filling. It just so happened that our-visit coincided with a divorce workshop.
After spending some time in the self-help room a few of us decided to observe the workshop.
Highlights:
While we were sitting in the self-help room, one of the court staff came out and announced who made it into the workshop and who did not. It seemed a bit impersonal and harsh to be called out by name, especially when everyone knows the issue associated with your use of the court. But maybe, that helps normalize the act of getting help?
The informational portion of the workshop consists of a 50 minute, screen-capture powerpoint presentation and narration. It was interesting that there were more spots for the video portion of the workshop than the 1:1 assistance portion of the workshop, even though the latter part feels more important towards the goal of filing a motion. This discrepancy between max capacity and serviceable capacity highlights the need for more staff.
The PowerPoint video described the technical legal terminology and processes surrounding divorce. While informative, the video didn’t seem to be helpful. Within the room, one couple talked over the video — trying to fill out their paperwork, as the video played. Most of the other viewers seemed to pay attention for the first five minutes before sliding into their chairs and waiting out the remainder of the video’s runtime.
The first problem with the video is that it is entirely in English. If you don’t speak English well, you’ve just wasted 50-minutes that could have been spent getting help.
The second problem with the video is that it is too long and lacked participant engagement. It’s important to be precise and informative, especially when dealing with legal matters; but the video consisted of a powerpoint and a voiceover. There was no color and few pictures. Furthermore, it did not actually help with the process of filling out the forms. Without interactivity, the video failed to provide actionable instructions — thus failing its purpose of providing help to individuals who needed assistance in filing for divorce.
The third problem with the video is that it is unaccessible. It cannot be accessed outside the workshop, and even within the workshop it cannot be paused, rewinded, etc. Thus, it fails it’s purpose of being a 1-stop-reference for all things divorce-related. Additionally, the video was poorly constructed in that a lot of the important facts were spoken but never transcribed on the slides themselves, even though the slides themselves were full of text.
Possible Language Access/Self-Help Solutions
After sitting through the workshop, I think there is a lot low hanging fruit here, i.e. small changes that can be made to improve outcomes and scale the program — even in the face of budgetary issues.
Solution 1 (Low Overhead): There are many computers in the workshop room. Instead of making everyone watch the PowerPoint video together, provide every workshop-attendee a pair of headphones, so that they can pause and rewind the video wherever they want.
Solution 2 (Low Overhead): Split the presentation into digestible chunks. After each video section have the workshop-attendees fill out the respective portion of the form. This tight coupling is often used in flipped classrooms and should make the process more self-directed.
Solution 3 (Low Overhead): Post the video and presentation online. Let people view the contents and fill out the form digitally at home.
Solution 4 (High Overhead): Translate the presentation into several key languages, i.e. Spanish, Vietnamese, Korean, Hindi, Mandarin. This is a one-time job but would improve accessibility tremendously.
Miscellaneous Observations
After experiencing the divorce workshop first hand, we decided to sit on a few of the court hearings that were open to the public. Before, we headed up the stairs to the court rooms, I stepped away to get some water. In the five minutes that I was gone, my teammates encountered a Latino women, who could not speak English well. She was asking, where she could find the police; and it was only after a few exchanges that my teammates realized that she was looking for “something to keep [a person] away”, i.e. a restraining order. They then showed her the route to the appropriate court office, but it was apparent that there needs to better outreach within local cultural and ethnic communities in both discussing the purpose of the court, the terminology surrounding the court, and the services that it can provide. This might help reduce friction for those seeking support, especially not native speakers. Perhaps outreach at libraries, churches, and grocery stores might help with this problem.
Overall, I was surprised to see how calm and collected the judges were at responding and guiding the proceedings. It seemed as if they really cared about both parties involved. The empathy demonstrated was quite moving, especially given how messy some of the court cases were.
For our Design for Justice: Language Access class, our teaching team made a canvas to help a design team craft a forward plan for the projects they have been working on to advance language access in the courts through technology. The canvas can be useful to have a one-page hand-off for a policy partner to understand what the team is proposing, and how it can be taken to the next stage of piloting and evaluation.
In late April 2018, Daniel Bernal and Margaret Hagan taught the first part of the d.school pop-up Design For Justice: Eviction. The class focused on how we might better empower people who have received eviction notices (specifically, in Arizona) to know their rights, their options, and to go to court to fight eviction.
In the class, our 2 teams focused on what intervention we might send in the mail to activate someone right after they have received an eviction notice, and what intervention we might point them to for greater support and guidance.
We worked in 2 phases. First, we did a recap of key insights, personas, players, and trends regarding the eviction process, user experience, and legal help resources in Arizona. We did this with calls to Arizona legal help leaders, a service designer who has been working on eviction help, and Daniel’s presentations on his research into eviction trends and strategies in Arizona.
Our second phase of work was brainstorming and prototyping. Our 2 teams focused on the different intervention points, to create an Idea Catalogue of possible ways to empower users through a mailer or a digital resource.
From this brainstorm, we critiqued the ideas with some help from our frequent collaborator, Kursat Ozenc, who is a design strategist. We will now write up these ideas, formalize them slightly, and invite a panel of legal, sociology, behavior change, technology, and design experts to give further feedback. From there, we will begin to develop first versions of several of the concepts that we will test with the public in our second half of the class.
In our classes, we map out different users’ journeys through the court. This is one of the Northeastern University student teams’ map, that abstracts different users’ journey through housing court in Boston.
A sketch from my notebook, while I was observing a waiting room in a Court Service center in Boston, for people who were waiting for help with housing cases.
Today we held our Prototyping Access to Justice class on-site at San Mateo County court house, specifically in and around the Self-Help Center and Family Law Facilitator.
The six student teams are all at the point where they have working prototypes that they want to test. They each have hypotheses about how they can make the legal system better for people without lawyers, and have embodied these hypotheses into a new tool — digital- or paper-based.
Instead of our usual class setting at a design studio at Stanford’s d.school, we created an impromptu class space in the Waiting Area on the 2nd floor of the Superior Court, where people are lining up to see people at the Self- Help Center, or are waiting to be called for an appointment. Some of the teams also set up testing spaces inside the Self Help Center, for when people had down-time after they had filled in forms or were waiting for next-steps.
The teams sought out people to give quick feedback, as well as longer experiential testing. They had interactive click-through prototypes of digital tools, paper mockups of new tools, posters and floor pathways for navigation, and tablets with new feedback forms. They had gift cards to give to user testers, to compensate for their time.
They tested their prototypes in small groups — with some taking notes (or translating into Spanish) and others leading the questions. They also had designer and developer coaches with them, to help them spot new opportunities and to run the testings.
Takewaways
So what were the takeaways? I was able to pull out some high-level insights during my debriefs with each team, as well as some specific points for improvement.
1. The forms are too many and too complex. This was a refrain that each team heard from users, no matter if their questions and prototype revolved around forms or not. If there is one big message that family law litigants have for courts, it is: make your forms easier to understand, and easier to complete.
There is an overload of paperwork, that is laid out in a way that does not make sense to people, and overwhelms them.
2. Little things about court — like parking, way finding, and security checks — have a big influence on people’s experience. Though we as lawyers might think about the legal procedure, forms, and hearings as the main determinants of people’s procedural justice and sense of fairness about court, there are other more pedestrian factors that shape their time with the legal system. If parking is difficult, expensive, or with a ticking timer, this puts an extra layer of pressure and confusion. If the security guards doing initial checks at the door are adversarial or cold, this raises the stress level of people and sets them off on a bad foot. If there is confusion about where they are going or how to get there, people lose confidence in themselves and feel that they are wasting time and not being strategic.
3. Pathways on the Floor should be implemented immediately. Our team Chukka-Ryorui, who are focused on improving navigation, put down a dotted red line from the building’s entrance to the Self Help Center on the 2nd floor. They used masking tape to make the line — and it took less than half an hour to implement. The feedback was universally positive. People were able to follow it and understand it without any complicated explanation. Users reported that they already are familiar with this pattern from hospitals, and appreciate having it here. They want bold color lines that they can follow easily, along with complementary signage.
We recommend that courts implement colored floor pathways for their most popular routes: Self Help Center, County Clerk, and Jury Services primarily. This is a relatively cheap intervention (vinyl floor paths are not that expensive) that can have a major impact.
4. “Out-of-Court Homework” Tasks must be Modeled + with reminders. As we heard from litigants and from staff, the most common fail points are around all the tasks that the litigant cannot do at the court, but must do outside. Getting service of process done, and done correctly, with the paperwork noted correctly with address, date, and other pieces of data is a very common failure. Also, remembering to get this done at the right time is also a fail point.
Some of the recommendations in this space is to have more reminder services that proactively reach out to the litigant to tell them they have to do this task before their service.
Also, the demand is for models of forms that have been done correctly, with annotations about why it is correct and how to do it right.
5. Be Mobile First, with guides and tools for the phone. The overwhelming majority of the people we spoke with have mobile phones, and are willing to use them to get legal tasks done. Tools must be built for phones, not desktops.
We are setting a bounty for the best new product that lets people understand processes and fill out forms using the phone. Even if this is not ideal — even if we wish that people would have the big screens of a desktop computer when they’re doing complex processes, they will be using mobile phones and paper handouts most of the time in practice.
Six. Maps are key. The team that was testing out a giant, slightly comic-based map of child custody process got great reviews. People responded well to their characters and to the map-based view.
They are thinking in terms of both a paper-based map (that could be a wall poster of a general ‘happy path’ of how the process works in the ideal, combined with a booklet of in-detail maps that include detours). And in terms of a digital map, that could be zoomed in.
People were able to instantly figure out the paper based map. They know how to use it. The digital map was harder — people were more hesitant to use it, and to know what to touch on the screen in order to see what would happen.
—
More insights to come as our class proceeds — stay tuned for more of our design work and proposals for making Self Help Centers and the legal system more user-friendly.
I am also quite excited about setting up a more regular pop-up design lab on site at courts and other points in the legal system. To create more relevant and interactive designs, having input directly from litigants and court professionals is highly valuable. And doing the prototyping in the environment also helps the designs better mesh with this particular context, and what affordances and opportunities already exist there.
In Spring 2016, Margaret Hagan and Janet Martinez taught a course at Stanford Law School, through the Policy Lab program, called Prototyping Access to Justice: Designing New Legal Services for Self-Help (see the official class description on Stanford Law’s site here).
In partnership with the California Judicial Council and Self-Help Centers in San Mateo and Santa Clara County, we ran through several design cycles to document how self-help centers currently welcome, orient, and provide services to people without lawyers, who are going through civil law (mainly family law) to solve their problems.
On this page, we document our process and some of our initial learnings. We will be writing up a more formal report and paper with our results. In the meantime, we present this as a more informal write-up, with lots of images.
Our Process
We followed a human centered design process to explore what new solutions would be possible. This meant that we had half of our classes at the University, in a workshop or lecture seven, in which the students learned design methods and then practiced them in teams. The other half of the classes were in the field. Our team worked in the courthouse in Redwood City, where the San Mateo County Self Help Center is located. We interviewed the staff that works there. We ran through the process of trying to find the Self Help Center and use it. We observed people as they were going through the service and interacting with the Center’s resources and staff. We did design reviews of how information was being communicated and how service was being provided.
Scoping out our Challenge
In our first class, our team spoke with our partners in the court and California judicial council in order to understand their perspective. We were particularly interested in how they framed the problem that our class would be addressing. We also asked them about previous efforts to improve self-help centers, as well as the legal, budget, and staff constraints that self-help centers must operate within.
From these conversations, our team focused on a few questions to guide our research at the court. We would focus on how to improve the welcome and orientation experiences of people who come to the court building in order to find help for their family law problems. In addition to this initial stage of engagement, we would also consider the experience of understanding the legal process and trying to follow it.
Our class, with its limited time frame, would not explore later steps, like negotiations or mediation, or enforcement of court orders. It would also not address the process of attending hearings and being strategic about them. Rather, we limited ourselves to the initial stages of the person figuring out the court system and using self-help resources to get started on their process.
In California, many people with issues around divorce, child custody, child support, and guardianship, try to navigate the legal system on their own without a lawyer. Often they seek out help from the self-help center in order to do it themselves. They seek out help in person as well as online. Our research would focus on those coming in person to the building, but we would also consider how to adapt our proposals to an online context.
In addition, many of the litigants in California do not speak English as their first language. There are also major issues with literacy in general, and literacy of legal jargon, written in English, in particular. We decided to sideline these issues of language access temporarily, and focus on litigants who at least could read English at the grade school level or above. But we recognize that any of our solutions, to truly serve access to justice, would need to adapt to foreign languages easily, as well as to low literacy situations as well.
We entered into our next phase of user and system research, with a scope down vision of which types of users we would be designing for, and what parts of court user experience we would focus on. We would target people with English literacy who were starting off their family law court journey, and who chose to do so in person and not online.
Design Observations on site
We had three sessions in the court building and the self-help center to uncover problem areas, successes, and other needs and requirements for the people seeking out help for family law problems. During the sessions, we used a range of design research methods to understand the current Service design and user experience. These methods included observation, service safaris, interviews with users, interviews with staff and other professionals, and feedback sessions on sacrificial ideas. Using these methods, we were able to figure out more precisely how we could improve the user experience of the courts
We identified key touchpoints where the current Self-Help Center design was failing, and where it could be improved. They are as follows.
Target areas for user welcome and orientation to the court:
Signage
Names, Terms, Language
Line and Triage
Welcome Experience
Physical Space
Resources
Directions
Referrals
Target areas for navigation of the legal process:
Filling in forms
Understanding basics of process
Knowing the detailed step-by-step
Supporting Homework and Legwork outside of the Center
Following through on the plan
Findings about our Target Users
Juicebox Gallery Id 27 has been deleted.
Signage isn’t supporting people
people get lost
people don’t read the signs, or after they read them, they go back to ask the security guard
no one is staffing the information booth
many signs outside the building, but with very different visual languages
importance of entry way right before the metal detectors
importance of after the metal detectors, that guard can refer to easily
fail point of at the escalator, whether to go up or to go straight
fail point of the self help center’s board that should be in the eyeline
all signs in English
Names and Terms don’t signify what they’re supposed to
importance of ‘start a divorce’ versus ‘help with a divorce’ — two different offices you might have to go to (clerk versus facilitator)
different signs refer to center by different names
clerk in office downstairs doesn’t consistently say same name
people in the know say “Family Law Facilitator”, few people say Self-Help
but confusion with “Family Law Facilitator” or “Family Law Services” or “Family Law Clerk”
nothing feels ‘natural’
The Pain of the Line
people don’t like waiting inside the actual center — too cramped, not private, really “feel the line”, anxious about making it through, it’s hot, it’s uncomfortable
“The Line” experience sets off on a negative foot, the line-ness makes them angry at each other, feel length
people don’t know what to expect, what they can get from here or not
can’t prep beforehand, get a jump on the line through online or phone channels, need to show up and hope for the best
Where is the Welcome?
feeling of confusion and lostness until they reach the center — undermines confidence, makes them feel lost
feeling of intimidation, whether they can even enter the door of the Center
Bulletin Board resources is not in their eyeline, often missed
No strong signal that “you’ve arrived in the right place”
not enough human people to guide — just the security guards
clerks in office downstairs could be a valuable welcoming point, to direct people to the center, but they do not give clear or enthusiastic referrals to Center
The Physical Space’s discomfort and exposure
Lack of privacy for very sensitive and confidential information about family problems
Really tough if you have an infant or toddler who’s not potty trained
Not enough chairs
Not enough physical space to do the interviews
Computers are not used (potentially should be removed completely?)
Lots of Resources, but not enough order to them
Many scattered notices and resources, without clear labels about who they’re for, and what their value is
Signs and resources are small, printed in too small a way
Staff verbally give directions and support (like with clerks downstairs), but without knowing how much people actually take in
Lots of next steps, hard to take in
Physical directions to other offices or buildings are verbally described, hard to remember
Early stage Concepts
Juicebox Gallery Id 26 has been deleted.
As soon as we began this design cycle, we took note of any ideas that we had. Even though there is a designated part our work cycle in which we would do brainstorming, we acknowledge that good ideas pop up even when we are supposedly focused on other things, like observing the space or doing interviews. Anytime an idea was suggested to us or something triggered an idea in us, we jotted it down, and then did regular debriefs as a team to share these ideas.
Many of the ideas are very sketchy, without much detail or thought about implementation. But this large collection of concepts helped us to have a very productive brainstorm, and gave us a lot of material with which to pattern and then prioritize the concepts we felt were most promising take forward. We also used many of these early concepts as sacrificial prototypes/provocations, that we could show to our users or to court staff, and get their feedback as well as their additional ideas.
Iterated, improved designs
By our final presentation to our Court partners in May 2016, we had made some decisions about which ideas to prioritize as suggested pilots in the Self-Help Center. We combined some ideas, and we staged them from the most immediate and do-able, to the most ambitious and resource-demanding.
We’d also point back to a Guardianship/Court Design workshop that we ran two years ago. This previous workshop was focused on self-help guardianships, and produced a range of concept designs to help litigants through this particular family law procedure. Those ideas are listed out on our workshop page here. We noticed some overlap between those proposals and ours here.
Court user experience can be heavy sometimes due to the entangled nature of court use cases and structures. This past week, our course participants took that challenge and conducted research in the field with court employees, and end users. When they were preparing to present their findings, we asked them to think them as highlight reels of a motion picture.
The discussion during the presentations was rich, teams realized the interconnectedness of their design briefs, and why they should be collaborating more in the upcoming weeks. Teams did an excellent job highlighting the detailed issues in their selected scopes. Moreover, they identified some overarching challenges such as courts’ fairness principle, technology bottlenecks regarding vendors, and the importance of feedback loops. What also fascinated me was the medium of some of the presentations. Using enactment and blueprinting/mapping were two creative reporting methods that helped the audience to visualize the insights and the context.
I would like to share two of these creative reels ~research reportings from the session.
First one was from the team who picked the prior-to court design brief. Team decided to present their early research findings through enactment. They enacted three different personas, including all-in techie, all-offline persona, and a tech-friendly but busy persona. Choosing enactment as reporting research helped us the audience to grasp the personas and their daily lives more clearly.
The team who picked the inside the court experience presented their findings using a blueprint of the building on the whiteboard. Each member then presented their experience when they tried to use the service ( walk-in-the mile method) and referred to the blueprint when they were presenting their findings. Again, the medium of presentation made the research more lively than a dry powerpoint presentation.
