People Lead the Way

Last month, Code for America saw the highest volume of SNAP applications ever submitted through GetCalFresh. Since the COVID-19 pandemic hit California, our qualitative research and client success teams have worked together to sift through thousands of messages in order to better inform product strategy and advocate for policy changes.

GetCalFresh’s research team has expanded the way we do research to meet this demand.

Our work has always been centered on putting people first. Our research team surfaces real, on-the-ground needs so that we can better understand how to build a more inclusive safety net.

 

But due to the pandemic, we currently don’t have the privilege of going into the field to conduct our typical methods of research, including in-person interviews, usability tests, and diary studies. However, we noticed that an increasing number of our clients were submitting feedback and comments via their GetCalFresh applications. We realized that these messages provided direct client feedback that could drive our research efforts.

Having real time feedback created an opportunity for the GetCalFresh qualitative research team. In working with our colleagues in client success, we’re starting to understand how we can do more “non-extractive” research (using the stories our clients are already sharing with us, rather than extracting more information from our clients by having them relive or dive deeper into their situations). Within the GetCalFresh application, our clients often tell us about their circumstances, including their familial, housing, and financial situations. From our client’s messages we can gather how to make changes to our product and continue to keep our work people-centered.

 

Collaboration between research and client success in the “normal” times

In a non-pandemic time, our research team would go out into the field and speak to people about their journey with CalFresh, California’s name for SNAP. These in-person interactions help us gain insights into everything from how clients hear about CalFresh, to how their interview went with county workers, to understanding their recertification process.

Before COVID-19, I would meet with our client success lead, Atzay, on a biweekly basis to understand and analyze the questions clients were asking through our live chat feature (a service that our client success team provides for real-time communication with clients via Intercom).

For example, in light of the federal public charge rule which states that undocumented immigrants could potentially be barred from citizenship if they use public benefits, our teams would analyze messages that mentioned an applicant’s immigration status to better measure the impact of the rule. We wanted to understand the experience of undocumented applicants better—what their fears were and what questions they had. These insights would better inform our application’s guidance around citizenship questions. It also ensured that we were addressing client’s concerns about how they may be affected by public charge rules.

 

Before COVID-19, our client success team was responding to about 430 people per day. Now, our team is responding to an average of 1,200 people a day. Since the research team won’t be able to have conversations with our clients in the field for the foreseeable future, these messages are the best alternative to guide our research efforts and ensure we are continuing to people first.

Pivoting our research methods

During the first few days of California’s mandated stay-at-home order, we saw thousands of messages come in through GetCalFresh that directly mentioned the pandemic and quickly began a daily review. Though we have used this method to inform our product and program, we had never worked with such a magnitude of client messages before. Before this pandemic, we would examine about 100 of these messages per research project.

Over the past two months, we’ve analyzed nearly 5,000 client messages. The analysis and insights of this research has been used to update California’s Department of Social Services (CDSS) on themes around client experience and needs, inform other states on what is to come, advocate for policy changes which make it easier for people to apply and stay on SNAP, and inform product changes on GetCalFresh. While we wish we could be out in the field with our clients, we’re slowly getting used to a new research flow.

Step 1: Upload daily client messages into Airtable

“I have been laid off due to the coronavirus. I am a single mom with 2 daughters and I am now home schooling. I receive help from their dad but he is also unemployed. My older daughter has special needs. We receive Medi-Cal and I have never applied for food stamps in the past. I hope you can consider my application as I am worried now about getting enough groceries in the house during this stay in place we are now restricted to.” —GetCalFresh client

Every morning, we download the previous day’s client messages—from live chat, open text fields in the application, and enrollment surveys—and upload the messages into Airtable. In qualitative research, we analyze (or “code”) messages, interviews, and data to identify concepts within the research. We code messages according to themes we’ve established through the research process, while also adding new ones every day. After we’ve uploaded and coded messages, we’ll look at the new codes to understand how the pandemic has changed people’s situations.

 

Emotional impact on researchers

Each day, Code for America researchers go through hundreds of messages that detail the traumas our clients are experiencing. The process can be extremely emotional, with our work days spent reading about the myriad challenges our clients are experiencing. Many are navigating the complexities of multiple government systems while urgently trying to get food on the table.

We also hear about how clients have been systematically excluded from benefits for reasons like not having access to a phone for a verification interview, or not having proper documentation to prove that they need help. We gain further insight into many of the structural ways the system was designed to keep people out, and every day we hear stories of how the system has failed in providing access to basic necessities like food.

“My husband lost one of his jobs two weeks ago due to the store closing, and his remaining job is suspended until the COVID-19 virus is contained. We will be homeless within the next two months without help.” —GetCalFresh client

Reading messages like these every day can cause vicarious and secondary traumatic stress for researchers. Even before the COVID-19 crisis, we learned that taking breaks and giving opportunities for our team can share what they’ve heard from clients is crucial to provide the space, clarity, and emotional support necessary to keep doing the work. That’s never been more true than it is now.

Step 2: Understand what we’re coding

We use the previously established codes to analyze what applicants are saying in their messages. We’ll add new codes to document new trends for that day. Once we’ve coded a certain amount of messages for the day, we’ll look at both old and new codes to understand if things have gotten worse, or stayed steady. We’ll also look at what percentage of clients are mentioning certain keywords, and so on. Once we’ve analyzed the messages, we send our GetCalFresh team an update. These updates are often sent to our government partners at the California Department of Social Services (CDSS) and national advocates.

“I just lost my job as a waitress due to COVID-19. My school has not been canceled but has been moved online. I just need help to get by until the restaurant I work for is able to reopen.” —GetCalFresh client

The biggest trend we’ve seen since the pandemic hit in our coded messages has been vast job loss. In fact, 80% of the messages we coded mentioned furlough, temporary job loss, or permanent job loss.

Step 3: Translate research insights into product changes

This spike in messages means we have had to adapt certain sections of the GetCalFresh application where clients are asked questions about their income or how many jobs they have. We synthesized the data and shared our recommendations with our designer, who used the analysis to implement changes that better reflected the current day-to-day realities of our applicants.

 

Step 4: “Hack” usability testing

California’s stay-at-home orders are keeping both our research team and our clients out of public spaces, which means we do not have the ability to carry out our pre-pandemic usability testing in person. Ideally, we would be able to sit next to a client while they went through a product flow, and have a conversation about what aspects of these new design changes are confusing, what changes they would want to see, and what parts of the applications do not make sense.

However, now we have to use client comments as an alternative form of usability testing. The comments help us to understand how the new designs are performing, i.e. “I don’t have a place to enter 0 jobs” or “I don’t know what pre-tax means.” We’ll document new questions clients are asking to understand where the designs are confusing and could give clients more guidance.

We’ve been tracking feedback by analyzing client messages regarding specific job and income questions, and any messages that mentioned income throughout the application with the new design implementation. This analysis is helping us understand what we need to work on next.

Non-extractive research

We need to turn to the avenues in which clients are already communicating and sharing their experiences, as opposed to extracting even more from them during this time and making them relive or stay in that trauma. We work with impacted and populations made vulnerable who for so long and continually have had to tell their story to explain to policymakers why changes need to be made. Historically, research has probed into people’s lives unethically and unnecessarily. This crisis will change many things about the way we conduct our lives, and hopefully one will be a change in the ways we practice non-extractive research—with and without a pandemic.

Putting people first

This pandemic has allowed our team to “hack” some research methods, and also understand the depths of how research and client success can work together to produce more nuanced insight into the experience of applying for food assistance through GetCalFresh. As stated in our recently released qualitative research guide, we believe that the experiences and needs of the people should always drive decisions. This cross-functional research helps us live out those principles by keeping the people we serve at the center.

Because our values and principles are human-centered, we are able to adapt our research practice to meet our client’s needs. As always, our team is guided by the people. Even without a full researcher’s toolkit, we are fully committed to continuing to deliver a people-centered service.