Closing the Experience Gap

Upon moving to the United States from Japan at the age of six, I couldn’t speak a word of English. Kids bullied me, and strangers often scoffed at my parents’ imperfect grammar and pronunciation. This experience taught me the feeling of being “othered” because of what we look like, or how we speak. But it also helped shape the researcher I am today. My passion lies in ensuring I represent the voices of users who are often marginalized throughout our product cycles. On the Anti-Discrimination and In-Home Accessibility teams, I get to do exactly that.

The Anti-discrimination team is dedicated to uncovering biases on our platform and making changes to help prevent and address discrimination. The In-Home Accessibility team works to ensure travelers with disabilities can find and confidently book homes that meet their mobility needs. As the lead researcher on both of these teams, I’ve spoken with marginalized groups of travelers – specifically with racial minorities in the United States and travelers with disabilities – about how they use Airbnb to book accommodations. Through these conversations, I’ve found that some marginalized travelers are taking many more steps to book an Airbnb compared to non-marginalized travelers. This gap in how marginalized groups and non-marginalized groups use our product is what our teams call the “experience gap” – and in order to close this gap, it must start with research.

Discrimination is a difficult, nebulous topic to address in a product, especially with some of it being systemic and embedded into a country’s history. Every milestone seems minute in the larger journey of solving discrimination, but it gets us one step closer to achieving equity. For us, this includes milestones such as launching Project Lighthouse, a tool that will allow us to measure discrimination on our platform. Another landmark was removing guest profile photos from reservation requests until after a booking is confirmed to encourage more objective decision making by hosts. Here are a few best practices that will help you uncover the experience gap that I’ve learned as we worked towards these milestones.

Look for inequity

Many product teams fail to recognize that the “average user” use cases do not apply to everyone, and continuing to only prioritize this segment will create larger experience gaps on the platform. As researchers we must look at this through the lens of inclusivity, and question whether everyone truly experiences our product in the same way.

1. Challenge your assumptions – Challenging the assumption that everyone uses your product in the same way can uncover ways in which some groups of users may be using your product in a completely different way. For example, we have assumed over the years that Airbnb guests would search for homes using the filters, browsing results, comparing a few homes, contacting hosts if they had questions, and finally booking the home. However, when we questioned whether this behavior was true for people with disabilities, I found in my research that these participants’ behaviors were vastly different – they carefully looked for photos of accessibility features to confirm the features exist in the home, and contacted the host with multiple questions about factors such as the dimensions of a doorway, the height of a threshold, or the terrain of the sidewalk to make sure the home would be accessible for them. Had we not questioned our assumptions about our users’ behavior, we would not have uncovered this experience gap.

2. Screen for diversity in every study – When recruiting participants, we make sure to ask about demographic information and intentionally choose participants from various backgrounds – age, gender, sexual orientation, race, and disabilities are just a few attributes you can ask about. This, as well as intersectionalities (i.e. how the combination of one’s multiple identities might result in different types of discrimination), will ensure a diverse set of perspectives for your study, thus helping you uncover the experience gaps that may exist. Note that some demographic questions may not be appropriate for different regions of the world, so be very thoughtful about which questions you ask.

3. Treat demographics in the same manner as user behaviors – In research, we always want to make sure we’re gathering feedback from a representative sample to inform our product. The question then becomes “How many people per demographic should I recruit?” In the same way you might want a mix of participants with different behaviors (e.g. new vs. experienced Airbnb users, mobile vs. desktop users), you want to make sure you’re speaking with a representative mix of participants from different backgrounds. If your screener does not yield enough participants from various backgrounds, even just having one or two participants can help shed light on any experience gaps that may exist.

Thoughtfully address sensitive topics

As researchers, we know that building rapport with our participants is foundational to conducting any qualitative study. This is especially important for sensitive topics like discrimination, where participants may not feel as comfortable opening up and sharing their experiences. Here are some thoughtful strategies you can use to build rapport, and thus create a safe space for the participant to talk about these topics.

1. Use screener questions as fodder for discussion – When intentionally discussing sensitive topics, your screener questions can be used as fodder for discussion in your interviews. For example, let’s say our team’s hypothesis is that marginalized groups are more likely to message hosts before booking as a way to gauge whether a host will be discriminatory. To recruit participants who exhibit this behavior, we may add a multiple-choice question in our screener such as “What are the top reasons you message hosts before booking?” and choose to speak with participants who answer “To gauge whether the host may be discriminatory.” We can then create space during the interviews for participants to explain their answer, rather than bringing up the topic of discrimination unexpectedly in an interview.

2. Be conscious of language – When speaking with different groups of users about sensitive topics, researchers must do their due diligence to understand appropriate language. There may be specific language that is more appropriate for certain groups (e.g. using people-first language in the U.S. for people with disabilities). Make sure you are aware and are respectful of these nuances, which can help build trust between you and your participants.

3. Be aware of the impact of the researcher’s identity – When moderating these discussions, we should be cognizant of how the researcher’s identity and background may affect how comfortable our participants (who may identify differently) feel about discussing these topics. You may consider reaching out to external researchers who come from similar backgrounds as your participants to help moderate these discussions

Find allies

Getting buy-in from stakeholders may be challenging for inclusive research, especially when  metrics like “belonging” and “inclusivity” are more subjective than the usual tactical business metrics. Finding allies who can uplevel and elevate your work is crucial to making sure your research is properly implemented to create more inclusive products.

1. Put stakeholders in your users’ shoes – One strategy I’ve used to gain allies is conducting studies using a methodology called “personal concierge”, through which we pair up stakeholders with travelers and have them act as the travelers’ personal concierge. For In-home accessibility, I paired up leadership members with travelers with disabilities. During the exercise, leadership members asked questions about travelers’ trip preferences, such as the accessibility features they’d need and the neighborhoods they’d want to stay in, then attempted to find the perfect Airbnb for their trip based on those needs. Through this exercise, stakeholders began to understand just how difficult it can be to find an accessible home on Airbnb, resulting in their support for this work.

Note that empathy-building exercises should be thoughtfully executed to increase stakeholders’ awareness of users’ pain points (e.g. finding a home that fits their accessibility needs), without pitying or reinforcing negative stereotypes of different user groups by simulating their personal travel experiences.

2. Present the impact on growth business metrics – Create a growth business case. Make the  argument that leaving out marginalized populations means missing out on a certain revenue value, or on certain key metrics. By including these populations, we are able to grow our business significantly while also creating an inclusive product.

3. Circle back to your mission and values – Our mission at Airbnb is “Belong Anywhere,” meaning our users feel like they are part of a welcoming community wherever they may be in the world. This work is thus foundational to fulfilling this mission. Ask yourself what company or team-level missions/values align with inclusion work. Using this narrative can also help to keep your leadership and stakeholders accountable to staying true to the mission.

How will you conduct inclusive research?

Take a moment to think about what research you can do to understand whether your product is putting certain groups of people at a disadvantage, especially those who have been systemically disadvantaged. Amplifying these users’ stories starts with research – without researchers, these stories would not be uncovered and the experience gap will only continue to grow. Once we put inclusive research into practice, only then can we begin to diminish that gap.