From the Ground Up

Shining a light on how research guides our product development

Categories: Behind the Scenes — Judd Antin

In May of 2015 I came to Airbnb to lead Research. I arrived to find a talented group of researchers doing great work within a collaborative product process. Designers, product managers, engineers, content strategists, data scientists and many others worked hip-to-hip with researchers to make great products. What I found were the raw materials for an enduring culture of research– a product environment in which research is baked-in in all the right ways. I also saw a huge opportunity be more specific about what the culture of research should look like at Airbnb, and ways for Research to develop a stronger, more consistent foundation.

Since then, Research at Airbnb has grown from 10 people to a team of 36. Our work now encompasses three complementary research disciplines: Experience Research, Product Specialists, and Survey Science. And, perhaps more importantly, we’ve made huge progress in building that strong foundation.

Here are two key ideas that help us continue to build the research culture at Airbnb.

 

Bulldozing

Early on in my time at Airbnb I met with a senior Product Manager to talk about research and process. His main feedback was that researchers sometimes did too much “bulldozing.”

“Tell me more about that,” I said, without a hint of judgment, trying to apply my best open-ended interviewing technique. Bulldozing, it turns out, is what happens when a researcher gets called in towards the end of a project. You know, just to make sure it gets in front of some users, to do some usability testing and validate the product ideas. And then the researcher finds that a bunch of the project’s assumptions and premises are off-base. When they give that honest report– Yes there are some usability issues we can fix but there are some fundamental issues as well— that’s bulldozing. The perception is that the researcher isn’t trying to help, but is instead trying to tear down what the team has built. In many ways that perception makes sense. The team has invested itself in building something, and now here’s someone telling them it’s not the right stuff.

This was a strong signal to me, not about the researchers’ behavior as much as a failure of process and the need for more deep and consistent engagement between researchers and product teams.

The simplest (but perhaps least effective) way to integrate research is to think of it as a tool for validation. If you think this way, then the end of a project is just the right time to bring in research. But that’s far too late. Massaging evidence to validate existing ideas is easy, while re-evaluating those ideas is hard. It’s not fair to rely on cross-functional partners, no matter how research-minded they are, to know when and how research should be involved. Every one of those partners needs a direct research point of contact, a trusted partner who is there from beginning to end, proactively contributing as the voice of the user. Ultimately, researchers need these relationships just as badly as product teams do in order to work effectively.

This environment just can’t exist unless researchers are deeply embedded into teams, as equal partners on the product team, forming strong and enduring relationships. Those are the table stakes.

Our researchers stick to a team for at least 12-18 months, building the trust that makes research effective. We participate in every stage of the product process, making sure we’re tackling real-world problems based on assumptions that have evidence behind them. We make sure the voices of guests and hosts are embedded in everything the team does. By making all this happen at the right point in the process, things move faster, not slower, because there’s more focus, more informed decisions and less backtracking. We’re proactive about communicating early and often in ways that integrate whole teams. When research is there from the beginning, team members also participate in research more, and the team is able to take ownership of the results.

Setting up this stable structure and building the team around it has been really successful. One of the things we’ve found is that when it comes at the right time “bulldozing” is a good thing. It doesn’t flatten the building, it flattens the ground on which we build.

 

Doors and corners

Many research teams rely on a relatively small set of qualitative methods.The problem with having such a small set is that it’s like having just one key and a narrow-beam flashlight. Many doors won’t open and most corners stay dark. In research, behind doors and around corners are where we find the profound complexities of humans interacting with each other and with our products. We need to shine a light in these places, and at Airbnb we do that by pushing on a diverse, holistic approach to research. We seek out every method, every perspective and every source of data.

As a team we cover the spectrum of research methods. Our researchers have experience that ranges from ethnographic-style research, deep content and case analysis, open-ended design exploration and usability all the way to rigorous survey design and statistical analysis with big data. Whatever tension might exist between qualitative and quantitative viewpoints– it’s not here. Airbnb also has a world-class Data Science team full of talented and eager partners. Building on each others’ strengths, Research and Data Science are constantly partnering to peer around corners and behind doors. It’s all about the research questions at hand because they define the approach, and it’s usually a collaborative one.

When it comes at the right time “bulldozing” is a good thing. It doesn’t flatten the building, it flattens the ground on which we build.

One concrete consequence of our holistic multi-methods approach is that it demands that every researcher be a generalist. Most researchers are more passionate and experienced in some subset of research methods, but everyone has to build a diverse set of research tools. We ask even the most qualitative researcher to learn applied statistics and to try using statistical programming software like R. Everyone learns the basics of rigorous survey design. Researchers teach each other by facilitating classes like “Moderating the Observer Room” and “How to Run a Brainstorm.”

Early in 2016 we took our commitment to this holistic point of view even further by bringing two new functions into the Research family. Product Specialists at Airbnb are experts who bridge the gap between Product and our customer support teams. They are helping us do more with the rich sources of data generated by the thousands of guests and hosts who contact us every day and the agents who spend all day working with them. We also brought in the team that runs our NPS (Net Promoter Score) survey programs, and we’re expanding their scope into Survey Science– Airbnb’s center of excellence for all things survey.

Airbnb isn’t just about technology– it’s about relationships that hosts and guests build. Relationships will always be complex. There will always be doors and corners. But we believe that our commitment to multi-method, multi-perspective research with as many data sources as we can get our hands on will mean fewer blind spots over time.

 

Sharing our journey

These two principles aren’t the only parts of our foundation, but they are key pillars. We’ve made a lot of progress working towards the culture of research we want, but there’s more to do. The foundation is still uneven, as it is likely to be in a company that’s growing quickly, always bringing in new and talented people and pushing the boundaries of travel and social interaction.

As we continue to work towards our vision of Airbnb Research, we want to do that with honesty and transparency. This post is a new beginning in our efforts to share more of our processes and our insights. In the coming months and years we’ll use blog posts, conferences and events to share what we’re learning and to learn from you, with as much real talk as we can muster.

Our challenges in the years ahead include not just the growth of our products, our team and our company, but the growth of the Airbnb community that spans the world. Our research needs to be increasingly global. Just as we benefit from a diverse internal perspective, we benefit from a diverse external perspective as well. Most of our hosts and guests have different perspectives, knowledge, experience and needs than we do. Expanding our understanding and our empathy across the globe is a huge challenge, but it’s one that we think is made easier by the strong foundation of embedded, holistic research that we’ve developed and continue to build upon.

 

Judd Antin is Director of Research at Airbnb, a PhD social psychologist, a lapsed professional chef, current food and drink enthusiast, and still the least hippie resident of Berkeley, California.

Up Next