Measuring Magic

“How can we measure magic?” That was the first question I was asked when I joined Airbnb as an Experience Researcher more than a year ago. At the time, I was part of a scrappy team pursuing a deceptively complex goal. Our CEO Brian Chesky had a vision: Make travel magical and easy. Now it was up to our team of 10 people to translate that idea into reality. We felt confident that we could develop a robust, quick, and user-friendly travel booking app. But “magic”? We weren’t even sure what that meant yet.

In November, Chesky announced our solution, Airbnb trips, a platform where you can reserve a place to stay and book unique experiences like truffle hunting with a Florentine and his dog, painting with a San Francisco native in her home studio, or shopping for records with a Motown expert in Detroit. The idea to empower people to host experiences, not just their homes, may seem obvious now, but it took careful research and collaboration to get there.

Along the way, I was punted outside of my digital comfort zone. Since travel is inherently offline, I could no longer just look at how people interact with screens. I had to figure out ways to study how they interact with others in the real world, which is an unpredictable, scary, and ultimately beautiful thing. Not surprisingly, I learned some lessons along the way.

Lesson 1: To ground a grand vision in reality, listen to your research participants.

First, we had to discover what magic was. Before I joined the experiences team, they created the most crazy, cool experiential trips to San Francisco for travelers from around the world. One of the first guests was Francisco, a gay Chilean man. Francisco felt like he didn’t quite belong in his home country, so we brought him to the Castro, the epicenter of gay culture in the US. Three people spent around 600 hours planning his itinerary, which included meeting locals who brought the city to life for him. The trip was literally life-changing. Francisco quit his architecture job, moved to Australia, and credits his experiences in San Francisco with making him feel like he could be himself in the world.

Clearly, the team had achieved some success, and it did feel magical. But they soon realized that they didn’t know exactly what made these trips magical or how to scale that magic. That’s when I was brought on. Several months after the initial guests returned home, I interviewed them on video calls and transcribed what they said. A pattern soon emerged. Everyone talked about gaining perspective and the most satisfied guests all mentioned that they grew as a person. One man said that meeting the local hosts had shifted his perspective on the world.

By carefully listening to our guests, we were able to shift our perspective, too. We began to realize that itineraries packed with expensive meals and exciting activities didn’t necessarily lead to magical trips. What made a trip magical was people — interesting community elders, engaging historians, friendly neighbors.

We had tracked down magic’s source and grounded it in the real world. Now we could focus on connecting travelers with locals in a meaningful way. In retrospect, empowering this new type of host unlocked the product development process, but it definitely didn’t feel like an epiphany at the time. Which leads me to my next lesson …

When working on a project with a high level of pressure and potential, researchers often wait for an aha moment. But for better or worse, that isn’t how most products are actually forged.

Lesson 2: Don’t hold your breath for a big aha moment.

When working on a project with such a high level of pressure and potential, researchers often wait for an aha moment. We want to reach a place where we feel like we’ve figured it all out. But for better or worse, that isn’t how most products are actually forged.

Refining the product required a phase of rapid iteration. This wasn’t sexy work. It took perseverance and repetitive testing. During this phase, I worked hard to represent the voices of the people I talked to. I ran a test trip almost every week, and on each trip I had the guests try out different technology and offline activities. We tried small groups, large groups, solo travelers. We tried dropping people mid-way into a group experience. We tested trips with story arcs. Basically, we tested every possible permutation of a trip. After each one, I debriefed the guests to try to understand what we should integrate into the final product.

As a team, we were learning more and more each week. Looking back now it would be easy to claim that we had one big aha moment. But instead we were deep in a process of accumulating a million little tweaks as we figured out what was working and where the momentum was leading us. At the time this didn’t feel quite so satisfying.  There was no moment where we thought, “This is it. Everything’s perfect. We’ve got it.” But over time, each of those small learnings helped get us to where we needed to be.

Lesson 3: Be prepared to reconnect with the larger company.

Eventually, the big moment arrived — it was time to scale and launch the product. We now faced new questions: How do we merge experiences into the main Airbnb app? How will people find them? Luckily, we had colleagues who had already researched similar questions for other products. We didn’t need to reinvent those wheels.

However, we did need to figure out how to bring those colleagues on board. For a year, we had been a team of 10 people, working in a little garage. Our co-workers hadn’t seen us, our project wasn’t on their roadmap, and now we were back saying, “This is what we’re doing now. Trust us.” This transition can be tricky.

Airbnb’s greater research team is a mix of 47 Survey Scientists, Product Specialists, and Experience Researchers. During this phase, I spent a lot time both sharing with these researchers and ultimately learning from them. And I found that the collaborative spirit inherent in research can help set the overall tone for functions, including engineering and design. Researchers with a wide range of expertise stepped up to help shape the new product. For example, a researcher who had spent a year studying how people search for homes was quick to help study how people want to search for experiences. And another Product Specialist, who had worked on launching Airbnb business travel, created a system for capturing feedback before, during, and after our launch.

Our incubated team had been focused and fast. I had been the only researcher in the room for many months. But it was time to shift from being the research pilot to becoming more of an air-traffic controller. The style of work shifted from hyperfocused iteration to more sharing and comparing notes. This work is often easy to ignore, but ultimately crucial to the success of research and the product. Once it’s time to plug into the rest of the company, you need to slow down and find out where people are coming from on a human level. Find out what your colleagues know, how they can help, and take time to patiently share what you know.

The power of human-centric research

This is the most human-centric project that I’ve ever been part of. Every piece of it was built by people, in cooperation with people. Personally, it forced me to step outside of controlled lab studies and the reassuring constraints of digital experiences, and jump into the messy, unpredictable real world. When you put four people in a room together — let alone on a bike trip in a new city — the guard rails just aren’t there. The group might not get along. It could start pouring rain. Something you never thought of might happen.

But with all that can go wrong and the setbacks that can follow, this human-centric approach leads to deep insights, and maybe a bit of magic, too. I can confidently say that the product is stronger than it would’ve been without this approach in the mix. We weren’t doing precision experiments, but we were learning. We just had to be open to the lessons being as unpredictable as people can be. I am proud of the product we have created, but I will be the first to tell you that I expect to be researching, iterating, and learning about it for years to come — championing people, not just technology.