How Zapier amplified learning during product development

If you’re unfamiliar with Zapier, it’s a bit like LEGO. We provide you with the building blocks and you combine them to automate some part of your work. For example, one block might be a new starred message in Slack (or your favorite chat app). Another block might be creating a task in your favorite to-do list. When you combine these 2 blocks, you can automate sending data from Slack to your to-do list.

Like LEGO, the options with Zapier are endless. You can combine any number of steps in any order from over 750 different apps. This is both awesome and a little daunting.

Until today, Zapier has been a single-player product, which has made it hard to share what you’ve built with your team. In the same way, it’s more fun to build LEGO with others—and we believe it’s more productive to use Zapier as a team.

The problem

Last year, Zapier launched multi-step Zaps. This was the biggest change to our product since we started in 2012. Before this, the product allowed you to only connect one trigger with one action. With this update we enabled users to connect multiple apps in a sequence. Since then, our customers have created millions of multi-step Zaps. Some workflows are almost 100 steps long!

Our user base matured along with the product. These new, complex workflows are no longer maintained by a single person. Instead, teams of people wanted to share responsibility for creating and maintaining workflows.

We also started to see a particular type of user emerge: the efficiency evangelist. These people were creating Zaps for other people in the organization. They’d become the go-to-person for automation in their company.

There was a problem though, best illustrated by this quote from one of our customers:

“I try to keep my team off Zapier so that I’m only one who uses it. Once, I went on vacation and came back to a dozen broken Zaps because of a small change one person accidentally made. It took 3 days to fix!”

Ouch! Not a glowing recommendation for an automation tool. Instead of being able to empower their colleagues, they needed to act as a gatekeeper. As we interviewed customers we started to see other problems emerge:

“I don’t want to be the only person responsible for this Zap. If I’m sick or on vacation, someone else on my team should be able to log in and update it.”

And from others we heard:

“I’d love to know how other people use Zapier. I’ve got a few Zaps set up, but I bet if we could share ideas we’d be able to really boost our productivity.”

There was no good way to share a Zap. The only solution was to share your login credentials, and this wasn’t a great solution for a few reasons. First, sharing credentials isn’t secure. Second, managing access becomes a headache. Third, there is no way to know who created what.

Zapier’s first product team

At this point, Zapier was still a small organization and we could only work on one thing at a time. Stories like this showed us that we needed to scale how we build the product.

Enter Zapier’s first product team. Based on successful product organizations, like Spotify and Intercom, we hired our first product manager (yours truly) to run this new team. This meant we could now work on multiple things in parallel. And we got to work right away.

When I joined, there was a lot of momentum in the organization behind making Zapier multiplayer. We started by collecting more actionable feedback from customers. Up until this point, most of our knowledge came from all-hands support and ad hoc user research. We placed a short survey right within the product. In under a year, we had more 3,000 people request the feature and fill out the survey.

The process for building Zapier for Teams took us over 18 months. During this time there were all kinds of tough decisions to make and problems to solve. We knew that our ability to learn during this process was going to be critical. So here’s a look at some behind-the-scenes tools we used to amplify our learning during product development.

How we learned during product development

1. Informational interviews

When we first started out, we knew there was a lot we didn’t know. Of course, we were talking with our customers all the way along. Sometimes, we weren’t even sure if we were asking the right questions.

So early on in the project, we decided to interview other product teams who had built team or business accounts into their product. We spoke with product teams at Typeform, Dropbox, Buffer, Github, Unbounce, and Trello. Our main question for those teams:

What do you wish you had known before you started designing and building your team features?

Here are a few of the main things we learned from those interviews:

1. Test the core value that teams will deliver for your customers. For any given product, the value of a multiplayer version will be different. Focus on learning this before trying to optimize revenue.
2. Optimize the invite flow and new member activation rate. The value of a team account typically grows as the number of active members grows. This should be an early focus for the product team.
3. Don’t invest in real-time collaboration too early—it’s not the core value. Thanks to Google Docs, we’ve all become accustomed to real-time, concurrent document editing. But early on your focus should be on the core value of teams accounts—and that may not be real-time editing.
4. Engineer flexibility into the system so you can experiment. We heard this a lot, particularly around pricing. Your first assumptions about pricing will probably be wrong, so try to avoid painting yourself into corner.
5. Most decisions you make will be easy to reverse. A few won’t. We heard a lot of painful stories about companies having to rework the way they handled user identities. The main advice: your customers don’t want this today, but when your user base is 10x the size, they will. And it may be tough to build that in later.

Informational interviews provide a lot of insight for very little investment. When you’re respectful of people’s time and you’re specific about what you’re looking for, a lot of people will say yes to a request for an interview. So give it a try!

2. Low-fidelity vs. high-fidelity prototypes

You’re reading the InVision Blog, so I’m pretty sure I don’t need to convince you of the importance of testing prototypes with your customers. Instead, I’ll share a way of thinking about your prototypes that we’ve found invaluable. The fidelity of your prototypes should align with the type of feedback you’re looking for.

Low-fidelity prototypes are primarily conceptual and contain only a few details about the interface and interactions. We found this extremely helpful in our early research when we were exploring the concepts and mental models behind sharing. In early stages, feedback on a quick sketch or wireframe can provide significant insights. It also prevents you from falling in love with your design prematurely.

Related: How to defend your work against terrible feedback

However, as you move along in your product development, the values of these low-fi prototypes start to diminish. High-fidelity prototypes look and work almost like the real thing. When we create these, we spend a lot more time making sure the prototype includes all the various screens, interactions, interface, and copy elements that the real thing would have.

While these require more investment, they’re still much cheaper to build than coding something up.

Of course, the highest level of fidelity isn’t a prototype but the product itself. As such, we also spend a lot of time watching our customers use our product. For this, we’ve found FullStory to be an invaluable tool.

Choosing the right fidelity for your prototypes can be a massive productivity hack for your UX research. Too much detail too early and you’ll get the wrong type of feedback. Too little detail in later stages won’t be an efficient use of your time—or your participants.

3. Feature switches

Shipping early and often is pretty common-place advice these days. We all know that the sooner we ship something, the sooner we can get feedback from our customers. But that can be easier said than done, particularly when you have millions of users.

One of the things that allowed us to do this confidently was something called “feature switches.” Feature switches let you enable a specific feature for just a subset of your users. Here are a few ways we used feature switches:

1. Releasing internally first. One of the big benefits we have is that our whole team uses our product on a daily basis. This meant we could ship a version of team features that we’d be comfortable with just our team seeing in a couple of weeks rather than wait months. Our team could then provide immediate feedback and we were able to prioritize the biggest issues we needed to solve next.
2. Running a private beta. We were able to release the feature to just a subset of users who had agreed to try the features, were okay with a bug here or there, and were willing to give us feedback.
3. Gradually releasing a feature. When we first released shared billing, we were pretty confident in what we’d built, but we used a feature switch to release the feature to only 10% of our user base. We waited to see if any issues came up, knowing we could hide the feature again if there was a problem. Over the course of a couple of weeks, we increased the volume and eventually released it to 100% of customers.

Using feature switches is something you might think only the big guns use. But they can be an invaluable tool for any product team. Most software frameworks will have a library or package that should help you with this.

4. Customer feedback systems

There isn’t really a point in shipping early and often if you don’t have systems in place to collect feedback from your customers. Also, don’t expect that customers will just reach out when they have problems or feedback. More likely, they’ll just silently stop using it.

Here are few of the systems we set up to collect feedback during product development:

Automated emails

Don’t underestimate the power of just asking for feedback. Every time someone signed up for a team account, I would wait a week and then send them an email asking for feedback.

Here’s what I wrote:

Subject: quick question about your Zapier team

Hey Mike,

Chris from Zapier here. I head up the group building Zapier for Teams. I just wanted to drop a note as I saw you created a team and I wanted to introduce myself. Since you’re using Zapier with your team, I’d be interested to hear any feedback or questions you have.

Specifically… what was your experience like when you created your team and invited team members? Is there anything we could improve?

I look forward to hearing from you!

Cheers,
Chris Geoghegan
Product Manager @ Zapier

This email got a pretty good response rate, and when asked, our customers were pretty open about what they liked and what they didn’t.

Since Zapier is an automation company, you won’t be surprised to hear that emails were automated. However, I wanted to make them feel as personal as possible. They would be automatically created as drafts, and then when I had time to respond—usually first thing in the morning on Tuesday or Thursday—I would send out a batch of emails. This way I’d be able to respond promptly.

Slack Advisory Group

During the product development process, we also set up a Slack team where we invited about a dozen beta customers to be involved in the development process in a more interactive way.

If we were facing a tough decision, we would put it out to the group and get immediate feedback. If we had a new prototype we were testing with customers, we’d also put it out the Advisory Group for their feedback. As soon as we released a new feature, typically behind a feature switch as described above, we’d let the advisory group know and we’d start collecting feedback and having conversations about it in real time.

Customer support

One of the things we do at Zapier is “all-hands support.” I absolutely love this. Even if you’re not a full-time customer champion, you’ll spend 4 hours every week answering support tickets. All-hands support lets me see first hand some of the questions and challenges our customers are having with the product.

These all sound great, but does it really make a difference? Yes. Let me share a quick anecdote. We had to make a decision early on about whether team accounts would be a new account that a customer would create or an upgrade to an existing account. Based on all the research and inputs we had at the time, we decided to make it a new account.

However, after shipping this we started hearing feedback (through the channels above) that this was confusing. So we decided to reverse that decision and make teams an upgrade. Have a look at what that did to the number of new teams we were adding each week:

The danger of relying entirely on quantitative metrics is we might have just thought that this was the norm. We might have confused what we saw with a lack of demand. The fact that we also had systems to collect qualitative feedback is what saved us from making that mistake.

5. Threat modeling

The vast majority of our product development is driven by things we learn from our customers. But as we were working on this project, we started to see a problem: a lot of our customers don’t think about the data privacy implications.

We never wanted our customers to be in a situation where they had shared access to data that they didn’t intend to. At the same time, there was no feedback from our customers asking for this feature or improvement.

We used something I’d never used before but will definitely include in my product development toolkit from this point on—threat modeling. I could write a whole blog post just on this topic, but in short here’s how it works:

1. Describe the current situation (e.g. How does data flow through your system?)
2. Brainstorm possible risks (e.g. How might data privacy be breached?)
3. Prioritize those risks (e.g. How common or severe are these issues?)
4. Identify possible solutions (e.g. How might we mitigate or eliminate the risk?)
5. Build the solutions

Here’s an example of an improvement we made based on this process:

One of the key ways data flow through Zapier is when you connect different apps you or your team can use. You have the option to share this connection with your team or keep it private. The problem we saw is that some customers might share a connected account mistakenly or without fully understanding the implications.

We decided to mitigate this issue by making all connected account private by default. If you wanted to share the connected account with your team, it was an explicit second step. As part of that explicit second step, we showed the customer a warning about exactly what they were sharing.

Takeaways

• Informational interviews with product teams that have traveled the road before can help you think about the right questions you and your team should be asking
• Low- and hi-fidelity prototypes serve different purposes. Make sure you’re using the right kind and the right time.
• Feature switches help you release on your own terms and encourage shipping early and often—which means you learn faster
• Make sure you develop systems for collecting feedback as early as possible. The product development process is only as good as the inputs your curate.
• As much as customers should drive your product development, sometimes you have to find features that protect your customers—even if they aren’t asking for it.