Design for relevance, then usability

Usability testing is great—when you’re trying to figure out if something is usable.

But what happens when you’re trying to understand whether a product or feature resonates? You’re likely using the wrong tool for the job at hand. 

The average usability test goes something like this:

You instruct a few tech-savvy shoppers to try a new product or service—like an online grocer—for the first time, and give them a discrete set of instructions: Find a cookie recipe on the online grocery website, then, using that recipe, add cookie ingredients to your cart. Finish by selecting a time for home delivery.  

You ask these users to think aloud as they navigate the website and discuss why they choose a particular brand of milk or eggs.

“Establish relevance first, and optimize for usability later.”

These shoppers are performing for you. Because they follow your instructions, your results nearly always come back skewed; the only thing you can measure is their ability to add groceries to their cart and click checkout. But you’ll never really know the drivers behind why they purchase online groceries, pick certain ingredients, or choose specific delivery windows. That’s understanding what’s relevant. 

We spend hours recruiting and screening users, defining business goals, creating tasks, composing scripts, crafting prototypes, and running usability tests—all while dancing around the question we really care about:

Relevance is finding product market fit. It’s seeing whether a user is even interested in a concept, product, or feature. It’s hard to measure relevance, but it’s possible—using digital ethnography, for example.

Imagine if you simply watched someone shop on Instacart. Then you shut up and observe them. With free reign to roam, the shopper might pull a recipe from another website, poke around Instacart, find some ingredients, and check their calendar for delivery. Your product’s experience may be linear, but your user’s experience with it won’t be.

In this test, the grocery shopper chooses what’s most relevant. And by watching them make choices on their own instead of following a set of directions, you’ll discover contextual clues that drive their purchase behaviors and the mental checklist people use when planning a grocery delivery.

You’ll gain empathy by observing natural user behaviors and, as a result, get a better read on usability. You’ll find patterns and unlock opportunities to improve your user experience.

But even by observing users, there’s still one missing piece: a clear picture on relevance.

A user typically needs just a few moments to determine whether your product, new feature, or idea is relevant to them, and just about 30 seconds to explain why. 

Over the past year, I’ve been working on an approach to give product designers rich “relevance” feedback that’s quick to gather and easy to process.

We’ve tested products in the idea stage, InVision prototypes, and newly launched mobile apps. Understanding relevance takes priority over usability and, even for mobile apps, usability plays just a small role in whether your product is relevant to users. There’s a reason Quettra shows that 77% of apps lose Daily Active Users within 3 days: Most products—from concept to executed ideas—just fail to resonate.

Everyone has a toolkit of apps, for personal and professional purposes. Take a moment to think about your toolkit; even if you have 50 apps on your phone, only a handful may be actually relevant in a given day. Here are a few in my toolkit:

• Hopper: I can shop around for flights, and get flight alerts and predictions from my phone
• InVision: I can create a prototype in minutes, and use the tool to collaborate with customers and colleagues
• Croissant: I can check into any co-working space when traveling and pay by the hour

But what happens when an already-relevant product introduces a new feature? I used dscout Sprint to determine whether Instagram Stories was relevant to real people. Here’s what they said:

“Instagram Stories is usable. Most anyone can open the app and view a story. The story feed seamlessly fits the Instagram UI. It’s easy to create a story, view a story, to tap back and forth between stories.” 

But are Stories relevant? Through their posts, Instagram users illustrate a story over time—creating a heavily-curated collection of memories that expresses each user’s favorite moments. There’s an entire ecosystem to help Instagram users craft the perfect photo and get the most likes. For users, swiping through Instagram is the equivalent of flipping through a magazine: You pause to read a caption, and you write in the margins of your favorites.

“Product teams shouldn’t misuse usability testing to infer relevance.”

It’s really easy to assemble an Instagram Story, but with Stories, it’s really hard to capture the Instagram-worthy moments that belong in a user’s main feed. Instagram, at its core, is about precious moments, not the fleeting, everyday moments that Snapchat harnessed into a billion-dollar product. The bar for good quality content is high enough that most users don’t even try to create a meaningful Instagram story. 

Usability is crucial to product design and development. But product teams shouldn’t misuse usability testing to infer relevance. They should perform usability tests to optimize validated, relevant features.

Establish relevance first, and optimize for usability later. If a feature—or product idea—isn’t relevant, ditch it. In the end, it’s possible that your product will be even more usable. 

So how do you test for relevance? There’s no boilerplate way to measure it. On the quantitative side, you can track email signups, whether people click buy buttons, and impressions. On the qualitative side, you can spend hours to recruit users and meet with them one-on-one, or you can take advantage of digital ethnography tools to test for relevancy in just a few moments. 

“Perform usability tests to optimize validated, relevant features.”

After a year’s worth of testing different ways to get feedback on concepts and determine relevance, I’ve settled on the following framework.

First, I give my test subjects a new prototype or app concept (even a one-pager works) with just enough direction to get them started. After they’ve explored it for a bit, I give them the following prompt:

You’ll often learn in 30 seconds what a usability test might show you in 15 minutes: whether your product is relevant

Thirty seconds forces the user to make a quick decision and only discuss what matters to them. Less is more.

Here’s what you’ll get in those 30 seconds:

• Hits: Learn what really resonates with users. Build around these for growth.
• Misses: Learn what falls flat. Expose where your product is weakest in the user’s mind. Discover holes in the experience.
• Wishes: Identify what would make the product relevant. Inform your roadmap by unlocking the features people think should come next.

These 30 seconds will also reveal what people don’t talk about. Most usability tests focus on watching a user’s ability to find specific aspects of a product. Why spend 15 minutes to discover whether an advanced feature is easy to access when you can simply ask whether it’s relevant in 30 seconds?

For example, you’d want to see if Uber jetpacks resonates before spending years to build the feature—and buy all that jetpack fuel. 

This framework won’t tell you everything; it’s a starting point for when a user examines a concept, plays with a prototype, or downloads and uses your app for the first time. It’s certainly worthwhile to observe users, to ask indirect questions, and poke at what excites users in the first place.

So when your team wants to run another usability test, ask yourself: Have you first established relevance?