The simple guide to product testing

At Five and Shoutem, this is what we tell every single user during our in-person testing sessions:

I want to start by saying that we’re testing [the product], and not you. You can’t do or say anything wrong, and nothing is going to hurt our feelings.

And we mean it. We’re not actually testing the user, right? We’re testing the product, regardless of the stage it’s in, from a concept to a healthy, released product.

When we’re in the very beginning of a new project and validating a product idea through testing, we’re miles away from proper usability testing (which we do later in the project). This is just one of the reasons we don’t refer to these sessions as usability testing or user testing—we instead call it product testing.

“You shouldn’t do product design without product testing.”

These guidelines aren’t just for teams already building products. They’re also for anyone who’s just starting to think about new product ideas and ways to validate them.

While there are multiple things to keep in mind when planning and conducting a product testing, here’s a simple list that can help you focus:

1. Start testing the very first week of the project
2. Do more than ‘just’ usability testing
3. 5 users per target group
4. Do it more than once
5. Take care of the details

1. Start testing the very first week of the project

As an agency, we start new projects with a short discovery/research phase followed by a design sprint. Sprints are great when kicking off new initiatives, testing new product ideas and features, etc. We end the workshop by creating a medium-fidelity prototype (a rapid process for us with tools like Sketch and InVision) that we then test with users. We want to validate the direction we’re heading right at the start.

Can you imagine how rewarding that is? There was almost nothing a couple a days ago—now there’s a concept, a business case, and actual feedback from real customers.

What a great way to start a project.

2. Do more than “just” usability testing

Priorities in testing sessions change as a project evolves. We might start out really heavy on validating the product idea and prioritizing features, but as we progress, we may focus less on that and more of how things come together.

Let’s discover how the testing focus shifts throughout the project:

1. Concept: Validate the product idea with the desired target audience
2. Features: Prioritize features (especially when building an MVP)
3. Information architecture: Test out different approaches to information architecture
4. User flow: How proposed flows affect the user experience and behavior
5. Visual design: A series of tests related to visual design, voice and tone, readability, branding, naming, micro-copy, etc.

3. 5 users per target group

You’ve heard this a million times: 5 participants in a testing session is enough to get the majority (cca 85%) of the feedback you need at that point.

What’s important to note is that’s actually 5 users per a target group, meaning if you have primary and secondary type of users, you need to multiply that number.

Identify people who match your demographics (or even better: match your personas and then screen for behavioral traits, attitudes, and goals that match those of your users.

–Nielsen Norman Group

We tend to always have 6 humans recruited (plus floaters available for morning and afternoon sessions). The tendency to do more than 5 comes from experience—sometimes users who show up aren’t a good match, sometimes there are equipment issues, or there’s a fire drill (it’s happened more than once!).

4. Do it more than once

A very important thing for the entire team to realize is that early concept testing is by no means enough. Even though a project is underway, there should still be regular testing sessions as the designs get more fidelity, longer flows, richer features… as the final UX prototype has been created, as we jump into development. 

“Early concept testing is by no means enough.”

The most important thing is to guide those tests towards a desired outcome. For example, in testing the product in the visual design phase, it’s important for the tests, assumptions, and goals to focus on items directly related to this phase. While I’m positive you’ll encounter feedback related to other things than what you intended to test (everything from product fit to the actual user experience), capture that feedback and pencil it into the product roadmap.

5. Take care of the details

As always in life, the details are not the details. We’ve seen completely different testing results when one or more details weren’t executed with care. It’s important to keep the story we’re telling to users alive and not to break the flow. Users need to stay in the shoes we’ve asked them to put on.

“Users need to stay in the shoes we’ve asked them to put on.”

1. Use real content in your prototype. These users were sourced because they are the target audience. For instance, they frequently order food on mobile, they have previously learned Spanish, etc. They will definitely look at content in your prototype and sometimes be blocked by an easily solvable gap or error.
2. Know thy flow. You’re effectively guiding the user through a path without them knowing where they are in the flow (e.g. how many steps or screens are left). You need to guide them, and more importantly, you need to have a mental map of the flow in your head if something goes unplanned (as it often does). It’s always important to know what’s the primary focus on each screen so that the study goals can be properly validated.
3. Know your hardware. Set yourself up for success and don’t let “typical” hardware behaviors get in the way. I’m talking about screensavers, brightness levels, notifications, other apps, etc.
4. Know your software. Prototyping tools and behind the scenes interactions can sometimes conflict with intended actions for our users. An example: Swiping on mobile is (obviously) intuitive for users, even though a prototyping software might interpret this as [navigate-to-next-screen] rather than interacting with the screen the desired way. Although you envisioned the study user flow in one way, this one simple “feature” can disrupt the organic navigation through the screens. Bare that in mind when creating the ideal setup. If you’re using InVision, you can work around this by disabling the “Explore All Screens” feature.

At the end of the day

The most striking truth of the curve is that zero users give zero insights.

If you’re doing product design, you should do product testing. We hope this guide helps you in this process.