User testing gone wild: A guide to course correction

So you have a usability test this afternoon, and you’ve thought of everything. You’ve built a prototype, written a script, gargled, flossed, and gone through some vocal warmups.

Excellent. You are so ready for this.

Most of the time, testing sessions go great and your team collects honest, unfiltered feedback. But sometimes the unexpected happens. No matter how much you prepare for an interview or usability test, you still need to stay on your toes and quell some hiccups along the way. These hiccups can come from you, your team, your materials, your environment, the users, etc. Some are obvious, and some are not so obvious.

Let’s go over the most common user testing problems—and ways to address them in the hopes of avoiding them altogether.

The participant clams up

They’re sweaty, their arms are crossed, and their hands shake when they touch the tablet. Or worse, they start to back away from you… slowly. Body language is the most important indicator of a person’s comfort. If they slowly roll their chair to the other end of the table as you speak to them, you need to rethink your approach.

How to avoid this:

• Build a rapport with the user before you transition into testing. If you’re at their home or place of business, allow them to greet you and show you around. If they traveled to your office, ask them how their trip was and if they need any refreshments. Shifting the focus onto them shows that you care about their needs and what they have to say.
• Learn how to read body language to empathize with your user. Once you become aware of what their body is telling you, you can start to adjust yours accordingly.

“User testing participants should feel comfortable making mistakes.”


• Know that usability tests can be distressing for users. They might not be as familiar with this process as you are. They can easily feel like they’re being watched, tested, and judged by a panel of professionals. Not a fun feeling. It’s your job to make them feel as comfortable as possible so that you can gain their trust. And having one person facilitate will help them open up and ultimately feel comfortable making mistakes.

During the session:

• Check your posture. Are you crowding them or leaning in too much? Loosen up and give them more breathing room.
• Check where you’re sitting. If you’re sitting across from them, swing around and sit next to them to show that you’re with them.
• Ask them if they need a break.
• Make them laugh! There’s no better way to ease the tension.

Too many people talking at once

Your team has gone rogue. Everyone started out on their best behavior. The facilitator facilitated, the observer observed, and the note-taker took notes. But there’s been a shift and everyone is now participating. Your allies are talking over each other, asking questions out of sync and in rapid fire. But wait! You were the one leading this.

How to avoid this:

• A usability test should include no more than 2 people from your team. Any more than that, and things can get uncomfortable for you and the user.
• Make sure everyone knows their role before user testing begins. One person facilitates and talks to the user while the other observes and takes notes. Anyone else is a quiet observer.

“A usability test should include no more than 2 people from your team.”


• If you’re an observer and have a question you’d like to ask, write it on a Post-it and pass it to the facilitator. Have just one person talk to the user to help them forget about the other people in the room. This will in turn make them more comfortable and more likely to open up.
• Debrief after each session. When you finish talking about what you learned from the person, switch to team execution and dynamics. Use this time to call out specific instances of interruptions and missed opportunities. Communicating frequently and openly makes these conversations feel less accusatory and more constructive.

During the session:

• Maintain eye contact with your team. The second you stop looking at each other to give and read cues, you’re disconnected and are more likely to act as individuals rather than as a unit.
• If a teammate is overexcited and talking more than the user, find a non-verbal way to tell them to ease up a little. A gentle arm squeeze and smile can go a long way.
• After the test, remind your team that the user should be doing 99% of the talking. We are merely there to listen and learn.

You ran out of questions

You went through that script way faster than intended. The user performed wonderfully and wasn’t stuck on anything. But you have a full 20 minutes left before the session ends. You start to get a little nervous, and now you’re asking one-off rhetorical questions that you know won’t tell you much.

How to avoid this:

• Loosen up your script. Write down topics you’d like to cover in addition to the questions you want to ask. Having topics allows you to go wherever the conversation leads, rather than sticking to questions that might be rendered irrelevant after the first couple of minutes.
• Incorporate a bonus section into your script that covers some nice-to-have questions. These are things that could go unanswered, but would be interesting to ask. For example, “If you had a magic wand that could do anything to this app, what would you do?”

During the session:

• Ask your user to show you their favorite apps: the ones they used on the way to the session, the ones they find simple or complicated, the ones they can’t live without. Have them perform tasks and explain to you what happens at each step. See if they have one that’s similar to what you’re testing and learn about what works and what doesn’t.
• They don’t have a smartphone? Then ask them to tell you some stories. Think about the tasks they’ll perform in your app, and ask how they do them now. Even asking about what they did yesterday could be helpful for your research.
• If you’re truly done with the test and feel like your questions have been answered, end early.

Your prototype is busted

But it worked in the office! We know. But if it depends on wifi, and you’re in a location where it’s not available, then you’ll have a problem.

How to avoid this:

• Host all of your files locally so you don’t need wifi. This includes scripts, images, libraries, etc.
• Test it yourself before you present it to someone else. Go through all the tasks to make sure it’s not broken.
• Bring the device you’ve been testing on. Testing on a new device can throw all sorts of unexpected wrenches.
• Bring print-outs of your screens as a backup plan. Have the user tap on elements and ask what they’d expect to happen next.

During the session:

• Turn to paper. If you don’t have print-outs with you, quickly sketch out the parts of the prototype that you’d like to test and ask questions about.
• If you don’t have the time to draw anything yourself, have the user do it. Ask them to draw from memory whatever they tool they are using now to get the job done. Just seeing what they remember will give so much insight to whatever you’re building. You can also ask them to draw the ideal interface they would want to use.
• As mentioned in the previous scenario, asking them to show you apps on their phone and to tell you stories can really help if your prototype is unsalvageable.

Your facilitator is leading the witness

They started out just fine. But they gradually spiraled into the land of leading questions with yes or no answers. Now they can’t stop starting questions with: “Wouldn’t it be great if…?”, “Do you like it when…?”, “Would you use this feature…?” These types of questions stifle conversation by making it difficult to ask follow-up questions.

How to avoid this:

• Spend time with your script and turn any leading questions into open-ended ones. “When was the last time you…?”, “What do you think will happen when you…”, “How often do you…?”.
• Conduct some practice interviews with your teammates. Two of you can take turns asking each other questions, while a moderator points out examples of open-ended or leading questions.
• It takes time to get into the habit of asking for stories rather than opinions. The more interviews you do, the better you’ll get!

During the session:

• Ask the user “why?” after some of their answers. This can help open the conversation a bit, and may naturally get things back on track.
• Communicate to your facilitator via Post-it notes. Scribble a gentle reminder asking for “open-ended” and “stories.”
• Write down some examples of their leading questions, so you can give them pointers after the interview. Having concrete examples makes it easier to give constructive feedback.

The user is stuck on fake data

In your prototype’s world, $2.00 + $5.00 = $10.00, and the user is having a hard time believing it.

How to avoid this:

• Be diligent about the fake data you put in. If you’re showing a shopping cart, make sure the prices add up. If you’re trying to show a history of orders, make sure that all the dates aren’t the same. If you don’t want to use any values, draw some scribbles and tell the user that it’s data.

Photo by #WOCinTech Chat. Creative Commons Attribution-ShareAlike 2.0 Generic.



• Don’t use lorem ipsum. Use real names and real(ish) descriptions. Making your data coherent can save you a lot of time during the test and prevent any confusion. It can also bring up things you didn’t realize were issues, like character length.
• Don’t use the pictures or names of people they may know, like celebrities, yourself, or your team members. They have the potential to create unwelcome distractions and assumptions about your product.

During the session:

• Take a moment to explain that because this is a prototype, the data is neither real nor relevant for today’s test. Be sure to mention this in an intro at the beginning to alleviate any questions or confusion.
• Keep redirecting their focus to the task at hand.
• If they really can’t see past the fake data, ask them why. Maybe it’s not about whether or not 2 + 5 = 10, but that they were expecting something else entirely.

When all else fails

Take a deep breath and remember that you’re all here with the same purpose: to make your product a success. The fact that you’re doing usability research shows that you care about your users and want them to love your product as much as you do.

So relax—and your users will, too.

This post was originally published on the thoughtbot blog.