Getting started with accessibility user testing

Updated Tuesday, November 13, 2018: We’ve added a few more bits of advice for those just getting started with accessibility user testing. Let us know what you think!

With accessibility user testing, the main goal is to verify whether your assumptions concerning accessibility features were right. The objective is to discover potential problems and opportunity areas based on the already working product or the prototype.

For the examples in this post, I’m going to focus on users with visual impairments. The recommendations I’ll share are based on my experience working at Polidea, a design and development studio that delivers digital projects.

Related: The ultimate guide to web content accessibility

I’m hoping your team can use this as a guide for getting started with accessibility testing, as it’s just so important that we’re creating products with all users in mind. So, let’s get started.

1. Set your goal

You’ve got to understand the end goals of all design actions you decide to perform. So before you begin accessibility user testing, ask yourself:

Why do I need this research? What do I have to verify?

Naturally, you already have some assumptions concerning certain features that may be problematic for test participants. Write them down, along with a set of focus points—these will be the most important elements in your project that you have to test carefully.

Choose the right methodology

Based on my experience with accessibility user testing, I recommend In-Depth Interviews (IDI) for 2 reasons:

1. IDIs help participants feel more at ease

Accessibility user testing needs to be done in a private setting—that’s really one of its core elements. It’s important to create a comfortable environment, allowing participants to relax and talk openly about their everyday habits and difficulties arising from their disabilities. Thanks to IDI, you’ll be able to elicit candid responses and put your participants at ease.

2. IDIs offer deeper insights

With one-on-one talks, you can devote full attention to each participant. You can observe their body language while they use the app. You’ll also have time to investigate new and interesting subjects that appear during the interviews. Each participant will have more time to share thoughts and feelings. Also, there will be no group influence factor.

Hint: Schedule all interviews for one day and prepare templates for your notes with focus points carefully underlined. After several interviews, you’ll notice that some things come up regularly.

Recruit wisely

Define your criteria

Be clear about the type of users you want to work with. You can use this open-source document prepared by Michael Margolis, a UX Research Partner at Google Ventures: Worksheet: Writing a User Research Participant Screener.

Remember that in order to test your product with representatives of the given group of users, there must be at least few of them.

Hint: Let participants use the platform they’re familiar with, especially if you test for accessibility. For example, iOS users usually face problems while using Android. For users with disabilities, it can be even more difficult and stressful.

Find your test users

Don’t hesitate to work with a recruiting company—it can save you a lot of time! Nevertheless, don’t forget that recruiting people with disabilities is more challenging. It may require contacting appropriate professional associations, community groups, and university clubs. Start the process well in advance and prepare a set of materials explaining the scope and objective of the research. Since accessibility is such an important aspect of all digital products’ creation, people are eager to help and cooperate.

Hint: Recruit more people than you need for your user testing session, just in case some users resign.

Hint 2: If your participants don’t speak English, you’ll need to provide them with translations.

Prepare the structure of an interview

Plan it carefully

Be ready to prepare the detailed plan of your interview and conduct it step-by-step later on. The following example may be of some inspiration for accessibility, as well as for regular user testing:

Introduction: introduce yourself and discuss the logistics. Answer basic questions, like:

• Who are we?
• Why are we here and what’s the goal of today’s meeting?
• Where are we?
• Who is involved?
• What will be everyone’s role?

When preparing your interviews, don’t forget about accessible materials!

Prepare a good invitation and give a proper welcome

Remember to include the address and the direct link to a map in the invitation. Try to be as specific as possible in providing your guests with helpful information:

• Which floor should they go to?
• Where is the intercom placed?
• Who should they call if they have problems getting in?

Try to be empathetic—think what they may need and what difficulties they could face.

Before the meeting, get acquainted with Disability Etiquette (opens PDF) and with these inclusive communication tips.

Related: How to get started with user testing

Introduce the setting and people involved

It may sound like this:

“We are in the observation room. A place dedicated for user testing. There are just 3 of us. While we will be talking, Anne will take notes and observe your performance. There is a one-way mirror in the room. At its other side sits Tom, who is ready to help us with tech issues if somethings goes wrong.”

To build a rapport with visually impaired participants, imagine that you’re their eyes now—you’re responsible for providing them with all the information they need. Also, remember not to involve too many people in the session—it’ll be more comfortable with a smaller group.

Hint: Show that you’re on the same team with participants, and you’re not just there for the designs you’re testing. Remember that you’re working for your users.

Then come the following questions:

• What will you be asked to do for us?
• How long will the testing session take?
• Are we going to record you? Do we plan to share the material from our session?

While providing the answers, try to be as clear as possible. Creating an atmosphere of comfort and mutual understanding is crucial.

Scenarios: Get straight to the point

Start by asking participants to perform concrete actions. Try not to refer to them as “tasks,” since they aren’t something you can succeed or fail at. Scenarios should have concrete focus points: things you’re interested in and elements that you feel can be challenging for the users. Get straight to the point.

Hint: It’s important to highlight the main idea of user testing, meaning: gathering feedback. There are no bad answers!

Although encouraging users to speak aloud while using the app is usually helpful, remember that during accessibility testing you have to be very sensitive. Don’t force your user to do that—instead, offer some alternatives in case some participants find this type of communication difficult. If you notice that someone feels uncomfortable, suggest they express their opinion after performing the task.

Ask follow-up questions to investigate the problematic areas. They may be inspired by the following list:

• Who told you about this?
• What made you click that?
• Why did you decide to go to the next screen?
• Where did you hear that?
• How would you change the app’s options?

Summary and closing

Thank all the participants for their time and help. Specify any follow-up information, and ask if they have any questions.

Synthesize and share

Keep the momentum going and write a concise report (visual presentation) with the recommended solutions right after the testing session. Carefully consider all the feedback, and avoid assuming that one person speaks for everyone.

Related: 7 guidelines for writing accessible microcopy

The report structure should be simple. Just write down:

• What you did
• What you heard
• What the design recommendations are
• What you suggest to do next

It’s not about writing an extensive report. Include a few of the most important insights you discovered, along with a short but strong argumentation based off what users said during the session.

Summary

Technology makes it possible for those with disabilities to be independent and active. So it’s crucial to create digital products with all users in mind.