Measure UX for better understanding

Grammar tables vs. real communication

One summer semester in college, I flew to Costa Rica to study Spanish and finish my degree minor. When I got off the plane, my program director informed me that I wasn’t allowed to use my native tongue. That would force me to pick up Spanish.

My host mom, despite having walked through a nasty thunderstorm, met me with a radiant smile. As the rain subsided, we walked between distant volcanoes to what would be my home for the next 3 months. She gave me my own set of keys and welcomed me into the living room. There we sat together over coffee—she stared at me and I stared at my coffee—in awkward silence.

I’d studied Spanish for years in high school and college, but only a childish, stumbling collection of words spilled out as I tried to speak with her. I panicked and the next thing I knew I was speaking a Spanish-Italian hybrid language. Frustrated by the grammar tables running through my mind, I felt like there was some barrier I couldn’t get past.

Over the next couple of days, as I listened to the family speak with one another, my mind adjusted. I stopped thinking in translations. One morning, walking into the kitchen, I decided to go for it and I told my host mom a joke. I’ll never forget her laughter—or her surprise—and my relief when we finally connected. At that moment, we began communicating and I was able to make the most of our time together.

Related: Can UX design be taught?

Last year I took another trip, this time to the Nielsen Norman Group’s UX Conference in Chicago. My goal was to expand my skills and learn strategy. Sitting in the Measuring UX course, I expected to hear a familiar lecture about key performance indicators, task timing, and completion rates. Instead, the instructor applied statistical calculations to those times and rates. She demonstrated how much more there was to learn.

As she began to explain confidence intervals and pull out Excel calculators, it hit me: this approach—UX measurement expressed as numbers—was the communication “tool” I had been looking for in all these years of working in design.

When “lost in translation” means lost opportunities

In the past, while speaking with decision-makers such as engineers, product managers, and vice presidents, I sometimes experienced that same awkward silence that I felt in Costa Rica. Those conversations felt like my struggle to communicate with my host mom. When I mentioned usability best practices, heuristics, or qualitative anecdotes, the stakeholders heard me but it wasn’t making an impact.

“Making calculations can help UX researchers paint the larger picture for stakeholders.”

Listening to the instructor in that Measuring UX class, it struck me that the use of applied statistics in UX is the perfect language metaphor. In learning a new language, there’s usually not a direct translation, but there is a common message to aim for. In UX, we tend to speak from a place of empathy while our stakeholders are speaking about conversion. Even though we come from different places, we can find a common understanding.

There are things that a UX researcher cannot quantify, such as how a human being—with all their hopes and dreams—will interact with a product or service. There’s no magic number for that. There are certain things that we can put into numbers like task times, completion rates, and satisfaction scores. As Alexandra Mack, a Senior Fellow at Pitney Bowes, likes to say, “Make it an economic argument.” Making calculations can help UX researchers paint the larger picture for stakeholders, to make a stronger impact.

The great statistician W. Edwards Deming said it best: “Without data, [I was] just another person with an opinion.” Stakeholders are often making $100,000+ decisions. It’s our job in UX to inform part of that decision, often with small sample sizes. I realized I’d been missing part of the message. When speaking with business and engineering teams, it’s more powerful to use numbers, while keeping the client population in mind.

Data provides a common language

With statistical calculations, I can now “translate” the message that I want to convey for stakeholders who need a measurable insight. I can now “translate” the behavior that I observed with a small sample and apply it with greater certainty to what we might observe in the larger client population.

This approach allows me to present my expectations to stakeholders in numbers. For example, I’ll say that I can expect a plausible success rate between 54–60%, with a 3% margin of error, and I’m 95% confident that more than half of our clients will complete a task successfully. This information, coupled with industry standards and best practices, gives me stronger footing for representing our client in broader, cross-disciplined meetings.

Qualitative data gives us insight into the ‘why’ of human behavior. And, when it’s available, quantitative data can be used to balance our findings, as these figures tell us “what” a client did, as well as the “how many/how often.” When we interview clients about how they solve a problem, what they say is qualitative data, and what they do and how many other clients do the same thing is quantitative data. The 2 perspectives complement each other and give us a more rounded view of our clients’ realities.

There is still a lot of work to be done to integrate this statistical approach into our current process. But I’m happy to have this additional “translation tool” to improve communication across product teams and between business units, and to help us build products based upon our clients’ wants and needs.

I’m looking forward to continuing to share this useful skill with other UX professionals. When we’re better able to connect with one another other, that translates to more productive meetings, better employee engagement, and better understanding of our clients. And I believe that truly drives innovation.