Learning velocity matters—here’s how to increase yours

How much do you really know about the people you serve as customers? In reality, probably not much.

Designers operate in an environment of uncertainty. We have to make decisions based on limited information.

Yet we can change the environment we operate in. By conducting experiments, we can learn more about the situational context people find themselves in when our product or service becomes relevant. We can use what we learn to create more customer and business value.

Related: How to coach design experiments

This article will give you a template you can use to design experiments. We call this template the Experiment Card.

Your experiments can focus on almost anything. The Experiment Card will help regardless of what you choose. What matters is that you use the card as a foundation to test, learn, and take action on the new insights you generate.

To introduce the card, let’s take you through the experiment design process.

An introduction to experiments

This may bring you back to high school, nostalgia and all. If you haven’t conducted an experiment since then, this will be fun.

An experiment is really just a process that enables you to support or refute a specific hypothesis. Typically, the process works something like this:

1. Define your focus
2. Isolate your variable(s)
3. Frame a clear hypothesis
4. Design the experiment
5. Execute the experiment
6. Collect data
7. Synthesize data and draw a conclusion

Our Experiment Card follows a slightly different process—a process we’ve found makes more sense in a business context. Throughout this article, we’ll explain how you can use the Experiment Card to design an experiment.

If you want a detailed explanation of all 7 steps, here it is.

So let’s get started by using the Experiment Card to design the first version of our experiment.

The Experiment Card

At >X, we use the Experiment Card as our frame of reference for the experiments we conduct internally and on behalf of our clients.

We print cards out, make them visible, and version control so we can track progress over time.

Below is a step-by-step guide for how you can use it to start designing your experiments.

Step 1: Start with why

This isn’t a Simon Sinek thing. This is where you set the context for your experiment. It’s how you define your focus.

We’ve learned that business objectives and customer insights are the best place to start. They help you focus on things your customers or business stakeholders really care about. They also directly inform the hypotheses you develop.

If you start your experiment design process with a business objective, make it explicit. Map it to OKRs (objectives and key results) or something tangible and broadly accepted.

Here’s an example:

In this example, we’ve set a business objective. Yet we’ve recently learned our current sales cycle and resources won’t enable us to achieve that objective. We need to fundamentally change our environment.

This will help us frame a hypothesis, conduct an experiment, and hopefully uncover new ways to achieve our business objective.

If you’re starting with some type of customer insight, try to focus on specific attitudes or behavior. Back it up with numbers where possible.

An example:

In this example, we’ve discovered there are 3 primary channels that drive almost all recent traffic to our website. This is interesting behaviorally, but what does it mean? What can we do about it? What’s our inference?

Again, this is an opportunity to frame a hypothesis and design a series of experiments to learn about how we can take advantage of this dynamic.

So let’s focus on the customer insight and progress to the next step.

Step 2: Here’s what we’ll try and prove

A hypothesis is really a guess. Based on your current environment and the information at your disposal, you think something might be true. But you need to find a way to support or refute what you think.

Hypotheses can sometimes seem like complex beasts, so let’s explore them a little further before continuing.

Hypotheses have categories. The most common are:

1. Simple hypothesis
2. Complex hypothesis
3. Empirical hypothesis
4. Null hypothesis
5. Alternative hypothesis
6. Logical hypothesis
7. Statistical hypothesis

Hypotheses also have independent and dependent variables. An independent variable is the cause—the thing you have control over. A dependent variable is the effect—the thing you’re going to observe.

But the key is really how you frame your hypothesis. Keep your hypothesis simple, and ensure it’s:

• Comprehensible
• Precise
• Testable

Feel free to do more reading, but this may be one of those times where KISS comes in handy. For us, the simpler we make this process the better. Although we’d love to think we’re mad scientists, really all we’re trying to do is find ways to increase our learning velocity. This doesn’t have to be complex.

With that said, let’s continue with our example.

We’ve taken our customer insight and started to think about what it might mean. We’ve always hoped our content has a positive impact, but we’ve never focused on proving it. This is our opportunity.

Step 3: Put on your lab coat

It’s time to start designing our experiment. Let’s keep it simple and focus on executing a process we are confident will help support or refute our hypothesis.

Here’s our example continued.

In this experiment our independent variable (our cause—the thing we can control) is the content we’re going to introduce to 50% of the sample population. Our dependent variable (the effect—the thing we’re trying to observe) is the outcome of the discussions we engage in.

What we’re trying to understand is whether there is a causal relationship between people reading our content and making the choice to work with us.

In an experiment like this, there’s only so much we can control for. It’s therefore pretty rough. The best we can realistically hope for is supporting or refuting a correlation.

So don’t be too hard on yourself. The mad scientist in you might want to optimize for every parameter, and control for everything you can. But the reality is you need to move fast. You need to learn fast. And you need to outpace your competitors by releasing valuable products and services to market.

Perfect is not the goal, so proceed with speed.

Step 4: Dust off the scales

Now that we’ve defined our focus, framed a hypothesis, and proposed a scope for our experiment, it’s time to define what data we need to capture.

Like the above, we like to keep this as simple as we can, regardless of what experiment we’re conducting. Keep in mind that these could be qualitative or quantitative indicators.

Just make sure you explicitly define what matters most.

Back to our example.

Fortunately, our imperfect experiment is focused on one key metric. And although we can’t control for all variables, we can measure what matters most.

Now it’s time to start defining the resources you need to conduct your experiment. You’ll also define what success looks like.

Related: The genius designer versus a mindset of experimentation

Step 5: The ask

Experiments always require resources. They cost money. Because of this there needs to be a way for you to justify the value of the investment that’s being made.

As an example, we might suggest that if the hypothesis is supported, we have a genuine reason to invest time, effort, and capital into specific content channels and partnerships. In doing this, we estimate there will be a 35% decrease in sales cycle time. We then of course have the opportunity to support or refute that estimate.

This section of the experiment card should support that type of thinking. It gives you a simple reference point for what success means, as well as what you’ll require to execute the experiment.

It’s just enough detail to give “the gist.” Further detail to support the actual cost, time allocation, and other things that matter to the business can reside elsewhere. An Experiment Card isn’t the entire plan, it’s a reference point. Something you can use to quickly sense check your experiment proposal. It’ll likely be refined before you decide to execute on it.

Which brings us to pointing fingers.

Step 6: Assign ownership

Someone’s got to take the bull by the horns. Give someone the agency to drive this experiment forward. When combined with your time-box, this completes the Experiment Card. It doesn’t complete the work, though. This is just the start.

In our case, ownership has been assigned to me. I’ve decided we should run this experiment over a 6-month period. I defined this time frame because of our sales cycle. Your time frame will be contextual to what you’re testing. If it’s a simple variable on your website, the time frame might be a day. If you’re selling to enterprises, maybe it’ll be longer.

So the card is done. I show it to Bianca. She questions why it’ll take so long. Surely there’s a different experiment we can conduct that enables us to learn faster?

She’s absolutely right—experiments should be fast. They should accelerate your learning velocity. It doesn’t mean my work is waste. The few minutes I spent defining this experiment gave us visibility of an option. We can now work together, using the same template, to define a faster, more appropriate experiment.

Once we’ve done that, we can get to work on the really fun part: executing.

The Experiment Card can help you quickly design experiments. The compound effect of conducting these experiments will change your environment. You will know more about the thing that matters most—your customers. And by knowing more, you’ll be in a position to create more value.

Better go get that lab coat. It’s experiment time.

How does your design team experiment?

Share your story on the InVision Blog—get in touch.

More from Nathan Kinch

• Hiring is hard, so here’s a model to make it easy
• Rationalize design through numbers, not analogy
• How to turn an error state into a conversation starter