Why companies with a culture of experimentation innovate faster than their competitors

A common superstition in the industry is that A/B testing and experimentation make companies slower.

The intuition is that if a company has to run A/B tests on top of everything else it already does, it will move more slowly.

But that is a simplistic view, and it leads to a conclusion that is the exact opposite of reality.

(Or would anyone argue that companies such as Google, Netflix, Microsoft, and Amazon, where experimentation is central to how they operate, innovate in their products more slowly than the rest of the market?)

When a culture of experimentation is embraced in the right way, innovation happens faster and at a lower cost.

Take a look at this chart to understand why:

What I illustrated here is a representation of what I have seen again and again during more than 10 years working in experimentation.

I have seen many companies in both scenarios. And I have had the pleasure of seeing companies move from the first scenario to the second.

Here's what the chart shows:

In a typical process, there is unavoidable bureaucracy involved in getting a new idea produced and tested. That is normal, since trying something new involves a production COST and the RISK of changing something.

Imagine, for example, what it would take for a newly hired designer at a large company to radically change a critical page on the website and test a new approach with users based on an idea they came up with.

It would be impossible to skip multiple rounds of leadership approval. There would be a significant design and development effort, which could easily cause the idea to rot in a backlog because there would never be room in a sprint to work on it, and so on.

At a company that embraces a culture of experimentation, that process gets faster in several ways. Here are a few:

Bureaucracy

The bureaucracy involved in getting leadership approval is much lighter. That is because the RISK is lower. If the change does not work, an A/B test will show it clearly and in real time. It is easy to take a step back. In some companies with a strong culture of experimentation and shared innovation, many ideas do not even need leadership approval before they are tested. In those companies, the blue rectangle in the chart does not exist.

Development

The COST is lower, too. As I said above, when you implement a change without a test, that change is just another ticket for designers and developers. Everything has to be created according to the company's full quality, documentation, and other requirements.

But when you validate an idea through an A/B test, you can, and should, develop it as an MVP, with the lowest possible cost to find out whether the idea works.

That philosophical difference in development is huge. It helps explain why small teams at companies with a culture of experimentation can deliver a much wider range of changes than a similarly sized team at a typical company.

Validating results

When you change an interface without an A/B test, it is difficult, perhaps impossible, to know whether it worked. Often, you have to pull a senior resource, such as an analyst or data scientist, away from other work to try to determine whether the change succeeded. And even then, the conclusion is often imprecise.

With an A/B test, however, the results are available in detail in the report from your testing tool. No one has to spend days stitching together data from all over the place.

Finally, when you have a clear result, if it is positive, it is much easier to present it to company leaders so that the learning can be scaled and put into practice permanently.