What is all the hype about experimentation?

Starting an article with Jeff Bezos making the very point you want to make is a huge help. I should do this more often.

And I hope that by the end of this article, you'll join our experimentation fan club.

You've probably been hearing the term "experimentation" increasingly often, especially if you subscribe to this newsletter. But you may not fully appreciate its importance, now and in the future.

In my view, experimentation is the most important concept in business today.

Think of any term, trend or approach you like. It doesn't have experimentation's potential.

For example, Avinash Kaushik, a leading authority on digital analytics, perhaps one of the terms you thought of in the previous paragraph, recently said:

Why is everyone so excited?

First, understand that those who use experimentation properly can dominate their markets, as some already have. Meanwhile, those who dismiss it as "just another hype" and ignore it will be left eating dust.

Think I'm exaggerating? Feeding the hype?

Then take a look at this chart:

Source: Experimentation Works, 2020

The lighter line shows the performance of S&P 500 shares. If you don't follow financial markets closely, the S&P 500 isn't just any list of shares. It's made up of the shares of the 500 most important companies in the world.

Yet those shares perform ridiculously worse than the other group in the chart, the "Experimenters Index".

You might ask: which shares make up the Experimenters Index?

They belong to companies recognised for having an exceptionally strong experimentation culture, perhaps the best in the world.

We're talking about names such as Microsoft, Google, Booking and Facebook.

Obviously, these companies don't owe all their success to their experimentation cultures. But they would never have achieved that success without them.

Come with me and let's understand why.

Too many people call anything an "experiment"

Not a day goes by without me seeing someone say they've run "an experiment" or "a test" when, whether they know it or not, they've done nothing of the sort.

Without going too far into the scientific method, here's a simplified explanation:

If you've created something new or explored a new strategy to improve your current results, you must have run an A/B test to call it an "experiment" or "test".

Otherwise, you've simply done something new. You don't get to use those terms for it. 🥲

Note: there are several forms of experiments and tests. For simplicity, this article refers to the simplest of them: the A/B test.

An A/B test also has a more technical, but very important, name: randomised controlled experiment.

That name matters because it highlights a fundamental requirement for something to qualify as an experiment: users must be assigned randomly to at least two groups:

• The "control" group sees the original version of your page, email, campaign, app, ad or whatever you're changing, without your modification.
• The "variant" or "treatment" group sees the system with the change you want to test.

This lets you isolate the variables and precisely measure the effect of what you're testing. Control and variant "compete" under the same conditions, with audiences of the same profile, during the same period and under the same external influences.

Make no mistake: NOTHING is more effective than experiments for understanding the real results of your work

I don't want to invalidate the efforts of people who implement changes without testing them. In some cases, there are ways to try to estimate their real impact afterwards.

But none of these even compares to experimentation, in either reliability or precision.

Ron Kohavi, a scientist who played important roles in the experimentation programmes at Amazon, Microsoft and Airbnb, describes A/B tests, or randomised controlled experiments, as the "gold standard" for establishing causality: understanding the impact caused by a particular element or action.

Notice that, from a scientific perspective, approaches such as "expert studies", which many companies consider satisfactory for understanding phenomena, have significant vulnerabilities. They carry a high risk of confusing correlation with causation.

Yes: analysing isolated data isn't enough to draw reliable conclusions about cause and effect. Want an example?

Look at this chart from 1949 showing the correlation between ice-cream sales in the US and the number of polio cases:

Curious, isn't it? Based on analysing these data, health experts asked people to stop buying ice cream to reduce the number of polio cases...

Of course, people eventually discovered that there was no causal relationship between those two data series. Polio transmission was more common in higher temperatures, during the warmer months, as were ice-cream sales.

Think that's an extreme example? Even today, it's very common to see marketers credit reduced product churn to the use of a particular feature without a reliable experiment to prove the claim.

It's exactly the same mistake, more than 70 years later: assuming causation where there may only be correlation.

The best companies are fanatical about experimentation. Why?

Think of any digital company that's a global leader in a fiercely competitive market.

You will very, VERY likely have thought of a company that's fanatical about experimentation, where everything that can be tested will be tested. More than that, the entire company embraces and encourages as many experiments as possible.

Airbnb, Amazon, Booking, Facebook, Google, LinkedIn, Microsoft, Netflix, Spotify, Uber...

Feel free to check how innovation and decision-making work at each of these companies and other giants today.

Let's understand some of the reasons they take experimentation so seriously, and why you should too.

1. Experiments prevent expensive mistakes

No matter how good you are, you'll sometimes be wrong when predicting the outcome of an action in your company.

No matter how strong your research or discovery work was, the action still has a real chance of failing.

Over the years, I've seen many companies, several of them excellent, get into trouble after making changes that seemed like "obvious" or thoroughly supported improvements, but ended up setting them back significantly.

Did you know that only about 10% of tests at brilliant companies such as Google and Microsoft are winners?

Now imagine if they made those changes without testing. Think how much time and money would be wasted on the 90% of initiatives that would actually take the company backwards.

One of experimentation's great strengths is that it prevents you from implementing changes that make your KPIs worse. The worst outcome is a losing test that leaves you at the same level.

2. Experiments accelerate innovation

Let's use this moment to destroy a huge, and very common, myth about experimentation.

Many people believe experiments will slow their company down.

"A/B test everything? Impossible. We'd be far too slow! It wouldn't work here. We're incredibly dynamic."

Sound familiar?

I understand and sympathise with the legitimate concern. In the digital economy, anyone who doesn't grow quickly gets eaten by competitors.

But I guarantee that adopting an experimentation culture properly will actually make your company more agile.

It's true that before a change becomes permanent, you need to test it with enough users to reach statistical significance. That can take days, a few weeks, a month or even a little longer.

However, once you adopt this process, the wait for statistical significance is more than offset by effects such as these:

• Experimentation speeds up decision-making. Knowing an initiative will go through reliable validation brings dramatic reductions in the discussions and bureaucracy needed to get it off the ground.
• Measuring the results of an initiative becomes much easier. Without an experiment, companies often need senior professionals or even entire teams to stop what they're doing and try to work out the real impact of a recent change. In an A/B test, the precise result is available to everyone.
• Finally, one of my favourites: with experimentation, you don't need to implement everything new with the highest possible code quality, translated into every language, perfect on every device, with documentation ready and so on. During the experiment, you implement the minimum needed to validate your hypothesis. You only invest in everything required for a finished release if the experiment shows that the hypothesis really works. This can dramatically improve a company's efficiency and productivity.

3. Experiments democratise innovation

In a company with a good experimentation culture, barriers to innovation are lower. More people will therefore try to innovate through experiments. And the more people experiment... Well, Jeff Bezos already said it at the start of this article: more experiments = more growth.

That's why the most innovative companies work so hard to give everyone room to run their experiments.

An interesting case happened at Booking.com, where around 75% of its thousands of technology and product employees actively use the company's experimentation tool.

In December 2017, an important month for businesses in this market, a designer proposed a test with a radical homepage redesign. The many elements at the top of the page that the company had spent years optimising, including accommodation lists, offers, photos and copy, would be replaced by a simple form asking where users wanted to go, when and with how many people.

The CEO and other executives doubted it would work. They even bet a bottle of champagne that the conversion rate would drop.

Yet the test was still allowed to run.

They were all-in on developing and democratising their experimentation culture. The agreement with every team was that anyone could test whatever they wanted without leadership's permission.

Would a test in a similar situation go live at your company?

That's innovation at scale, with everyone taking part through experimentation.

Oh, you want to know whether that crazy test worked? Just look at Booking.com's homepage years later to find out. 🙂

4. Experimentation is the best way to understand your customers

Don't get me wrong. I love, and practise, techniques such as:

• Product discovery
• User interviews
• Usability testing
• Looking for opportunities in analytics tools
• Audience surveys

But none has an experiment's precision and reliability.

For example, a few years ago Intuit, a provider of financial management software for businesses, interviewed 20 users to find out whether they wanted a faster way to pay employees when setting up their accounts. The interviews said no, unanimously.

In an A/B test, however, the faster payment option performed better and proved excellent over the long term, consistently increasing Intuit's revenue.

There will always be differences between what people say, what experts understand from what they say, and what they actually do.

Experimentation focuses on the last of these. It assesses, without noise, what users really prefer, what makes them stay longer, return more often, spend more and so on.

Every test result, win or loss, gives you a valuable lesson about your customer.

Experiments are truth serum injected straight into your customers' veins. 👀

That's why a famous maxim is absolutely true:

A visitor who hasn't been in an A/B test is a wasted learning opportunity.

Anyone serious about dominating their market takes that maxim very seriously. That's why, according to some internal reports, every time you open Facebook you have a 99% chance of participating in at least 10 tests.

OK. How do I start bringing experimentation into my company?

Great question. There's a lot to discuss about introducing and developing experimentation in a company. I'll bring related content to this newsletter from time to time. For now, let's focus on some important general points you need to know.

In my view, high-level experimentation requires a company to take good care of three pillars: people, technology and culture.

People

Unfortunately, experimentation isn't the simplest subject in the world. Technical mistakes in its application can be very costly because company decisions start being guided by test results.

That's why it's essential to have people who can guide experimentation properly.

At the same time, as you've seen in this article, experimentation only reaches its full potential when the entire company embraces it and every team runs tests.

The organisational model I've seen work best at leading companies therefore combines centralisation and a distributed structure:

In this model, a central expert team takes overall responsibility for making the experimentation programme work well: processes, technology, advocacy, communication and so on. You can outsource this team to a specialist provider or bring together the right people internally if you already have them.

The team should train and support all the other teams or squads in the company so they can practise experimentation.

Understand that this team may not even run its own A/B tests. That isn't its main role. It exists so the rest of the company can run theirs.

One of the best things about this structure is that it scales and avoids the fantasy that every team can efficiently put experimentation at the heart of its processes without help.

If you'd like me to explore people and team structure further in future newsletter editions, let me know. 🙂

Technology

If you want reliable test results and a scalable experimentation programme, paying attention to your experimentation platform is essential.

Your technological needs will depend heavily on your level of maturity. But you'll face questions such as:

• Should we buy an external tool or build our own?
• How do we integrate our business data with the testing tool? Many popular A/B testing tools, hello Google Optimize, make this difficult or impossible for many companies' requirements.
• How do we run server-side tests?
• How do we run simultaneous tests reliably?

Remember: if your platform gives you incorrect results, you may condemn the entire company's future through bad decisions. If it doesn't let you move quickly, competitors can make you easy prey.

For these and other reasons, don't underestimate the attention a sound technology setup deserves.

Culture

A broad name for a broad subject. Experimentation culture involves many things, and nothing works without it.

For example, your company needs to be genuinely data-driven. Test results must be taken seriously and carry far more weight than anyone's opinion. At the same time, there need to be barriers preventing decisions without data.

The culture also needs to ensure every team understands experimentation's importance and has the freedom to experiment. Remember the Booking.com homepage example earlier?

And of course, another key part of culture is communication. As I've said, lessons from experiments are extremely valuable, sometimes even more valuable than the measured conversion-rate increase. How do teams share what they learn? How do they discuss ideas for new experiments? How do they prioritise? A good culture will have answers to questions like these.