Most CRO and A/B testing case studies are unreliable

And it is not necessarily because the people sharing them are acting in bad faith.

Unfortunately, most people who analyze test results do not have the statistical knowledge needed to do it properly.

So when a testing tool says a variation is a “winner,” it is hard to resist the temptation to share the case, right?

Testing tools have a strong interest in making you believe that a test produced a winner. As a result, they often declare “winners” when that is not necessarily what happened and the result is far more complex than the simplified report suggests.

This post is not meant to cover every factor you need to consider to make a result reliable. That would take much more than one post.

My goal is to open people's eyes and encourage caution about the case studies we see.

The problem is not limited to CRO case studies

Did you know that most of the “scientific studies” you hear about may also be unreliable?

A figure in this image is a good example.

A 2015 study published in Science analyzed 100 prominent experiments in psychology.

About 66% of those studies FAILED when researchers tried to replicate them. In other words, most of the results reported by prominent scientific studies could not be confirmed in replication attempts.

And the problem is widespread. Even studies in classic books, written by NOBEL PRIZE WINNERS, can be unreliable (as this example shows).

(With that in mind, think twice before sharing every neuromarketing study you come across, too.)

Scientific studies receive much more rigorous planning and analysis than, say, simple tests on landing pages. And even then, most of those studies have questionable conclusions.

I think you can now see how large the problem is with the A/B testing results we see online.

Conclusion

One important point: in general, I do not think people who publish unreliable CRO case studies or scientific studies are “trying to pull a fast one.” I made the same mistake early in my career, without understanding that I was presenting something questionable. In my mind, I had something as scientifically proven as Darwinism.

Understanding what a test or study result means statistically is difficult. More than that, sometimes there is no right or wrong conclusion, only a wide gray area dominated by “it depends.”

Even so, as in scientific research, there are interesting initiatives to define a minimum standard of reliability for published A/B test case studies. For example, Ron Kohavi, formerly at Microsoft, Amazon, and Airbnb, proposes a set of rules.

Probably more than 90% of case studies online would fail those rules, which is sad. And remember, the problem is not limited to A/B tests. The amount of junk presented at talks or on news sites under the misleading label of “science” is impossible to count.

On the other hand, this shows us that experimentation is not a walk in the park. If you want your company or project to be truly data-driven, you need to understand that.

As I always say, it is worse to THINK you are data-driven based on misleading data than to not be data-driven at all.