90% of A/B tests fail, and the wrong mindset is a major reason

The digital industry generally tends to test changes that are far too subtle to make a meaningful positive difference to a business.

Changing a button colour, rearranging some information, swapping an image... These tests USUALLY produce no improvement in strategic results. When they do, the results are almost invisible.

But if very subtle tests have such a high chance of being useless, why do most companies still run so many of them?

1) Because humans don't like taking risks. However "risk-friendly" a company may be, nobody wants to be responsible for an expensive, complex test idea that dramatically reduces conversion when it goes live. So most ideas that actually move forward are more "cosmetic". Less risky.

2) Because articles such as "Booking.com made US$10 million by changing an icon" are incredibly appealing, and everyone wants to copy the idea. But they forget that their websites almost never operate at Booking's scale. That "US$10 million" amounts to a 0.01% conversion increase for Booking. What would it mean for you?

3) Because very few people calculate beforehand whether a small change on their interface, which may consequently produce only a small conversion increase, has any chance of yielding a reliable result with statistical significance around 90%.

4) Because the concept of "rapid experimentation", which isn't necessarily wrong, is being interpreted by many people in an extremely dangerous way: "only test things that are quick to build". Follow that rule and you're embracing mediocrity and stagnation.

5) Because under the pressure of deadlines and targets, many people end up treating the patient before knowing what the disease is. An alarming number of tests launch with little or no supporting data. People try to solve problems without even knowing whether they exist. These hypotheses are based solely on opinions or internal brainstorming. Nobody makes time for a proper diagnosis. I'll write more specifically about this "cancer" in the digital industry soon.

One final caveat: in some situations, "timid" but quick tests are highly recommended. On interfaces with enormous numbers of users, even the smallest increase can produce an attractive financial result. There will also be enough traffic to identify even tiny KPI increases with statistical confidence.

But if that isn't your situation, very subtle experiments should be the exception, not the rule.