It is hard to stay skeptical when an experiment tells us what we want to hear

In a recent article, I warned against believing every test result or “scientific experiment” we come across. Even when the experiment comes from a Nobel Prize winner (a real case).

Today, an article in O Globo, originally published in The New York Times, illustrates a classic problem with poorly designed experiments.

For decades, the media and popular wisdom have said that wine, or “a little alcohol” every day, is good for your health, based on SCIENTIFIC STUDIES.

Meanwhile, some scientists, exercising the skepticism every person should practice, were already warning that these studies were not all that reliable.

Those studies said that groups of patients who drank alcohol every day had fewer heart problems and lived longer than the other group, made up of people who did not drink.

(This is a wonderful “truth” for everyone who drinks, and the alcohol industry, to hear, right? That is why it spread so easily. It is also much harder to be skeptical of a “truth” we like.)

But the skeptics I mentioned pointed to several potential problems with those studies. For example:

What if the group that drank wine or other alcoholic beverages every day was wealthier? What if that meant they ate more vegetables and fiber, or had better access to health care?

What if the group that did not drink included former alcoholics who had stopped drinking because they had already developed health problems?

And so on.

And now, studies conducted with more rigor and less funding from the alcohol industry show that ANY amount of alcohol is harmful to health?

That is right. Forget the comforting story. Based on what we know today, the most accurate claim is that alcohol is not only useless for “taking care of your heart,” it also increases your chances of heart problems and cancer.

In the world of online A/B testing, this reminds us how essential it is to truly randomize the groups in an experiment.

If something makes your Control group more likely to include, for example, users with faster connections or returning users, your result is probably invalid, even if it reaches statistical significance.