Criticism of Studies with the “Wrong” Outcomes

It seemed during Covid everyone had studies that proved their point. It was a case of “my study can beat up your study.”

The meme-world had some fun with this.

In 2005, John Ioannidis, a research scientist at Stanford, published a paper called “Why Most Published Research Findings are False”. The paper has been cited somewhere between 5-10,000 times.

He says that are a lot of research findings are false for the following reasons:

• When the studies are smaller

• When effect sizes are smaller

• When there is a greater number and lesser preselection of tested relationships

• Where there is a greater flexibility in designs, definitions, outcomes an analytical models

• When there is greater financial interest or prejudice

• When more teams are involved in a scientific field in chase of statistical significance

You have to think there are many studies that are incorrect. Often you can find studies that contradict other studies. One of them has to be wrong.

Yet everyone relies on these studies for their arguments about what works and what doesn’t. I’m just as guilty of it as the next guy.  Many of the families of my assisted living residents and I love  to swap stories of the latest podcast we just heard. The podcast cited some study or other reinforcing our beliefs.

Those studies will make us try a new supplement or therapy that sounds very promising.

Usually the results don’t measure up. It can be very disheartening. It makes me think studies are more about product marketing than true scientific research in many cases.

I decided to type into Google “what to do for your health when there are so many false studies published”. Here is what the Google AI replied:

Not sure about you, but Covid made me distrust the government health bureaucracies a whole lot more than before. Not to mention that I regularly see better health outcomes by ignoring a lot of government advice such as the old food pyramid.

So what to do?

One part of the Google AI response I can support is to ‘focus on foundational healthy habits - balanced diet, exercise and sleep.’ I don’t need a whole lot of studies to convince me that eating well, exercising and sleeping well make a difference.

I can look around the general population and see those foundational habits work.

Those activities are where you should focus. Like a great piece of music, they will stand the test of time. The effective treatments usually stick around.

Penicillin is still an effective antibiotic. I understand antibiotics are a big problem if overused. But they do work as designed. It’s been around since 1928.

Exercise, nutrition and good sleep have been recommended for a lot longer.

So when you see a new therapy or new study go viral, take a breath and ask yourself. Is this real or is it curated by some very good marketers? There’s a reason for the saying “only time will tell”.