What's in a Study

Why you can't necessarily trust "scientific" research

Sarabeth Matilsky
July 24, 2002

"Reports show that eggs raise cholesterol levels." "A new study claims that eggs are one of the healthiest foods on earth." "A revolutionary new discovery heralds the scientific discovery of this year's Miracle Food, which will make you healthier, thinner, sexier, and have a better sense of humor than ever before."

Sometimes studies like these can be amusing, interesting, and even informative. But there are some reasons to be wary:

• The only really universal prescription for health is to Eat Right and Exercise.
• The best way to Eat Right and Exercise is different for everyone.
• Figuring out your personal Eating Right and Exercising plan is a life-long process, and there's probably not only one best way for you, anyway.
• Eating Right and Exercising are not available in a pill, no matter how badly advertisers wish you would believe otherwise. There is, as they say, no substitute.
• "Science" and "Scientific" are two of the most misused words I can think of.

Science, to paraphrase Douglas Adams, is simply a way to try to understand life, the universe, and everything—while minimizing our human propensity to be biased observers. Just because a study claims to be "scientific," however, doesn't mean that its conclusions are valid and worthy of life-changing action.

I've heard it explained this way: Imagine that at a particular hospital, it's discovered that most patients who die of lung cancer are also found to have lighters in their pockets. Perhaps the number of deaths-with-lighters-in-pockets is high, say 95%. An observational study might report that scientifically researched fact, and then hypothesize: "our study shows that lighter fluid may be carcinogenic."

In order to link cause and effect, however, a controlled, double-blind study would then have to occur. A large group of people would have to carry lighters around in their pockets all day, every day; half of the people's lighters would contain lighter fluid, the other half would contain a placebo. At the end of a certain period of time, this controlled study would probably find that lighter-fluid containing lighters were no more likely to cause cancer deaths than the placebo. The study would be back to square one, and maybe this time they'd examine other items in the lung cancer victims' pockets, such as their cigarettes.

Unfortunately, the media usually reports on observational and controlled studies indistinguishably, which misleads people who don't know the difference. And of course, even the controlled studies can only look at a minute number of variables, so they have to be taken with some grains of salt too.