Everything in Your Fridge Causes and Prevents Cancer

Thinking critically about science in the news is a recurring topic here, so I want to highlight a new study that grabbed some headlines — including in the American Psychological Association’s magazine, Monitor on Psychology.

Here, in a small nutshell, is what the new study found: among a group of 14,000 Finnish people who were tracked for 40 years, those who used saunas frequently were way less likely to develop dementia.

Big if true. And that’s an impressively huge study! But let’s start peeling back the layers.

First of all, it wasn’t every sauna enthusiast who reaped the supposed protective effect against dementia; it was specifically those who used a sauna 9-12 times a month. Sauna bathers who hit the wooden bench 5-8 times a month — sorry, no effect. And those who went more than 12 times a month — again, no luck.

That should raise a caution flag in your head. When only a very specific subpopulation in a study experiences a benefit, it may indeed be that there is some extremely nuanced sweet spot. But it is more likely that the researchers collected a lot of data, which in turn allowed them to analyze many different correlations between sauna use and dementia; the more different analyses they can do, the more likely some of those analyses will generate false positives, just by statistical chance. And then, of course, those titillating positive results are the ones that end up at the top of the paper, and in the press release.

The protection against dementia for that 9-12-times-per-month set was big over the first 20 years of the study, and then faded later, so now we’re slicing and dicing data not only by sauna-frequency, but by time. And then there's temperature. The Finns who saunaed in temperatures hotter than 100°C (212°F) actually had a higher risk of dementia than those who sauna-and-chilled below 80°C (176°F).

So if you want to prevent dementia, hit the sauna precisely 9-12 times a month at below 80°C — and, unfortunately, expect the protective effect to diminish in the future. Whew!

I regularly talk to UC Irvine research psychologist Drew Bailey about this sort of thing, because he happens to be a gadfly for research methods. He likes to use a sports analogy to help people gain at least a bit of intuition for the statistical problem here.

If you’ve watched a lot of NFL games, you’ve probably heard a commentator at some point trot out a stat like: “The Chicago Bears are undefeated in division games following a bye week when they wear their alternate jerseys.”

It is possible that the combination of extra rest and unusual attire gets the Bears amped for division rivals, thus improving their performance — just as it is possible that using the sauna precisely 9-12 times per month at a particular temperature protects against dementia (at first). It is far more likely, however, that given a large database of games and conditions, there will be loads of correlations that occur just by chance; the more you slice and dice the data, the more you’ll find. So I wouldn't head to Vegas and bet the farm on the Bears in alternate jerseys just yet; you can always come up with stats that do a great job at predicting the past and a terrible job at predicting the future.

I Have Wonderful and Terrible Science-Backed News About Your Diet

There is a sad and hilarious paper that famously makes a point about spurious correlations — the “everything in your fridge causes and prevents cancer” paper.

If you look at the figure below, you see various foods listed on the Y-axis, and relative risk of developing cancer on the X-axis. (A relative risk of 2 means you’re twice as likely as a typical person to get cancer.) Each dot is the result of a particular study, indicating whether that study found that the food on the Y-axis increases or decreases cancer risk.

Nearly every food listed has been shown by certain scientific studies to cause cancer, and by others to prevent it. (Sorry bacon lovers! I guess you can hold out for more data??) Milk has been shown alternately to increase or decrease cancer risk nearly five-fold! So what’s the reality?

The reality is probably that for most foods, we just don’t know. The reality is also that many of these studies probably had a large helping of what is known as “researcher degrees of freedom.” That is, there were many different ways that the scientists could group and analyze their data, which in turn meant there were many false positive correlations waiting to be found. And false positives are more likely to get published (and then grab headlines) than true findings that say a food has little or no impact on cancer.

Here’s the point I want to hammer home: when you see a tantalizing health headline — like that saunas prevent dementia — keep an eye out for indications that the effect only applies to specific subgroups of the study population. Even if the headline is very authoritative, revealing nuggets are often buried lower in the story.

I want to stress that you shouldn’t assume the sauna results can’t possibly be true. But when you see Bears-undefeated-in-alternate-jerseys type conclusions — and someone is claiming one thing causes the other — you should hold out for more evidence. (A testable theory for why such an unusual result might be true would also be nice.)

We can't say that alternate jerseys (or saunas) definitely don't have an impact, but humans (and scientists are humans) tend to look for the most interesting effects. In doing so, we often focus on results that — even if true — are exaggerated by statistical chance to look more interesting.

As demonstrated by a cheeky study which showed that listening to the Beatles’ “When I’m Sixty-Four” makes you younger, (see table 3), it is easy to alight on false positives when researchers have loads of options for how to subdivide and analyze their data.

David’s Digressions

-Christie Aschwanden, one of my favorite writers, has done incredible reporting on the kind of statistical troubles I’m highlighting here. In one article, she detailed the time that 29 research teams were given identical soccer data, and asked to determine whether dark-skinned players were more likely to get red cards. The research teams analyzed the same data but came up with a wide variety of results.

-If you’re a stathead and itching for the really-academic treatment, check out “The Garden of Forking Paths” paper, named after a Jorge Luis Borges short story. The more I read, the more I realize how many roads (er, paths?) lead back to Borges.

-Finally, informed skepticism is important, but rejecting all statistics out of hand is no way to live — or make better decisions. Here’s a five-minute video from one of my favorite thinkers, economist Tim Harford, on a few simple habits of mind that anyone can use to get a better sense of whether statistical claims are true. There’s much more where that came from in Tim’s outstanding book, The Data Detective. The paperback is out next week. It's both entertaining and important. Highly recommend.

Thank you for reading. If a friend sent this to you, you can subscribe here. If you think this post might help someone else parse science headlines, please share it — that's the only way it spreads. (If you’re reading in email, here’s the link.) Until next week...

David

P.S. The Range ebook is again on sale for $2.99.