Keep Calm and Research On: How to separate hype from fact in biomedical research

By Eric Verdin, President and CEO of the Buck Institute

As scientists, at times we lament the fact that much of our work is done behind closed doors, our data published behind paywalls, and our conclusions rarely seen by the public eye. However, on occasion the opposite occurs. A recently published study garners a slew of media attention and suddenly is promoted everywhere from the local news station to the New York Times to our social media accounts. While we often celebrate the sharing of our hard work and carefully crafted studies with the world, it can sometimes be frustrating when we see our findings misconstrued or even misrepresented in the media. Harder still is, as a reader, how do you evaluate these reports with a critical eye? How can you separate fact from hyperbole so that you can be informed?

Take for instance a research article earlier this year on drinking which spun such stress-inducing headlines as “No Alcohol Safe to Drink” and “All Alcohol is Bad”. Upon careful reading of this article, which was a meta-analysis of several previously published studies on alcohol consumption and various health factors, it became clear that, as pointed out by the New York Times and others, the research had been widely overstated and overblown by the popular media, and to an extent, the original researchers themselves. The study looks at the global population and correlates drinking alcohol with health outcomes. They conclude that “the safest level of drinking is none,” I suppose in much the way the safest level of driving is none, or the safest level of leaving your house is none.

Certainly, drinking alcohol can incur some risks, especially if those consuming alcohol are also engaging in risky behaviors which often correlate with drinking, such as smoking or driving under the influence. As the study is a large, meta-analysis (a study combining data from multiple previous studies) looking at global populations, they did not take into account many of the possible confounders, such as previous health issues, environment, or socioeconomic status, and only observed risk by age, location, and sex. In the end, the study shows a very small increase in health risk with increasing alcohol consumption, which is statistically significant due to the large data size (about 700 studies), not necessarily because these differences are biologically relevant. For someone consuming 1 drink per day, these data indicate a 0.5 percent increase in risk for a health issue. To put this into perspective, if you had 100,000 people drinking at that level, only 4 are likely to experience any health decline. Indeed, it is only at relatively heavy levels of drinking that substantial risks for disease are observed.

The data presented in this paper also show a slight decrease in the risk for ischemic heart disease and diabetes in both males and females consuming moderate amounts of alcohol (approximately 1 drink per day), echoing previous research findings. In the end, these types of observational studies can be very difficult to interpret and can only tell us about how one thing (in this case drinking) correlates with another (health outcomes), and not whether there is a causal relationship. So for studies like these, I would take the findings with a grain of salt, or perhaps a glass of wine.

Perhaps unsurprisingly, aging research is not immune from similar over-statements in the press or by companies attempting to sell products to consumers. Many times, large, observational nutrition studies are used to suggest that eating a certain diet will improve longevity despite a lack of randomized controls, or that taking a supplement will decrease your risk of disease and improve overall health, even if no research study or clinical trial has demonstrated that effect in humans.

So how are we to know what is hype and what is real? My best recommendation is always to go to the original source. Most press articles, if they are properly researched, should link back to the original paper or study to which they are referencing. While some of these articles may be behind a paywall or written in technical jargon that is incomprehensible to a non-scientist (or even scientists in another field), usually the abstract or summary statement will be available and the main conclusions stated most clearly there. If the claims made in the news do not align with the conclusions drawn by the original study authors, be wary. It is possible that the media source has attempted to sensationalize the article to drive up readership. Be especially aware that correlation does not equal causation. In the alcohol study, researchers found that increasing alcohol consumption correlated with poor health outcomes, but they presented no evidence to support that consuming alcohol caused any incidence of disease or mortality. These concepts are frequently conflated in the popular press.

In addition to reading the original article, there are several more intermediary sources between the study and the news that may provide a more accurate and modest report of the findings. These include the website or press page of the author’s university or study funder, and scientific news sources such as Science Daily or New Scientist. I have found that the reporting of studies on these sites tends to be truer to the findings of the original article than the popular press’s rendition, while putting it into greater context. If all else fails, you can always ask a scientist! Our scientists here at the Buck are experts in their fields. You can use our social media pages (Twitter, Facebook, LinkedIn) to find sources for the latest research news and ask us questions directly at blog@buckinstitute.org. Maybe we will feature your question in an upcoming post. Research on!