In 2003, researchers writing in the American Journal of Medicine discovered something that should change how you think about medical news. They looked at 101 studies published in top scientific journals between 1979 and 1983 that claimed a new therapy or medical technology was very promising. Only five, they found out, made it to market within a decade. Only one (ACE inhibitors, a pharmaceutical drug) was still extensively used at the time of their publication.
One.
But you'd never know that from reading the press. Take a recent miracle procedure for multiple sclerosis. MS is a degenerative disease with no cure. In sufferers, the immune system attacks the protective layer around the nerves, disturbing the communication between brain and body — and causing a cascade of devastating symptoms: unsteady and jerking movements; loss of vision, bladder and bowel control; and eventually, early death.
n 2009, a breakthrough: a charming Italian researcher, Dr. Paolo Zamboni, claimed to have cured his wife's MS by "unblocking" the veins in her neck. He theorized MS wasn't an autoimmune disorder but a vascular one. The research was counterintuitive, it gave people with the disease hope, and it had an appealing personal tale behind it, involving one man's quest to save his wife. It was catnip for health reporters, who hailed "liberation therapy" as a romance-fueled medical triumph.
Sadly, however, Zamboni's discovery was more hype than breakthrough. What didn't get as much attention as his romantic quest was the fact that his study was small and badly designed. Other researchers who attempted to replicate his findings failed. Soon, anecdotes of patient complications and relapses emerged.
This cycle recurs again and again. An initial study promises a miracle. News stories hype the miracle. Researchers eventually disprove the miracle.
This cycle recurs again and again. An initial study promises a miracle. News stories hype the miracle. Researchers eventually disprove the miracle.
"There's a big, big, difference between how the media think about news and how scientists think about news," Naomi Oreskes, a Harvard professor of the history of science, recently told me in an interview. "For you, what makes it news is that it's new — and that creates a bias in the media to look for brand new results. My view would be that brand new results would be the most likely to be wrong."
Most medical studies are wrong
It’s a fact that all studies are biased and flawed in their own unique ways. The truth usually lies somewhere in a flurry of research on the same question. This means real insights don't come by way of miraculous, one-off findings or divinely ordained eureka moments; they happen after a long, plodding process of vetting and repeating tests, and peer-to-peer discussion. The aim is to make sure findings are accurate and not the result of a quirk in one experiment or the biased crusade of a lone researcher.
As science is working itself out, we reporters and our audiences seize on "promising findings." It's exciting to hear about a brand new idea that maybe — just maybe — could revolutionize medicine and stop some scourge people suffer through. We're often prodded along byoverhyping scientists like Zamboni, who are under their own pressure to attract research funding and publications.
We don't wait for scientific consensus; we report a little too early, and we lead patients and policymakers down wasteful, harmful, or redundant paths that end in dashed hope and failed medicine.
We don't wait for scientific consensus; we report a little too early, and we lead patients and policymakers down wasteful, harmful, or redundant paths that end in dashed hope and failed medicine.
This tendency could be minimized if we could only remember that the overwhelming majority of studies in medicine fail.
Forbes health writer Matthew Herper recently unpicked a new Vice documentary about a "miraculous" cancer cure. While the experimental therapies featured in the film seem to be the holy grail in cancer treatment at the moment, they're also the latest in a long line of seemingly "revolutionary" fixes. According to one of Herper's sources, in fact, there have been more than 200 failures of supposed cancer breakthroughs in recent years.
Forbes health writer Matthew Herper recently unpicked a new Vice documentary about a "miraculous" cancer cure. While the experimental therapies featured in the film seem to be the holy grail in cancer treatment at the moment, they're also the latest in a long line of seemingly "revolutionary" fixes. According to one of Herper's sources, in fact, there have been more than 200 failures of supposed cancer breakthroughs in recent years.
A highly regarded service that vets new studies for clinicians finds — on average — only 3,000 of 50,000 new journal articles published each year are well-designed and relevant enough to inform patient care. That's 6 percent.
More often than not, single studies contradict one another — such as the research on foods that cause or prevent cancer. The truth can be found somewhere in the totality of the research, but we report on every study in isolation underneath flip-flopping headlines. (Red wine will add years to your life one week, and kill you quicker the next.)
For a study on whether everything we eat is associated with cancer, academics randomly selected 50 ingredients from recipes in The Boston Cooking-School Cook Book. Most foods had studies behind them claiming both positive and negative results.
For a study on whether everything we eat is associated with cancer, academics randomly selected 50 ingredients from recipes in The Boston Cooking-School Cook Book. Most foods had studies behind them claiming both positive and negative results.
Researchers cannot always replicate the findings of other researchers, and for various reasons many don't even try. All told, an estimated 85 percent — or $200 billion — of annual global spending on research is wasted on badly designed or redundant studies.
This means early medical research will mostly be wrong until maybe eventually, if we're lucky, it's right. More tangibly, only a tiny fraction of new science will lead to anything that’s useful to humans.
This means early medical research will mostly be wrong until maybe eventually, if we're lucky, it's right. More tangibly, only a tiny fraction of new science will lead to anything that’s useful to humans.
No comments:
Write comments