Lies, Damned Lies and Medical Research

SLAM: debunk creationism, pseudoscience, and superstitions. Discuss logic and morality.

Moderator: Alyrium Denryle

Post Reply
User avatar
Ace Pace
Hardware Lover
Posts: 8456
Joined: 2002-07-07 03:04am
Location: Wasting time instead of money
Contact:

Lies, Damned Lies and Medical Research

Post by Ace Pace »

The Atlantic has published an article looking at medical research and flawed results. Since the article is long, I'll quote some highlights and let people read the rest. :)

In 2001, rumors were circulating in Greek hospitals that surgery residents, eager to rack up scalpel time, were falsely diagnosing hapless Albanian immigrants with appendicitis. At the University of Ioannina medical school’s teaching hospital, a newly minted doctor named Athina Tatsioni was discussing the rumors with colleagues when a professor who had overheard asked her if she’d like to try to prove whether they were true—he seemed to be almost daring her. She accepted the challenge and, with the professor’s and other colleagues’ help, eventually produced a formal study showing that, for whatever reason, the appendices removed from patients with Albanian names in six Greek hospitals were more than three times as likely to be perfectly healthy as those removed from patients with Greek names.
One of the researchers, a biostatistician named Georgia Salanti, fired up a laptop and projector and started to take the group through a study she and a few colleagues were completing that asked this question: were drug companies manipulating published research to make their drugs look good? Salanti ticked off data that seemed to indicate they were, but the other team members almost immediately started interrupting. One noted that Salanti’s study didn’t address the fact that drug-company research wasn’t measuring critically important “hard” outcomes for patients, such as survival versus death, and instead tended to measure “softer” outcomes, such as self-reported symptoms (“my chest doesn’t hurt as much today”). Another pointed out that Salanti’s study ignored the fact that when drug-company data seemed to show patients’ health improving, the data often failed to show that the drug was responsible, or that the improvement was more than marginal.
And before long he discovered that the range of errors being committed was astonishing: from what questions researchers posed, to how they set up the studies, to which patients they recruited for the studies, to which measurements they took, to how they analyzed the data, to how they presented their results, to how particular studies came to be published in medical journals.
He zoomed in on 49 of the most highly regarded research findings in medicine over the previous 13 years, as judged by the science community’s two standard measures: the papers had appeared in the journals most widely cited in research articles, and the 49 articles themselves were the most widely cited articles in these journals. These were articles that helped lead to the widespread popularity of treatments such as the use of hormone-replacement therapy for menopausal women, vitamin E to reduce the risk of heart disease, coronary stents to ward off heart attacks, and daily low-dose aspirin to control blood pressure and prevent heart attacks and strokes. Ioannidis was putting his contentions to the test not against run-of-the-mill research, or even merely well-accepted research, but against the absolute tip of the research pyramid. Of the 49 articles, 45 claimed to have uncovered effective interventions. Thirty-four of these claims had been retested, and 14 of these, or 41 percent, had been convincingly shown to be wrong or significantly exaggerated. If between a third and a half of the most acclaimed research in medicine was proving untrustworthy, the scope and impact of the problem were undeniable.
But even for medicine’s most influential studies, the evidence sometimes remains surprisingly narrow. Of those 45 super-cited studies that Ioannidis focused on, 11 had never been retested. Perhaps worse, Ioannidis found that even when a research error is outed, it typically persists for years or even decades. He looked at three prominent health studies from the 1980s and 1990s that were each later soundly refuted, and discovered that researchers continued to cite the original results as correct more often than as flawed—in one case for at least 12 years after the results were discredited.
Brotherhood of the Bear | HAB | Mess | SDnet archivist |
User avatar
Molyneux
Emperor's Hand
Posts: 7186
Joined: 2005-03-04 08:47am
Location: Long Island

Re: Lies, Damned Lies and Medical Research

Post by Molyneux »

Well...fuck.
Can't say that anything comes to mind as far as a workable way to limit this, really, though it is disgusting to read.
Ceci n'est pas une signature.
User avatar
Singular Intellect
Jedi Council Member
Posts: 2392
Joined: 2006-09-19 03:12pm
Location: Calgary, Alberta, Canada

Re: Lies, Damned Lies and Medical Research

Post by Singular Intellect »

The medical field finally entering the digital information age certainly couldn't have come soon enough. It is rather unnerving that our medical science was such a hit and miss system for all this time.
"Now let us be clear, my friends. The fruits of our science that you receive and the many millions of benefits that justify them, are a gift. Be grateful. Or be silent." -Modified Quote
User avatar
montypython
Jedi Master
Posts: 1128
Joined: 2004-11-30 03:08am

Re: Lies, Damned Lies and Medical Research

Post by montypython »

It reminds me of how some of my professors would say about scientists and researchers being as much filled with human bickering and shortcomings as anyone else, despite notions of being paragons of rationality to the contrary. :|
User avatar
Alyrium Denryle
Minister of Sin
Posts: 22224
Joined: 2002-07-11 08:34pm
Location: The Deep Desert
Contact:

Re: Lies, Damned Lies and Medical Research

Post by Alyrium Denryle »

montypython wrote:It reminds me of how some of my professors would say about scientists and researchers being as much filled with human bickering and shortcomings as anyone else, despite notions of being paragons of rationality to the contrary. :|

The issue here is that we are not actually dealing with scientists, but MDs (though in some areas, like pesticide testing, the problem is just as bad). Medical schools have been little more than trade schools for 20 years or so, and they do not teach students how to do good research, even though hospitals require a certain number of publications.

Combine this with the profit motive being present in the testing phase during medical research, and it gets even worse. The doctors on the review panels for journals do not actually have the expertise to analyze the statistics used for example.
GALE Force Biological Agent/
BOTM/Great Dolphin Conspiracy/
Entomology and Evolutionary Biology Subdirector:SD.net Dept. of Biological Sciences


There is Grandeur in the View of Life; it fills me with a Deep Wonder, and Intense Cynicism.

Factio republicanum delenda est
User avatar
Edi
Dragonlord
Dragonlord
Posts: 12458
Joined: 2002-07-11 12:27am
Location: Helsinki, Finland

Re: Lies, Damned Lies and Medical Research

Post by Edi »

Kaljamaha, who is a member here, though posts only very infrequently, works with cancer research and we've talked about stuff like this a few times. Problem being, biology is still such an unknown frontier where research can take you nearly anywhere. One problem he told me about was getting non-biologists to understand that no, you can't just go and research X in order to get Y, because more often than not, you first have to research whether X is even possible in the first place and to chart out what requirements it has and only then you can take a guess whether Y is anywhere in the vicinity.

There is no map and in order to get from point A to point B, you have to make it, except you could just as well end up at point Z when you start out.

So this article does not surprise me at all.
Warwolf Urban Combat Specialist

Why is it so goddamned hard to get little assholes like you to admit it when you fuck up? Is it pride? What gives you the right to have any pride?
–Darth Wong to vivftp

GOP message? Why don't they just come out of the closet: FASCISTS R' US –Patrick Degan

The GOP has a problem with anyone coming out of the closet. –18-till-I-die
User avatar
PainRack
Emperor's Hand
Posts: 7569
Joined: 2002-07-07 03:03am
Location: Singapura

Re: Lies, Damned Lies and Medical Research

Post by PainRack »

Singular Intellect wrote:The medical field finally entering the digital information age certainly couldn't have come soon enough. It is rather unnerving that our medical science was such a hit and miss system for all this time.
What makes you think that digital information would allow researchers to be more effective?
Let him land on any Lyran world to taste firsthand the wrath of peace loving people thwarted by the myopic greed of a few miserly old farts- Katrina Steiner
User avatar
Singular Intellect
Jedi Council Member
Posts: 2392
Joined: 2006-09-19 03:12pm
Location: Calgary, Alberta, Canada

Re: Lies, Damned Lies and Medical Research

Post by Singular Intellect »

PainRack wrote:
Singular Intellect wrote:The medical field finally entering the digital information age certainly couldn't have come soon enough. It is rather unnerving that our medical science was such a hit and miss system for all this time.
What makes you think that digital information would allow researchers to be more effective?
Due to the exponential progress curve that is expected and being observed when any field enters the modern information age.
"Now let us be clear, my friends. The fruits of our science that you receive and the many millions of benefits that justify them, are a gift. Be grateful. Or be silent." -Modified Quote
User avatar
Lagmonster
Master Control Program
Master Control Program
Posts: 7719
Joined: 2002-07-04 09:53am
Location: Ottawa, Canada

Re: Lies, Damned Lies and Medical Research

Post by Lagmonster »

Singular Intellect wrote:
PainRack wrote:
Singular Intellect wrote:The medical field finally entering the digital information age certainly couldn't have come soon enough. It is rather unnerving that our medical science was such a hit and miss system for all this time.
What makes you think that digital information would allow researchers to be more effective?
Due to the exponential progress curve that is expected and being observed when any field enters the modern information age.
Or else, because making information about results public and widespread may lead to more qualified examinations of the findings.
Alyrium Denryle wrote:The issue here is that we are not actually dealing with scientists, but MDs (though in some areas, like pesticide testing, the problem is just as bad). Medical schools have been little more than trade schools for 20 years or so, and they do not teach students how to do good research, even though hospitals require a certain number of publications.

Combine this with the profit motive being present in the testing phase during medical research, and it gets even worse. The doctors on the review panels for journals do not actually have the expertise to analyze the statistics used for example.
What gives you the idea that all of the studies reviewed were performed and reviewed only by physicians?

You paint a broad negative portrait of medical research that could be misinterpreted by people unfamiliar with the field as a whole. My wife, just out of university, worked in medical research ethics (specifically, the Ottawa Hospital Research Institute), and I can attest by way of her experience that most prominent researchers connected to the Institute are Ph.Ds, MDs, or both, but in all cases their work is overseen by individuals who are dedicated to proper scientific practices.

Modern medical research does not only exist at the corporate shyster level either; organizations such as the OHRI have Technology Transfer and Business Development groups dedicated to the responsible commercialization of research results.

There will always be exceptions which become breaches of research ethics and accuracy, but they are not invulnerable to review and change. It's irresponsible to wave one's hand and speak about incompetence and financial interests as though one should infer that they were driving a widely corrupted field populated by well-meaning ninnies.
Note: I'm semi-retired from the board, so if you need something, please be patient.
User avatar
Alyrium Denryle
Minister of Sin
Posts: 22224
Joined: 2002-07-11 08:34pm
Location: The Deep Desert
Contact:

Re: Lies, Damned Lies and Medical Research

Post by Alyrium Denryle »

What gives you the idea that all of the studies reviewed were performed and reviewed only by physicians?
Because many of them are. It depends on what stage of the research process you are in of course. The basic research is often done by universities and is usually top notch. Much of the issues come once you have applied research, such as an MD devising a surgical technique, or during clinical trials of a drug.
GALE Force Biological Agent/
BOTM/Great Dolphin Conspiracy/
Entomology and Evolutionary Biology Subdirector:SD.net Dept. of Biological Sciences


There is Grandeur in the View of Life; it fills me with a Deep Wonder, and Intense Cynicism.

Factio republicanum delenda est
Post Reply