Sunday, October 24, 2010
Why Medical Research (and Economics Research) is Wrong?
There is an Atlantic article entitled, "Lies, Damn Lies, and Medical Science". It's about the finding of a small group of researchers led by Dr. John Ioannidis who are showing how flawed the academic biomedical research establishment's findings are. This is not a bad thing. Science only progresses through self examination, self reflection, and checking assumptions. Any field of science or academic research will have a percentage of incorrect results being published. The question is, what is that percentage? The "harder" the science, the lower the percentage of bad or flawed publications. The softer the science or the more perverse the incentives to succeed, the more publications will be bad, flawed, or fraudulent. The following is from a comment I posted on NakedCapitalism.
The Atlantic article appears to support Sturgeon’s Revelation that 90% of everything humans produce is crud. I once had a discussion with my thesis advisor about scientific results due to a scientific misconduct case occurring in the early 1990’s. His belief was that in that particular Nobel Prize winner’s lab, that the pressure to produce was so great that at least 30% of results emanating from that lab were either flawed or fabricated. The problem is that if caught fabricating evidence, the researcher’s career is effectively over. In biomedical research, it is easier to fabricate evidence in obscure fields or areas where one’s results are not likely to be checked. The more prestigious the result, the less likely false or misleading evidence will go unnoticed because others will try to reproduce the results and fail. This is how cheaters are caught.
Even then, some studies are flawed due to environmental factors. Scientists who work with mice found out that different treatment results from the same treatment with different mouse strains could be minimized if they limited food intake shortly before the study began. Different mouse strains giving different results for the same treatment go back 70 years or more. Recently, they discovered that having mice in different cages affected study results.
With medical papers, it’s more difficult to catch frauds. Add in the uncertainty of mice studies and multiply the effect with humans. Every person is unique in genotype and phenotype. We are not at all like inbred mouse strains. Add in insufficient statistical sample sizes, bad statistical analysis, sloppy methodology, environmental and psychological effects, and it’s difficult to tell if the author is incompetent or a fraud. Generally, with frauds, the results are too good to be true, and the methodology is sound, but the results are unreproducible. But, the poor quality of clinical medical articles seems to have been a given for some time.
What the Atlantic article didn’t discuss is the difference in publications between researchers in fields with dedicated funding such as Germany versus America. German researchers generally don’t have to worry about publishing to obtain funding. Their funding is dedicated, so the publish or perish linkage is broken. Therefore, there is less pressure to be “right” or prove others wrong for career or professional advancement. Since the US publishes a great deal more research than any other nation or even group of nations, the results will be skewed by our publish or perish system. It would be informative to know who is getting correct results rather than who is getting it wrong for prestige or profit. Until the incentives are fixed and proper methodologies followed, nothing will change in medicine, economics, or any other field of human endeavor.
The Atlantic article appears to support Sturgeon’s Revelation that 90% of everything humans produce is crud. I once had a discussion with my thesis advisor about scientific results due to a scientific misconduct case occurring in the early 1990’s. His belief was that in that particular Nobel Prize winner’s lab, that the pressure to produce was so great that at least 30% of results emanating from that lab were either flawed or fabricated. The problem is that if caught fabricating evidence, the researcher’s career is effectively over. In biomedical research, it is easier to fabricate evidence in obscure fields or areas where one’s results are not likely to be checked. The more prestigious the result, the less likely false or misleading evidence will go unnoticed because others will try to reproduce the results and fail. This is how cheaters are caught.
Even then, some studies are flawed due to environmental factors. Scientists who work with mice found out that different treatment results from the same treatment with different mouse strains could be minimized if they limited food intake shortly before the study began. Different mouse strains giving different results for the same treatment go back 70 years or more. Recently, they discovered that having mice in different cages affected study results.
With medical papers, it’s more difficult to catch frauds. Add in the uncertainty of mice studies and multiply the effect with humans. Every person is unique in genotype and phenotype. We are not at all like inbred mouse strains. Add in insufficient statistical sample sizes, bad statistical analysis, sloppy methodology, environmental and psychological effects, and it’s difficult to tell if the author is incompetent or a fraud. Generally, with frauds, the results are too good to be true, and the methodology is sound, but the results are unreproducible. But, the poor quality of clinical medical articles seems to have been a given for some time.
What the Atlantic article didn’t discuss is the difference in publications between researchers in fields with dedicated funding such as Germany versus America. German researchers generally don’t have to worry about publishing to obtain funding. Their funding is dedicated, so the publish or perish linkage is broken. Therefore, there is less pressure to be “right” or prove others wrong for career or professional advancement. Since the US publishes a great deal more research than any other nation or even group of nations, the results will be skewed by our publish or perish system. It would be informative to know who is getting correct results rather than who is getting it wrong for prestige or profit. Until the incentives are fixed and proper methodologies followed, nothing will change in medicine, economics, or any other field of human endeavor.
Labels: bad science versus fraudulent science
