Saturday, December 29, 2007

A lot of bias and possibly some reality

picking lucky clover
A sample of twenty medical studies shows three times more significant results than null findings which gives me the strong impression that selective picking and publishing of results is the cause rather than real links between factors relevant to health.

During the holidays, I am taking a few days off from blogging, and I have used some of this time to think about what I am doing here. One big challenge for anyone reporting medical research findings is John Ioannidis' landmark essay, published two years ago at PLoS Medicine: "Why most published research findings are false". He gives a number of reasons and uses a bunch of math to prove them, and I must admit that I cannot follow the math. But the logic is convincing. Ioannidis comes to the following conclusion:

"There is increasing concern that most current published research findings are false. The probability that a research claim is true may depend on study power and bias, the number of other studies on the same question, and, importantly, the ratio of true to no relationships among the relationships probed in each scientific field. In this framework, a research finding is less likely to be true when the studies conducted in a field are smaller; when effect sizes are smaller; when there is a greater number and lesser preselection of tested relationships; where there is greater flexibility in designs, definitions, outcomes, and analytical modes; when there is greater financial and other interest and prejudice; and when more teams are involved in a scientific field in chase of statistical significance. Simulations show that for most study designs and settings, it is more likely for a research claim to be false than true. Moreover, for many current scientific fields, claimed research findings may often be simply accurate measures of the prevailing bias. In this essay, I discuss the implications of these problems for the conduct and interpretation of research."
Let's suppose a field of research where there are no associations at all. The level of significance, by generally accepted convention, is 5 percent - the probability that an association has been detected where in reality there is none. That is, for every hundred trials we are likely to see five associations where there is none. This would be no problem as long as every finding, be it positive or null, would have the same chance of being published. But “something found" is much more likely to be published than “nothing found". And with this publication bias, false findings are more likely to be published than true findings.

In search of publication bias

Yesterday I have taken a look into this matter. Of course it is not a scientific research but I try to be as accurate as possible. I did a Medline search with the terms "nhanes association mortality diet" in order to find studies that link various diet factors to diseases and mortality. I only included original studies (no meta analysis), and I limited my search to twenty publications. I have found that significant results strongly outnumber the non-significant ones by a factor of about three:
  • 9 non-significant findings where all but one are presented together with significant findings in subgroups or different associations,
  • 26 significant findings.
For instance, potassium intake is not linked to stroke mortality in most people but to a lower stroke mortality in hypertensive men. This is a clear example of publication bias: The finding in the hypertensive subgroup has been published because it has been found significant, it would not have been published if non-significant, just as the non-significant finding in hypertensive women has been omitted in this study.

According to Ioannidis, the relationship between true and not true results is 1:1 in a well-powered randomized controlled trial in a well explored scientific field. In epidemiologic studies like the twenty I have selected, this relationship is only 1:10. The relationship between published significant and non-significant findings is 3:1 where it is expected to be 1:10, hence I have found a publication bias of 30. In other words, a significant finding may be up to 30 times more likely to be published than a non-significant one. This concludes my own little research. Let's turn to some more conclusions of Ioannidis.

Less power, more false results

Picking and publishing subgroup results increases the rate of false results not only by publication bias but also by weakening the statistical power. A subgroup of cases is always smaller than the whole group, and by laws of statistics the fault rate must increase.

How many results may be true?

Ioannidis has calculated the percentage of study results that are likely to be true. This is the case only for a well-powered randomized controlled trial with little bias and 1:1 odds that a true result may be found, or for a meta-analysis of good randomized controlled trials where results have to be confirmed. In both cases, the results are 85 percent likely to be true.

Single epidemiological studies that are done for testing a hypothesis (e.g. about fat intake and heart disease) will be true only in 20 percent of the cases, even if well done. A meta-analysis of small, inconclusive studies is still less likely (41 percent) to be true than false.

Photo credit:

No comments: