THE TYPICAL FALSITY OF BIOMEDICAL RESEARCH FINDINGS

 


Readers are urged to review an important paper by Dr. John P. A. Ioannidis1 published back in 2005 (but even more relevant today given the problematic origin of certain novel diseases and the unbridled politicization of public health care), the opening paragraph of which includes the unsettling statement: [colour emphasis added]

It can be proven that most claimed research findings are false.2

Ioannidis takes issue with ostensibly conclusive research results being based exclusively on formal statistical significance (i.e., a \(p\)-value usually less than \(0.05\) of a single study),3,4,5 his focus being on “relationships that investigators claim exist, rather than null findings.” 6 He continues in the section titled, Modeling the Framework for False Positive Findings,7 establishing the \(2 \textrm{ x 2}\) table (Table1) from which he obtains the positive predictive value,8 \[\textrm{PPV}=\dfrac{(1-β)R}{(R-βR+α)},\] where \(R\) is the “ratio of the number of true relationships to no relationships,” \(α\) is the Type I error,9 and \(β\) is the Type II error.10 He concludes that “[a] research finding is thus more likely to be true than false if \((1-β)R>α \)”, further clarifying that “[s]ince usually the vast majority of investigators depend on \(α=0.05\), this means that a research finding is more likely true than false if \((1-β)R>0.05\).” 11

Ioannidis then points out and models two factors that may further reduce the probabilities of research findings being true: (i) Bias, and (ii) Testing by Several Independent Teams.12

Under the section titled, Bias, Ioannidis defines bias as “the combination of various design, data, analysis, and presentation factors that tend to produce research findings when they should not be produced.” In that context, he then formally defines bias as \(u\) — “the proportion of probed analyses that would not have been ‘research findings,’ but nevertheless end up presented and reported as such, because of bias.” He establishes the \(2 \textrm{ x 2}\) table (Table 2) influenced by bias from which he obtains \[\textrm{PPV}=\]

\[\dfrac{([1-β]R+uβR)}{(R+α-βR+u-uα+uβR)},\]

further clarifying that “\(\textrm{PPV}\) decrease with increasing \(u\), unless \(1-β≤α\), i.e., \(1-β≤0.05\) for most situations.” He concludes the formal or analytical part of the section (i.e., the 1st paragraph) on the influence of bias by stating: “Thus, with increasing bias, the chances that a research finding is true diminish considerably. This is shown for different levels of power and for different pre-study odds in Figure 1.” 13,14

Under the section titled, Testing by Several Independent Teams, Ioannidis explains that while many independent teams worldwide may be addressing the same research questions, a claimed research finding by a single team often receives unilateral attention [emphasis added]. He states: [...] “For \(n\) independent studies of equal power, the \(2 \textrm{ x 2}\) table is shown in Table 3:\[\textrm{PPV}=\]\[\dfrac{R(1-β ^n)}{(R+1-[1-α]^n-Rβ ^n)},\] (not considering bias).”15 He further states: “With increasing number of independent studies, \(\textrm{PPV}\) tends to decrease, unless \(1-β<α\), i.e., \(1-β<0.05\). This is shown for different levels of power and for different pre-study odds in Figure 2.” 16 [...]

Based on the above influences, Ioannidis deduces and explains (mentioning several biomedical fields and providing a practical example in Box 1) the following six corollaries in respect of the probability of any particular research finding being true:17 [colour emphasis added]

Corollary 1: The smaller the studies conducted in a scientific field, the less likely the research findings are to be true.

Corollary 2: The smaller the effect sizes in a scientific field, the less likely the research findings are to be true.

Corollary 3: The greater the number and the lesser the selection of tested relationships in a scientific field, the less likely the research finding are to be true.

Corollary 4: The greater the flexibility in designs, definitions, outcomes, and analytical modes in a scientific field, the less likely the research findings are to be true.

Corollary 5: The greater the financial and other interests and prejudices in a scientific field, the less likely the research findings are to be true.

Corollary 6: The hotter a scientific field (with more scientific teams involved), the less likely the research findings are to be true.

Finally, Ioannidis provides a recapitulation under the following statement headings:18 [colour emphasis added]

Most Research Findings Are False for Most Research Designs and for Most Fields

Under the above statement, he includes a table (Table 4) that “... provides the results of simulations using the formulas developed for the influence of power, ratio of true to non-true relationships, and bias, for various types of situations that may be characteristic of specific study designs and settings.” 19

Claimed Research Findings May Often Be Simply Accurate Measures of the Prevaiing Bias

Under the above statement, Ioannidis summates [1st paragraph]: “As shown, the majority of modern biomedical research is operating in areas with very low pre- and post-study probability for true findings.” [...] He subsequently elaborates on this concept and incldes an example.20

Ioannidis concludes his paper with a section under the question: How can we improve the situation? To that question, he offers three suggestions that include the following statements: 1. [2nd paragraph] [...] “Large-scale evidence should be targeted for research questions where the pre-study probability is already considerably high, so that a significant research finding will lead to a post-test probability that would be considered quite definitive. Large-scale evidence is also particularly indicated when it can test major concepts rather than narrow, specific questions.” [...] 2. [3rd paragraph] [...] “... the principles and developing and adhering to a protocol could be more widely borrowed from randomized controlled trials.” 3. [4th paragraph] “... instead of chasing statistical significance, we should improve our understanding of the range of \(R\) values—the pre-study odds ... .” [...] 21

Denouement

As pointed out on our web page titled, Science and Scientism, an in-depth coverage of the extent to which scientific dogma and scientism had all but eclipsed real science by the early twenty-first century is provided in the book by Paul & Phillip Collins titled, The Ascendancy of the Scientific Dictatorship 22 — a book that we highly recommend. Especially from that perspective but equally in and of itself, the paper by Dr. John P. A. Ioannidis weighs heavily on the peril of real science in our day and should sound the alarm for anyone considering serious research in virtually any scientific field.

The highly politicized public health care response to novel diseases of problematic origin has typically included the trite invocation of the mantra, “follow the science.” If we are to buy into that mantra, we need to ensure that the science we are following is actually grounded in reality.


— FINIS —


  1. John P. A. Ioannidis, Why Most Published Research Findings Are False, PLOS Medicine, 2 (8) e124: 0696–0701 (August 2005).↩️

  2. Ibid., p. 0696.↩️

  3. Loc. cit.↩️

  4. See for example, P-Value: What It Is, How to Calculate It, and Why It Matters. The \(p\)-value can be defined as: “[a] statistical measure used to determine the liklihood that an observed outcome is the result of chance.”↩️

  5. Cf. WolframAlpha — Input Interpretation: P-value. Definition: “The probability that a variate would assume a value greater than or equal to the observed value strictly by chance: \(P\) \((z≥z_{observed})\).” The variate \(z\) (i.e., the z-score) is defined as follows: See WolframAlpha — Input Interpretation: z-score. Definition: “A z-score, also called a “standard score,” is the difference from the mean divided by the standard deviation.” Detailed Definition: “The \(z\)-score associated with the \(i\)th observation of a random variable \(x\) is given by\[z_i≡\frac{x_i-x̄}{σ},\] where \(x̄ \) is the mean and \(σ\) the standard deviation of all observations \(x_1, ...,x_n\).”↩️

  6. Ioannidis, loc. cit.↩️

  7. Ibid., pp. 0696–0697.↩️

  8. WolframAlpha — Input Interpretation: predictive value. Definition: “The positive predictive value is the probability that a test gives a true result for a true statistic. The negative predictive value is the probability that a test gives a false result for a false statistic.”↩️

  9. WolframAlpha — Input Interpretation: type I error. Definition: “An error in a statistical test which occurs when a false hypothesis is accepted (a false positive in terms of the null hypothesis).”↩️

  10. WolframAlpha — Input Interpretation: type II error. Definition: “An error in a statistical test which occurs when a true hypothesis is rejected (a false negative in terms of the null hypothesis).”↩️

  11. Ioannidis, op. cit., pp. 0696–0697.↩️

  12. Ibid., p. 0697.↩️

  13. Loc. cit.↩️

  14. Ibid., Figure 1, p. 0698.↩️

  15. Ibid., p. 0697 and Table 3, p. 0698.↩️

  16. Ibid., p. 0697 and Figure 2, p. 0699.↩️

  17. Ibid., pp. 0697–0699.↩️

  18. Ibid., pp. 0699–0700.↩️

  19. Ibid., p. 0699 and Table 4, p. 0700.↩️

  20. Ibid., p. 0700.↩️

  21. Ibid., pp. 700–0701.↩️

  22. Phillip Darrell Collins and Paul David Collins, The Ascendancy of the Scientific Dictatorship: An Examination of Epistemic Autocracy, From the 19th to the 21st Century (BookSurge Publishing, 2006).↩️

 
Edwin Wright