Young 2008 PLoS Med
|Young NS, Ioannidis JPA, Al-Ubaydli O (2008) Why current publication practices may distort science. PLoS Med 5:e201. doi: 10.1371/journal.pmed.0050201
Abstract: The current system of publication in biomedical research provides a distorted view of the reality of scientific data that are generated in the laboratory and clinic. This system can be studied by applying principles from the field of economics. The “winner’s curse,” a more general statement of publication bias, suggests that the small proportion of results chosen for publication are unrepresentative of scientists’ repeated samplings of the real world. The self-correcting mechanism in science is retarded by the extreme imbalance between the abundance of supply (the output of basic science laboratories and clinical investigations) and the increasingly limited venues for publication (journals with sufficiently high impact). This system would be expected intrinsically to lead to the misallocation of resources. The scarcity of available outlets is artificial, based on the costs of printing in an electronic age and a belief that selectivity is equivalent to quality. Science is subject to great uncertainty: we cannot be confident now which efforts will ultimately yield worthwhile achievements. However, the current system abdicates to a small number of intermediates an authoritative prescience to anticipate a highly unpredictable future. In considering society’s expectations and our own goals as scientists, we believe that there is a moral imperative to reconsider how scientific data are judged and disseminated.
• Bioblast editor: Gnaiger E
- This essay makes the underlying assumption that scientific information is an economic commodity, and that scientific journals are a medium for its dissemination and exchange. While this exchange system differs from a conventional market in many senses, including the nature of payments, it shares the goal of transferring the commodity (knowledge) from its producers (scientists) to its consumers (other scientists, administrators, physicians, patients, and funding agencies). The function of this system has major consequences. Idealists may be offended that research be compared to widgets, but realists will acknowledge that journals generate revenue; publications are critical in drug development and marketing and to attract venture capital; and publishing defines successful scientific careers. Economic modelling of science may yield important insights.
- The Winner’s Curse: In auction theory, under certain conditions, the bidder who wins tends to have overpaid. .. As with individual bidders in an auction, the average result from multiple studies yields a reasonable estimate of a “true” relationship. However, the more extreme, spectacular results (the largest treatment effects, ..) may be preferentially published. Journals serve as intermediaries and may suffer minimal immediate consequences for errors of over- or mis-estimation, but it is the consumers of these laboratory and clinical results (other expert scientists; ..) who are “cursed” if these results are severely exaggerated—overvalued and unrepresentative of the true outcomes of many similar experiments.
- Among of the 49 most-cited papers on the effectiveness of medical interventions, published in highly visible journals in 1990–2004, showed that a quarter of the randomised trials and five of six non-randomised studies had already been contradicted or found to have been exaggerated by 2005 . The delay between the reporting of an initial positive study and subsequent publication of concurrently performed but negative results is measured in years [10,11].
- For most published papers, “publication” often just signifies “final registration into oblivion”. .. Only 73 of the many thousands of articles ever published by the 187 BMC-affiliated journals had over 10 000 accesses through their journal Web sites in the last year .
- Impact factors are widely adopted as criteria for success, despite whatever qualms have been expressed [27–32].
- A common excuse for rejection is selectivity based on a limitation ironically irrelevant in the modern age—printed page space.
- Manuscripts are assessed with a fundamentally negative bias: how they may best be rejected to promote the presumed selectivity of the journal.
- Promote rapid, digital publication of all articles that contain no flaws, irrespective of perceived “importance”.
- Adopt preferred publication of negative over positive results; require very demanding reproducibility criteria before publishing positive results.
- Select articles for publication in highly visible venues based on the quality of study methods, their rigorous implementation, and astute interpretation, irrespective of results.
- Adopt formal post-publication downward adjustment of claims of papers published in prestigious journals.
- Promote critical reviews, digests, and summaries of the large amounts of biomedical data now generated.
- To exorcise the winner’s curse, the quality of experiments rather than the seemingly dramatic results in a minority of them would be the focus of review, but is this feasible in the current reality?
- Gnaiger E (2021) Beyond counting papers – a mission and vision for scientific publication. Bioenerg Commun 2021.5. https://doi:10.26124/BEC:2021-0005