The COVID-19 Research Horse Has Left the Barn
Reading How to Identify Flawed Research Before It Becomes Dangerous by Michael B. Eisen and Robert Tibshirani in the July 20 New York Times had me feeling quite nostalgic for my younger days when I was managing research into scientific publishing practices via grants and contracts from agencies like the National Science Foundation and the National Library of Medicine.
In those pre-Internet days of decades ago “preprints” were already key media for disseminating research findings and data before they completed the formal “peer review” process. Federal agencies were investigating how to speed up both scientific research and the dissemination of findings. Electronic publishing was viewed as a means to improve both speed and cost. Much Federal money was being pumped into the efforts of scientific societies to improve indexing and bibliographic control. It was understood early on that combining both indexing and document access — something we now take for granted — would spread access to research data far and wide — and maybe even speed up the delays inherent in peer review.
One thing that became clear to anyone researching such topics was that you had to take into account not only the physical media being used (journals, preprints, photocopies, microfiche, the US Mail) but also more importantly their relationship to “informal” scientific communication practices around which influence and information are exchanged. Such communication took place via phone and in the bar at night over drinks following presentations at scientific meetings. Good luck trying to control (or even investigate) the influence such communication really had in spreading data and research findings — and I tried!
Even back then once a finding was announced it could spread like wildfire through informal communication channels and cross over to popular or public media like TV, radio, newspapers, and magazines. That could not be controlled. All you could really hope for is that the reporters responsible for breaking the news to the public would actually understand what was being reported and its context. Sometimes they did and sometimes they didn’t.
Fast forward to today and Eisen and Tibshirani’s lament in the NYT that COVID-19 research data are being released prematurely and in some cases “preprints” of research still undergoing peer review are being released prematurely.
One possible impact is that the data being reported are in error and these errors have not yet surfaced in the peer review process as journal editors review the draft publications already distributed via preprints. Another possibility is that the data and findings are actually “correct” but will be misinterpreted by those scientifically unqualified or by those intent on “cherry picking” findings to support a political viewpoint.
Eisen and Tibshirani’s solution is to put more control around the preprint process:
That is why we and a group of over 100 scientists are calling for American scientists and journalists to join forces to create a rapid-review service for preprints of broad public interest. It would corral a diverse contingent of scientists ready to comment on new preprints and to be responsive to reporters on deadline. This would provide journalists reliable access to independent scientists to help deal with today’s growing stream of preprints.
Despite the good intentions of this proposal I fear it can never cover 100% of the problem situations that arise surrounding preprint release prior to completion of the peer review process. Science does not stand still. Neither do the forces pushing for more open access to research findings as they occur. Simply putting more control mechanisms in place on top of those that are already in effect will not solve the problems associated with witting or unwitting misinterpretation.
Given the ubiquity and immediacy of personal communication via the web there will always be routes for data and research to be disseminated outside formal research communication channels. I would argue that the benefits of this outweigh the costs even in the current pandemic crisis. In fact, I would argue that, rather than put another control mechanism in place on top of those that already exists, it would make more sense to strengthen the systems that already exist rather than attempt to replace or bypass them.
Copyright (c) 2020 by Dennis D. McDonald