December 30, 2005

An Insider’s Assessment of Medical Journals and Peer Review

The author, Thomas Stossel, is American Cancer Society Professor at Harvard Medical School and co-director of the division of hematology at Brigham and Women’s Hospital. This is an excerpt of his column in the December 30 Wall Street Journal (requires subscription). It confirms the “passes the smell test (maybe)” perception that “peer review” deserves discussed at this previous post (“What Does ‘Peer Review’ Mean?”), and notes that the term does not deserve the high ground it currently holds:

The (media) message in all this is clear: Medical academics are saints — devoted selflessly to patient care — and corporate people are sinners, morally blinded by greed. But having worked in academic medicine for over 35 years and consulted for companies, this Manichean duality is inconsistent with my experience and a woeful distortion of reality. In a Sept. 8 article in the New England Journal of Medicine, I reported that no systematic evidence exists that corporate sponsorship of academic research contributes to misconduct, bias, public mistrust or poor research quality.

….. But unbeknownst to the media, the journals at the top got there because of herd behavior by researchers, not because they are better than lower-tier journals at vetting research quality. Here’s why: Researchers submit their best work to the top journals, which can therefore afford to maintain their prestige by rejecting, not publishing, many high quality papers. That’s brand creation — not science. Most of their editorial effort goes into deciding which submitted papers are sufficiently newsworthy. Anonymous peer review by jealous competitors has its merits, but it has a tendency to select for fashionable if relatively unoriginal and inoffensive papers. Top medical journals compete for papers describing large clinical trials reporting small effects of treatments for diseases affecting many people, although these reports often do not substantively advance scientific knowledge, and many subsequently are invalidated.

And no description of medical research in a medical journal comes close to the detail level or intense scrutiny imposed by the FDA on companies’ documentation of drug or device development before approval. Space constraints for readability and cost-savings preclude journals from publishing detailed information on the order of what companies file with the FDA, and unpaid journal peer reviewers, not to mention practicing doctors, would never read it anyway. The recent Korean cloning fiasco, in which the leading science journals published blatantly fraudulent papers, wasn’t the first such incident to afflict prestige journals, and it could never happen under conditions of FDA review. Indeed, doctors should take all studies published in “prominent medical journals” with “skeptical caution.”

The lower stringency of journals compared to the FDA is a good thing, because academic biomedical research would come to a screeching halt if subjected to anything even approximating FDA examination. Scientific knowledge advances reasonably efficiently, and new technologies emerge, despite the looseness of journals. And researchers’ craving for prestige goads them to greater efforts.

If reporters understood that journals are magazines, not Holy Scripture, we might not be witnessing ever more onerous regulations inhibiting interactions between academic and industry science. Prestigious biomedical journals are good for our health — provided they stick to their core business of facilitating imperfect communication between researchers. Leave drug and device monitoring to the FDA — and theology to theologians.

Share

1 Comment

  1. Well spoken, Dr. Stossel. Thanks for posting his comments – I hadn’t seen the article. By the way, I think BizzyBlog’s coverage of this scandal has really demonstrated a great learning experience about peer review. As a scienctist and teacher, I had to write about it. Keep up the great work.

    Comment by wulf — January 2, 2006 @ 10:17 am

RSS feed for comments on this post.

Sorry, the comment form is closed at this time.