December 17, 2005

This Weekend’s Single Unanswered Question (121705): What Does “Peer Review” Mean?

Filed under: Corporate Outrage,Taxes & Government,TWUQs — Tom @ 7:26 am

Another installment in a nearly-regular series of mysteries and pseudo-mysteries (usually 3-4, but this time just one) this inquiring mind would like to have answers for:

QUESTION: Did this shake your faith in scientific reports and research like it shook mine?

It turns out that a major research paper relating to embryonic stem-cell research was faked:

Cloning pioneer to withdraw paper, doctor says

SEOUL (AP) — A doctor who provided human eggs for research by cloning pioneer Hwang Woo-suk said in a broadcast Thursday that the South Korean scientist agreed to withdraw a key research paper because most of the stem cells produced for the article were faked. (Related item: Independent probe sought).

Huh? I thought an “independent probe” is always done when major scientific findings are reported.

So let’s move to USAT’s “independent review” article linked above:

Science is at the center of this issue because important research papers are peer-reviewed and published in the prestigious journal.

….. In a letter released Tuesday by Science, eight prominent stem cell researchers — including Ian Wilmut, who cloned Dolly the sheep in 1997 — called for an independent verification of the Hwang lab’s cell-cloning feats.

Again, huh? Why the need for an “independent verification” if the paper was already “peer-reviewed”?

Mystery solved (HT Amy Ridenour)–”peer review” gives none of the assurance a layman would expect from the term (the excerpt concerns climate science, but blows open the fallacy in the general public’s understanding of all scientific “peer review”):

In business, “full, true and plain disclosure” is a control on stock promoters. While it may not always be successful, it gives an enforcement mechanism. There is no such standard in climate science. (or “science” in general–Ed.) ….. In fairness, the journals do not require authors to warrant full, true and plain disclosure and there is little guidance to such authors as to what is required reporting and what is not required.

I’ve found that scientists strongly resent any attempt to verify their results. One of the typical reactions is: don’t check our studies, do your own study. I don’t think that businesses like being checked either, but one of the preconditions of being allowed to operate is that they are checked. Many of the most highly paid professionals in our society – securities lawyers, auditors – earn much of their income simply by verifying other people’s results. Businesses developed checks and balances because other peoples’ money was involved, not because businessmen are more virtuous than academics.

Then there is this comment at the same post:

I don’t think that it’s practical for journal peer reviewers to check every piece of data and every calculation – nobody would ever do reviews. But people should realize that even “peer reviewed” journal articles are (in business terms) unaudited. It doesn’t mean that they are wrong; it only means that they are unaudited. The crunch comes if people rely on them as though they were audited.

So the next time you hear the term “peer-reviewed,” I would substitute these words: “passed the smell test (maybe, and if the person submitting the work is ethical and conducted his/her work conscientiously and honorably).”

Given the ever-larger dollars, very often tax dollars, that are based on the reliability of scientific work, standards must be raised, even if it costs money up-front (auditors, if you will) to raise them, and even if scientists’ egos are bruised in the process.

Maybe we as citizens should demand that the entire scientific community be brought under the heavy hand of Sarbanes Oxley.

Full Disclosure: I have recently posted on what I see as disparate media treatment between embryonic stem cell research and adult stem cell research (with the press reporting more favorably, and more often, on embryonic developments), and my personal belief that long-run success in using stem cells will come from the adult variety.

UPDATE: BizzyBlog links, you decide–”Cloning Doctor Finally Answers Critics — South Korean stem cell pioneer insists his groundbreaking research is still credible”

UPDATE 2, Dec. 26: What is probably the last word — “Stem cells in disgraced scientist’s paper did not exist: SKorean report”



  1. Peer review can be interpreted in many different ways. As a lead engineer, I am responsible for reviewing the work of subordinates. A peer review to me, is to ensure that an engineer has appropriately followed processes, ensure that from a high-level an engineer has created output with a sufficient level of scientific discipline, and determine if the particular solution integrates into a pre-existing architecture. What I will not do is scrutinize the small details. These sorts of peer reviews make employees redundant and waste productivity.

    Comment by Kevin Irwin — December 17, 2005 @ 1:31 pm

  2. #1 – points well taken.

    I think the difference is this: You and your engineer are on the same team and there are powerful disincentives not to cook the results, depending on the type of engineering. If it’s civil, it’s so what you build doesn’t come crashing down, you all lose your jobs, etc., which of course could happen if you’re caught before the structure is built too. So I think in general the level of review in a situation like yours is fine.

    The incentives in research for public and government consumption are totally different. Now as long as you have competent people with integrity and objectivity, that’s fine. The problem is that time and again studies have been cooked for the purpose of getting public funding or influencing the public debate, and the initial peer review process hasn’t screened out the offenders.

    When it gets to the point that how hot or cold it is today is a political issue (which it is), and you realize the ethical breakdowns that have occurred all over the place (not just science, obviously), and you realize the billions (and the results of political races) that are at stake, I think it’s obvious that peer review to the level described in the post and elaborated on in your comment won’t suffice any more. It’s too bad, because it’s costly, as you noted, but look at what this guy pulled off for a while. And how many others are slipping through, only to (hopefully) be discovered as frauds many years and millions or billions of dollars later? The post I linked to for the peer review support had to do with climate change. There is SOOOOO much garbage out there on climate change, yet it all gets treated equally once it has that magic “peer review” sticker placed on it.

    One more thing: I remember all this hype from the 1980s about electromagnetic fields from utility towers leading to cancers and other things. The studies were debunked because they were found to be cooked many, many years down the road (the primary purveyor finally admitted to the cooking). But years later, the public still largely believes that they’re courting death if they build anything close to one of these towers. This is unacceptable, and if it takes peer audits to ferret out BS like this, I am (reluctantly, because of money) for it.

    Comment by TBlumer — December 17, 2005 @ 1:50 pm

  3. It’s not just a matter of money — even in uncontroversial areas the peer review can be a little sloppy. I saw that with some published math journal articles — there’s too much specialization sometimes and it can be extremely difficult to follow someone else’s results… especially as there are certain institutional biases involved. No one’s being intentionally dishonest, but there’s not much incentive to work as hard on someone else’s research as on your own.

    Comment by meep — December 18, 2005 @ 5:28 am

  4. #3, Good point. I guess that’s just another point as to why there shouldn’t be teh kind of halo effect from peer review that most people give it.

    Comment by TBlumer — December 18, 2005 @ 11:36 am

RSS feed for comments on this post.

Sorry, the comment form is closed at this time.