The peer review process is central to most research disciplines and is used in the selection of papers for publication and research proposals for funding.

A new paper by Stefan Thurner and Rudolf Hanel develops an agent-based model of the scientific peer review process, Peer-review in a world with rational scientists: Toward selection of the average.

“… we are interested in the effects of rational referees, who might not have any incentive to see high quality work other than their own published or promoted. We find that a small fraction of incorrect (selfish or rational) referees can drastically reduce the quality of the published (accepted) scientific standard. We quantify the fraction for which peer review will no longer select better than pure chance. Decline of quality of accepted scientific work is shown as a function of the fraction of rational and unqualified referees. We show how a simple quality-increasing policy of e.g. a journal can lead to a loss in overall scientific quality, and how mutual support-networks of authors and referees deteriorate the system.”

Their agent model has several reviewers types:

• The correct: Accepts good and rejects bad papers.
• The stupid: This referee can not judge the quality of a paper (e.g. because of incompetence or lack of time) and takes a random decision on a paper.
• The rational: The rational referee knows that work better than his/her own might draw attention away from his/her own work. For him there is no incentive to accept anything better than one’s own work, while it might be fine to accept worse quality.
• The altruist: Accepts all papers.
• The misanthropist: Rejects all papers.

I’ve known them all, as I am sure many of us have. As an editor or program chair I’ve met a few other types, including these:

• The Bartleby: His or her response to an invitation is always “I would prefer not to.”
• The Black Hole: Messages go in and nothing ever comes out.
• The Gary Cooper: A person of few words, even when many are called for.
• The Perseverator: Sees all sides of any decision and keeps all carefull in balance. Usually recommends “major revision”.

I am sure I’ve overlooked some — suggest your own via a comment.

(h/t Shlomo Argamon)

This sounds like a book worth reading, Proofiness – The Dark Arts of Mathematical Deception by Charles Seife. It is reviewed in tomorrow’s New York Times — Fibbing With Numbers.

It goes without saying that what you learn in a book like this should only be used for defensive purposes. Do not turn to the Dark Side!

The book reminds me of the classic How to Lie with Statistics published in the 1950s.