In praise of the referee (guest post)

Nicolas Chopin sent me this piece after reading Larry’s radical proposal. And my post on it. This is a preliminary version, so feel free to comment!

In a provocative column in the latest ISBA Bulletin, Larry Wasserman calls for “a world without referees”. This is an interesting read, not devoid of mala fide arguments (“We are using a refereeing system that is almost 350 years old. If we used the same printing methods as we did in 1665 it would be considered laughable.”), but this is hard to avoid, given how passionate this subjects is for many scientists. In this article, I’d like to propose a defense of the too often derided referee, arguing that a system that has served us so well for 350 years cannot be ditched so easily.

To start with, talking about getting rid of peer-reviewing is a bit of idle talk, as we all know it is never going to happen. In fact, it’s a perfect example of the prisoner’s dilemma. Stopping to send papers to journals would make sense only if a majority of scientists would decide to do so simultaneously. But, for many of us, so much depends on our publication record (including jobs, promotions, grants, even salaries in certain institutions) that very few would dare to “shoot first”. And, even if we assume a given field would be ready to do so (say all the statisticians), would that such a move make any sense, without all the other fields of Science doing it also at the same time? The prisoner’s dilemma simply moves to a higher level, as scientific fields are also competing for grants, jobs openings and so on.

I think US scientists would be surprised to see how regular, yet basic and dumbly quantitative is the evaluation of research by governments and Universities in most countries. This is certainly unfortunate, but it reinforces the prisoner’s dilemma I am talking about.

The previous section seems to make the usual argument that refereeing is a necessary evil. We believe on the contrary it is a necessary good. Yes, certain referees are annoying, or even aggressive or too dismissive about our work. Of course, like Larry, we can tell several horror stories about referees completely missing the point, or even perhaps being outright dishonest.

But, ego bruising and venting aside, all this is beside the point. The real question is: does refereeing increase the quality of our papers? or is it just “noise”, as stated by Larry?

My personal experience is that all my papers have benefited from refereeing. The most extreme example I can share is that of a very obnoxious referee which was obviously doing all his best to get my paper rejected (while making me add irrelevant references, probably his), and managed to point out a mistake in my rejection algorithm during the third revision. How glad I am to have had such a nitpicking referee in this case. Publishing a wrong paper is much more damaging in the long run than getting rejected. Plus, getting rejected is not such a big deal. If the paper is worth it, we always find the energy to submit it elsewhere, hopefully in a better shape.

In fact, the only referee I fear is the sloppy one who quickly reads the paper and thinks he “gets it”. But usually associate editors are good at spotting these, so this does occur so often.

I also believe that our papers get improved “preemptively” by refereeing; that is, we write better papers because we know it is going to be evaluated by colleagues. We go the extra mile, chase typos, think more carefully about real examples, and so on. And, finally, refereeing simply filters the very long tail of bad papers. How glad I am that referees guard the gates of my favorite journals against them. Please do not ask me instead to check every other paper on arXiv. I don’t have the time, when I do it, I’m strongly biased in favor of authors I know well (like Larry), so this is not fair, and it does not make sense in the first place that all of us replicate this filtering. Plus, from the papers I review, I can see that quite a small proportion of submitted papers are actually send first to arXiv; if all papers were sent there, we would be even less able to deal with this deluge of papers.

Finally, some people seem to be outraged that referees ask certain modifications, because they consider that, as “authors”, they should have an inalienable right to decide the exact form of their texts. But we are not artists (who actually rarely obtain this right anyway), we are scientists, and science (in particular hard science, especially mathematics) works better through consensus on the validity and correctness of the proposed research.

As a final note, Larry’s letter seems to be part of a wider movement of rethinking the way we are dealing with scientific publishing, especially in lights of the “disruptive” impact of the Internet, whatever that oft-used word means. Such questioning is of course welcome. But, as most things, our mental energy comes in limited supply, and should be targeted first at the most glaringly obvious drawbacks of the current system. I am talking of course of the issue of unethical publishers, which is a polite word for thugs. The problem is well-known: we, the scientists, write the papers, evaluate the papers, do the editorial work, everything for free, yet these publishers take our work from us, and charge ridiculous, and ever-increasing, prices for accessing it. Harvard, of all places, cannot afford anymore to pays for journal subscriptions (3.5 millions a year, see an official memorandum). Can you imagine how is the situation in less endowed Universities? This is not sustainable, and we should do our best to get rid of these thugs. This is another subject, but may we quickly urge our readers to sign the current boycott of Elsevier, and prefer to submit to open access journals, such as our beloved Bayesian Analysis‘. Let us push together to get rid of this non-sense.

When this is done, we may return to questioning the refereeing system, and perhaps try to improve on it. One thing I’d like to experiment with would be to reveal the names of referees, not only to the authors, but also to the public, by mentioning their on each publication. That way, referees cannot be either too complacent, or too negative. And at the same time, this would be good recognition of the role of referees, in their help of publishing better research. I would go as far as saying that this would help us to recognize them as co-authors. Not so bad for the poor referee we have been venting at for the last 350 years.