Archive for scientific journals

In praise of the referee (guest post)

Posted in Books, Statistics, University life with tags , , , , on May 3, 2012 by xi'an

Nicolas Chopin sent me this piece after reading Larry’s radical proposal. And my post on it. This is a preliminary version, so feel free to comment!

In a provocative column in the latest ISBA Bulletin, Larry Wasserman calls for “a world without referees”. This is an interesting read, not devoid of mala fide arguments (“We are using a refereeing system that is almost 350 years old. If we used the same printing methods as we did in 1665 it would be considered laughable.”), but this is hard to avoid, given how passionate this subjects is for many scientists. In this article, I’d like to propose a defense of the too often derided referee, arguing that a system that has served us so well for 350 years cannot be ditched so easily.

To start with, talking about getting rid of peer-reviewing is a bit of idle talk, as we all know it is never going to happen. In fact, it’s a perfect example of the prisoner’s dilemma. Stopping to send papers to journals would make sense only if a majority of scientists would decide to do so simultaneously. But, for many of us, so much depends on our publication record (including jobs, promotions, grants, even salaries in certain institutions) that very few would dare to “shoot first”. And, even if we assume a given field would be ready to do so (say all the statisticians), would that such a move make any sense, without all the other fields of Science doing it also at the same time? The prisoner’s dilemma simply moves to a higher level, as scientific fields are also competing for grants, jobs openings and so on.

I think US scientists would be surprised to see how regular, yet basic and dumbly quantitative is the evaluation of research by governments and Universities in most countries. This is certainly unfortunate, but it reinforces the prisoner’s dilemma I am talking about.

The previous section seems to make the usual argument that refereeing is a necessary evil. We believe on the contrary it is a necessary good. Yes, certain referees are annoying, or even aggressive or too dismissive about our work. Of course, like Larry, we can tell several horror stories about referees completely missing the point, or even perhaps being outright dishonest.

But, ego bruising and venting aside, all this is beside the point. The real question is: does refereeing increase the quality of our papers? or is it just “noise”, as stated by Larry?

My personal experience is that all my papers have benefited from refereeing. The most extreme example I can share is that of a very obnoxious referee which was obviously doing all his best to get my paper rejected (while making me add irrelevant references, probably his), and managed to point out a mistake in my rejection algorithm during the third revision. How glad I am to have had such a nitpicking referee in this case. Publishing a wrong paper is much more damaging in the long run than getting rejected. Plus, getting rejected is not such a big deal. If the paper is worth it, we always find the energy to submit it elsewhere, hopefully in a better shape.

In fact, the only referee I fear is the sloppy one who quickly reads the paper and thinks he “gets it”. But usually associate editors are good at spotting these, so this does occur so often.

I also believe that our papers get improved “preemptively” by refereeing; that is, we write better papers because we know it is going to be evaluated by colleagues. We go the extra mile, chase typos, think more carefully about real examples, and so on. And, finally, refereeing simply filters the very long tail of bad papers. How glad I am that referees guard the gates of my favorite journals against them. Please do not ask me instead to check every other paper on arXiv. I don’t have the time, when I do it, I’m strongly biased in favor of authors I know well (like Larry), so this is not fair, and it does not make sense in the first place that all of us replicate this filtering. Plus, from the papers I review, I can see that quite a small proportion of submitted papers are actually send first to arXiv; if all papers were sent there, we would be even less able to deal with this deluge of papers.

Finally, some people seem to be outraged that referees ask certain modifications, because they consider that, as “authors”, they should have an inalienable right to decide the exact form of their texts. But we are not artists (who actually rarely obtain this right anyway), we are scientists, and science (in particular hard science, especially mathematics) works better through consensus on the validity and correctness of the proposed research.

As a final note, Larry’s letter seems to be part of a wider movement of rethinking the way we are dealing with scientific publishing, especially in lights of the “disruptive” impact of the Internet, whatever that oft-used word means. Such questioning is of course welcome. But, as most things, our mental energy comes in limited supply, and should be targeted first at the most glaringly obvious drawbacks of the current system. I am talking of course of the issue of unethical publishers, which is a polite word for thugs. The problem is well-known: we, the scientists, write the papers, evaluate the papers, do the editorial work, everything for free, yet these publishers take our work from us, and charge ridiculous, and ever-increasing, prices for accessing it. Harvard, of all places, cannot afford anymore to pays for journal subscriptions (3.5 millions a year, see an official memorandum). Can you imagine how is the situation in less endowed Universities? This is not sustainable, and we should do our best to get rid of these thugs. This is another subject, but may we quickly urge our readers to sign the current boycott of Elsevier, and prefer to submit to open access journals, such as our beloved Bayesian Analysis‘. Let us push together to get rid of this non-sense.

When this is done, we may return to questioning the refereeing system, and perhaps try to improve on it. One thing I’d like to experiment with would be to reveal the names of referees, not only to the authors, but also to the public, by mentioning their on each publication. That way, referees cannot be either too complacent, or too negative. And at the same time, this would be good recognition of the role of referees, in their help of publishing better research. I would go as far as saying that this would help us to recognize them as co-authors. Not so bad for the poor referee we have been venting at for the last 350 years.

down with referees, up with ???

Posted in Books, Statistics, University life, Wines with tags , , , , , , , , , , on April 18, 2012 by xi'an

Statisfaction made me realise I had missed the latest ISBA Bulletin when I read what Julyan posted about Larry’s tribune on a World without referees. While I agree on many of Larry’s points, first and foremost on his criticisms of the refereeing process which seems to worsen and worsen, here are a few items of dissension…

The argument that the system is 350 years old and thus must be replaced may be ok at the rethoretical level, but does not carry any serious weight! First, what is the right scale for a change: 100 years?! 200 years?! Should I burn down my great-grand-mother’s house because it is from the 1800’s and buy a camping-car instead?! Should I smash my 1690 Stradivarius and buy a Fender Stratocaster?! Further, given the intensity and the often under-the-belt level of the Newton vs. Leibniz dispute, maybe refereeing and publishing in the Philosophical Transactions of the Royal Society should have been abolished right away from the start. Anyway, this is about rethoric, not matter. (Same thing about the wine store ellipse. It is not even a good one:  Indeed, when I go to a wine store, I have to rely on (a) well-known brands; (b) brands I have already tried and appreciated; (c) someone else’s advice, like the owner, or friends, or Robert Parker…. In the former case, it can prove great or disastrous. But this is the most usual way to pick wines as one cannot hope [dream?] to sample all wines in the shop.)

My main issue with doing away with referees is the problem of sifting through the chaff. The amount of research documents published everyday is overwhelming. There is a maximal amount of time I can dedicate to looking at websites, blogs, twitter accounts like Scott Sisson’s and Richard Everitt’s, and such. And there clearly is a limited amount of trust I put in the opinions expressed in a blog (e.g., take the ‘Og where this anonymous X’racter writes about everything, mostly non-scientific stuff, and reviews papers with a definite bias!) Even keeping track of new arXiv postings sometimes get overwhelming. So, Larry’s “if you don’t check arXiv for new papers every day, then you are really missing out” means to me that missing arXiv for a few days and I cannot recover. One week away at an intense workshop or on vacations and I am letting some papers going by forever, even though I carry them in my bag for a while…. Noll’s suggestion to publish only on one’s own website is even more unrealistic: why should anyone bother to comment on poor or wrong papers, except when looking for ‘Og’s fodder?! So the fundamental problem is separating the wheat from the chaff, given the amount of chaff and the connected tendency to choke on it! Getting rid of referees and journals to rely on depositories like [the great, terrific, essential] arXiv forces me to also rely on other sources for ranking, selecting, and eliminating papers. Again with a component of arbitrariness, subjectivity, bias, variation, randomness, peer pressure, &tc. In addition, having no prior check of papers means reading a new paper a tremendous chore as one would have to check the references as well, leading to a sort of infinite regress… and forcing one to rely on reputation and peer opinions, once again! And imagine the inflation in reference letters! I already feel I have to write too many reference letters at the moment, but a world without (good and bad) journals would be the Hell of non-stop reference letters. I definitely prefer to referee (except for Elsevier!) and even more being a journal editor, because I can get an idea of the themes in the field and sometimes spot new trends, rather than writing over and over again about an old friend’s research achievements or having to assess from scratch the worth of a younger colleague’s work…

Furthermore, and this is a more general issue, I do not believe that the multiplication of blogs, websites, opinion posts, tribunes, &tc., is necessarily a “much more open, democratic approach”: everyone voicing an opinion on the Internet does not always get listened to and the loudest ones (or most popular ones) are not always the most reliable ones. A complete egalitarian principle means everyone talks/writes and no one listens/reads: I’d rather stick to the principles set by the Philosophical Transactions of the Royal Society!

Anyway, thanks to Larry for launching a worthwhile debate into discovering new ways of making academia a more rational and scientific place!