preprints promote confusion and distorsion, and don’t blame journalists!

“…anyone considering publicizing a preprint have a responsibility.”

On my way to the airport, flying to B’ham, I read an older issue of Nature that contained this incredible editorial entry from Tom Sheldon Tim Horton, calling for regulation of preprints or worse, for the reason that journalists could misunderstand their contents and over-hype a minor or worse wrong claim. Taking as mistaken illustration the case of the Séralini et al. paper, about the Monsanto maize, which happened to be published under “embargo” conditions and reproduced in most media before a scientific storm erupted on the lack of significance of the samples. This call is unbelievably cheeky and downright absurd as it shifts the responsibility away from the journalists to the scientific community, throwing the “check your sources” principle of investigative journalism down the drain. As if the only reason for immediately publishing front-page discoveries is not to beat the competition and attract more readers…

The irony of seeing this piece in Nature is that a few pages later, there is a news entry on German and Swedish institutions breaking negotiations with Elsevier, as the publisher refuses to join a global package of open source publications. Nothing seems amiss about this nice aspect of scientific publishing with the author of this editorial, nor with the further reports of retraction of published paper in the same issue. Presumably because journalists have already moved to the next hot discovery by the time the retractions at last appear…! And to answer the final question of “Should all preprints be emblazoned with a warning aimed at journalists that work has not been peer reviewed?”, no, no, and no: preprints are not written for journalists or the general public. Unsurprisingly, the tribune induced outraged reactions from Nature readers.

agent-based models

An August issue of Nature I recently browsed [on my NUS trip] contained a news feature on agent- based models applied to understanding the opioid crisis in US. (With a rather sordid picture of a drug injection in Philadelphia, hence my own picture.)

To create an agent-based model, researchers first ‘build’ a virtual town or region, sometimes based on a real place, including buildings such as schools and food shops. They then populate it with agents, using census data to give each one its own characteristics, such as age, race and income, and to distribute the agents throughout the virtual town. The agents are autonomous but operate within pre-programmed routines — going to work five times a week, for instance. Some behaviours may be more random, such as a 5% chance per day of skipping work, or a 50% chance of meeting a certain person in the agent’s network. Once the system is as realistic as possible, the researchers introduce a variable such as a flu virus, with a rate and pattern of spread based on its real-life characteristics. They then run the simulation to test how the agents’ behaviour shifts when a school is closed or a vaccination campaign is started, repeating it thousands of times to determine the likelihood of different outcomes.

While I am obviously supportive of simulation based solutions, I cannot but express some reservation at the outcome, given that it is the product of the assumptions in the model. In Bayesian terms, this is purely prior predictive rather than posterior predictive. There is no hard data to create “realism”, apart from the census data. (The article also mixes the outcome of the simulation with real data. Or epidemiological data, not yet available according to the authors.)

In response to the opioid epidemic, Bobashev’s group has constructed Pain Town — a generic city complete with 10,000 people suffering from chronic pain, 70 drug dealers, 30 doctors, 10 emergency rooms and 10 pharmacies. The researchers run the model over five simulated years, recording how the situation changes each virtual day.

This is not to criticise the use of such tools to experiment with social, medical or political interventions, which practically and ethically cannot be tested in real life and working with such targeted versions of the Sims game can paradoxically be more convincing when dealing with policy makers. If they do not object at the artificiality of the outcome, as they often do for climate change models. Just from reading this general public article, I thus wonder at whether model selection and validation tools are implemented in conjunction with agent-based models…

peer reviews on-line or peer community?

Nature (or more precisely some researchers through Nature, associated with the UK Wellcome Trust, the US Howard Hughes Medical Institute (hhmo), and ASAPbio) has (have) launched a call for publishing reviews next to accept papers, one way or another, which is something I (and many others) have supported for quite a while. Including for rejected papers, not only because making these reviews public diminishes on principle the time involved in re-reviewing re-submitted papers but also because this should induce authors to revise papers with obvious flaws and missing references (?). Or abstain from re-submitting. Or publish a rejoinder addressing the criticisms. Anything that increases the communication between all parties, as well as the perspectives on a given paper. (This year, NIPS allows for the posting of reviews of rejected submissions, which I find a positive trend!)

In connection with this entry, I am still most sorry that I could not pursue the [superior in my opinion] project of Peer Community in computational statistics, for the time requested by Biometrika editing is just too important [given my current stamina!] for me to handle another journal (or the better alternative to a journal!). I hope someone else can take over the project and create the editorial team needed to run it.

And yet again in connection with this post (!), Andrew posted an announcement about the launch of res3archers.one, an on-line publication forum launched by Harry Crane and Ryan Martin, where the authors handle the peer review process from A to Z, including choosing the reviewers, whose reviews may be public or not, taken into account or not. Once published, the papers are open to comments from users, which constitutes a form of post-publication peer-review. Albeit a weak one in my opinion as the weakness of all such open depositories is the potential lack of interest of and reaction from the community. Incidentally, there is a $10 fee per submission for maintenance. Contrary to Peer Community in… the copyright is partly transferred to res3archers.one, which apparently prevents further publication in another journal.

unrejected null [xkcd]

graph of the day & AI4good versus AI4bad

Apart from the above graph from Nature, rendering in a most appalling and meaningless way the uncertainty about the number of active genes in the human genome, I read a couple of articles in this issue of Nature relating to the biases and dangers of societal algorithms. One of which sounded very close to the editorial in the New York Times on which Kristian Lum commented on this blog. With the attached snippet on what is fair and unfair (or not).

The second article was more surprising as it defended the use of algorithms for more democracy. Nothing less. Written by Wendy Tam Cho, professor of political sciences, law, statistics, and mathematics at UIUC, it argued that the software that she develops to construct electoral maps produces fair maps. Which sounds over-rosy imho, as aiming to account for all social, ethnic, income, &tc., groups, i.e., most of the axes that define a human, is meaningless, if only because the structure of these groups is not frozen in time. To state that “computers are impervious to the lure of power” is borderline ridiculous, as computers and algorithms are [so far] driven by humans. This is not to say that gerrymandering should not be fought by technological means, especially and obviously by open source algorithms, as existing proposals (discussed here) demonstrate, but to entertain the notion of a perfectly representative redistricting is not only illusory, but also far from democratic as it shies away from the one person one vote  at the basis of democracy. And the paper leaves us on the dark as to whom will decide on which group or which characteristic need be represented in the votes. Of course, this is the impression obtained by reading a one page editorial in Nature [in an overcrowded and sweltering commuter train] rather than the relevant literature. Nonetheless, I remain puzzled at why this editorial was ever published. (Speaking of democracy, the issue contains also warning reports about Hungary’s ultra-right government taking over the Hungarian Academy of Sciences.)

scientific societies start to address sexual harassement

As ISBA releases a letter of her president to the members about the decision by the ISBA Board [taken in Edinburgh] to exclude three of its members following multiple complaints of harassment, the ASA publishes an update on the activities of the task force created to address this issue last November. And Nature reports on the report published by the US academies of Sciences, Engineering, and Medicine, which points out the limited impact of the current policies and mechanisms at play in US institutions.

“The analysis concludes that policies to fight the problem are ineffective because they are set up to protect institutions, not victims.” Nature, June 12, 2018

A common feature between the ASA and the Academy approaches is to rely on a survey of their respective members, soon to come for ASA members. Another feature of major relevance is the issue of anonymous reporting and counselling. So that victims and witnesses of harassment can trust the procedure strongly enough to report  a case without being afraid of being known to a large number of people. In my opinion, having identified individuals that represent the diversity of a scientific society such as ISBA, rather than an anonymous email account or a web form, is more likely to induce testimonies or complaints.

Nature snippets

Besides this remarkable picture of a fox and an eagle fighting for a rabbit, posted in Nature of 7 June, I noticed [in Nature 24 May] an editorial by Richard McEalreath, author of the remarkable Statistical Rethinking, about a paper by González-Forero & Gardner developing a model for brain vs body growth, incorporating social and ecological challenges. The goal was to fit the actual growth in body mass and brain mass. As in the one below.Without reading the supplementary material, I cannot tell how much statistics was involved in preventing the “best fit” to turn to overfitting. But Richard McEalreath points out that this modelling goes away and presumably beyond the “purely statistical”, including regression approaches, without elaborating more on the methodological aspects.