5 ways to fix statistics?!

In the last issue of Nature (Nov 30), the comment section contains a series of opinions on the reproducibility crisis, by five [groups of] statisticians. Including Blakeley McShane and Andrew Gelman with whom [and others] I wrote a response to the seventy author manifesto. The collection of comments is introduced with the curious sentence

“The problem is not our maths, but ourselves.”

Which I find problematic as (a) the problem is never with the maths, but possibly with the stats!, and (b) the problem stands in inadequate assumptions on the validity of “the” statistical model and on ignoring the resulting epistemic uncertainty. Jeff Leek‘s suggestion to improve the interface with users seems to come short on that level, while David Colquhoun‘s Bayesian balance between p-values and false-positive only address well-specified models. Michèle Nuitjen strikes closer to my perspective by arguing that rigorous rules are unlikely to help, due to the plethora of possible post-data modellings. And Steven Goodman’s putting the blame on the lack of statistical training of scientists (who “only want enough knowledge to run the statistical software that allows them to get their paper out quickly”) is wishful thinking: every scientific study [i.e., the overwhelming majority] involving data cannot involve a statistical expert and every paper involving data analysis cannot be reviewed by a statistical expert. I thus cannot but repeat the conclusion of Blakeley and Andrew:

“A crucial step is to move beyond the alchemy of binary statements about ‘an effect’ or ‘no effect’ with only a P value dividing them. Instead, researchers must accept uncertainty and embrace variation under different circumstances.”

7 Responses to “5 ways to fix statistics?!”

  1. Many important scientific questions involve binary decisions. On those questions it is imperative we decide whether we should act or abstain from acting.

    Any idea on how we “accept uncertainty and embrace variation” without treating all attempts to answer those questions as an alchemy?

    • Econostatistician

      I recommend making decisions using decision analysis, based on the costs and benefits of different options, along with their posterior probabilities. We discuss this a bit in the longer article (of which Christian Robert is a coauthor) to which we refer in our Nature article. Formal decision analysis using statistical inference is fine; I just don’t think it makes sense to do this using p-values.

  2. Another major problem is: how to educate editors that there will be no closure in a paper. Everything is a tentative conclusion. How to educate an editor that an effect with 95% credible interval [2,42] is no more informative than an effect with interval [-2, 38]. They’d consider the former informative and publishable but the latter not. To get the editor to the point that they see their mistake (and which editor will say, oh snap, I’m a world authority in psychobabble but I didn’t understand this fundamental point in my 30 year career—foolish me) seems like an impossibility to me.

  3. I am still new enough to embracing uncertainty that I remember my pre-Bayesian self. I think my pre-2013 self would be puzzled by vague demands from statisticians to embrace uncertainty. The paradox is that for people to understand what Andrew and you are saying here, they have to know something about statistics, which you correctly point out is wishful thinking. So the protestations of knowledgeable scientists like you will forever be treated by the great unwashed masses as incoherent rantings of wild-eyed men.

    • hmmm, in the very same week I have been qualified of cleverer than f**k and accused of incoherent wild-eyed rantings…

      • I’m not accusing you of this at all! :)

        I am making the observation that that’s what people outside of stats think, that’s why they just ignore your concerns.

      • xi’an: If you want to appear prestigious among those who actually don’t know that they don’t know much about statistics (not saying you do or should) – just put on an eye patch and become a one-eyed man ;-)


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.

%d bloggers like this: