how many academics does it take to change… a p-value threshold?
“…a critical mass of researchers now endorse this change.”
The answer to the lightpulp question seems to be 72: Andrew sent me a short paper recently PsyarXived and to appear in Nature Human Behaviour following on the .005 not .05 tune we criticised in PNAS a while ago. (Actually a very short paper once the names and affiliations of all authors are taken away.) With indeed 72 authors, many of them my Bayesian friends! I figure the mass signature is aimed at convincing users of p-values of a consensus among statisticians. Or a “critical mass” as stated in the note. On the next week, Nature had an entry on this proposal. (With a survey on whether the p-value threshold should change!)
The argument therein [and hence my reservations] is about the same as in Val Johnson’s original PNAS paper, namely that .005 should become the reference cutoff when using p-values for discovering new effects. The tone of the note is mostly Bayesian in that it defends the Bayes factor as a better alternative I would call the b-value. And produces graphs that relate p-values to some minimax Bayes factors. In the simplest possible case of testing for the nullity of a normal mean. Which I do not think is particularly convincing when considering more realistic settings with (many) nuisance parameters and possible latent variables where numerical answers diverge between p-values and [an infinity of] b-values. And of course the unsolved issue of scaling the Bayes factor. (This without embarking anew upon a full-fledged criticism of the Bayes factor.) As usual, I am also skeptical of mentions of power, since I never truly understood the point of power, which depends on the alternative model, increasingly so with the complexity of this alternative. As argued in our letter to PNAS, the central issue that this proposal fails to address is the urgency in abandoning the notion [indoctrinated in generations of students that a single quantity and a single bound are the answers to testing issues. Changing the bound sounds like suggesting to paint afresh a building on the verge of collapsing.
Xi'an's Og 
August 24, 2017 at 8:45 pm
From the maxim “the perfect is the enemy of the good”, this proposal, while maybe not “perfect”, may be “good enough” to help.
August 24, 2017 at 8:50 pm
Our point is that moving the cursor on a flawed tool does not help.
August 25, 2017 at 1:27 pm
And many argue it actually will hurt and those will appear soon if not already.
Keith O’Rourke
August 23, 2017 at 3:33 am
I like the idea of power insofar as it gives a framework to ask questions like “how big would a signal need to be for me to find it in data that looks like x”, which I think is an important thing to understand. I don’t like the idea of power in any other way, though.
August 22, 2017 at 3:35 pm
I like the paint analogy, but I would make it more dramatic by first scraping all the old paint off and then painting with a very thin diluted single coat of paint (that may will hasten the collapse).
Keith O’Rourke
August 22, 2017 at 4:20 pm
Yes, indeed, but I am not renowned for my practical skills…!