I first spotted Peter Coles’ great post title “Frequentism: the art of probably answering the wrong question” (a very sensible piece by the way!, and mentioning a physicist’s view on the Jeffreys-Lindley paradox I had intended to comment) and from there the following site jumping occured:

“I confess that in my early in my career as a physicist I was rather cynical about sophisticated statistical tools, being of the opinion that “if any of this makes a difference, just get more data”. That is, if you do enough experiments, the confidence level will be so high that the exact statistical treatment you use to evaluate it is irrelevant.” John Butterworth, Sept. 15, 2014

**A**fter Val Johnson‘s suggestion to move the significant level from .05 down to .005, hence roughly from 2σ up to 3σ, John Butterworth, a physicist whose book Smashing Physics just came out, discusses in The Guardian the practice of using 5σ in Physics. It is actually induced by Louis Lyons’ arXival of a recent talk with the following points (discussed below):

- Should we insist on the 5 sigma criterion for discovery claims?
- The probability of A, given B, is not the same as the probability of B, given A.
- The meaning of p-values.
- What is Wilks Theorem and when does it not apply?
- How should we deal with the `Look Elsewhere Effect’?
- Dealing with systematics such as background parametrisation.
- Coverage: What is it and does my method have the correct coverage?
- The use of p0 versus p1 plots.