is the p-value a good measure of evidence?

“Statistics abounds criteria for assessing quality of estimators, tests, forecasting rules, classification algorithms, but besides the likelihood principle discussions, it seems to be almost silent on what criteria should a good measure of evidence satisfy.” M. Grendár

A short note (4 pages) appeared on arXiv a few days ago, entitled “is the p-value a good measure of evidence? an asymptotic consistency criterion” by M. Grendár. It is rather puzzling in that it defines the consistency of an evidence measure ε(H₁,H₂,Xⁿ) (for the hypothesis H₁ relative to the alternative H₂) by

$\lim_{n\rightarrow\infty} P(H_1|\epsilon(\neg H_1,H_2,X^n)\in S) =0$

where S is “the category of the most extreme values of the evidence measure (…) that corresponds to the strongest evidence” (p.2) and which is interpreted as “the probability [of the first hypothesis H₁], given that the measure of evidence strongly testifies against H₁, relative to H₂ should go to zero” (p.2). So this definition requires a probability measure on the parameter spaces or at least on the set of model indices, but it is not explicitly stated in the paper. The proofs that the p-value is inconsistent and that the likelihood ratio is consistent do involve model/hypothesis prior probabilities and weights, p(.) and w. However, the last section on the consistency of the Bayes factor states “it is open to debate whether a measure of evidence can depend on a prior information” (p.3) and it uses another notation, q(.), for the prior distribution… Furthermore, it reproduces the argument found in Templeton that larger evidence should be attributed to larger hypotheses. And it misses our 1992 analysis of p-values from a decision-theoretic perspective, where we show they are inadmissible for two-sided tests, answering the question asked in the quote above.

This entry was posted on November 30, 2011 at 12:11 am and is filed under Statistics, University life with tags Bayes factors, Bayesian decision theory, Bayesian inference, Bayesian model choice, consistency, evidence, hypothesis testing, p-values. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.

4 Responses to “is the p-value a good measure of evidence?”

Harmonic means, again again « Xi'an's Og Says:
January 10, 2012 at 12:13 am

[...] by the review). Indeed, it considers a series of competing computational methods for approximating evidence, aka marginal [...]

Reply
jimmy Says:
November 30, 2011 at 3:33 am

hi christian, sorry for the stupid question. (if it is too stupid, don’t feel as if you have to answer.) but what does admissibility mean?

Reply
- xi'an Says:
  November 30, 2011 at 8:17 am
  
  Admissibility and its opposite inadmissibility are two notions used in game theory and in statistical decision theory to evaluate estimators. An inadmissible estimator is dominated everywhere by another estimator, hence should not be used (if the domination criterion is relevant for the problem at hand). Wald in the 1950′s showed that the admissible estimators are more or less the Bayes estimators. This is covered in Chapter 3 of The Bayesian Choice.
  
  Reply
Paulo Marques Says:
November 30, 2011 at 2:55 am

Also, this other kind of incoherence makes p-values unbearable:

http://www.jstor.org/pss/2684655

Reply

Xi'an's Og