Measuring statistical evidence using relative belief [book review]

“It is necessary to be vigilant to ensure that attempts to be mathematically general do not lead us to introduce absurdities into discussions of inference.” (p.8)

This new book by Michael Evans (Toronto) summarises his views on statistical evidence (expanded in a large number of papers), which are a quite unique mix of Bayesian  principles and less-Bayesian methodologies. I am quite glad I could receive a version of the book before it was published by CRC Press, thanks to Rob Carver (and Keith O’Rourke for warning me about it). [Warning: this is a rather long review and post, so readers may chose to opt out now!]

“The Bayes factor does not behave appropriately as a measure of belief, but it does behave appropriately as a measure of evidence.” (p.87)

First, Evans’ perspective on continuous models and measurability issues is that those are only approximations  of true models which can only be on finite sets. (I know this is also Keith’s perspective, so he should appreciate!) Any measure theoretic  inconsistency like the Dickey-Savage paradox (central to Evans’ approach) can then be attributed to continuous features that vanish in the finite case. Easy does it! There is even an Appendix on “The definition of a density” following Rudin’s (1974) definition of densities as limits. And hence much more topological than the standard Lebesgue’s definition. Which makes those densities continuous for instance and avoid the selection of a “nice” version of the density.

“It seems clear that there is only one aspect of a statistical investigation that can ever truly be claimed to be objective, namely, the observed data (…) it can be claimed that the data are objective [when] the control over the data selection process is entirely through the random system.” (p.12)

The first chapter has a quite fascinating discussion about objectivity and subjectivity, under the heading of empirical criticism, and I tend to side with Evans’ arguments on the inherent subjectivity of statistical analyses and the need of empirically checking every aspect of those analyses. For instance, the way frequentism is handled. There is just the point made in the quote above that seems unclear, in that it implies an intrinsic belief in the model, which should be held with with the utmost suspicion! That is, the data is almost certainly unrelated with the postulated model since all models are wrong et cetera…

“First, randomness has nothing to do with probability. Second, there is no statistical test for randomness.” (p.49)

It is sort of getting rarer and rarer to see statistics books exposing the various concepts or meaning of probability, rather than merely presenting (not so) standard measure theory. But this is what Chapter 2 in Evans’ book does, discussing all sorts of axiomatics for defining probability. And including all sorts of paradoxes like the Monty Hall problem or the Borel paradox as examples. (Speaking of paradoxes, the Jeffreys-Lindley paradox is discussed in the next chapter and blamed on the Bayes factor and its lack of “calibration as a measure of evidence” (p.84). The book claims a resolution of the paradox on p.132 by showing confluence between the p-value and the relative belief ratio. This simply shows confluence with the p-value in my opinion.)

“It is not clear how one checks a model using the pure likelihood (…) Overall, pure likelihood theory does not lead to a fully satisfactory theory of inference.” (p.58)

“…a common misconception [seems to be] that Bayesian inferences are only based on the posterior but, with the exception of probability statements as determined by the principle of conditional probability, there is nothing to support this view.” (p.73)

After presenting the classical approaches from “pure” likelihood to p-values and Neyman-Pearson tests to Bayesian inferences (mind the s!), including his explanation as to why the likelihood principle (L) does not hold as a consequence of the sufficiency (S) and conditionality (C) principles (basically because the conditionality principle is not an equivalence relation), i.e.,

S∪C ⊂ L ⊂ S∪C,

including this rather puzzling quote about Bayesian inference(s)—with which I cannot agree—Michael Evans defines his own version of evidence, namely the relative belief ratio that could also be called the Savage-Dickey ratio, being the ratio of the posterior over the prior density at a  specific parameter value, and he expands on the various properties of this ratio. The estimator he advocates in association with this evidence is the maximum relative belief estimator, maximising the relative belief ratio, another type of MAP then. With the same drawbacks as the MAP depends on the dominating measure and is not associated with a loss function in the continuous case. Even in the finite case, the associated loss is an indicator function divided by the prior, which sounds highly counter-intuitive.

“A [frequentist] theory of inference suffers from two main defects. First, there does not seem to be a good answer to the question of why it is necessary to consider the frequency properties of statistical procedures. Without this justification, basing statistical procedures on the principle of frequentism seems like a weak foundation (…) Second, it seems almost misleading to refer to the frequentist theory of inference because it does not exist in the sense that such a theory can be applied to statistical problems [without] a guaranteed sensible answer or, for that matter, even an answer.” (p.71)

A major surprise for me when reading the book is that Evans ends up with a solution [for assessing the strength of the evidence] that is [acknowledged to be] very close (or even equivalent) to Murray Aitkin’s integrated likelihood approach! An approach much discussed on the ‘Og. And in a Statistics and Risk Analysis paper with Andrew and Judith. Indeed, the strength of a relative belief ratio expressed as a ratio of marginals is a posterior p-value associated with this ratio. Which again has the drawbacks of not being defined a priori, of using the data twice, and of being defined on one model versus the other. Solving the Lindley-Jeffreys paradox—in the understanding of a clash between the Bayesian and frequentist answers—this way is then no major surprise, for this was one major argument in Murray Aitkin’s support of his approach (as seen for instance in the 1991 Read Paper).

“Currents attempts at developing a theory based upon improper priors have close connections with frequentist ideas but, as already discussed, there are issues with the principle of frequentism itself that remain unsolved.” (p.170)

Pursuing this unique mix of Bayesian and extra-Bayesian principles, Evans also acknowledges a connection with Mayo and Spanos error–statistics philosophy of science and severity tests, although since he relies on a marginal likelihood for this purposes, this should clash with the authors’ arguments.

“There are a variety of problems associated with improper priors. Perhaps the most obvious one is that there is no guarantee that [the marginal is finite] (…) Also, when PI is improper, it cannot be the case that m represents a probability distribution and so all applications of the prior predictive that rely on it are lost.” (p.173)

Since the ratio is defined in terms of densities, the whole approach does not allow for improper priors. A fact acknowledged in the chapter on model and prior checking. Some of the criticisms are standard, including the marginalisation paradox and the impossibility to define marginal priors. Some less, as the above that the constraint of finiteness makes the prior data dependent (!) [no it should be imposed uniformly, excluding for instance Haldane’s prior] or the conclusion of that section that “it is not clear how to measure evidence in such a context.” (p.176) Looking at the bigger picture, this pessimistic conclusion is in line with (a) the global perspective adopted in the book that everything is finite, which leaves little room or use for infinite mass priors, and (b) the general difficulty in handling improper priors in testing. Although this was precisely the reason for Murray Aitkin to introduce his integrated likelihood paradigm, close to the current proposal (p.119).

“Our preference is to approach all our inference problems using the relative belief ratio. At least a part of the motivation for this lies with a desire to avoid the prescription that, with continuous models, the Bayes factor must be based on the mixture prior as in Jeffreys’ definition.” (p.146)

The above remark is a very interesting point and one bound to appeal to critics of the mixture representation, like Andrew Gelman. However, given that the Dickey-Savage ratio is a representation of the Bayes factor in the point null case, I wonder how much of a difference this constitutes. Not mentioning the issue of defining a prior for the contingency of accepting the null. So in the end I do not see much of a difference. (In connection with the Bayes factor, and the use of the mixture prior, there is an inconsequential typo in Example 4.5.6, p.128, with a missing Dirac mass.)

“Some may argue that the sanctity of the prior is paramount, as it represents beliefs and any assessment of the suitability of the prior, or worse, attempts to modify the prior based on the results of this assessment, is incoherent.” (p.188)

I was eagerly and obviously waiting for the model choice chapter, but it somewhat failed to materialise! Chapter 5 is about model and prior checking, a sort of meta-goodness of fit check, but as far as I can see, the relative belief ratio methodology does not extend to the comparison of non-embedded models. The part on model checking is very limited and involves computing a p-value for the likelihood of a “discrepancy” statistic, conditional on the minimal sufficient statistic. Why minimal sufficient? Because otherwise the p-value would depend on the unknown parameter, or the discrepancy would have to be chosen ancillary. Which requires a certain degree of understanding about the model and excludes reasonably complex models, unless one uses only rank statistics or the like, which cannot be good for efficiency. I think the criticism therein about posterior checks rather unfortunate, because using the predictive has the strong appeal to (a) bypassing the dependence on the unknown parameter and (b) avoiding adhoqueries like the choice of discrepancies. I also have a general difficulty with using ancillaries and sufficient statistics because, even when they are non-trivial and well-identified, they remain a characteristic of the model: using those to check the model thus sounds fraught with danger.

“The developments concerning the assessment of bias and the checking of the ingredients is certainly very close to a fequentist approach. There is nothing contradictory about this, as there is no role for the posterior in these issues (…) While various approaches to Bayesian theory share features with relative belief, there are also key differences, such as not being decision based and making a sharp distinction between belief and evidence.” (p.212)

The second part on prior checking is quite original and challenging. While the above quote is somewhat imbalanced in its use of religious terms like sanctity and beliefs (!), I tend to concur with the idea that priors can be checked and compared. This is for instance the role of Bayesian robustness. Or of Bayes factors.  Here, the way to identify prior-data conflict in practice and the consequence of a rejection of the prior remain vague, as it seems hard to avoid the data influencing the modification of the prior (besides the obvious, namely that identifying conflict leads to a modification). The only reasonable way is to set both a collection or family of priors and a modus vivendi for changing the prior, before the checking is done. But even then, assessing the coherence of the resulting construction is a huge question mark… There is however little incentive in using a marginal p-value (m-value?) especially when several priors are under comparison, since it requires to define a reference or preference prior. Or would it relate to a baseline model in the spirit of the revolutionary proposal of Simpson et al?

“Evidence is what causes beliefs to change and so evidence is measured by changes in belief.” (p.244)

Overall, and even though I would not advocate this approach to evidence, I find Michael Evans’ Measuring statistical evidence using relative belief a fascinating book, in that it defines a coherent and original approach to the vexing question of assessing and measuring evidence in a statistical problem. And spells out most vividly the issue of prior checking. As clear from the above, I somehow find the approach lacking in several foundational and methodological aspects, maybe the most strident one being that the approach is burdened with a number of arbitrary choices, lacking the unitarian feeling associated with a regular Bayesian decisional approach. I also wonder at the scaling features of the method, namely how it can cope with high dimensional or otherwise complex models, without going all the way to ask for an ABC version! In conclusion, the book is a great opportunity to discuss this approach and to oppose it to potential alternatives, hopefully generating incoming papers and talks.

[Disclaimer about potential self-plagiarism: this post or an edited version will eventually appear in my Books Review section in CHANCE.]