posterior predictive distributions of Bayes factors

Once a Bayes factor B(y)  is computed, one needs to assess its strength. As repeated many times here, Jeffreys’ scale has no validation whatsoever, it is simply a division of the (1,∞) range into regions of convenience. Following earlier proposals in the literature (Box, 1980; García-Donato and Chen, 2005; Geweke and Amisano, 2008), an evaluation of this strength within the issue at stake, i.e. the comparison of two models, can be based on the predictive distribution. While most authors (like García-Donato and Chen) consider the prior predictive, I think using the posterior predictive distribution is more relevant since

1. it exploits the information contained in the data y, thus concentrates on a region of relevance in the parameter space(s), which is especially interesting in weakly informative settings (even though we should abstain from testing in those cases, dixit Andrew);
2. it reproduces the behaviour of the Bayes factor B(x) for values x of the observation similar to the original observation y;
3. it does not hide issues of indeterminacy linked with improper priors: the Bayes factor B(x) remains indeterminate, even with a well-defined predictive;
4. it does not separate between errors of type I and errors of type II but instead uses the natural summary provided by the Bayesian analysis, namely the predictive distribution π(x|y);
5. as long as the evaluation is not used to reach a decision, there is no issue of “using the data twice”, we are simply producing an estimator of the posterior loss, for instance the (posterior) probability of selecting the wrong model. The Bayes factor B(x) is thus functionally  independent of y, while x is probabilistically dependent on y.

Note that, even though probabilities of errors of type I and errors of type II can be computed, they fail to account for the posterior probabilities of both models. (This is the delicate issue with the solution of García-Donato and Chen.) Another nice feature is that the predictive distribution of the Bayes factor can be computed even in complex settings where ABC needs to be used.

4 Responses to “posterior predictive distributions of Bayes factors”

1. I’m not 100% sure I understand what’s being discussed here. (I do have some idea, but I could be mistaken.) I suggest that the post could be improved by inserting a simple example, perhaps between, “I think using the posterior predictive distribution is more relevant…” and the subsequent list of characteristics.

• No issue with this: I am not sure either..! This was a piece of blog I wrote a write ago, maybe during O’Bayes in Duke last winter, and somehow could not connect back to when I saved it from oblivion.

2. Dan Simpson Says:

Predictive distributions also have the particularly pleasing property that they’re easy to compute, unlike Bayes factors…

• Also – I’m not sure why Andrew doesn’t like testing (it shouldn’t be the default stats action, but it’s not crazy), but I agree that weakly informative priors are terrible for testing. You get that sharp null problem with the large sample asymptotics of the Bayes factors that is impossible to avoid without being strongly informative (that the alternative prior as almost no mass near the null)

This site uses Akismet to reduce spam. Learn how your comment data is processed.