ABC for model criticism

An ABC paper by Oliver Ratmann, Christophe Andrieu, Carsten Wiuf and Sylvia Richardson has just appeared on PNAS Early Edition, (It will also be presented at the “ABC in Paris” meeting of June 26.) It is about the use of the ABC approximation error in an altogether different way, namely as a tool assessing the goodness of fit of a given model. It reminds me of Richard Wilkinson’s earlier ABC paper, discussed on that post, but the scope is somehow different. The fundamental idea is to use as an additional parameter of the model, simulating from a joint posterior distribution

where is the data and plays the role of the likelihood. (The are obviously the priors on and .) In fact, is the prior predictive density of given and when is distributed from . The authors then derive an ABC algorithm they call ABCμ to simulate an MCMC chain targeting this joint distribution, replacing with a non-parametric kernel approximation. For each model under comparison, the marginal posterior distribution on the error is then used to assess the fit of the model, the logic of it being that this posterior should include 0 in a reasonable credible interval. (Contrary to other ABC papers, can be negative and multidimensional in this paper.)

As written above, this is a very interesting paper, full of innovations, that should span new directions in the way one perceives ABC. It is also quite challenging, partly due to the frustrating constraints PNAS imposes on the organisation (and submission) of papers. The paper thus contains a rather sketchy main part, a Materials and Methods addendum, and a Supplementary Material file! Flipping back and forth between those files certainly does not improve reading. I have never understood why PNAS was is so rigid about a format that does not suit non-experimental sciences…

Given the wealth of innovations contained in the paper, I will certainly post again on it, but let me add here that, while the authors stress they use the data once (a point always uncertain to me), they also define the above target by using simultaneously a prior distribution on and a conditional distribution on the same —that they interpret as the likelihood in . The product being most often defined as a density in it can be simulated from, but I have trouble seeing this as a regular Bayesian problem, especially because it seems the prior on significantly contributes to the final assessment (but is not particularly discussed in the paper, except in the S1.10 section).

Another Bayesian conundrum is the fact that both and are taken to be the sames across models. In a sense, I presume can be completely different, but using the same prior on over all models under comparison is more of an issue…