## ABC for model criticism

**A**n ABC paper by Oliver Ratmann, Christophe Andrieu, Carsten Wiuf and Sylvia Richardson has just appeared on **PNAS**** Early Edition**, (It will also be presented at the “ABC in Paris” meeting of June 26.) It is about the use of the ABC approximation error in an altogether different way, namely as a tool assessing the goodness of fit of a given model. It reminds me of Richard Wilkinson’s earlier ABC paper, discussed on that post, but the scope is somehow different. The fundamental idea is to use as an additional parameter of the model, simulating from a joint posterior distribution

where is the data and plays the role of the likelihood. (The are obviously the priors on and .) In fact, is the prior predictive density of given and when is distributed from . The authors then derive an ABC algorithm they call **ABCμ** to simulate an MCMC chain targeting this joint distribution, replacing with a non-parametric kernel approximation. For each model under comparison, the marginal posterior distribution on the error is then used to assess the fit of the model, the logic of it being that this posterior should include 0 in a reasonable credible interval. (Contrary to other ABC papers, can be negative and multidimensional in this paper.)

**A**s written above, this is a very interesting paper, full of innovations, that should span new directions in the way one perceives ABC. It is also quite challenging, partly due to the frustrating constraints * PNAS* imposes on the organisation (and submission) of papers. The paper thus contains a rather sketchy main part, a

*Materials and Methods*addendum, and a

*Supplementary Material*file! Flipping back and forth between those files certainly does not improve reading. I have never understood why

*was is so rigid about a format that does not suit non-experimental sciences…*

**PNAS****G**iven the wealth of innovations contained in the paper, I will certainly post again on it, but let me add here that, while the authors stress they use the data *once* (a point always uncertain to me), they also define the above target by using simultaneously a prior distribution on and a conditional distribution on the same —that they interpret as the likelihood in . The product being most often defined as a density in it can be simulated from, but I have trouble seeing this as a regular Bayesian problem, especially because it seems the prior on significantly contributes to the final assessment (but is not particularly discussed in the paper, except in the * S1.10* section).

**A**nother Bayesian conundrum is the fact that both and are taken to be the *sames* across models. In a sense, I presume can be completely different, but using the same prior on over all models under comparison is more of an issue…

January 14, 2015 at 8:55 pm

Any thoughts on the use of “discrepancy variables” for “Model checking using posterior predictive simulation”, from Gelman and Meng in Gilks, Richardson, Spiegelhalter (eds), MCMC IN PRACTICE? It looks a lot like a setup for a generic set of summary statistics, , for ABC, but, then, perhaps you’ve written on that some place?

March 14, 2012 at 12:15 am

[…] assessments also are available is quite correct, as demonstrated in the multicriterion approach of Olli Ratmann and co-authors. This is simply another approach, not followed by most geneticists so […]

January 31, 2011 at 12:17 am

[…] six steps concluding with preferring M_1 to M_0 avoids the problem. This is why ABC solutions like Ollie Ratmann‘s or others based on predictive performances would be of huge interest to bypass the delicate […]

January 27, 2011 at 12:16 am

[…] the approximation of Bayes factor. The only solution seems to be using discrepancy measures as in Ratmann et al. (2009), ie (empirical) model criticism rather than (decision-theoretic) model choice. Bayes […]

April 14, 2010 at 12:03 am

[…] in 2/(2+d). There should thus be a way to link this decision-theoretic approach with the one of Ratmann et al. since the latter take h to be part of the parameter […]

December 16, 2009 at 12:06 am

[…] Indian restaurant closed!, I enjoy very much reading this very rich and unusual thesis. As posted earlier, I have disagreements with some of the choices made in this thesis, in particular the […]

November 16, 2009 at 8:52 am

[…] argument in the criticisms of predictive Bayes inference. I have difficulties with the concept in general and, in the present case, there is no difficulty with using to predict the distribution of […]

October 1, 2009 at 2:13 am

[…] on ABC for model criticism As noted in an earlier post, the ABCμ paper by Oliver Ratmann, Christophe Andrieu, Carsten Wiuf and Sylvia Richardson that […]