checking ABC convergence via coverage

Dennis Prangle, Michael Blum, G. Popovic and Scott Sisson just arXived a paper on diagnostics for ABC validation via coverage diagnostics. Getting valid approximation diagnostics for ABC is clearly and badly needed and this was the last slide of my talk yesterday at the Winter Workshop in Gainesville. When simulation time is not an issue (!), our DIYABC software does implement a limited coverage assessment by computing the type I error, i.e. by simulating data under the null model and evaluating the number of time it is rejected at the 5% level (see sections 2.11.3 and 3.8 in the documentation). The current paper builds on a similar perspective.

The idea in the paper is that a (Bayesian) credible interval at a given credible level α should have a similar confidence level (at least asymptotically and even more for matching priors) and that simulating pseudo-data with a known parameter value allows for a Monte-Carlo evaluation of the credible interval “true” coverage, hence for a calibration of the tolerance. The delicate issue is about the generation of those “known” parameters. For instance, if the pair (θ_, y) is generated from the joint distribution prior x likelihood, and if the credible region is also based on the true posterior, the average coverage is the nominal one. On the other hand, if the credible interval is based on a poor (ABC) approximation to the posterior, the average coverage should differ from the nominal one. Given that ABC is always wrong, however, this may fail to be a powerful diagnostic. In particular, when using insufficient (summary) statistics, the discrepancy should make testing for uniformity harder, shouldn’t it? 

PLoS topic page on ABC

A few more comments on the specific entry on ABC written by Mikael Sunnåker et al…. The entry starts with the representation of the posterior probability of an hypothesis, rather than with the posterior density of a model parameter, which seems to lead the novice reader astray. After all, (a) ABC was not introduced for conducting model choice and (b) interchanging hypothesis and model means that the probability of an hypothesis H as used in the entry is actually the evidence in favour of the corresponding model. (There are a few typos and grammar mistakes, but I assume either PLoS or later contributors will correct those.) When the authors state that the “outcome of the ABC rejection algorithm is a set of parameter estimates distributed according to the desired posterior distribution”, I think they are misleading the readers as they forget the “approximative” aspect of this distribution. Further below, I would have used the title “Insufficient summary statistics” rather than “Sufficient summary statistics”, as it spells out more clearly the fundamental issue with the potential difficulty in using ABC. (And I am not sure the subsequent paragraph on “Choice and sufficiency of summary statistics” should bother with the sufficiency aspects… It seems to me much more relevant to assess the impact on predictive performances.)

Although this is most minor, I would not have made mention of the (rather artificial) “table for interpretation of the strength in values of the Bayes factor (…) originally published by Harold Jeffreys^[6] “. I obviously appreciate very much that the authors advertise our warning about the potential lack of validity of an ABC based Bayes factor! I also like the notion of “quality control”, even though it should only appear once. And the pseudo-example is quite fine as an introduction, while it could be supplemented with the outcome resulting from a large n, to be compared with the true posterior distribution. The section “Pitfalls and remedies” is remarkable in that it details the necessary steps for validating a ABC implementation: the only entry I would remove is the one about “Prior distribution and parameter ranges”, in that this is not a problem inherent to ABC… (Granted, the authors present this as a “general risks in statistical inference exacerbated in ABC”, which makes more sense!) It may be that the section on the non-zero tolerance should emphasize more clearly the fact that ε should not be zero. As discussed in the recent Read Paper by Fearnhead and Prangle when envisioning ABC as a non-parametric method of inference.

At last, it is always possible to criticise the coverage of the historical part, since ABC is such a recent field that it is constantly evolving. But the authors correctly point out to (Don) Rubin on the one hand and to Diggle and Graton on the other. Now, I would suggest adding in this section links to the relevant softwares like our own DIY-ABC…

(Those comments have also been posted on the PLoS Computational Biology wiki.)

internal workshop in Montpellier

Today was a meeting day for our research (ANR) network EMILE and I flew to Montpellier in the early morning, barely catching my 7am flight by a mere 8 minutes, thanks to a huge unannounced gap (more than 30mn!) in the distribution of the metro trains… Anyway, it was a very nice day with interesting talks on on-going researchs by several members of the network, including a new type of (non-ABC) approximation for phylogenetic trees, INLA on genotype distribution, Bayesian tree estimation with SNP data, and the new version of the DIYABC software. (Jean-Michel Marin and I also presented our recent work on ABC model choice and advertised the incoming Read Paper on ABC methods to the group, as they could contribute to the discussion.) One of the talks involved the pseudo-Bayes factors (CPO) of Geisser and Eddy discussed recently in connection with the book reviews of both Bayesian ideas and data analysis and Bayesian modeling using WinBUGS. Unfortunately, again estimated by an harmonic mean…

ABC in nine steps

Bertorelle, Benazzo and Mona have published a nice survey about ABC in Molecular Ecology. It is available on-line and can be downloaded. The nine steps are summarised in the above graph. Step 6 corresponds to ABC model selection, but Step 7 follows the suggestion found in DIYABC to generate pseudo-observed data (pods) to evaluate the “type I – type II” errors of the procedure. In Step 9, a second quality control is proposed via ABC posterior predictive tests. The survey also makes an interesting link with an early version of ABC published by Weiss and von Haessler (1998, Genetics).

My only criticisms are, beyond the lack of confidence in ABC model selection, about an unclear discussion of the “controversial results” about ABC-MCMC and ABC-PMC.  The fact that reference tables as those produced by DIYABC can be recycled with “limited additional effort” is however a good point in favour of Monte Carlo evaluations of estimation and model selection procedures.

ABC in London

After the very exciting and I think quite successful ABC in Paris meeting two years ago, Michael Stumpf from Imperial College London suggested a second edition in London along the same lines. Michael kindly associated me with the planning of this meeting. It is (logically) called ABC in London (or ABCiL) and will take place in Imperial on May 5, 2011. The website is now available for registration. Since this is a one-day intense workshop, registration is compulsory and comes on a first-come first-serve basis. You may take a look at the program to realise how much of an exciting and full day this is going to be! Poster sessions will also be an opportunity to discover new packages and softwares, like the incoming second version of DIYABC. (It would be interesting to see if this trend is going to be pursued with another ABC meeting in a European capital, like Berlin, Bern or even Reykjavik!)