## Confidence (bis repetita placent?)

**D**ue to the simultaneous presence of Jean-Michel Marin and Natesh Pillai in Paris, we were able to write within a few days a new revision of our lack of confidence in ABC model choice paper, currently submitted to PNAS. Following my earlier comments on the reviews, we now consider that indeed different statistics are needed for model choice in that estimation statistics are generally unable to distinguish between the models. The collection of “other statistics” being rather wide means we are still at a loss about which statistics to pick in practive, but the principle is worth stressing.

**H**ere is an illustration of the difficulty to separate (by ABC) between models in a setting suggested by one referee, i.e. when testing a location normal distribution versus a location Laplace distribution (both with variance 1). The (frequentist) distribution of the ABC estimate of the posterior probability of the normal model is centred at the same position when simulating data from the normal and from the Laplace distributions and when using a Euclidean distance involving the mean, the median and the variance as summary statistics.

**T**he new version has now been (re)resubmitted and re-arXived. The changes are toward a more cautionary tone, stating that all we know about the ABC approximation to the Bayes factor is that… we do not know whether or not it constitutes a converging approximation to the Bayes factor (mostly not) or to the Bayes decision (presumably yes with “enough” statistics). We are currently working in exploring that direction…

July 17, 2011 at 9:09 am

[…] I just got the following email from PNAS about our Lack of confidence in ABC model choice. […]

July 8, 2011 at 12:13 am

[…] is rather unlikely in most realistic settings (this is noted in the Discussion as well as in our PNAS paper). So the term sufficient should not be used as in Figure 3 for instance. Overall, the method of […]

June 28, 2011 at 2:04 am

[…] my earlier posts on the revision of Lack of confidence, here is an interesting outcome from the derivation of the exact marginal likelihood in the Laplace […]

June 26, 2011 at 12:16 am

[…] conjunction with the normal-Laplace comparison mentioned in the most recent post about our lack of confidence in ABC model choice, we have been working on the derivation of the […]

June 24, 2011 at 5:13 am

Hello,

How would you compare the failure of ABC to choose between models in this setting to its success in selecting the correct demographic model in the Wright-Fisher setting? It seems that the choice between the Laplace and the normal is not equivalent to the choice between two demographic models. In the latter case, one assumes that the population dynamics of both demographic models are described by the WF process and the mutational dynamics correspond to the infinite-alleles or model. In the former case one is choosing between two different distribution. Is this qualitative difference between the two model choice settings relevant to the performance of summary statistics for differentiating between the models? Is this difference even real? Thank you.

-Sameer

June 24, 2011 at 7:31 am

The difference may feel substantial in that one setting proposes two explanations for demographic evolution while the other setting discusses noise. There is therefore a difference in the complexity of the models and hence of the question. In my opinion, both settings are nonetheless

in fineof the same nature in that several models are proposed for a given dataset and a Bayes factor is then computed to decide which model is more likely for this dataset.June 23, 2011 at 8:17 pm

This plot is more illustrative and it is nice to see the difference in models. By the way I was reading this new review on ABC

http://onlinelibrary.wiley.com/doi/10.1111/j.1461-0248.2011.01640.x/full

I consider this is really poor in the sense that they do not illustrate the difficulties while using the algorithm. There are some other technical issues like “for continuous variables, the probability of observing exactly the same outcome is infinitesimally small”. I would like to know your opinion about it.

June 23, 2011 at 10:12 pm

Rosana: thank you for signaling this survey on ABC (surveys seem to sprout like weeds these days!!!), I will read it and report on it shortly. My interpretation of the graph above is not as optimistic as yours. There is a slight difference in the distribution of the posterior probabilities, but, overall, they spread the same region, hence give little confidence in trusting this particular version of ABC model choice…