Lack of confidence in ABC model choice

Over the past weeks, we have worked a population genetic illustration to our ABC model choice paper. Jean-Marie Cornuet and Jean-Michel Marin set up an experiment where two scenarios including three populations were compared, two populations having diverged 100 generations ago and the third one resulting of a recent admixture between the first two populations (scenario 1) or simply diverging from population 1 (scenario 2) at the same time of 5 generations in the past. In scenario 1, the admixture rate is 0.7 from population 1. Pseudo observed datasets (100) of the same size as in experiment 1 (15 diploid individuals per population, 5 independent microsatellite loci) have been generated assuming an effective population size of 1000 and mutation rates of 0.0005. There are six parameters (provided with the corresponding priors): admixture rate (U[0.1,0.9]), three effective population sizes (U[200,2000]), the time of admixture/second divergence (U[1,10]) and the time of the first divergence (U[50,500]). Although this is rather costly in computing time, the posterior probability can nonetheless be estimated by importance sampling, based on 1000 parameter values and 1000 trees per parameter value, based on the modules of Stephens and Donnelly (JRSS Series B, 2000).  The ABC  approximation is obtained from DIYABC, using a reference sample of two million parameters and 24 summary statistics. The result of this experiment is shown above, with a clear divergence in the numerical values despite stability in both approximations. Taking the importance sampling approximation as the reference value, the error rates in using the ABC approximation to choose between scenarios 1 and 2 are 14.5\% and 12.5\% (under scenarios 1 and 2), respectively. Although a simpler experiment with a single parameter and the same 24 summary statistics shows a reasonable agreement between both approximations, this result comes an additional support to our earlier warning about a blind use of ABC for model selection. The article written with Jean-Marie Cornuet, Jean-Michel Marin and Natesh Pillai is now posted on arXiv (and submitted to PNAS).

20 Responses to “Lack of confidence in ABC model choice”

  1. […] paper “Lack of confidence in approximate Bayesian computation model choice“, with  Jean-Marie Cornuet, Jean-Michel Marin, and Natesh S. Pillai, has now appeared in the […]

  2. […] are now available in Nature Precedings. A neat concept by the way!) This paper of them builds on our earlier warning about (unfounded) ABC model selection to propose a selection of summary statistics that partly […]

  3. […] Judith Rousseau and myself) are currently making progress towards an indicator of worth for the ABC Bayes factor approximation,  these exploratory tools are quite valuable to defend an ABC approach to model evaluation and to […]

  4. […] conjunction with the normal-Laplace comparison mentioned in the most recent post about our lack of confidence in ABC model choice, we have been working on the derivation of the exact Bayes factor and I derived an easy formula for […]

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

Join 705 other followers