Confidence (bis repetita placent?)

Due to the simultaneous presence of Jean-Michel Marin and Natesh Pillai in Paris, we were able to write within a few days a new revision of our lack of confidence in ABC model choice paper, currently submitted to PNAS. Following my earlier comments on the reviews, we now consider that indeed different statistics are needed for model choice in that estimation statistics are generally unable to distinguish between the models. The collection of “other statistics” being rather wide means we are still at a loss about which statistics to pick in practive, but the principle is worth stressing.

Here is an illustration of the difficulty to separate (by ABC) between models in a setting suggested by one referee, i.e. when testing a location normal  distribution versus a location Laplace distribution (both with variance 1).  The (frequentist) distribution of the ABC estimate of the posterior probability of the normal model is centred at the same position when simulating data from the normal and from the Laplace distributions and when using a Euclidean distance involving the mean, the median and the variance as summary statistics.

The new version has now been (re)resubmitted and re-arXived. The changes are toward a more cautionary tone, stating that all we know about the ABC approximation to the Bayes factor is that… we do not know whether or not it constitutes a converging approximation to the Bayes factor (mostly not) or to the Bayes decision (presumably yes with “enough” statistics). We are currently working in exploring that direction…

8 Responses to “Confidence (bis repetita placent?)”

  1. [...] I just got the following email from PNAS about our Lack of confidence in ABC model choice. [...]

  2. [...] is rather unlikely in most realistic settings (this is noted in the Discussion as well as in our PNAS paper). So the term sufficient should not be used as in Figure 3 for instance. Overall, the method of [...]

  3. [...] my earlier posts on the revision of Lack of confidence, here is an interesting outcome from the derivation of the exact marginal likelihood in the Laplace [...]

  4. [...] conjunction with the normal-Laplace comparison mentioned in the most recent post about our lack of confidence in ABC model choice, we have been working on the derivation of the [...]

  5. Hello,

    How would you compare the failure of ABC to choose between models in this setting to its success in selecting the correct demographic model in the Wright-Fisher setting? It seems that the choice between the Laplace and the normal is not equivalent to the choice between two demographic models. In the latter case, one assumes that the population dynamics of both demographic models are described by the WF process and the mutational dynamics correspond to the infinite-alleles or model. In the former case one is choosing between two different distribution. Is this qualitative difference between the two model choice settings relevant to the performance of summary statistics for differentiating between the models? Is this difference even real? Thank you.

    -Sameer

    • The difference may feel substantial in that one setting proposes two explanations for demographic evolution while the other setting discusses noise. There is therefore a difference in the complexity of the models and hence of the question. In my opinion, both settings are nonetheless in fine of the same nature in that several models are proposed for a given dataset and a Bayes factor is then computed to decide which model is more likely for this dataset.

  6. This plot is more illustrative and it is nice to see the difference in models. By the way I was reading this new review on ABC

    http://onlinelibrary.wiley.com/doi/10.1111/j.1461-0248.2011.01640.x/full

    I consider this is really poor in the sense that they do not illustrate the difficulties while using the algorithm. There are some other technical issues like “for continuous variables, the probability of observing exactly the same outcome is infinitesimally small”. I would like to know your opinion about it.

    • Rosana: thank you for signaling this survey on ABC (surveys seem to sprout like weeds these days!!!), I will read it and report on it shortly. My interpretation of the graph above is not as optimistic as yours. There is a slight difference in the distribution of the posterior probabilities, but, overall, they spread the same region, hence give little confidence in trusting this particular version of ABC model choice…

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

Join 678 other followers