Confidence (bis repetita placent?)

Due to the simultaneous presence of Jean-Michel Marin and Natesh Pillai in Paris, we were able to write within a few days a new revision of our lack of confidence in ABC model choice paper, currently submitted to PNAS. Following my earlier comments on the reviews, we now consider that indeed different statistics are needed for model choice in that estimation statistics are generally unable to distinguish between the models. The collection of “other statistics” being rather wide means we are still at a loss about which statistics to pick in practive, but the principle is worth stressing.

Here is an illustration of the difficulty to separate (by ABC) between models in a setting suggested by one referee, i.e. when testing a location normal  distribution versus a location Laplace distribution (both with variance 1).  The (frequentist) distribution of the ABC estimate of the posterior probability of the normal model is centred at the same position when simulating data from the normal and from the Laplace distributions and when using a Euclidean distance involving the mean, the median and the variance as summary statistics.

The new version has now been (re)resubmitted and re-arXived. The changes are toward a more cautionary tone, stating that all we know about the ABC approximation to the Bayes factor is that… we do not know whether or not it constitutes a converging approximation to the Bayes factor (mostly not) or to the Bayes decision (presumably yes with “enough” statistics). We are currently working in exploring that direction…