Still not confident…

About 45 days after submitting our revised version of lack of confidence in ABC model choice to PNAS, we have received the reviews and they still ask for more clarity in our conclusions. In particular, one referee does not buy the distinction between ABC point (and confidence) estimation and ABC model choice, namely that ABC may equally go wrong in the former case when using a poor selection of summary statistics.  This is correct, even though a stepwise ABC estimation à la Marjoram could bring some measure of confidence in the set of summary statistics used for estimation. This referee’s conclusion is then that “the statistics we use for model checking must be different from those used in inference“. This is sound when considering the simple normal example we include in the paper, however in realistic situations, there is no sufficiency to aim at and therefore it is impossible to know how good or how different those summary statistics are. The second referee maintains her or his earlier position that there is nothing wrong with ABC when using a sufficiently large collection of summary statistics. The proof is given in our paper by the fact that for the larger dataset  we “get statistically reliable results using ABC for model choice (without using sufficient statistics)”. First, this is only one experiment, with no reason to extrapolate to other philogenetic trees and even less to models outside population genetics. Second, we only argue that the ABC Bayes factor does not have any reason to converge to the true Bayes factor. It may well be a consistent quantity for model selection, but theoretical arguments are currently missing (although an extension of the perspective adopted in Fearnhead and Prangle and in Dean et al., namely to see ABC as an inference method per se rather than an approximation to a Bayesian inaccessible methodology, could be considered here as well). In conclusion, this means that we have to go for yet another run of revision that will tone even further down our conclusion and presumably soften the distinction between point estimation and hypothesis testing… A good thing Natesh starts his visit to Paris today!