summary statistics for ABC model choice

A few days ago, Dennis Prangle, Paul Fernhead, and their co-authors from New Zealand have posted on arXiv their (long-awaited) study of the selection of summary statistics for ABC model choice. And I read it during my trip to England, in trains and planes, if not when strolling in the beautiful English countryside as above.

As posted several times on this ‘Og, the crux of the analysis is that the Bayes factor is a good type of summary when comparing two models, this result extending to more model by considering instead the vector of evidences. As in the initial Read Paper by Fearnhead and Prangle, there is no true optimality in using the Bayes factor or vector of evidences, strictly speaking, besides the fact that the vector of evidences is minimal sufficient for the marginal models (integrating out the parameters). (This was a point made in my discussion.) The implementation of the principle is similar to this Read Paper setting as well: run a pilot ABC simulation, estimate the vector of evidences, and re-run the main ABC simulation using this estimate as the summary statistic. The paper contains a simulation study using some of our examples (in Marin et al., 2012), as well as an application to genetic bacterial data.

ISBA 2012 [#1]

The empirical likelihood session was the first one I attended in the morning. As I had slept fairly little the past night, I had alas trouble (more than usual!) to stay awake during the talks! They covered the application of a mix of empirical likelihood and Bayesian tools to missing data and survey data. My overall impression is however that there was not much and not enough discussion about the validation of the approach, i.e.~in connection with my lecture of the previous day, whether or not it was Bayesian, and whether or not it leads to a coherent type of inference, albeit asymptotically.

The following session was ABC #1 that I organised, with Scott Sisson chairing. As a biased organiser, I though it went on well and presented some current viewpoints on model choice and summary statistic selection, all of which have been discussed on this ‘Og! In particular, Dennis Prangle exposed his extension of the Read Paper to the model choice issue, raising interesting questions about the notion of sufficiency in this setting. Similarly, Chris Drovandi discussed the choice of pseudo-model in indirect inference, using a model fit as the selection tool, which does not seem an obvious solution to me as what matters is rather the different behaviour of the corresponding estimator in a collection of models…

After a very pleasant lunch with Ed George in what sounded like a very local and secluded restaurant, where we prepared tomorrow’s memorial session by mostly exchanging stories and memories about George Casella, I went to the adaptive Monte Carlo session, where exhaustion got the better of my genuine interest in the topic (despite Pierre prodding me awake from time to time!)… The more relevant my call for contributions to those impressions of ISBA 2012 from all volunters!

semi-automatic ABC [reply]

When I came back from LGM2012 in Trondheim, I found the latest issue of Series B on my desk. It is much thicker than in “my” days, with about 250 pages in this June 2012 issue! (One reason is that it contains two Read Papers with their discussions, amounting to 110 pages of the journal.) The first Read Paper was “Catching up faster by switching sooner” by van Erven, Grünwald and de Rooij, that we discussed with Nicolas Chopin. There are also comments (among others!) from Stephen Lauritzen, Iain Murray, and Aki Vehtari, who also spoke about Bayesian model evaluation tools at LGM2012. The second Read Paper is Fearnhead’s and Prangle’s semi-automatic ABC that I discussed last December. I have already posted about this Read Paper and used some of the discussion in preparing my ABC PhD class in Roma.  However, the remark we made in our discussion with Jean-Michel Marin that the Bayes factor would not be a pertinent summary statistic for model choice is wrong, as shown by Dennis Prangle in his poster at the workshop in Bristol. And, when reading the reply by Paul Fearnhead and Dennis Prangle, I do not see a satisfactory answer to my demand of more formal conditions for Theorem 2 and its corollary, the convergence of the noisy ABC posterior to the true parameter (page 425), to apply. (Such results exist in indirect inference.)

still confronting intractability in Bristol…

Another definitely interesting and intense day at the Confronting Intractability in Statistical Inference workshop in Bristol: all talks there had a high informational content for me and even those I had heard previously [in no time difference and hence much less chances of my dozing during talks, which, alas!, now gets into an almost certainty for US conferences!) For instance, I am still coming to terms with Gareth’s importance sampling for continuous diffusions. (This was the first time I was hearing Arnaud’s talk on the estimation of the score vector and I definitely to hear it again, given its technicality!) Sumeet Singh gave a talk mixing ABC with maximum likelihood estimation for HMMS, in connection with his earlier paper, and I got more convince  by the idea of using a sequence of balls for keeping pseudo-data close to the true data when I realised it could be implemented sequentially. Nial Friel’s talk on the double intractable likelihoods was covering graphical models and social network models, maybe calling for a comparison with ABC, as done in the recent paper by Richard Everitt. I had too many slides and thus presumably failed to deliver an intelligible message about the selection of ABC summary statistics for testing, even though the population genetics new illustration presumably helped. In connection with our ABC paper, Dennis Prangle and Paul Fernhead presented a poster on using the Bayes factor as a summary statistics in this setup, in the spirit of their Read Paper of last December. And Richard Wilkinson concluded the day with a more philosophical talk on the dual nature of ABC inference, in a quite pleasant perspective (that related to the way ABC was received by econometricians during my talk in Princeton last week). The day ended up quite pleasantly in a south-Indian thali restaurant, a good preparation for Glasgow’s Ashoka tomorrow night!

Large-scale Inference

Large-scale Inference by Brad Efron is the first IMS Monograph in this new series, coordinated by David Cox and published by Cambridge University Press. Since I read this book immediately after Cox’ and Donnelly’s Principles of Applied Statistics, I was thinking of drawing a parallel between the two books. However, while none of them can be classified as textbooks [even though Efron's has exercises], they differ very much in their intended audience and their purpose. As I wrote in the review of Principles of Applied Statistics, the book has an encompassing scope with the goal of covering all the methodological steps  required by a statistical study. In Large-scale Inference, Efron focus on empirical Bayes methodology for large-scale inference, by which he mostly means multiple testing (rather than, say, data mining). As a result, the book is centred on mathematical statistics and is more technical. (Which does not mean it less of an exciting read!) The book was recently reviewed by Jordi Prats for Significance. Akin to the previous reviewer, and unsurprisingly, I found the book nicely written, with a wealth of R (colour!) graphs (the R programs and dataset are available on Brad Efron’s home page).

“I have perhaps abused the “mono” in monograph by featuring methods from my own work of the past decade.” (p.xi)

Sadly, I cannot remember if I read my first Efron’s paper via his 1977 introduction to the Stein phenomenon with Carl Morris in Pour la Science (the French translation of Scientific American) or through his 1983 Pour la Science paper with Persi Diaconis on computer intensive methods. (I would bet on the later though.) In any case, I certainly read a lot of the Efron’s papers on the Stein phenomenon during my thesis and it was thus with great pleasure that I saw he introduced empirical Bayes notions through the Stein phenomenon (Chapter 1). It actually took me a while but I eventually (by page 90) realised that empirical Bayes was a proper subtitle to Large-Scale Inference in that the large samples were giving some weight to the validation of empirical Bayes analyses. In the sense of reducing the importance of a genuine Bayesian modelling (even though I do not see why this genuine Bayesian modelling could not be implemented in the cases covered in the book).

“Large N isn’t infinity and empirical Bayes isn’t Bayes.” (p.90)

The core of Large-scale Inference is multiple testing and the empirical Bayes justification/construction of Fdr’s (false discovery rates). Efron wrote more than a dozen papers on this topic, covered in the book and building on the groundbreaking and highly cited Series B 1995 paper by Benjamini and Hochberg. (In retrospect, it should have been a Read Paper and so was made a “retrospective read paper” by the Research Section of the RSS.) Frd are essentially posterior probabilities and therefore open to empirical Bayes approximations when priors are not selected. Before reaching the concept of Fdr’s in Chapter 4, Efron goes over earlier procedures for removing multiple testing biases. As shown by a section title (“Is FDR Control “Hypothesis Testing”?”, p.58), one major point in the book is that an Fdr is more of an estimation procedure than a significance-testing object. (This is not a surprise from a Bayesian perspective since the posterior probability is an estimate as well.)

“Scientific applications of single-test theory most often suppose, or hope for rejection of the null hypothesis (…) Large-scale studies are usually carried out with the expectation that most of the N cases will accept the null hypothesis.” (p.89)

On the innovations proposed by Efron and described in Large-scale Inference, I particularly enjoyed the notions of local Fdrs in Chapter 5 (essentially pluggin posterior probabilities that a given observation stems from the null component of the mixture) and of the (Bayesian) improvement brought by empirical null estimation in Chapter 6 (“not something one estimates in classical hypothesis testing”, p.97) and the explanation for the inaccuracy of the bootstrap (which “stems from a simpler cause”, p.139), but found less crystal-clear the empirical evaluation of the accuracy of Fdr estimates (Chapter 7, ‘independence is only a dream”, p.113), maybe in relation with my early career inability to explain Morris’s (1983) correction for empirical Bayes confidence intervals (pp. 12-13). I also discovered the notion of enrichment in Chapter 9, with permutation tests resembling some low-key bootstrap, and multiclass models in Chapter 10, which appear as if they could benefit from a hierarchical Bayes perspective. The last chapter happily concludes with one of my preferred stories, namely the missing species problem (on which I hope to work this very Spring).