an arithmetic mean identity

A 2017 paper by Ana Pajor published in Bayesian Analysis addresses my favourite problem [of computing the marginal likelihood] and which I discussed on the ‘Og, linking with another paper by Lenk published in 2012 in JCGS. That I already discussed here last year. Lenk’s (2009) paper is actually using a technique related to the harmonic mean correction based on HPD regions Darren Wraith and myself proposed at MaxEnt 2009. And which Jean-Michel and I presented at Frontiers of statistical decision making and Bayesian analysis in 2010. As I had only vague memories about the arithmetic mean version, we discussed the paper together with graduate students in Paris Dauphine.

The arithmetic mean solution, representing the marginal likelihood as the prior average of the likelihood, is a well-known approach used as well as the basis for nested sampling. With the improvement consisting in restricting the simulation to a set Ð with sufficiently high posterior probability. I am quite uneasy about P(Ð|y) estimated by 1 as the shape of the set containing all posterior simulations is completely arbitrary, parameterisation dependent, and very random since based on the extremes of this posterior sample. Plus, the set Ð converges to the entire parameter space with the number of posterior simulations. An alternative that we advocated in our earlier paper is to take Ð as the HPD region or a variational Bayes version . But the central issue with the HPD regions is how to construct these from an MCMC output and how to compute both P(Ð) and P(Ð|y). It does not seem like a good idea to set P(Ð|x) to the intended α level for the HPD coverage. Using a non-parametric version for estimating Ð could be in the end the only reasonable solution.

As a test, I reran the example of a conjugate normal model used in the paper, based on (exact) simulations from both the prior and  the posterior, and obtained approximations that were all close from the true marginal. With Chib’s being exact in that case (of course!), and an arithmetic mean surprisingly close without an importance correction:

> print(c(hame,chme,came,chib))
[1] -107.6821 -106.5968 -115.5950 -115.3610

Both harmonic versions are of the right order but not trustworthy, the truncation to such a set Ð as the one chosen in this paper having little impact.

a remarkably simple and accurate method for computing the Bayes factor &tc.

This recent arXiv posting by Martin Weinberg and co-authors was pointed out to me by friends because of its title! It indeed sounded a bit inflated. And also reminded me of old style papers where the title was somehow the abstract. Like An Essay towards Solving a Problem in the Doctrine of Chances… So I had a look at it on my way to Gainesville. The paper starts from the earlier paper by Weinberg (2012) in Bayesian Analysis where he uses an HPD region to determine the Bayes factor by a safe harmonic mean estimator (an idea we already advocated earlier with Jean-Michel Marin in the San Antonio volume and with Darren Wraith in the MaxEnt volume). An extra idea is to try to optimise [against the variance of the resulting evidence] the region over which the integration is performed: “choose a domain that results in the most accurate integral with the smallest number of samples” (p.3). The authors proceed by volume peeling, using some quadrature formula for the posterior coverage of the region, either by Riemann or Lebesgue approximations (p.5). I was fairly lost at this stage and the third proposal based on adaptively managing hyperrectangles (p.7) went completely over my head! The sentence “the results are clearly worse with O(∞) errors, but are still remarkably better for high dimensionality”(p.11) did not make sense either… The method may thus be remarkably simple, but the paper is not written in a way that conveys this impression!

Oxford, Miss. [Le Monde travel guide]

The weekend edition of Le Monde has a pseudo-travel guide written by a local writer about his or her town. It is necessarily partial and subjective, but often interesting. It also sometimes mentions towns one would never dream of visiting. This week (18/02/2012), this tribune most unexpectedly focus on Oxford, Mississippi, that I visited two and a half years ago for MaxEnt 2009. (The writer in charge is Tom Franklin. Not that I ever heard of him…) I find it quite puzzling that Le Monde spends two pages on this little town where the only attraction worth mentioning is Faulkner’s family home, now turned into a museum, and where the (decent) local bookstore is the only place in town one can buy the New York Times. Unsurprisingly, the highlights are local bars and cafés… I wonder if any Le Monde reader will be induced by the guide to travel to this place.

A revised assessment of nested sampling

As announced earlier on that post, the MaxEnt2009 meeting was a very good opportunity to revise a second time our nested sampling evaluation, written with Nicolas Chopin. The new version is now posted on arXiv as well as resubmitted to Biometrika. The changes in the text are presumably less important than those in our (my?) understanding of the method. I indeed think the nested sampling method belongs to the general category of importance sampling methods and that potential improvements lie in modifying the evaluation of the prior mass between slices of the likelihood.I still have to understand better why nested sampling would be the only importance solution in a given problem, as argued for instance in this recent posting on arXiv by Livi Pártay, Albert Bartók, and Gábor Csányi, from Cambrdige University, which was presented earlier this week at the MaxEnt2009 meeting, but this will have to wait till the Fall, I am afraid…

MaxEnt2009 impressions

As I am getting ready to leave Oxford and the MaxEnt2009 meeting, a few quick impressions. First, this is a meeting like no other I have attended in that the mix of disciplines there is much wider and that I find myself at the very (statistical) end of the spectrum. There are researchers from astronomy, physics, chemistry, computer science, engineering, and hardly any from statistics. Second, the audience being of a decent (meaning small enough) size, the debates are numerous and often focus on the foundations of Bayesian statistics, a feature that has almost disappeared from the Valencia meetings. Some of the talks were actually mostly philosophical on the nature of deduction and inference, and I could not always see the point, but this is also enjoyable (once in a while). For instance, during the poster session, I had a long and lively discussion with David Blower on the construction of Jeffreys versus Haldane priors, as well as another long and lively discussion with Subhadeep Mukhopadhyay on fast exploration stochastic approximation algorithms. It was also an opportunity to reflect on nested sampling, given the surroundings and the free time, and I now perceive this technique primarily as a particular importance sampling method. So, overall, an enjoyable time! (Since MaxEnt2010 will take place in Grenoble, I may even attend the next conference.)