Archive for MaxEnt2009

an arithmetic mean identity

Posted in Books, pictures, R, Statistics, Travel, University life with tags , , , , , , , , , , , , on December 19, 2019 by xi'an

A 2017 paper by Ana Pajor published in Bayesian Analysis addresses my favourite problem [of computing the marginal likelihood] and which I discussed on the ‘Og, linking with another paper by Lenk published in 2012 in JCGS. That I already discussed here last year. Lenk’s (2009) paper is actually using a technique related to the harmonic mean correction based on HPD regions Darren Wraith and myself proposed at MaxEnt 2009. And which Jean-Michel and I presented at Frontiers of statistical decision making and Bayesian analysis in 2010. As I had only vague memories about the arithmetic mean version, we discussed the paper together with graduate students in Paris Dauphine.

The arithmetic mean solution, representing the marginal likelihood as the prior average of the likelihood, is a well-known approach used as well as the basis for nested sampling. With the improvement consisting in restricting the simulation to a set Ð with sufficiently high posterior probability. I am quite uneasy about P(Ð|y) estimated by 1 as the shape of the set containing all posterior simulations is completely arbitrary, parameterisation dependent, and very random since based on the extremes of this posterior sample. Plus, the set Ð converges to the entire parameter space with the number of posterior simulations. An alternative that we advocated in our earlier paper is to take Ð as the HPD region or a variational Bayes version . But the central issue with the HPD regions is how to construct these from an MCMC output and how to compute both P(Ð) and P(Ð|y). It does not seem like a good idea to set P(Ð|x) to the intended α level for the HPD coverage. Using a non-parametric version for estimating Ð could be in the end the only reasonable solution.

As a test, I reran the example of a conjugate normal model used in the paper, based on (exact) simulations from both the prior and  the posterior, and obtained approximations that were all close from the true marginal. With Chib’s being exact in that case (of course!), and an arithmetic mean surprisingly close without an importance correction:

> print(c(hame,chme,came,chib))
[1] -107.6821 -106.5968 -115.5950 -115.3610

Both harmonic versions are of the right order but not trustworthy, the truncation to such a set Ð as the one chosen in this paper having little impact.

a remarkably simple and accurate method for computing the Bayes factor &tc.

Posted in Statistics with tags , , , , , , , , on February 13, 2013 by xi'an

This recent arXiv posting by Martin Weinberg and co-authors was pointed out to me by friends because of its title! It indeed sounded a bit inflated. And also reminded me of old style papers where the title was somehow the abstract. Like An Essay towards Solving a Problem in the Doctrine of Chances… So I had a look at it on my way to Gainesville. The paper starts from the earlier paper by Weinberg (2012) in Bayesian Analysis where he uses an HPD region to determine the Bayes factor by a safe harmonic mean estimator (an idea we already advocated earlier with Jean-Michel Marin in the San Antonio volume and with Darren Wraith in the MaxEnt volume). An extra idea is to try to optimise [against the variance of the resulting evidence] the region over which the integration is performed: “choose a domain that results in the most accurate integral with the smallest number of samples” (p.3). The authors proceed by volume peeling, using some quadrature formula for the posterior coverage of the region, either by Riemann or Lebesgue approximations (p.5). I was fairly lost at this stage and the third proposal based on adaptively managing hyperrectangles (p.7) went completely over my head! The sentence “the results are clearly worse with O() errors, but are still remarkably better for high dimensionality”(p.11) did not make sense either… The method may thus be remarkably simple, but the paper is not written in a way that conveys this impression!

Oxford, Miss. [Le Monde travel guide]

Posted in Books, Travel, University life with tags , , , , , on February 18, 2012 by xi'an

The weekend edition of Le Monde has a pseudo-travel guide written by a local writer about his or her town. It is necessarily partial and subjective, but often interesting. It also sometimes mentions towns one would never dream of visiting. This week (18/02/2012), this tribune most unexpectedly focus on Oxford, Mississippi, that I visited two and a half years ago for MaxEnt 2009. (The writer in charge is Tom Franklin. Not that I ever heard of him…) I find it quite puzzling that Le Monde spends two pages on this little town where the only attraction worth mentioning is Faulkner’s family home, now turned into a museum, and where the (decent) local bookstore is the only place in town one can buy the New York Times. Unsurprisingly, the highlights are local bars and cafés… I wonder if any Le Monde reader will be induced by the guide to travel to this place.

A revised assessment of nested sampling

Posted in Statistics with tags , , , , on July 13, 2009 by xi'an

As announced earlier on that post, the MaxEnt2009 meeting was a very good opportunity to revise a second time our nested sampling evaluation, written with Nicolas Chopin. The new version is now posted on arXiv as well as resubmitted to Biometrika. The changes in the text are presumably less important than those in our (my?) understanding of the method. I indeed think the nested sampling method belongs to the general category of importance sampling methods and that potential improvements lie in modifying the evaluation of the prior mass between slices of the likelihood.I still have to understand better why nested sampling would be the only importance solution in a given problem, as argued for instance in this recent posting on arXiv by Livi Pártay, Albert Bartók, and Gábor Csányi, from Cambrdige University, which was presented earlier this week at the MaxEnt2009 meeting, but this will have to wait till the Fall, I am afraid…

MaxEnt2009 impressions

Posted in Statistics, University life with tags , , , , on July 9, 2009 by xi'an

As I am getting ready to leave Oxford and the MaxEnt2009 meeting, a few quick impressions. First, this is a meeting like no other I have attended in that the mix of disciplines there is much wider and that I find myself at the very (statistical) end of the spectrum. There are researchers from astronomy, physics, chemistry, computer science, engineering, and hardly any from statistics. Second, the audience being of a decent (meaning small enough) size, the debates are numerous and often focus on the foundations of Bayesian statistics, a feature that has almost disappeared from the Valencia meetings. Some of the talks were actually mostly philosophical on the nature of deduction and inference, and I could not always see the point, but this is also enjoyable (once in a while). For instance, during the poster session, I had a long and lively discussion with David Blower on the construction of Jeffreys versus Haldane priors, as well as another long and lively discussion with Subhadeep Mukhopadhyay on fast exploration stochastic approximation algorithms. It was also an opportunity to reflect on nested sampling, given the surroundings and the free time, and I now perceive this technique primarily as a particular importance sampling method. So, overall, an enjoyable time! (Since MaxEnt2010 will take place in Grenoble, I may even attend the next conference.)

MaxEnt2009

Posted in Statistics with tags , , , , on July 6, 2009 by xi'an

The revision of my talk for the MaxEnt2009 conference is now available on slideshare as

The major difference with the talks I gave in Montréal, Edinburgh, Warwick or Rimini is the additional experiment we ran with Darren Wraith on the banana targets already used in the “PMC for cosmologist” paper. As for the mixture benchmark used in the paper with Nicolas Chopin, we found that implementing nested sampling by the book, ie based on the lighthouse code provided in the original papers, led to the right value on average but with a lot more variability than for importance sampling solutions (with a comparable number of iterations).

When running this morning on the campus (i.e. wading through water particles without a snorkel), I thought—helped by comments from Olivier Cappé—that the best explanation for nested sampling is one of importance sampling, the weight of points being the prior mass of the current upper likelihood region. Obviously, this is not the whole story since those constrained priors have a smaller support than the posterior, but this may help in a better evaluation of nested sampling.

La partenza per MaxEnt 2009

Posted in Books, Statistics, Travel with tags , , , , on July 4, 2009 by xi'an

Just to let this indefinable perfume of Italy linger for a few more posts, but there is no objective reason to switch to Italian… I am actually off to Oxford, Mississippi, at the MaxEnt 2009 meeting, which is the 29th workshop on Bayesian Inference and Maximum Entropy Methods in Science and Engineering. Despite the strong connections with my research interests, I have never attended one of those conferences and this sounds exciting. Especially given my talk there on computational methods, which covers in particular nested sampling with the critical views summarised in this post. This promises for interesting exchanges, since the inventor of nested sampling, John Skilling, is one of the founding fathers of the maximum entropy community.

NS1

This will also give me the opportunity to go over the new revision of the paper before sending it back to Biometrika. The reviews were so lukewarm that I was ready to pack up and go. Fortunately, Nicolas Chopin overrode my depressive tendencies and launched into the revision! All I have to do in the coming days is thus to compile once again the seven pages of the Biometrika style manual to make sure we comply with every item of it. (The editorial requests of Biometrika go beyond most other journals’, even PNAS, which is a pain when writing a paper, because the time spent of complying with the stylistic restrictions will be wasted if the paper is rejected!, but which is also most enjoyable at the reader’s level.)

Oxford is the home town of William Faulkner and. as such, enjoys a literary atmosphere (whatever that means) with several genuine bookstores, including Square Books. Given that I have never been in the South (Florida being a separate geographical entity!), this is also an interesting opportunity. Even though I am not looking forward the hot, clammy, humid weather (humidity is currently 78% at 11pm…). And considering that MaxEnt 2003 was in Jackson Hole, Wyoming…