[Here is the reply on my post from Michael Bétancourt, detailed enough to be promoted from comment to post!]
As Dan notes this is meant as an introduction for those without a strong mathematical background, hence the focus on concepts rather than theorems! There’s plenty of maths deeper in the references. ;-)
I am not sure I get this sentence. Either it means that an expectation remains invariant under reparameterisation. Or something else and more profound that eludes me. In particular because Michael repeats later (p.25) that the canonical density does not depend on the parameterisation.
What I was trying to get at is that expectations and really all of measure theory are reparameteriztion invariant, but implementations of statistical algorithms that depend on parameterization-dependent representations, namely densities, are not. If your algorithm is sensitive to these parameterization dependencies then you end up with a tuning problem — which parameterization is best? — which makes it harder to utilize the algorithm in practice.
Exact implementations of HMC (i.e. without an integrator) are fully geometric and do not depend on any chosen parameterization, hence the canonical density and more importantly the Hamiltonian being an invariant objects. That said, there are some choices to be made in that construction, and those choices often look like parameter dependencies. See below!
“Every choice of kinetic energy and integration time yields a new Hamiltonian transition that will interact differently with a given target distribution (…) when poorly-chosen, however, the performance can suffer dramatically.”
This is exactly where it’s easy to get confused with what’s invariant and what’s not!
The target density gives rise to a potential energy, and the chosen density over momenta gives rise to a kinetic energy. The two energies transform in opposite ways under a reparameterization so their sum, the Hamiltonian, is invariant.
Really there’s a fully invariant, measure-theoretic construction where you use the target measure directly and add a “cotangent disintegration”.
In practice, however, we often choose a default kinetic energy, i.e. a log density, based on the parameterization of the target parameter space, for example an “identify mass matrix” kinetic energy. In other words, the algorithm itself is invariant but by selecting the algorithmic degrees of freedom based on the parameterization of the target parameter space we induce an implicit parameter dependence.
This all gets more complicated when we introducing the adaptation we use in Stan, which sets the elements of the mass matrix to marginal variances which means that the adapted algorithm is invariant to marginal transformations but not joint ones…
The explanation of the HMC move as a combination of uniform moves along isoclines of fixed energy level and of jumps between energy levels does not seem to translate into practical implementations, at least not as explained in the paper. Simulating directly the energy distribution for a complex target distribution does not seem more feasible than moving up likelihood levels in nested sampling.
Indeed, being able to simulate exactly from the energy distribution, which is equivalent to being able to quantify the density of states in statistical mechanics, is intractable for the same reason that marginal likelihoods are intractable. Which is a shame, because conditioned on those samples HMC could be made embarrassingly parallel!
Instead we draw correlated samples using momenta resamplings between each trajectory. As Dan noted this provides some intuition about Stan (it reduced random walk behavior to one dimension) but also motivates some powerful energy-based diagnostics that immediately indicate when the momentum resampling is limiting performance and we need to improve it by, say, changing the kinetic energy. Or per my previous comment, by keeping the kinetic energy the same but changing the parameterization of the target parameter space. :-)
In the end I cannot but agree with the concluding statement that the geometry of the target distribution holds the key to devising more efficient Monte Carlo methods.
Yes! That’s all I really want statisticians to take away from the paper. :-)
A cyclist was convicted of “wanton or furious driving” last week in London after hitting a pedestrian crossing the street, leading to her death a few days later. The main legal argument for the conviction was that the cyclist was riding a “fixie”, a bike with no front brake and fixed-gear, as used in track cycling. Which is illegal in Britain and, I just found out, in France too. (He was actually facing manslaughter, for which he got acquitted.) This is a most tragic accident, alas leading to a loss of a human life, and I did not look at the specifics, but I do not get the argument about the brakes and the furious driving: if the rider was going at about 28 km/h, which seems a reasonable speed in low density areas [and is just above my average speed in suburban Paris], and if the pedestrian stepped in his path six meters ahead, he had less than a second to react. Front brake or not, I am certainly unable to react and stop in this interval. And braking hard with the front brake will invariably lead to going over the bars: happens to me every time I have to stop for a car with my road bike. And would if I had to stop for a pedestrian.
When leaving 
