parallel MCMC via Weirstrass sampler

During O’Bayes 2013, Xiangyu Wang and David Dunson arXived a paper (with the above title) that David then presented on the 19th.  The setting is quite similar to the recently discussed embarrassingly parallel paper of Neiswanger et al., in that Xiangyu and David start from the same product representation of the target (posterior). Namely,

However, they criticise the choice made by Neiswanger et al to use MCMC approximations to each component of the product for the following reasons:

1. Curse of dimensionality in the number of parameters p
2. Curse of dimensionality in the number of subsets m
3. Tail degeneration
4. Support inconsistency and mode misspecification

While I agree on point 1 (although there may be other ways than kernel estimation to mix samples from the terms in the product, terms Neiswanger et al. called the subposteriors), which is also a drawback with the current method, point 2 is not such a clearcut drawback. I first had the same reaction about the T^m explosion in the number of terms in a product of m sums of T terms, but Neiswanger et al. use a clever trick to avoid the combinatoric explosion, namely to operate one mixture at a time. Having non-manageable targets is not such an issue in the post-MCMC era, isn’t it?! Point 3 is formally correct, in that the kernel tail behaviour induces the kernel estimate tail behaviour, most likely disconnected from the true target tail behaviour, but this feature is true for any non-parametric estimate, I believe, even for the Weierstraß transform, and hence maybe not so relevant in practice. In fact, my operational intuition is that by lifting the tails up, the simulation from the subposteriors should help in visiting the tails of the true target. (Proofs of convergence and bound rely on L1 norms, not particularly interested in tails.) At last, point 4 does not seem to life-threatening: assuming the true target can be computed up to a normalising constant, the value of the target for every simulated parameter could be computed, eliminating those outside the support of the product and highlighting modal regions. (The paper does mention the improvement brought by computing those target values.)

The Wierstraß transform of a density f is a convolution of f and of an arbitrary kernel K. The authors propose to simulate from the product of the Wierstraß transform, using a multi-tiered Gibbs sampler. This way, the parameter is only simulated once and from a controlled kernel, while the random effects from the convolution are related with each subposterior. While the method requires coordination between the parallel threads, the components of the target are separately computed on a single thread. (I do not get the refinement in Section 4.2, about how “we can regard [θ] as a hyper-parameter, which allows to draw samples of [random effects] hierarchically”. Once a cycle is completed, all threads must wait for the common θ to be updated.) Maybe the clearest perspective on the Wierstraß transform is the rejection sampling version (Section 4.3) where simulations from the subpriors are merged together into a normal proposal on θ to be accepted with a probability depending on the subprior simulations.

There is something puzzling and vaguely familiar about the Wierstraß transform, something I cannot really put the finger on. It reminds me of the (great) convolution trick Jacques Neveu used to prove the Central Limit Theorem in his École Polytechnique notes. But it also reminds me of West and Harrison’s notion of time-evolving parameter (notion that took [me] a while to seep in…). Definitely interesting!