dominating measure
Yet another question on X validated reminded me of a discussion I had once with Jay Kadane when visiting Carnegie Mellon in Pittsburgh. Namely the fundamentally ill-posed nature of conjugate priors. Indeed, when considering the definition of a conjugate family as being a parameterised family Þ of distributions over the parameter space Θ stable under transform to the posterior distribution, this property is completely dependent (if there is such a notion as completely dependent!) on the dominating measure adopted on the parameter space Θ. Adopted is the word as there is no default, reference, natural, &tc. measure that promotes one specific measure on Θ as being the dominating measure. This is a well-known difficulty that also sticks out in most “objective Bayes” problems, as well as with maximum entropy priors. This means for instance that, while the Gamma distributions constitute a conjugate family for a Poisson likelihood, so do the truncated Gamma distributions. And so do the distributions which density (against a Lebesgue measure over an arbitrary subset of (0,∞)) is the product of a Gamma density by an arbitrary function of θ. I readily acknowledge that the standard conjugate priors as introduced in every Bayesian textbook are standard because they facilitate (to a certain extent) posterior computations. But, just like there exist an infinity of MaxEnt priors associated with an infinity of dominating measures, there exist an infinity of conjugate families, once more associated with an infinity of dominating measures. And the fundamental reason is that the sampling model (which induces the shape of the conjugate family) does not provide a measure on the parameter space Θ.
Xi'an's Og 
March 21, 2019 at 5:10 am
The non-uniqueness of conjugate priors is interesting and should be taken into account when teaching this concept. However, even outside of standard Bayesian textbooks, have conjugate priors ever been really advocated as default priors?
The standard scenario would be that, if you can find values for the hyperparameters of ANY family of conjugate priors such that the resulting prior adequately represents your prior beliefs, then you can (hopefully) obtain an analytically tractable posterior density. This could be viewed as a reasonably practical scenario, for example, in the case of a beta conjugate prior for a binomial likelihood, but less so for other combinations of conjugate priors and likelihood functions.
From this subjective Bayesian perspective, the key of course is believing in your prior. If you do not believe in your prior, then do something else.
March 21, 2019 at 2:38 pm
Thank you. From a perfect subjective perspective, it is clear that the worry expressed by this column is not relevant since the perfect subjectivist is knowledgeable and agreeable about the entire prior. I however see conjugate priors as a poor person’s substitute where only hyperparameters need be specified on a subjective basis, hence the closest to non-informative one can get without turning improper. Something very close to maximum entropy priors, actually.
March 22, 2019 at 1:39 am
I think I’m not following you here. If you have a family of priors parameterized by hyperparameters, you can decide to use that family with a prior on hyperparameters as your overall prior. You can do this regardless of whether or not this is a conjugate family. (Though obviously this may affect the ease of computation.) So how is this issue relevant beyond the practicalities of computations?
March 22, 2019 at 11:08 am
I think we actually agree: once given a parameterised family which hyperparameters can be specified through (prior) moment or generalised moment equations, one can automatically proceed with Bayesian inference, modulo computational hindrances of a more or less severe nature. Conjugate priors may have computational advantages in favour themselves, although mileage varies. What I find fascinating though (but may be alone in this!) is that there exists an infinity of conjugate families and that only one is standard or textbook-material, without anyone questioning the role of the dominating measure.