the unbounded likelihood problem

Following my maths of the Lindley-Jeffreys paradox post, Javier (from Warwick) pointed out a recent American Statistician paper by Liu, Wu and Meeker about Understanding and addressing the unbounded likelihood problem. (I remember meeting some of the authors when visiting Ames three years ago.) As often when reading articles in The American Statistician, I easily find reasons do disagree with the authors. Here are some.

“Fisher (1912) suggest that a likelihood defined by a product of densities should be proportional to the probability of the data.”

First, I fail to understand why an unbounded likelihood is an issue. (I also fail to understand the above quote: in a continuous setting, there is no such thing as the probability of the data. Only its density.) Especially when avoiding maximum likelihood estimation. The paper is quite vague as to why this is a statistical problem. They take as one category discrete mixture models. While the likelihood explodes around each observation (in the mean direction) this does not prevent the existence of convergent solutions to the likelihood equations. Or of Bayes estimators. Nested sampling itself manages this difficulty.

Second, I deeply dislike the baseline that everything is discrete or even finite, including measurement and hence continuous densities should be replaced with probabilities, called correct likelihood in the paper. Of course, using probabilities removes any danger of hitting an infinite likelihood. But it also introduces many layers of arbitrary calibration, incl. the scale of the discretisation. Like, I do not think there is any stability of the solution when the discretisation range Δ goes to zero, if the limiting theorem of the authors holds. But they do not seem to see this as an issue. I think it would make more sense to treat Δ as another parameter.

As an aside, I also find surprising the classification of the unbounded likelihood models in three categories, one being those “with three or four parameters, including a threshold parameter”. Why on Earth 3 or 4?! As if it was not possible to find infinite likelihoods with more than four parameters…

One Response to “the unbounded likelihood problem”

1. > continuous densities should be replaced with probabilities
Agree, that would foolish.

“Statistical procedures are abstractly defined in terms of mathematics [here one would be foolish to limit mathematical approaches in this task] but are used, in conjunction with scientific models and methods, to explain observable phenomena. … When we use a statistical model to make a statistical inference [address applied problems] we implicitly assert … the theoretical world corresponds reasonably well to the real world.”

There will a lack of correspondence between the continuity of possible outcomes in the model with actual outcomes observed. (As CS Peirce put, actualities break the continuum of possibilities.)

One just needs to be aware of this disconnect and not mistakenly make a big deal out of it – one way [inconsistent MLEs will arise in applications involving just a single parameter, be wary] or an other [you should always use probabilities when calculating likelihoods]

p.s. I will have to wait until I am in the library to actually read Liu, Wu and Meeker’s paper