Archive for maximum likelihood estimation

A misleading title…

Posted in Books, R, Statistics, University life with tags , , , , , , , , , , on September 5, 2011 by xi'an

When I received this book, Handbook of fitting statistical distributions with R, by Z. Karian and E.J. Dudewicz,  from/for the Short Book Reviews section of the International Statistical Review, I was obviously impressed by its size (around 1700 pages and 3 kilos…). From briefly glancing at the table of contents, and the list of standard distributions appearing as subsections of the first chapters, I thought that the authors were covering different estimation/fitting techniques for most of the standard distributions. After taking a closer look at the book, I think the cover is misleading in several aspects: this is not a handbook (a.k.a. a reference book), it does not cover standard statistical distributions, the R input is marginal, and the authors only wrote part of the book, since about half of the chapters are written by other authors…

Read more »

10 Comments »

Feedback on data cloning

Posted in Books, Statistics, Travel, University life with tags , , , , , , , , , , , , , on September 22, 2010 by xi'an

Following some discussions I had last week at Banff about data cloning, I re-read the 2007 “Data cloning” paper published in Ecology Letters by Lele, Dennis, and Lutscher. Once again, I see a strong similarity with our 2002 Statistics and Computing SAME algorithm, as well as with the subsequent (and equally similar) “A multiple-imputation Metropolis version of the EM algorithm” published in Biometrika by Gaetan and Yao in 2003—Biometrika to which Arnaud and I had earlier and unsuccessfully submitted this unpublished technical report on the convergence of the SAME algorithm… (The SAME algorithm is also described in detail in the 2005 book Inference in Hidden Markov Models, Chapter 13.)

Read more »

3 Comments »

Keynes’ derivations

Posted in Books, Statistics with tags , , , , , on March 29, 2010 by xi'an

Chapter XVII of Keynes’ A Treatise On Probability contains Keynes’ most noteworthy contribution to Statistics, namely the classification of probability distributions such that the arithmetic/geometric/harmonic empirical mean/empirical median is also the maximum likelihood estimator. This problem was first stated by Laplace and Gauss (leading to Laplace distribution in connection with the median and to the Gaussian distribution for the arithmetic mean). The derivation of the densities f(x,\theta) of those probability distributions is based on the constraint the likelihood equation

\sum_{i=1}^n \dfrac{\partial}{\partial\theta}\log f(y_i,\theta) = 0

is satisfied for one of the four empirical estimate, using differential calculus (despite the fact that Keynes earlier derived Bayes’ theorem by assuming the parameter space to be discrete). Under regularity assumptions, in the case of the arithmetic mean, my colleague Eric Séré showed me this indeed leads to the family of distributions

f(x,\theta) = \exp\left\{ \phi^\prime(\theta) (x-\theta) - \phi(\theta) + \psi(x) \right\}\,,

where \phi and \psi are almost arbitrary functions under the constraints that \phi is twice differentiable and f(x,\theta) is a density in x. This means that \phi satisfies

\phi(\theta) = \log \int \exp \left\{ \phi^\prime(\theta) (x-\theta) + \psi(x)\right\}\, \text{d}x\,,

a constraint missed by Keynes.

While I cannot judge of the level of novelty in Keynes’ derivation with respect to earlier works, this derivation therefore produces a generic form of unidimensional exponential family, twenty-five years before their rederivation by Darmois (1935), Pitman (1936) and Koopman (1936) as characterising distributions with sufficient statistics of constant dimensions. The derivation of the distributions for which the geometric or the harmonic means are MLEs then follows by a change of variables, y=\log x,\,\lambda=\log \theta or y=1/x,\,\lambda=1/\theta, respectively. In those different derivations, the normalisation issue is treated quite off-handedly by Keynes, witness the function

f(x,\theta) = A \left( \dfrac{\theta}{x} \right)^{k\theta} e^{-k\theta}

at the bottom of page 198, which is not integrable in x unless its support is bounded away from 0 or \infty. Similarly, the derivation of the log-normal density on page 199 is missing the Jacobian factor 1/x (or 1/y_q in Keynes’ notations) and the same problem arises for the inverse-normal density, which should be

f(x,\theta) = A e^{-k^2(x-\theta)^2/\theta^2 x^2} \dfrac{1}{x^2}\,,

instead of A\exp k^2(\theta-x)^2/x (page 200). At last, I find the derivation of the distributions linked with the median rather dubious since Keynes’ general solution

f(x,\theta) = A \exp \left\{ \displaystyle{\int \dfrac{y-\theta}{|y-\theta|}\,\phi^{\prime\prime}(\theta)\,\text{d}\theta +\psi(x) }\right\}

(where the integral ought to be interpreted as a primitive) is such that the recovery of Laplace’s distribution, f(x,\theta)\propto \exp-k^2|x-\theta| involves setting (page 201)

\psi(x) = \dfrac{\theta-x}{|x-\theta|}\,k^2 x\,,

hence making \psi a function of \theta as well. The summary two pages later actually produces an alternative generic form, namely

f(x,\theta) = A \exp\left\{ \phi^\prime(\theta)\dfrac{x-\theta}{|x-\theta|}+\psi(x) \right\}\,,

with the difficulties that the distribution only vaguely depends on \theta, being then a step function times exp(\psi(x)) and that, unless \phi is properly calibrated, A also depends on \theta.

Given that this part is the most technical section of the book, this post shows why I am fairly disappointed at having picked this book for my reading seminar. There is no further section with innovative methodological substance in the remainder of the book, which now appears to me as no better than a graduate dissertation on the probabilistic and statistical literature of the (not that) late 19th century, modulo the (inappropriate) highly critical tone.

2 Comments »

Follow

Get every new post delivered to your Inbox.

Join 342 other followers