the Hyvärinen score is back

Posted in pictures, Statistics, Travel with tags , , , , , , , , , , , , , on November 21, 2017 by xi'an

Stéphane Shao, Pierre Jacob and co-authors from Harvard have just posted on arXiv a new paper on Bayesian model comparison using the Hyvärinen score

\mathcal{H}(y, p) = 2\Delta_y \log p(y) + ||\nabla_y \log p(y)||^2

which thus uses the Laplacian as a natural and normalisation-free penalisation for the score test. (Score that I first met in Padova, a few weeks before moving from X to IX.) Which brings a decision-theoretic alternative to the Bayes factor and which delivers a coherent answer when using improper priors. Thus a very appealing proposal in my (biased) opinion! The paper is mostly computational in that it proposes SMC and SMC² solutions to handle the estimation of the Hyvärinen score for models with tractable likelihoods and tractable completed likelihoods, respectively. (Reminding me that Pierre worked on SMC² algorithms quite early during his Ph.D. thesis.)

A most interesting remark in the paper is to recall that the Hyvärinen score associated with a generic model on a series must be the prequential (predictive) version

\mathcal{H}_T (M) = \sum_{t=1}^T \mathcal{H}(y_t; p_M(dy_t|y_{1:(t-1)}))

rather than the version on the joint marginal density of the whole series. (Followed by a remark within the remark that the logarithm scoring rule does not make for this distinction. And I had to write down the cascading representation

\log p(y_{1:T})=\sum_{t=1}^T \log p(y_t|y_{1:t-1})

to convince myself that this unnatural decomposition, where the posterior on θ varies on each terms, is true!) For consistency reasons.

This prequential decomposition is however a plus in terms of computation when resorting to sequential Monte Carlo. Since each time step produces an evaluation of the associated marginal. In the case of state space models, another decomposition of the authors, based on measurement densities and partial conditional expectations of the latent states allows for another (SMC²) approximation. The paper also establishes that for non-nested models, the Hyvärinen score as a model selection tool asymptotically selects the closest model to the data generating process. For the divergence induced by the score. Even for state-space models, under some technical assumptions.  From this asymptotic perspective, the paper exhibits an example where the Bayes factor and the Hyvärinen factor disagree, even asymptotically in the number of observations, about which mis-specified model to select. And last but not least the authors propose and assess a discrete alternative relying on finite differences instead of derivatives. Which remains a proper scoring rule.

I am quite excited by this work (call me biased!) and I hope it can induce following works as a viable alternative to Bayes factors, if only for being more robust to the [unspecified] impact of the prior tails. As in the above picture where some realisations of the SMC² output and of the sequential decision process see the wrong model being almost acceptable for quite a long while…

Domaine de Montcalmès

Posted in Statistics with tags , , , , , , , , on November 20, 2017 by xi'an

a lifetime word limit…

Posted in Books, Kids, pictures, University life with tags , , , on November 20, 2017 by xi'an

“Exceptions might have to be made for experts such as statisticians and bioinformaticians whose skills are required on many papers.”

One of these weird editorials periodically occurring in Nature. By Brian Martinson, suggesting that the number of words allotted to a scientist should be capped. Weird, indeed, and incomprehensible that Nature wastes one of its so desperately sought journal page on such a fantastic (in the sense of fantasy, not as in great!) proposal. With sentences like “if we don’t address our own cognitive biases and penchant for compelling narratives, word limits could exacerbate tendencies to publish only positive findings, leading researchers to explore blind alleys that others’ negative results could have illuminated” not making much sense even in this fantasy academic world… As for the real world, the list of impossibilities and contradictions stemming from this proposal would certainly eat all of my allotted words. Even those allotted to a statistician. The supreme irony of the (presumably tongue-in-cheek) editorial is that the author himself does not seem particularly concerned by capping his own number of papers! (Nice cover, by the way!)

Arcad’yaaawn… [book review]

Posted in Books, pictures, Travel with tags , , , , , , , , , , , , , on November 19, 2017 by xi'an

“How does it do this? Pears, not traditionally a science fiction writer, employs some commonly used devices of the genre to create a mind-bending but wholly satisfying tale…” Robin’s Books

“Indeed, Arcadia seems to be aimed at the lucrative crossover point between the grownup and YA markets, even if it lacks the antic density of the Harry Potter series or the focused peril of The Hunger Games.” Steven Poole, The Guardian

The picture above is completely unrelated with the book if not the title. (And be at rest: I am not going to start an otter theme in the spirit of Andrew’s cats… Actually a cat plays a significant role in this book.) But Pears’ Arcadia is a fairly boring tale and an attempt at a rather dry play on the over-exploited theme of time-travel. Yaaawny, indeed!

I am fairly disappointed by this book, the more because Pears’ An Instance at the Fingerpost is a superb book, one of my favourites!, with a complexity of threads and levels, while maintaining a coherence of the plot that makes the final revelation a masterpiece. The Dream of Scipio also covers several historical periods of French Provence with a satisfactory plot and deep enough background (fed by a deep knowledge of the area and the eras…). The background, the broader perspective, the deep humanity of the characters, all these qualities of Pears’ books are lost in Arcadia, which sums up as an accumulation of clichés on dystopias, time-travel, and late 1950’s Oxford academics. [Warning, spoilers ahoy!] The parallel (and broadly medieval) universe to which the 20th century characters time-travel has some justifications for being a new type of Flatland: it is the creation of a single Oxonian academic, a mix of J.R. Tolkien and Eric Ambler. But these 20th century characters are equally charicaturesque. And so are the oppressors and the rebels in the distant future. (Set on the Isle of Mull, of all places!) And the mathematics of the time-travel apparatus are carefully kept hidden (with the vague psychomathematics there reminding me of the carefully constructed Asimov’s psychohistory.)

There is a point after which pastiches get stale and unattractive. And boring, so Yawn again. (That the book came to be shortlisted for the Arthur C. Clarke award this year is a mystery.)

highway to Hell [so long, Malcom!]

Posted in Kids, pictures with tags , , , , , , on November 18, 2017 by xi'an

ackward citation style

Posted in Statistics with tags , , , , , , on November 18, 2017 by xi'an

When submitting a paper to WIREs, I was asked to use the APA style for citations. This is rather unpleasant as it requires all kinds of fixes and even then returns an unseemly outcome, quoting sometimes authors with their first name and at a point ignoring the parentheses for \citep citations… Maybe all those annoying bugs are on purpose, as APA stands for the American Psychological Association, presumably eager to experiment on new subjects!

long journey to reproducible results [or not]

Posted in Books, pictures, Statistics, University life with tags , , , , , , , , , on November 17, 2017 by xi'an

A rather fascinating article in Nature of last August [hidden under a pile of newspapers at home!]. By Gordon J. Lithgow, Monica Driscoll and Patrick Phillips. About their endeavours to explain for divergent outcomes in the replications [or lack thereof] of an earlier experiment on anti-aging drugs tested on roundworms. Rather than dismissing the failures or blaming the other teams, the above researchers engaged for four years (!) into the titanic and grubby task of understanding the reason(s) for such discrepancies.

Finding that once most causes for discrepancies (like gentle versus rough lab technicians!) were eliminated, there were still two “types” of worms, those short-lived and those long-lived, for reasons yet unclear. “We need to repeat more experiments than we realized” is a welcome conclusion to this dedicated endeavour, worth repeating in different circles. And apparently missing in the NYT coverage by Susan Dominus of the story of Amy Cuddy, a psychologist at the origin of the “power pose” theory that got later disputed for lack of reproducibility. Article which main ideological theme is that Cuddy got singled-out in the replication crisis because she is a woman and because her “power pose” theory is towards empowering women and minorities. Rather than because she keeps delivering the same message, mostly outside academia, despite the lack of evidence and statistical backup. (Dominus’ criticisms of psychologists with “an unusual interest in statistics” and of Andrew’s repeated comments on the methodological flaws of the 2010 paper that started all are thus particularly unfair. A Slate article published after the NYT coverage presents an alternative analysis of this affair. Andrew also posted on Dominus paper, with a subsequent humongous trail of comments!)