Archive for Idaho

399 safe[[r] for now]

Posted in Kids, Mountains, pictures, Travel with tags , , , , , , , , , , , on October 14, 2018 by xi'an

hierarchical models are not Bayesian models

Posted in Books, Kids, Statistics, University life with tags , , , , , , , on February 18, 2015 by xi'an

When preparing my OxWaSP projects a few weeks ago, I came perchance on a set of slides, entitled “Hierarchical models are not Bayesian“, written by Brian Dennis (University of Idaho), where the author argues against Bayesian inference in hierarchical models in ecology, much in relation with the previously discussed paper of Subhash Lele. The argument is the same, namely a possibly major impact of the prior modelling on the resulting inference, in particular when some parameters are hardly identifiable, the more when the model is complex and when there are many parameters. And that “data cloning” being available since 2007, frequentist methods have “caught up” with Bayesian computational abilities.

Let me remind the reader that “data cloning” means constructing a sequence of Bayes estimators corresponding to the data being duplicated (or cloned) once, twice, &tc., until the point estimator stabilises. Since this corresponds to using increasing powers of the likelihood, the posteriors concentrate more and more around the maximum likelihood estimator. And even recover the Hessian matrix. This technique is actually older than 2007 since I proposed it in the early 1990’s under the name of prior feedback, with earlier occurrences in the literature like D’Epifanio (1989) and even the discussion of Aitkin (1991). A more efficient version of this approach is the SAME algorithm we developed in 2002 with Arnaud Doucet and Simon Godsill where the power of the likelihood is increased during iterations in a simulated annealing version (with a preliminary version found in Duflo, 1996).

I completely agree with the author that a hierarchical model does not have to be Bayesian: when the random parameters in the model are analysed as sources of additional variations, as for instance in animal breeding or ecology, and integrated out, the resulting model can be analysed by any statistical method. Even though one may wonder at the motivations for selecting this particular randomness structure in the model. And at an increasing blurring between what is prior modelling and what is sampling modelling as the number of levels in the hierarchy goes up. This rather amusing set of slides somewhat misses a few points, in particular the ability of data cloning to overcome identifiability and multimodality issues. Indeed, as with all simulated annealing techniques, there is a practical difficulty in avoiding the fatal attraction of a local mode using MCMC techniques. There are thus high chances data cloning ends up in the “wrong” mode. Moreover, when the likelihood is multimodal, it is a general issue to decide which of the modes is most relevant for inference. In which sense is the MLE more objective than a Bayes estimate, then? Further, the impact of a prior on some aspects of the posterior distribution can be tested by re-running a Bayesian analysis with different priors, including empirical Bayes versions or, why not?!, data cloning, in order to understand where and why huge discrepancies occur. This is part of model building, in the end.

Error in ABC versus error in model choice

Posted in pictures, Statistics, University life with tags , , , , on March 8, 2011 by xi'an

Following the earlier posts about our lack of confidence in ABC model choice, I got an interesting email from Christopher Drummond, who is a postdoc at University of Idaho, working on an empirical project with the landscape genetics of tailed frogs. Along the lines of the empirical test we advocated at the end of our paper, Chris evaluated the type I error (or the false allocation rate) on a controlled ABC experiment with simulated pseudo-observed data (pods) for validation, and ended up with an large overall error on the order of 10% across four different models, ranging from 5-25% for each.. He further reported that “there is not much improvement of an exponentially decreasing rate of improvement in predictive accuracy as the number of ABC simulations increases” and then extrapolated about the huge [impossibly large] number of ABC  simulations [hence the value of the ABC tolerance] that is required to achieve, say, a 5% error rate. This was a most  interesting extrapolation and we ended up exchanging a few emails around this theme… My main argument in the ensuing discussion was that there is a limiting error rate that presumably is different from zero simply because Bayesian procedures are fallible, just like any other statistical procedure, unless the priors are highly differentiated from one model to the next.

Chris also noticed that calibrating the value of the Bayes factor in terms of the false allocation rate itself rather than an absolute scale like Jeffrey’s might provide some trust about the actual (log10) ABC Bayes factors recovered for the models fit to the actual data he observed, since validation simulations indicated no wrong allocation for values above log(10) BF > 5, versus log10(BF) ~ 8 for the model that best fit the observed data collected from real frogs. Although this sounds like a Bayesian p-value, it illustrates very precisely our suggestion in the conclusion of our paper of turning to empirical measures as such to calibrate the ABC output without overly trusting the ABC approximation of the Bayes factor itself.