Comments on: on using the data twice…

By: xi'an

xi'an — Mon, 16 Jan 2012 11:13:38 +0000

Thanks for the precision! Even though I had handled a previous version of this paper, I did not remember the re-run part (and I have not re-checked the paper so far). Now, this is entering a grey area albeit far from the Mordor of “using the data twice” (to keep with the Tolkienian theme!). In a Bayesian analysis with different models, the global posterior distribution is the weighted sum of all the posteriors for the different models. Once a model is selected based on this global posterior, we do not rerun the chosen model since it already is part of the global model. I would thus say it’s mostly kosher.

By: Matthieu Authier

Matthieu Authier — Mon, 16 Jan 2012 10:47:44 +0000

Hello,

thanks very much for the answer. I was asking about the horseshoe because a part of the Biometrika paper deals with thresholding, and conditional on the threshold dropping some variables. I was feeling that using the horseshoe prior and reporting posteriors from this model is enough. That is, we don’t need to re-run a model with only the predictors that were selected before, since that would be using the data twice (but given my experience so far with the publication process in the field of ecology, this is probably what will be asked by reviewers).
Hence my question about the horseshoe. I feel that it could fall also in your category (e) in practice but really is fine on its own (d).
Or in Tolkienian prose, the horseshoe may be then some sort of “One model to find them all and in the prior shrink them”…
Sorry for bugging you with probably naïve questions and mille mercis d’avoir pris le temps de répondre.
Cheers
Matthieu

By: xi'an

xi'an — Sat, 14 Jan 2012 11:20:03 +0000

Matthieu: thanks for the comments. There are several issues:
(a) “using the data twice” is not related to Bayesian model selection per se in that we can run proper Bayesian model selection w/o using the data twice. It is also possible to “use the data twice” in non-Bayesian settings.
(a’) by the way variable selection is model choice [for me].
(b) Lasso is a penalised likelihood technique for variable comparison / nested model comparison. It has a vague Bayesian flavour that was precised by Park and Casella (2008).
(c) AIC, BIC, &C, Lasso, are on principle free from the sin of “using the data twice”, being all of the likelihood ratio type.
(c’) this is less clear for DIC!
(d) Carvahlo, Scott and Polson have reanalysed the Bayesian lasso by picking another prior on the penalty coefficients. This is strictly Bayesian, hence does not use the data twice.
(e) in practice, people use mixes of Bayesian and non-Bayesian techniques, like pluggin-hyperparameters, which end up using the data more than once…

Merci et à bientôt!

By: xi'an

xi'an — Sat, 14 Jan 2012 07:53:06 +0000

Thanks! Using a Bayesian approach where all unknowns are endowed with prior probabilities (for models) or prior distributions (for parameters), and where the errors are evaluated under a loss function, the data is only used once in the (global) posterior, which is then the only entity needed to conduct the decision process. A genuine Bayesian approach cannot therefore “use the data twice”. Only approximations to Bayesian procedures open the Pandora box of re-using the data, some of them in obviously detrimental fashions… Correcting for this can only be done in an un-Bayesian way, using asymptotics and indeed the sampling distribution.

By: mayod@vt.edu

mayod@vt.edu — Sat, 14 Jan 2012 02:34:46 +0000

Rather than make it an “all or nothing” principle, one needs an epistemological criterion to distinguish pejorative from unpejorative double counting. That is what I hope the “severity principle” achieves, at least for someone who cares to evaluate procedures according to how well or poorly specific errors are controlled/ruled out. When double counting does warrant an adjustment, we appeal essentially to the sampling distribution. But I’m not sure what the Bayesian justification for caring is. Unless one foists it on the prior, but I’d still want to know the grounds, then the evidential relationship between data and hypothesis, for likelihoodists, should have nothing to do with looking at or using up the data—isn’t that part of the Bayesian magic?

By: Authier Matthieu

Authier Matthieu — Fri, 13 Jan 2012 11:45:07 +0000

Hello,
as an ecologist, I am very often confronted to the issue of model selection. Most of the time, this is done by some automated procedure that computes the AIC of all possible variable combination. I suspect then what we’re doing more variable selection than model selection. Anyway, the process is tedious and looks a bit like an overkill : most of the models considered have not been thought about (after all, R does this on its own without thinking) and some are just not plausible.
I read recently the Biometrika paper of Carvahlo, Scott and Polson about the horsehoe estimator for sparse signals. I would be very curious to know your thoughts about this procedure. I have used the horseshoe in a logistic regression about clutch size in a seabird. It was very useful in shrinking some spurious signals (evaluated from posterior predictive checks… I guess this is using the data twice…).
My more focus question is then : do you think the horseshoe prior can solve this “using the data twice” problem?
Yours sincerely
Matthieu