Thanks for the precision! Even though I had handled a previous version of this paper, I did not remember the re-run part (and I have not re-checked the paper so far). Now, this is entering a grey area albeit far from the Mordor of “using the data twice” (to keep with the Tolkienian theme!). In a Bayesian analysis with different models, the global posterior distribution is the weighted sum of all the posteriors for the different models. Once a model is selected based on this global posterior, we do not rerun the chosen model since it already is part of the global model. I would thus say it’s mostly kosher.

]]>thanks very much for the answer. I was asking about the horseshoe because a part of the Biometrika paper deals with thresholding, and conditional on the threshold dropping some variables. I was feeling that using the horseshoe prior and reporting posteriors from this model is enough. That is, we don’t need to re-run a model with only the predictors that were selected before, since that would be using the data twice (but given my experience so far with the publication process in the field of ecology, this is probably what will be asked by reviewers).

Hence my question about the horseshoe. I feel that it could fall also in your category (e) in practice but really is fine on its own (d).

Or in Tolkienian prose, the horseshoe may be then some sort of “One model to find them all and in the prior shrink them”…

Sorry for bugging you with probably naïve questions and mille mercis d’avoir pris le temps de répondre.

Cheers

Matthieu

Matthieu: thanks for the comments. There are several issues:

(a) “using the data twice” is not related to Bayesian model selection per se in that we can run proper Bayesian model selection w/o using the data twice. It is also possible to “use the data twice” in non-Bayesian settings.

(a’) by the way variable selection *is* model choice [for me].

(b) Lasso is a penalised likelihood technique for variable comparison / nested model comparison. It has a vague Bayesian flavour that was precised by Park and Casella (2008).

(c) AIC, BIC, &C, Lasso, are on principle free from the sin of “using the data twice”, being all of the likelihood ratio type.

(c’) this is less clear for DIC!

(d) Carvahlo, Scott and Polson have reanalysed the Bayesian lasso by picking *another prior* on the penalty coefficients. This is strictly Bayesian, hence does not use the data twice.

(e) in practice, people use mixes of Bayesian and non-Bayesian techniques, like pluggin-hyperparameters, which end up using the data more than once…

Merci et à bientôt!

]]>Thanks! Using a Bayesian approach where all unknowns are endowed with prior probabilities (for models) or prior distributions (for parameters), and where the errors are evaluated under a loss function, the data is only used once in the (global) posterior, which is then the only entity needed to conduct the decision process. A genuine Bayesian approach cannot therefore “use the data twice”. Only approximations to Bayesian procedures open the Pandora box of re-using the data, some of them in obviously detrimental fashions… Correcting for this can only be done in an un-Bayesian way, using asymptotics and indeed the sampling distribution.

]]>as an ecologist, I am very often confronted to the issue of model selection. Most of the time, this is done by some automated procedure that computes the AIC of all possible variable combination. I suspect then what we’re doing more variable selection than model selection. Anyway, the process is tedious and looks a bit like an overkill : most of the models considered have not been thought about (after all, R does this on its own without thinking) and some are just not plausible.

I read recently the Biometrika paper of Carvahlo, Scott and Polson about the horsehoe estimator for sparse signals. I would be very curious to know your thoughts about this procedure. I have used the horseshoe in a logistic regression about clutch size in a seabird. It was very useful in shrinking some spurious signals (evaluated from posterior predictive checks… I guess this is using the data twice…).

My more focus question is then : do you think the horseshoe prior can solve this “using the data twice” problem?

Yours sincerely

Matthieu ]]>