chateau l’insoumise [not a political message]

model selection and multiple testing

Ritabrata Dutta, Malgorzata Bogdan and Jayanta Ghosh recently arXived a survey paper on model selection and multiple testing. Which provides a good opportunity to reflect upon traditional Bayesian approaches to model choice. And potential alternatives. On my way back from Madrid, where I got a bit distracted when flying over the South-West French coast, from Biarritz to Bordeaux. Spotting the lake of Hourtain, where I spent my military training month, 29 years ago!

“On the basis of comparison of AIC and BIC, we suggest tentatively that model selection rules should be used for the purpose for which they were introduced. If they are used for other problems, a fresh justification is desirable. In one case, justification may take the form of a consistency theorem, in the other some sort of oracle inequality. Both may be hard to prove. Then one should have substantial numerical assessment over many different examples.”

The authors quickly replace the Bayes factor with BIC, because it is typically consistent. In the comparison between AIC and BIC they mention the connundrum of defining a prior on a nested model from the prior on the nesting model, a problem that has not been properly solved in my opinion. The above quote with its call to a large simulation study reminded me of the paper by Arnold & Loeppky about running such studies through ecdfs. That I did not see as solving the issue. The authors also discuss DIC and Lasso, without making much of a connection between those, or with the above. And then reach the parametric empirical Bayes approach to model selection exemplified by Ed George’s and Don Foster’s 2000 paper. Which achieves asymptotic optimality for posterior prediction loss (p.9). And which unifies a wide range of model selection approaches.

A second part of the survey considers the large p setting, where BIC is not a good approximation to the Bayes factor (when testing whether or not all mean entries are zero). And recalls that there are priors ensuring consistency for the Bayes factor in this very [restrictive] case. Then, in Section 4, the authors move to what they call “cross-validatory Bayes factors”, also known as partial Bayes factors and pseudo-Bayes factors, where the data is split to (a) make the improper prior proper and (b) run the comparison or test on the remaining data. They also show the surprising result that, provided the fraction of the data used to proper-ise the prior does not converge to one, the X validated Bayes factor remains consistent [for the special case above]. The last part of the paper concentrates on multiple testing but is more tentative and conjecturing about convergence results, centring on the differences between full Bayes and empirical Bayes. Then the plane landed in Paris and I stopped my reading, not feeling differently about the topic than when the plane started from Madrid.

Côtes de Bourg 1994

cadillac [of wines]

dans le noir

Yesterday night, we went to a very special restaurant in down-town Paris, called “dans le noir” where meals take place in complete darkness (truly “dans le noir”!). Complete in the sense it is impossible to see one’s hand and one’s glass. The waiters are blind and the experiment turns them into our guides, as we are unable to progress or eat in the dark! In addition to this highly informative experiment, it was fun to guess the food (easy!) and even more to fail miserably at guessing the colour of the wine (a white Minervois made from Syrah that tasted very much like a red, either from Languedoc-Roussillon or from Bordeaux…!) The food was fine if not outstanding (the owner told us how cooking too refined a meal led to terrible feedbacks from the customers as they could not guess what they were eating) and the wine very good (no picture for the ‘Og, obviously!). This was my daughter’s long-time choice for her 18th birthday dinner and a definitely outstanding idea! So if you have the opportunity to try one of those restaurants (in Barcelona Paseo Picasso, London Clerkenwell, New York, Paris Les Halles, or Saint-Petersbourg), I strongly suggest you to make the move. Eating will never feel the same!

Beaulort (Sauternes)

Château Tayac