Error and Inference [#2]

(This is the second post on Error and Inference, again being a raw and naive reaction to a linear reading rather than a deeper and more informed criticism.)

“Allan Franklin once gave a seminar under the title `Ad Hoc is not a four letter word.'”—J. Worrall, p.130, Error and Inference, 2010

The fourth chapter of Error and Inference, written by John Worrall, covers the highly interesting issue of “using the data twice”. The point has been debated several times on Andrew’s blog and this is one of the main criticisms raised against Aitkin’s posterior/integrated likelihood. Worrall’s perspective is both related and unrelated to this purely statistical issue, when he considers that “you can’t use the same fact twice, once in the construction of a theory and then again in its support” (p.129). (He even signed a “UN Charter”, where UN stands for “use novelty”!) After reading both Worrall’s and Mayo’s viewpoints,  the later being that all that matters is severe testing as it encompasses the UN perspective (if I understood correctly), I afraid I am none the wiser, but this led me to reflect on the statistical issue.

From first principles, a Bayesian approach should use the data only once, namely when constructing the posterior distribution on every unknown component of the model(s).  Given this all-encompassing posterior, all inferential aspects are the consequences of a sequence of decision-theoretic steps in order to select optimal procedures. This is the ideal setting while, in practice,  relying on a sequence of posterior distributions is often necessary, each posterior being a consequence of earlier decisions, which makes it the result of a multiple use of the data… For instance, the process of Bayesian variable selection is on principle clean from the sin of “using the data twice”: one simply computes the posterior probability of each of the variable subsets and this is over. However, in a case involving many (many) variables, there are two difficulties: one is about building the prior distributions for all possible models, a task that needs to be automatised to some extent; another is about exploring the set of potential models. Resorting to projection priors as in the intrinsic solution of Pèrez and Berger (2002, Biometrika, a much valuable article!), while unavoidable and a “least worst” solution, means switching priors/posteriors based on earlier acceptances/rejections, i.e. on the data. Second, the path of models truly explored [which will be a minuscule subset of the set of all models] will depend on the models rejected so far, either when relying on a stepwise exploration or when using a random walk MCMC algorithm. Although this is not crystal clear (there is actually plenty of room for arguing the opposite!), it could be argued that the data is thus used several times in this process…

“Although the [data] set as a whole both fixes parameter values and (unconditionally) supports, no particular element of the data set does both.“—J. Worrall, p.140, Error and Inference, 2010

One paragraph in Worrall’s discussion intersects with the previous post [#1], while getting away from the “using the data twice” discussion. It compares two theories with a different number of “free” parameters, hence (seemingly) getting different amounts of support from a given dataset (“n lots of confirmation [versus] n-r lots”, p. 140). The distinction sounds a wee too arithmetic in that the algebraic dimension of a parameter may be only indirectly related to its information dimension, as illustrated by DIC.  Other than that, a notion like the fractional Bayes factor of Tony O’Hagan shows that the whole dataset may contribute both to the model selection and to the parameter estimation without the above dichotomy to occur.

Ps-There is an interesting workshop taking place in Madrid next december 15-16 on The Controversy About Hypothesis Testing, involving José Bernardo and Aris Spanos among others.