Boxplots essentially sum up to three central numbers, plus two extremes, while giving a 2-d impression of spread. Very often, when plotting the histogram of the corresponding data, the resulting impression and comparison of populations is quite different.

]]>There’s a chance of a Peskun ordering in “toy” problems (such as comparing a M-H method with its pseudo-marignal variant) or there is the recurrence time idea of Nichols, Fox and Watt, but I can’t really see that being a successful ranking strategy for even a problem as simple as the type of poisson regression with log-mean modelled by covariates and correlated (CAR) region-based effects that you’d see in disease mapping applications.

Is there a method for comparing MCMC algorithms (or heaven forfend MCMC and non-sampling-based strategies) that doesn’t use simulation studies that I don’t know about? (If it helps, my preferred answer to this question is “yes” for obvious reasons!)

]]>Or the two methods solve slightly different problems (such as problems with different priors).

Or that the comparison is between one method that is designed to solve a specific problem, and one explicitly designed to solve a different problem.

I could go on (and, for that matter, provide references), but it seems pointless.

While I think working out “how to numerically compare general statistical methods” is a REALLY GOOD IDEA, it wouldn’t have fixed any of these.

]]>