Now, that’s really a recommendation of what not to do if you’re interested in statistical modeling issues regarding mixture models. If instead you’re interested in computational methods for estimating marginal likelihoods, then it’s not a terrible example – the numbers you get are total nonsense from a scientific standpoint, but for a given model and prior, there is a correct mathematical answer, and it’s of some interest to know whether a proposed method gets this correct answer, and if so how fast. A flaw in many such tests, however, is that they don’t actually know what the correct answer is, and hence may not recognize that their method doesn’t get it.

Of course, it would be even better to test computational methods on problems where the answer is of scientific interest. It’s possible that the best method depends on the problem, in which case we’d prefer a method that does well on sensible problems, not one that does well only on ridiculous problems.

]]>