Peirce also defined “ideal-realism” as “a metaphysical doctrine which combines the principles of idealism and realism.” from http://suo.ieee.org/email/msg13155.html

Not that I have more than just dabbled in this to get some helpful ideas for applying statistics (hopefully) more thoughtfully…

]]>This sounds much more like some kind of idealism or empiricism, than as a special branch of realism… Not that I claim any expertise in philosophy!

]]>Dan: You might be interested in Larry’s post – http://normaldeviate.wordpress.com/2013/08/19/statistics-and-dr-strangelove/

Where I commented –

Though a more direct connection would be Peirce’s theory of realism where as you can’t ever get at what’s “real” or know it you can instead define what an ongoing community of (infinite) investigators would eventually settle on as being what to take as the truth.

But I am not convinced approximation has such a important limitation.

]]>I (conditionally) agree with you that if we can improve our MCMC without limit then there is no issue here. I just don’t think that we can.

There are fundamental problems with “attainable accuracy” in numerical algorithms that, for hard problems, may induce a significant “false accept/reject rate” and therefore lead to an “uncertainty principle” type thing.

This is particularly true for spatial stats (my great and enduring love), where the condition number of the covariance matrix grows polynomially (sometimes exponentially) with the number of locations. This means that it’s not uncommon to get condition number >> 10^15 which is a problem as, pretty much, every digit in the condition number corresponds to one fewer digit of attainable accuracy.

The question is “does this ever dominate the Cramer-Rao-type bounds?” And I’m not sure what the answer is (in non-pathological/limiting cases)

]]>I consider there is a distinction to be made between approximation and inference. While the methods we study are producing samples from approximately the posterior distribution rather than the exact posterior itself, we do not draw inference about the unavailable posterior the same way we try to infer about the distribution of the data or about some small number of parameters of interest. The distinction is that, while we can theoretically improve our MC or MCMC approximation with no limit, hence reduce the MC(MC) variability almost arbitrarily, the error resulting from the data cannot get under the Cramer-Rao bound or its equivalent. Monte Carlo methods add to the noise and error, but I find it hard to call them inference…

]]>The point that I was (clumsily) trying to make is that the inference that we make fundamentally depends on the approximation scheme we use for inference. (And you may still disagree with me on this and you may be right – I’ve been wrong before!)

It is typical to treat MCMC as a “ground truth” for Bayesian stats (especially when comparing with other methods) and the reality is that we are computing things and making statistical decisions based on samples from some other distribution.

In practice, the chains we use cannot even target the correct distribution if we run them to infinity. Think of the rejection step. The day 1 lesson of scientific computing is that that type of decision cannot be implemented exactly in floating point.

So basically, all Bayesian computing is Approximate and it’s probably instructive to treat it that way. Or at least find situations where it doesn’t matter.

For example, making incorrect decisions in the Hastings step doesn’t matter too much *as long as you don’t run the chain for too long*! (In the sense that in stationarity you are coupled to an ergodic Markov chain that targets the correct distribution for, on average O(epsilon^{-1}) steps where epsilon is the error)

So you will make the correct decisions, but it’s a challenge to call it Bayesian in any pure sense because you’re doing it all based on possibly non-convergent approximations.

]]>This is like saying business is business accounting because accounting has to be done.

I did have a group of graduate Epidemiology students once ask me to explain MCMC inference (also known as Bayesian inference in less technical literature).

Today we have to use MCMC for most applications, someday maybe just MC and maybe someday systematic approximation (super smart numerical integration) or at least a mix.

]]>