ABC in nine steps

Bertorelle, Benazzo and Mona have published a nice survey about ABC in Molecular Ecology. It is available on-line and can be downloaded. The nine steps are summarised in the above graph. Step 6 corresponds to ABC model selection, but Step 7 follows the suggestion found in DIYABC to generate pseudo-observed data (pods) to evaluate the “type I – type II” errors of the procedure. In Step 9, a second quality control is proposed via ABC posterior predictive tests. The survey also makes an interesting link with an early version of ABC published by Weiss and von Haessler (1998, Genetics).

My only criticisms are, beyond the lack of confidence in ABC model selection, about an unclear discussion of the “controversial results” about ABC-MCMC and ABC-PMC.  The fact that reference tables as those produced by DIYABC can be recycled with “limited additional effort” is however a good point in favour of Monte Carlo evaluations of estimation and model selection procedures.

One Response to “ABC in nine steps”

  1. Christopher Drummond Says:

    Agreed the Bertorelle et al. paper is a nice review… but the use of posterior predictive simulations seems more useful for evaluating the quality of parameter estimation rather than model selection. I’ve been considering these as two very different processes, although Cornuet et al. 2010 (Table 1) note that unsupported models have a higher frequency of extreme p-values.

    There could be stark differences in posterior predictive results among models, but I suspect in many cases those differences will be difficult to interpret on their own… and anyway should be reflected in the initial evaluation of models using something like multinomial logistic regression, since a scenario that consistently generates summary statistics farther from the observed data will be downweighted in the rejection step during model selection.

    Also unsure about the borrowed frequentist terminology of ‘type I’ and ‘type II’ error for model selection. Maybe it is better to think of false positives and false negatives with respect to each ‘true’ model based on pseudo-observed data from prior predictive simulations, rather than invoking a null-hypothesis framework?

    Regarding parameter estimation, graphical inspection of posterior predictive tests (comparing simulated vs. observed summary statistics) at least permits some judgment of whether a particular model can plausibly generate the observed data. But the predictive coverage (e.g., proportion of posterior predictive pseudo-observations that fall within the 90% HPD estimate for each pod simulation) seems more helpful.

    Likewise, the uniformity of posterior predictive quantiles illustrated in Wegmann et al. 2010 etc. could facilitate more nuanced evaluation of the posterior shape. For example, one might find that posterior distributions are biased toward larger or smaller values, but still provide an adequate estimate of credibility intervals (e.g., ca. 90% of pods fall within their estimated 90% HPDs)… or even a conservative estimate (e.g., ca. 95% of pods within their estimated 90% HPDs).

    Best,
    Chris

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.