Archive for legal statistics

Bayes, reproducibility, and the quest for truth

Posted in Books, Kids, Statistics, University life with tags , , , , , , , , , on September 2, 2016 by xi'an

“Avoid opinion priors, you could be held legally or otherwise responsible.”

Don Fraser, Mylène Bedard, Augustine Wong, Wei Lin, and Ailana Fraser wrote a paper to appear in Statistical Science, with the above title. This paper is a continuation of Don’s assessment of Bayes procedures in earlier Statistical Science [which I discussed] and Science 2013 papers, which I would qualify with all due respect of a demolition enterprise [of the Bayesian approach to statistics]…  The argument therein is similar in that “reproducibility” is to be understood therein as providing frequentist confidence assessment. The authors also use “accuracy” in this sense. (As far as I know, there is no definition of reproducibility to be found in the paper.) Some priors are matching priors, in the (restricted) sense that they give second-order accurate frequentist coverage. Most are not matching and none is third-order accurate, a level that may be attained by alternative approaches. As far as the abstract goes, this seems to be the crux of the paper. Which is fine, but does not qualify in my opinion as a criticism of the Bayesian paradigm, given that (a) it makes no claim at frequentist coverage and (b) I see no reason in proper coverage being connected with “truth” or “accuracy”. It truly makes no sense to me to attempt either to put a frequentist hat on posterior distributions or to check whether or not the posterior is “valid”, “true” or “actual”. I similarly consider that Efron‘s “genuine priors” do not belong to the Bayesian paradigm but are on the opposite anti-Bayesian in that they suggest all priors should stem from frequency modelling, to borrow the terms from the current paper. (This is also the position of the authors, who consider they have “no Bayes content”.)

Among their arguments, the authors refer to two tragic real cases: the earthquake at L’Aquila, where seismologists were charged (and then discharged) with manslaughter for asserting there was little risk of a major earthquake, and the indictment of the pharmaceutical company Merck for the deadly side-effects of their drug Vioxx. The paper however never return to those cases and fails to explain in which sense this is connected with the lack of reproducibility or of truth(fullness) of Bayesian procedures. If anything, the morale of the Aquila story is that statisticians should not draw definitive conclusions like there is no risk of a major earthquake or that it was improbable. There is a strange if human tendency for experts to reach definitive conclusions and to omit the many layers of uncertainty in their models and analyses. In the earthquake case, seismologists do not know how to predict major quakes from the previous activity and that should have been the [non-]conclusion of the experts. Which could possibly have been reached by a Bayesian modelling that always includes uncertainty. But the current paper is not at all operating at this (epistemic?) level, as it never ever questions the impact of the choice of a likelihood function or of a statistical model in the reproducibility framework. First, third or 47th order accuracy nonetheless operates strictly within the referential of the chosen model and providing the data to another group of scientists, experts or statisticians will invariably produce a different statistical modelling. So much for reproducibility or truth.

The case of Lucia de Berk

Posted in Statistics, University life with tags , , , , on September 8, 2010 by xi'an

The posting of a paper by Richard Gill, Piet Groeneboom, and Peter de Jong on arXiv today reminded me of a conference of Richard Gill in Ottawa two years ago where he vehemently defended the Dutch nurse Lucia de Berk. (She has been exonerated from all murder accusation this year, after spending several years in jail.) The current paper gives a very simple explanation of the lack of strong (statistical) evidence against this nurse, which makes the earlier conviction based solely on statistical arguments the more puzzling. (As in earlier cases, the fact that the statistical arguments were delivered by a non-statistician is also very surprising, This shows that judges should both get some basic training in Statistics, rather than considering forbidding statistical argument in court, which I think also is the position of the French courts, and that they should involve statisticians as experts.)

Anonymous fame: all wrong!

Posted in Statistics, University life with tags , , on January 30, 2010 by xi'an

Yesterday, the French daily Liberation ran a story about an appeal trial. As I happened to contribute to an expertise about this murder, I went pestering my office neighbours in Dauphine with the story and my two words of “fame” («très probablement»). The funniest thing is that our conclusion is quoted all wrong! We determined (by a simple Bayesian conjugate analysis of a Binomial experiment) that a cell-phone call was very unlikely to have been given from the place the main suspect said she was, given the antenna that got this call, while Liberation [and maybe the court] reports that the suspect was very likely to be at the location of the murder… No wonder when considering that statistics nor probability is taught in Law schools. Nor directly used in trials, as far as I know.

Ps- It is also comes as a surprise to me that this trial is still going on when considering that the murder took place in 1997 and that we sent our report in early 2000.