Archive for the Books Category

beyond subjective and objective in Statistics

Posted in Books, Statistics, University life with tags , , , , , , , , , , , , , on August 28, 2015 by xi'an

“At the level of discourse, we would like to move beyond a subjective vs. objective shouting match.” (p.30)

This paper by Andrew Gelman and Christian Henning calls for the abandonment of the terms objective and subjective in (not solely Bayesian) statistics. And argue that there is more than mere prior information and data to the construction of a statistical analysis. The paper is articulated as the authors’ proposal, followed by four application examples, then a survey of the philosophy of science perspectives on objectivity and subjectivity in statistics and other sciences, next to a study of the subjective and objective aspects of the mainstream statistical streams, concluding with a discussion on the implementation of the proposed move.

“…scientists and the general public celebrate the brilliance and inspiration of greats such as Einstein, Darwin, and the like, recognizing the roles of their personalities and individual experiences in shaping their theories and discoveries” (p.2)

I do not see the relevance of this argument, in that the myriad of factors leading, say, Marie Curie or Rosalind Franklin to their discoveries are more than subjective, as eminently personal and the result of unique circumstance, but the corresponding theories remain within a common and therefore objective corpus of scientific theories. Hence I would not equate the derivation of statistical estimators or even less the computation of statistical estimates to the extension or negation of existing scientific theories by scientists.

“We acknowledge that the “real world” is only accessible to human beings through observation, and that scientific observation and measurement cannot be independent of human preconceptions and theories.” (p.4)

The above quote reminds me very much of Poincaré‘s

“It is often said that experiments should be made without preconceived ideas. That is impossible. Not only would it make every experiment fruitless, but even if we wished to do so, it could not be done. Every man has his own conception of the world, and this he cannot so easily lay aside.” Henri Poincaré, La Science et l’Hypothèse

The central proposal of the paper is to replace `objective’ and `subjective’ with less value-loaded and more descriptive terms. Given that very few categories of statisticians take pride in their subjectivity, apart from a majority of Bayesians, but rather use the term as derogatory for other categories, I fear the proposal stands little chance to see this situation resolved. Even though I agree we should move beyond this distinction that does not reflect the complexity and richness of statistical practice. As the discussion in Section 2 makes it clear, all procedures involve subjective choices and calibration (or tuning), either plainly acknowledged or hidden under the carpet. Which is why I would add (at least) two points to the virtues of subjectivity:

  1. Spelling out unverifiable assumptions about the data production;
  2. Awareness of calibration of tuning parameters.

while I do not see consensus as necessarily a virtue. The following examples in Section 3 are all worth considering as they bring more details, albeit in specific contexts, to the authors’ arguments. Most of them give the impression that the major issue stands with the statistical model itself, which may be both the most acute subjectivity entry in statistical analyses and the least discussed one. Including the current paper, where e.g. Section 3.4 wants us to believe that running a classical significance test is objective and apt to detect an unfit model. And the hasty dismissal of machine learning in Section 6 is disappointing, because one thing machine learning does well is to avoid leaning too much on the model, using predictive performances instead. Furthermore, apart from Section 5.3, I actually see little in the paper about the trial-and-error way of building a statistical model and/or analysis, while subjective inputs from the operator are found at all stages of this construction and should be spelled out rather than ignored (and rejected).

“Yes, Bayesian analysis can be expressed in terms of subjective beliefs, but it can also be applied to other settings that have nothing to do with beliefs.” (p.31)

The survey in Section 4 about what philosophy of sciences says about objectivity and subjectivity is quite thorough, as far as I can judge, but does not expand enough the issue of “default” or all-inclusive statistical solutions, used through “point-and-shoot” software by innumerate practitioners in mostly inappropriate settings, with the impression of conducting “the” statistical analysis. This false feeling of “the” proper statistical analysis and its relevance for this debate also transpire through the treatment of statistical expertises by media and courts. I also think we could avoid the Heisenberg principle to be mentioned in this debate, as it does not really contribute anything useful. More globally, the exposition of a large range of notions of objectivity is as often the case in philosophy not conclusive and I feel nothing substantial comes out of it… And that it is somehow antagonistic with the notion of a discussion paper, since every possible path has already been explored. Even forking ones. As a non-expert in philosophy, I would not feel confident in embarking upon a discussion on what realism is and is not.

“the subjectivist Bayesian point of view (…) can be defended for honestly acknowledging that prior information often does not come in ways that allow a unique formalization” (p.25)

When going through the examination of the objectivity of the major streams of statistical analysis, I get the feeling of exploring small worlds (in Lindley‘s words) rather than the entire spectrum of statistical methodologies. For instance, frequentism seems to be reduced to asymptotics, while completely missing the entire (lost?) continent of non-parametrics. (Which should not be considered to be “more” objective, but has the advantage of loosening the model specification.) While the error-statistical (frequentist) proposal of Mayo (1996) seems to consume a significant portion [longer than the one associated with the objectivist Bayesianism section] of the discussion with respect to its quite limited diffusion within statistical circles. From a Bayesian perspective, the discussions of subjective, objective, and falsificationist Bayes do not really bring a fresh perspective to the debate between those three branches, apart from suggesting we should give up such value loaded categorisations. As an O-Bayes card-carrying member, I find the characterisation of the objectivist branch somehow restrictive, by focussing solely on Jaynesmaxent solution. Hence missing the corpus of work on creating priors with guaranteed frequentist or asymptotic properties. Like matching priors. I also find the defence of the falsificationist perspective, i.e. of Gelman and Shalizi (2013) both much less critical and quite extensive, in that, again, this is not what one could call a standard approach to statistics. Resulting in an implicit (?) message that this may the best way to proceed.

In conclusion, on the positive side [for there is a positive side!], the paper exposes the need to spell out the various inputs (from the operator) leading to a statistical analysis, both for replicability or reproducibility, and for “objectivity” purposes, although solely conscious choices and biases can be uncovered this way. It also reinforces the call for model awareness, by which I mean a critical stance on all modelling inputs, including priors!, a disbelief that any model is true, applying to statistical procedures Popper’s critical rationalism. This has major consequences on Bayesian modelling in that, as advocated in Gelman and Shalizi (2013) , as well as Evans (2015), sampling and prior models should be given the opportunity to be updated when they are inappropriate for the data at hand. On the negative side, I fear the proposal is far too idealistic in that most users (and some makers) of statistics cannot spell out their assumptions and choices, being unaware of those. This is in a way [admitedly, with gross exaggeration!] the central difficulty with statistics that almost anyone anywhere can produce an estimate or a p-value without ever being proven wrong. It is therefore difficult to perceive how the epistemological argument therein [that objective versus subjective is a meaningless opposition] is going to profit statistical methodology, even assuming the list of Section 2.3 was to be made compulsory. The eight deadly sins listed in the final section would require expert reviewers to vanish from publication (and by expert, I mean expert in statistical methodology), while it is almost never the case that journals outside our field make a call to statistics experts when refereeing a paper. Apart from banning all statistics arguments from a journal, I am afraid there is no hope for a major improvement in that corner…

All in all, the authors deserve a big thank for making me reflect upon those issues and (esp.) back their recommendation for reproducibility, meaning not only the production of all conscious choices made in the construction process, but also through the posting of (true or pseudo-) data and of relevant code for all publications involving a statistical analysis.

The Traitor Spy Trilogy

Posted in Books, Kids with tags , , , on August 23, 2015 by xi'an

“Add the new threat along with a mystery or two. Cook slowly for three novels, increasing temperature slowly. Pay attention. Threats and mysteries tend to disintegrate all of a sudden if you cook them too long after they are resolved, and all you will get is a bland, disappointing mess.” T. Canavan, Orbit Newsletter

When Trudi Canavan published her first (?) trilogy, The Black Magician, I enjoyed it very much. This second trilogy, The Traitor Spy, is a sequel, taking place in the same universe with almost the same characters 20 years later, i.e., one generation later… Recycling the universe (and the cover) of a previous trilogy is fine provided enough novelty is infused into the new series and this is simply not the case here… There are several stories interleaved in the novels, at different locations, with different characters, and I for once regret the classical (at the time of Corneille and Racine) rule for unity of place, time, and action..! Most characters sound incredibly childish and immature all along the three novels, including the senior black mage Sonea who was the heroin of the previous series and had enough of a strong mind to overcome the difficulties that faced her promotion to magician. And to make “black magic” acceptable to the magicians’ community. Here, when she should be a clear leader of this community, she hardly contributes to the major debates taking place in the first volume and does not seem able to argue against changes she does see as prejudicial. The same issue applies to other senior characters, who seem to spend their time wondering about others’ sentiments. And [spoiler alert!] I have not yet mentioned the Traitors’ uprising against the Sachakan regime, which is managed by an handful of individuals, managing to topple an entire society by a single street battle. As quoted above, the author wrote a self-mocking recipe for writing a sequel on the Orbit Newsletter: either she did not follow the recipe properly or there is something fundamentally flawed with the recipe…

STAN [no dead end]

Posted in Books, Statistics, Travel with tags , , on August 22, 2015 by xi'an

stanmoreMichael Betancourt found this street name in London and used it for his talk in Seattle. Even though he should have photoshopped the dead end symbol, which begged for my sarcastic comment during the talk…

Einvígið [book review]

Posted in Books, Kids, Mountains, pictures, Travel with tags , , , , , , , , on August 17, 2015 by xi'an

Reykjavik2In Roissy (De Gaulle) airport, prior to catching my flight to Seattle, I noticed a “new” Indriðason‘s novel, Le Duel (Einvígið), that has not yet been translated into English. But just translated into French! This is a most unusual novel in the Erlendur series, in that the central character of the series only appears as a young cop in the final lines of the novel. Instead, the mentor of Erlendur, Marion Biem, is conducting an inquiry as to who had killed a young man in an almost empty Reykjavik cinema. Where almost all spectators seemed to have something to hide, if not always a murder… A classical whodunnit?! Not really because this happens in 1972, during the famous Fisher-Spassky duel, and that duel is unrelated to the murder, while the Icelandic police seems overwrought by the event and the presence of Russian and American double-agents in Reykjavik…

I found the whole exercise interesting, creating a sort of genealogy in the Erlendur series, with Marion’s mentor playing a side role and his early training in Glasgow (of all places!), with the re-creation of a 1972 Iceland and the chess match between Fisher and Spassky at the height of the Cold War. Plus a reminder about the tuberculosis epidemics of the 1930’s, where  The detective side of the novel is however less convincing than usual, with clues and fingerprints appearing at the most convenient times. And a fairly convoluted resolution. Still worth reading, especially on a long flight!

Saffron and Brimstone [book review]

Posted in Books, Kids, Travel with tags , , , , , , , , on August 15, 2015 by xi'an

I cannot really remember how I came across this book, when selecting Amazon (free) books to collect from Andrew on my last trip to New York… (Thanks to ‘Og readers!) Presumably the name popped out of a list of recommended books. The cover was intriguing enough to stop by and to spot that the author was Elizabeth Hand, whose horror/fantasy trilogy I had liked very much in the late 80’s… So I ordered the book and brought it back from New York. Only to realise that this was an altogether different Elizabeth Hand, whose book Available Dark I had read a little while ago. And did not like so much. However, since the book is a collection of short and less short stories, I gave it a try.

As it happens, this Saffron and Brimstone truly is a great collection of short stories, fantastic in a completely different frame than those of the fantasy books I usually review here. It is a fantastic that borders reality, sometimes hardly fantastic, but with a constant feeling of something being not fully natural, not completely normal. The subtitle of “strange stories” is quite pertinent, as the feeling of strangeness hits the reader (or this reader) almost instantaneously from the beginning of each story. I enjoyed all of the eight stories for different reasons, from a reminiscence of an “Alfred Hitchcock presents” short story called the Cocoon that terrified me [as a pre-teen] when I read it late at night!, to variations around Greek myths that brings them beautifully into the modern era. And always with a central female character who brings another degree of strangeness and surreality to the tale.  I do not think it matters the least that those novels are or are not fantasy or fantastic. They are simply gems of contemporary literature. Superb. (Which makes the rather unexceptional Available Dark the more inexplicable!)

JSM 2015 [day #3]

Posted in Books, Statistics, University life with tags , , , , , , , , , , on August 12, 2015 by xi'an

My first morning session was about inference for philogenies. While I was expecting some developments around Kingman’s  coalescent models my coauthors needed and developped ABC for, I was surprised to see models that were producing closed form (or close enough to) likelihoods. Due to strong restrictions on the population sizes and migration possibilities, as explained later to me by Vladimir Minin. No need for ABC there since MCMC was working on the species trees, with Vladimir Minin making use of [the Savage Award winner] Vinayak Rao’s approach on trees that differ from the coalescent. And enough structure to even consider and demonstrate tree identifiability in Laura Kubatko’s case.

I then stopped by the astrostatistics session as the first talk by Gwendolin Eddie was about galaxy mass estimation, a problem I may actually be working on in the Fall, but it ended up being a completely different problem and I was further surprised that the issue of whether or not the data was missing at random was not considered by the authors.searise3

Christening a session Unifying foundation(s) may be calling for trouble, at least from me! In this spirit, Xiao Li Meng gave a talk attempting at a sort of unification of the frequentist, Bayesian, and fiducial paradigms by introducing the notion of personalized inference, which is a notion I had vaguely thought of in the past. How much or how far do you condition upon? However, I have never thought of this justifying fiducial inference in any way and Xiao Li’s lively arguments during and after the session not enough to convince me of the opposite: Prior-free does not translate into (arbitrary) choice-free. In the earlier talk about confidence distributions by Regina Liu and Minge Xie, that I partly missed for Galactic reasons, I just entered into the room at the very time when ABC was briefly described as a confidence distribution because it was not producing a convergent approximation to the exact posterior, a logic that escapes me (unless those confidence distributions are described in such a loose way as to include about any method f inference). Dongchu Sun also gave us a crash course on reference priors, with a notion of random posteriors I had not heard of before… As well as constructive posteriors… (They seemed to mean constructible matching priors as far as I understood.)

The final talk in this session by Chuanhei Liu on a new approach (modestly!) called inferential model was incomprehensible, with the speaker repeatedly stating that the principles were too hard to explain in five minutes and needed an incoming book… I later took a brief look at an associated paper, which relates to fiducial inference and to Dempster’s belief functions. For me, it has the same Münchhausen feeling of creating a probability out of nothing, creating a distribution on the parameter by ignoring the fact that the fiducial equation x=a(θ,u) modifies the distribution of u once x is observed.

Bayesian model averaging in astrophysics [guest post]

Posted in Books, pictures, Statistics, University life with tags , , , , , , , , , , , , , on August 12, 2015 by xi'an

.[Following my posting of a misfiled 2013 blog, Ewan Cameron told me of the impact of this paper in starting his own blog and I asked him for a guest post, resulting in this analysis, much deeper than mine. No warning necessary this time!]

Back in February 2013 when “Bayesian Model Averaging in Astrophysics: A Review” by Parkinson & Liddle (hereafter PL13) first appeared on the arXiv I was a keen, young(ish) postdoc eager to get stuck into debates about anything and everything ‘astro-statistical’. And with its seemingly glaring flaws, PL13 was more grist to the mill. However, despite my best efforts on various forums I couldn’t get a decent fight started over the right way to do model averaging (BMA) in astronomy, so out of sheer frustration two months later I made my own soapbox to shout from at Another Astrostatistics Blog. Having seen PL13 reviewed recently here on Xian’s Og it feels like the right time to revisit the subject and reflect on where BMA in astronomy is today.

As pointed out to me back in 2013 by Tom Loredo, the act of Bayesian model averaging has been around much longer than its name; indeed an early astronomical example appears in Gregory & Loredo (1992) in which the posterior mean representation of an unknown signal is constructed for an astronomical “light-curve”, averaging over a set of constant and periodic candidate models. Nevertheless the wider popularisation of model averaging in astronomy has only recently taken place through a variety of applications in cosmology: e.g. Liddle, Mukherjee, Parkinson & Wang (2006) and Vardanyan, Trotta & Silk (2011).

In contrast to earlier studies like Gregory & Loredo (1992)—or the classic review on BMA by Hoeting et al. (1999)—in which the target of model averaging is typically either a utility function, a set of future observations, or a latent parameter of the observational process (e.g. the unknown “light-curve” shape) shared naturally by all competing models, the proposal of cosmological BMA studies is to produce a model-averaged version of the posterior for a given ‘shared’ parameter: a so-called “model-averaged PDF”. This proposal didn’t sit well with me back in 2013, and it still doesn’t sit well with me today. Philosophically: without a model a parameter has no meaning, so why should we want to seek meaning in the marginalised distribution of a parameter over an entire set of models? And, practically: to put it another way, without knowing the model ‘label’ to which a given mass of model-averaged parameter probability belongs there’s nothing much useful we can do with this ‘PDF’: nothing much we can say about the data we’ve just analysed and nothing much we can say about future experiments. Whereas the space of the observed data is shared automatically by all competing models, it seems to me to be somehow “un-Bayesian” to place the further restriction that the parameters of separate models share the same scale and topology. I say “un-Bayesian” since this mode of model averaging suggests a formulation of the parameter space + prior pairing stronger than the statement of one’s prior beliefs for the distribution of observable data given the model. But I would be happy to hear arguments from the other side in the comments box below … ! Continue reading

Follow

Get every new post delivered to your Inbox.

Join 903 other followers