a football post?!

I am not interested in football, neither as a player (a primary school trauma when I was the last being picked!) or as a fan, contrary to my dad (who was a football referee in his youth) and my kids, but Gareth Roberts (University of Warwick) and Jeff Rosenthal wrote a paper on football draws for the (FIFA) World Cup, infamously playing in Qatar by the end of the year, which Gareth presented in a Warwick seminar.

For this tournament, there are 32 teams, first playing against opponent teams supposedly drawn from a uniform distribution over all draw assignments, within 8 groups of 4 teams, with constraints like 1-2 EU teams per group, 0-1 from the other regions. As done at the moment and on TV, the tournament is filled one team at time by drawing from Pot 1, then Pot 2, then Pot 3, & Pot 4. &tc.. Applying the constraints one draw at a time, conditional on the past draws and the constraints, rather obviously creates non-uniformity! Uniformity would be achievable by rejection sampling (with a success probability of 1/540!) But this is not televisesque enough…

A debiasing solution is found by using several balls for each team in the right proportion, correcting for the sequential draws. Still impractical when requiring 10¹⁴ balls…!

The fun in their paper is that the problem can be formulated as a particle filter, estimating the right probabilities by randomising the number of balls [hidden randomness] and estimating the probability for team j to be included by a few thousands draws. With some stratified sampling on the side to minimise randomness. Removing the need for the (intractable?) distribution is thus achieved by retrospective sampling, as in pseudo-marginal MCMC. Alternatively, one could swap pairs of teams by a simplistic MCMC algorithm, with no worry about stationarity and the possibility of on-screen draws. (Jeff devised a Java applet to simulate an actual draw.) Obviously, it is still a far stretch that this proposal will be implemented for the next World Cup. If so, I will watch it!

a statistic with consequences

In the latest Significance, there was a flyer with some members updates, an important one being that Sylvia Richardson had been elected the next president of the Royal Statistical Society. Congratulations to my friend Sylvia! Another item was that the publication of the 2018 RSS Statistic of the Year has led an Australian water company to switch from plastic to aluminum. Hmm, what about switching to nothing and supporting a use-your-own bottle approach? While it is correct that aluminum cans can be 100% made of recycled aluminum, this water company does not seem to appear to make any concerted effort to ensure its can are made of recycled aluminum or to increase the recycling rate for aluminum in Australia towards achieving those of Brazil (92%) or Japan (86%). (Another shocking statistic that could have been added to the 90.5% non-recycled plastic waste [in the World?] is that a water bottle consumes the equivalent of one-fourth of its contents in oil to produce.) Another US water company still promotes water bottles as one of the most effective and inert carbon capture & sequestration methods”..! There is no boundary for green-washing.

“extremely damaging and slanderous blog article”

Yesterday was a first for the ‘Og in that my university legal department received a complaint from a company about one of the posts, that among other things was considered as an “offending” and “extremely damaging and slanderous commentary” on the services proposed by this company, including posting the unsolicited marketing email it has sent me earlier and which induced this comment of mine.  Written in superb legalese of course. As my point had been made when the blog was posted, a while ago, and as I saw no point in bothering my legal department representatives or wasting further time on such nonsense, I removed this terribly damaging entry and hope the poor dears have recovered by now… It’s “a mere attempt at bloggin’, nothing more”, right?!

“UK outmoded universities must modernise”

[A rather stinky piece in The Guardian today, written by a consultant self-styled Higher Education expert… No further comments needed!]

“The reasons cited for this laggardly response [to innovations] will be familiar to any observer of the university system: an inherently conservative and risk-averse culture in most institutions; sclerotic systems and processes designed for a different world, and a lack of capacity, skills and willingness to change among an ageing academic community. All these are reinforced by perceptions that most proposed innovations are over-hyped and that current ways of operating have plenty of life left in them yet.”

numbersense (book review)

While I got an advance reader’s copy of numbersense, Kaiser Fung’s latest book, sent to me by the publisher McGraw-Hill, I did not managed to write a review until the book had been out for two months. The title of the book is clear enough about the purpose of the author, but the subtitle “How to use Big Data to your advantage” stresses it even further. And includes the sesame “Big Data”, much more likely to appeal to the general reader than “statistics”…!

“I wouldn’t blame you if you are ready to burn this book, and vow never to talk to the lying statisticians ever again.” (p.4)

So why did it take me such a long while to compose this review?! Besides the break induced by The Accident (I took the book to the hospital but ended up reviewing R for Dummies instead!), I figure I got rather taken aback by the style and intended audience of numbersense, given my earlier reading and enjoying Numbers rule your world. While the book remains of interest for statisticians (and other CHANCE readers!), providing examples to use in the classroom, the statistical connection is all but visible to the casual reader who may well conclude that numbersense is a form of numerical common sense of about fighting innumeracy, rather than modelling uncertainty thru statistical models.

“In analyzing data, there is no way to avoid having theoretical assumptions (…) The world has never run out of theoreticians; in the era of Big Data, the bar of evidence is reset lower, making it tougher to tell right from wrong.” (p.11)

Overall, the intended audience of numbersense seems even further away from statistically savy readers than Numbers rule your world. The book is divided into four sections: social data (Chap. 1 & 2), marketing data (Chap. 3-5), economic data (Chap. 6 & 7), and sport data (Chap. 8). Plus a prologue on the Simpson paradox (in marketing), involving Howard Wainer whose Uneducated Guesses: Using Evidence to Uncover Misguided Education Policies I reviewed a while ago. The first (more marketing than social) chapter is about doctoring admission policies against GPA and LSAT scores (whatever that means!) to improve the ranking of a school. This does not sound such a major numerical issue (once the trick is uncovered) and the chapter meanders too much to my taste. The second chapter goes back to Quetelet‘s impossible average man. Asking the reader to question the role of indices in definitions (like obesity). And mentioning the “significant result” bias in medical journals in passing. As well as causality. As in the previous chapter, I finished it waiting for a conclusion that never came. Chapters 3 and 4 focus on Groupon, Without much of a statistical model (except maybe a second-order Simpson paradox?). Chapter 6 is about how companies like Amazon target their suggestions to customers. Not elaborating on the logit or whatever model is behind, though, and drifting aside on the breach of data secrecy by most of “those” companies.  The economics chapters are more to my liking, presumably because they are more standard, covering the subtleties of unemployment and inflation (official) statistics. They fall into what I call the Gini index branch of statistics. At last, the sport chapter is about fantasy football (FF) and not about Moneyball (even though it has links, obviously). I did not go father than a quick perusal at the chapter as I did not understand (most of) the point of the chapter (or of playing FF). For instance, the conclusion seemed quite distanced from the actual story…

“today’s computers do not understand languages. All they  do is match text: they can tell me whether the words “empirical Bayes model” are found on a specific Web page.” (p. 209)

The epilogue is of a different nature as it describes two examples of the tasks undertaken by Kaiser Fung as a data analyst. A nasty data transfer. And a manual classification of some Google queries. This may be the part of numbersense that I enjoyed the most. Again, let me stress I have no scientific complaint about the book: it just sounds too low-tech’ for my taste. And I find it is not helping readers to go beyond the first level of scepticism about raw and processed data. Because they are not data-analysts.