## Texan black swan

Posted in Books, pictures with tags , , , , , , , , on September 12, 2017 by xi'an

“Un événement improbable aux conséquences d’autant plus désastreuses que l’on ne s’y est pas préparé.”

This weekend, there was a short article in Le Monde about the Harvey storm as a Texan illustration of Taleb’s black swan. An analysis that would imply every extreme event like this “once-in-a-thousand year” event (?) can be called a black swan… “An improbable event with catastrophic consequences, the more because it had not been provisioned”, as the above quote translates. Ironically, there is another article in the same journal, about the catastrophe being “ordinary” and “not unexpected”! While such massive floods are indeed impacting a huge number of people and companies, because the storm happened to pour an unusual amount of rain right on top of Houston, they indeed remain within the predictable and not so improbable in terms of the amount of water deposited in the area and in terms of damages, given the amount and style of construction over flood plains. For instance, Houston is less than 50 feet above sea level, has fairly old drainage and pipe systems, and lacks a zoning code. With mostly one or two-story high buildings rather than higher rises. (Incidentally, I appreciated the juxtaposition of the article with the add for Le Monde des Religions and its picture of a devilesque black goat!)

## book based on a single well-known concept [smbc repost]

Posted in Books, Kids with tags , , , , , on December 26, 2016 by xi'an

## the latest Significance: Astrostats, black swans, and pregnant drivers [and zombies]

Posted in Books, Kids, pictures, Statistics, Travel, University life with tags , , , , , , , , , , on February 4, 2015 by xi'an

Reading Significance is always an enjoyable moment, when I can find time to skim through the articles (before my wife gets hold of it!). This time, I lost my copy between my office and home, and borrowed it from Tom Nichols at Warwick with four mornings to read it during breakfast. This December issue is definitely interesting, as it contains several introduction articles on astro- and cosmo-statistics! One thing I had not noticed before is how a large fraction of the papers is written by authors of books, giving a quick entry or interview about their book. For instance, I found out that Roberto Trotta had written a general public book called the Edge of the Sky (All You Need to Know About the All-There-Is) which exposes the fundamentals of cosmology through the 1000 most common words in the English Language.. So Universe is replaced with All-There-Is! I can understand and to some extent applaud the intention, but it nonetheless makes for a painful read, judging from the excerpt, when researcher and telescope are not part of the accepted vocabulary. Reading the corresponding article in Significance let me a bit bemused at the reason provided for the existence of a multiverse, i.e., of multiple replicas of our universe, all with different conditions: multiplying the universes makes our more likely, while it sounds almost impossible on its own! This sounds like a very frequentist argument… and I am not even certain it would convince a frequentist. The other articles in this special astrostatistics section were of a more statistical nature, from estimating the number of galaxies to the chances of a big asteroid impact. Even though I found the graphical representation of the meteorite impacts in the past century because of the impact drawing in the background. However, when I checked the link to Carlo Zapponi’s website, I found the picture was a still of a neat animation of meteorites falling since the first report.

## on the Piketty craze [not a book review]

Posted in Books, University life with tags , , , , , , on June 15, 2014 by xi'an

The controversy about the data in Piketty’s Capital in the 21st Century fed a dozen articles in the NYT last week, not to mention other newspapers… To the point of Bloomberg producing this parody of a glitzy popular press cover. I have not read the book, nor intend to read it, but I find the buzz and controversies rather surprising. (Thomas Piketty will actually give a public BBC interview in Paris-Dauphine and in English two weeks from now, on June 19.) Indeed, the book sounds to me like a data analysis of extensive (and elaborately processed) datasets rather than the construct of a new economic theory. And I could not see how the criticisms from the Financial Time were anywhere beyond the obvious ideological.  I was thus pleased to read Francis Diebold’s blog on this craze. And to see the repeated acknowledgements that Piketty’s data being made public was a great thing. Although somewhat dismayed that such a painstakenly gathered dataset was not submitted to a more advanced econometric/statistical processing than the one offered by Excel… Definitely looking forward a more [statistically] involved analysis  of the data. (Unsurprisingly, Nassim Taleb reanalysed the whole craze as missing the fat tail features of wealth accumulation. With new aphorisms like “the fastest road to bankruptcy in foreign exchange was an economics degree”.)

## bias in estimating bracketed quantile contributions

Posted in Books, R, Statistics, University life with tags , , , , , , on May 16, 2014 by xi'an

“Vilfredo Pareto noticed that 80% of the land in Italy belonged to 20% of the population, and vice-versa, thus both giving birth to the power law class of distributions and the popular saying 80/20.”

Yesterday, in “one of those” coincidences, I voluntarily dropped Nassim Taleb’s The Bed of Procrustes in a suburban café as my latest contribution to the book-crossing (or bXing!) concept and spotted a newly arXived paper by Taleb and Douadi. Paper which full title is “On the Biases and Variability in the Estimation of Concentration Using Bracketed Quantile Contributions” and which central idea is that estimating

$\kappa_\alpha = \alpha\mathbb{E}[X|X>q_\alpha]\big/\mathbb{E}[X]$

(where qα is the α-level quantile of X) by the ratio

$\sum_{i=1}^n \mathbb{I}_{X_i>\hat{q_\alpha}} X_i \big/ \sum_{i=1}^n X_i$

can be strongly biased. And that the fatter the tail (i.e. the lower the power β for a power law tail), the worse the bias. This is definitely correct, if not entirely surprising given that the estimating ratio involves a ratio of estimators, plus an estimator of qα. And that both numerator and denominator have finite variances when the power β is less than 2.  The paper contains a simulation experiment easily reproduced by the following R code

#biased estimator of kappa(.01)
alpha=.01 #tail
omalpha=1-alpha
T=10^4    #simulations
n=10^3    #sample size
beta=1.1  #Pareto parameter
moobeta=-1/beta

kap=rep(0,T)
for (t in 1:T){
sampl=runif(n)^moobeta
quanta=quantile(sampl,omalpha)
kap[t]=sum(sampl[sampl>quanta])/sum(sampl)
}


What is somewhat surprising though is that the paper deems it necessary to run T=10¹² simulations to assess the bias when this bias is already visible in the first digit of κα. Given that the simulation experiment goes as high as n=10⁸, this means the authors simulated 10²⁰ Pareto variables to exhibit a bias a few thousand replicas could have produced. Checking the numerators and denominators in the above collection of ratios also shows that they may take unbelievably large values.)

“…some theories are built based on claims of such increase’ in inequality, as in Piketti (2014), without taking into account the true nature of κ, and promulgating theories about the variation’ of inequality without reference to the stochasticity of the estimation—and the lack of consistency of κ across time and sub-units.”

The more relevant questions about this issue of estimating κα are, in my opinion, (a) why this quantity is of enough practical importance to consider its estimation and to seek estimators that would remain robust as the power β varies arbitrarily close to 1; (b) in which sense there is anything more to the phenomenon than the difficulty in estimating β itself;  and (c) what is the efficient asymptotic variance for estimating κα (since there is no particular reason to only consider the most natural estimator). Despite the above quote, that the paper constitutes  a major refutation of Piketty’s Capital in the Twenty-First Century is rather unlikely!