Archive for E.T. Jaynes

Jaynes’ re-read

Posted in Books, Statistics, University life with tags , , , , , , , on March 21, 2011 by xi'an

On many technical issues we disagree strongly with de Finetti. It appears to us that his way of treating infinite sets has opened up a Pandora’s box of useless and unecessary paradoxes.”  E.T. Jaynes, PT, p.xxi

On Friday, despite the cancellation of the reading seminar on Jaynes’ Probability Theory, I completed my slides on Chapters 4 (Elementary hypothesis testing) to 14 (Simple applications of decision theory), plus of course Chapter 20 (Model comparison). I skipped Chapter 15 (Paradoxes of probability theory), despite its extensive and exciting coverage of the marginalisation paradoxes which saw Jaynes opposing David, Stone, and Zidek (and even the whole Establishment, page 470), as it would have taken me another morning at the very least… (Next year, maybe, if the seminar resumes?!)

Continue reading

Bertand’s paradox [R details]

Posted in Books, R, Statistics with tags , , , , , , , on March 20, 2011 by xi'an

Some may have had reservations about the “randomness” of the straws I plotted to illustrate Bertrand’s paradox. As they were all going North-West/South-East. I had actually made an inversion between cbind and rbind in the R code, which explained for this non-random orientation. Above is the corrected version, which sounds “more random” indeed. (And using wheat as the proper, if weak, colour!) The outcome of a probability of 1/2 has not changed, of course. Here is the R code as well:


for (t in 1:10^3){

 #distance from O to chord

 while (dchord>1){
 #Generate "random" straw in large box till it crosses unit circle


 #endpoints outside the circle
 if ((sum(a^2)>1)&&(sum(b^2)>1)){


 #chord inside
 if (max(abs(theta),abs(thetb))<pi/2)

 if (runif(1)<.1) lines(rbind(a,b),col="wheat")


As a more relevant final remark, I came to the conclusion (this morning while running) that the probability of this event can be anything between 0 and 1, rather than the three traditional 1/4, 1/3 and 1/2. Indeed, for any distribution of the “random” straws, hence for any distribution on the chord length L, a random draw can be expressed as L=F⁻¹(U), where U is uniform. Therefore, this draw is also an acceptable transform of a uniform draw, just like Bertrand’s three solutions.

Jaynes’ seminar cancelled!

Posted in Statistics with tags , , on March 17, 2011 by xi'an

The reading seminar on Jaynes’ Probability Theory that was planned to start on next Monday at CREST. is cancelled for lack of enough registered students. I am thus alas forced to cancel it, although I had now reached the most interesting sections of the book. I think I will finish reading it and writing my slides in preparation for next year when I plan to propose the reading seminar in conjunction with Massimiliano Gubinelli from Dauphine who would then provide a probabilist look at the book. The slides will thus get updated as I proceed through:

I am obviously sorry this seminar did not attract more students but neither this is a major disaster—especially when considering the dire events that took place this week in Japan—as it led me to read Jaynes’ Probability Theory carefully.

Bertrand’s paradox

Posted in Books, Statistics, Travel, University life with tags , , , on March 17, 2011 by xi'an

While progressing through Jaynes’ Probability Theory for my classes next week, I reached the end of Chapter 12 where he proposed a “resolution” of Bertrand’s paradox. Jaynes gives the following description of the paradox:

Bertrand’s problem was stated originally in terms of drawing a straight line `at random’ intersecting a circle (…) we do no violence to the problem if we suppose we are tossing straws onto the circle (…) What is the probability that the chord thus defined [by a random straw] is longer than the side of the inscribed equilateral triangle?

My understanding of the paradox is that it provides a perfect illustration of the lack of meaning of “random” and of the need for a proper definition of the σ-algebra leading to a probabilised space. Different σ-algebras lead to different probabilities, e.g., 1/4, 1/3, 1/2… However, Jaynes considers there is a “correct” answer and endeavours to construct an invariant distribution on the location of the centre of the chord, achieving Borel’s distribution

p(x) = \frac{x}{\sqrt{1-x^2}}

on the chord proportion L/2R. When illustrating this approach, I tried to bring an empirical vision and generated “random” straws by picking both ends at random on the (-10,10)^2 square. Here is a subsample of 10³ such straws interesting with the unit circle. The empirical distribution of the chord proportion is actually quite in agreement with Borel’s distribution, with a probability of being longer than the side of 1/2, as shown below but this does not validate Jaynes’ argument, simply illustrates that I picked the same σ-algebra as his. (Every σ-algebra considered by Bertrand could as well have been used for the simulation.)

Reading seminar on Jaynes’ Probability Theory

Posted in Statistics with tags , , on March 14, 2011 by xi'an

The reading seminar on Jaynes’ Probability Theory is planned to start on next Monday at CREST. (All welcome, just register with Nadine Guedj at I have so far covered Chapters 4-8 and I need to speed up if I want to keep my schedule. So far, reading carefully through the book has induced neither a burst of criticisms (as it occurred with Keynes’ Treatise on Probability) since there is no strong departure from traditional Bayesian inference, nor a foundational shift in my understanding of the field. The chapter on testing is quite well-written and starting with testing rather than estimation is a brilliant idea. The only annoying part is the permanent bashing of measure theory, which leads to an informal processing of improper priors. Here are the first slides written up to now:

Jaynes’ re-read

Posted in Books, Statistics, University life with tags , , , on July 2, 2010 by xi'an

Following the two past PhD courses on JeffreysTheory of Probability and KeynesTreatise On Probability, I will propose next year a reading course at CREST on Jaynes’s Probability Theory.

Jeffreys and Jaynes share a lot in common as physicists who both significantly contributed to Bayesian statistical theory and as writers of books with almost identical titles and with very ambitious and similar scopes. It is thus no surprise that Jaynes dedicates his book to Jeffreys. There are also differences, the most obvious one being that Jeffreys published his foundational book before his 50th birthday, while Jaynes’ book came out more than ten years after his death (under the scholarly supervision of Larry Bretthorst). The time difference between both books is not that revelant, however, in that Jaynes’s Probability Theory is what Persi Diaconis call “wonderfully out of date” in his review. (Meaning “Jaynes’s focus on basic philosophical issues [rather than on computational techniques] is timeless“, with the conclusion that  “this is the best introduction to Bayesian statistics that I know”!)

I plan to cover in the lectures what I consider to be the most significant aspects of Jaynes’s work. The corpus of work corresponding to the logical foundations of probability theory and the opposition of Jaynes to (Feller’s) measure theory, Bourbakism, Kolmogorov’s axioms, (Feller’s) countable additivity, de Finetti’s principles, and other probabilistic paradoxes will not be adressed, even though a second course by a probabilist colleague of mine at Dauphine may follow this one. The lectures will focus on

  1. the meaning and motivation of prior distributions (Chapter 6), culminating in the definition of the entropy principle (Chapter 11)
  2. the rules of hypothesis testing (Chapter 4) and the central role of evidence (Chapters 9 and 18)
  3. the special case of transformation groups (Chapter 12) and the debate about marginalisation paradoxes (Chapter 15)
  4. Bayesian estimation (Chapter 6) and the criticisms on decision theory (Chapters 13 and 14)
  5. Model comparison (Chapter 20) and the pathologies of orthodox methods (Chapters 16 and 17)

The dates of the course are already set: March 21, 24, 28, 31 and April 04 [2011]…

“Not only defended but also applied”

Posted in Statistics with tags , , , , on June 30, 2010 by xi'an

On page 124 of his superb Introduction to Probability Theory book (volume 1), William Feller has this strange remark about Bayesian inference:

“Unfortunately Bayes’ rule has been somewhat discredited by metaphysical applications of the type described above. In routine practice, this kind of argument can be dangerous. A quality control engineer is concerned with one particular machine and not with an in nite population of machines from which one was chosen at random. He has been advised to use Bayes’ rule on the grounds that it is logically acceptable and corresponds to our way of thinking. Plato used this type of argument to prove the existence of Atlantis, and philosophers used it to prove the absurdity of Newton’s mechanics. In our case it overlooks the circumstance that the engineer desires success and that he will do better by estimating and minimizing the sources of various types of errors in predicting and guessing.The modern method of statistical tests and estimation is less intuitive but more realistic. It may be not only defended but also applied.”

When we were discussing about this great book, Andrew Gelman pointed out to me this strong dismissal of Bayesian techniques (note that I had overlooked so far) and, given that it is still quoted as an argument against a Bayesian approach to inference, we ended up writing [well, mostly Andrew!] a short note on the motivations and implications of this remark, now published on arXiv. One of the points is that Feller’s sentence has the interesting feature that it is actually the opposite of the usual demarcation: typically it is the Bayesian who makes the claim for inference in a particular instance and the frequentist who restricts claims to infinite populations of replications. Another point is the naïve faith in the classical Neyman-Pearson theory to solve practical problems in statistics.

Actually, Persi Diaconis took a (deeper) look at Feller’s stance as well, as mentioned in this review of Jaynes’s Probability Theory. Using Amazon Look Inside tool,  I spotted Feller being mentioned more than 30 times in Jaynes’s book, one of the best quotes being “The date was 1956 when the author met Willy Feller“! More to the point, Jaynes identifies Feller’s dismissal of the “old wrong ways” (volume 2, p.76), which is to be opposed to the “modern method” above. (Persi Diaconis and Susan Holmes also wrote a nice piece entitled “A Bayesian peek into Feller volume 1” that does not relate directly to this issue.) In a loosely related point, Persi’s warning that he sees “a strong trend against measure theory in modern statistics departments: [he] had to fight to keep the measure theory requirement in Stanford’s statistics graduate program“, to which I completely subscribe, should be heard more widely…


Get every new post delivered to your Inbox.

Join 980 other followers