Bayesian program synthesis

Last week, I—along with Jean-Michel Marin—got an email from a journalist working for Science & Vie, a French sciences journal that published a few years ago a special issue on Bayes’ theorem. (With the insane title of “the formula that deciphers the World!”) The reason for this call was the preparation of a paper on Gamalon, a new AI company that relies on (Bayesian) probabilistic programming to devise predictive tools. And spent an hour skyping with him about Bayesian inference, probabilistic programming and machine-learning, at the general level since we had not heard previously of this company or of its central tool.

“the Gamalon BPS system learns from only a few examples, not millions. It can learn using a tablet processor, not hundreds of servers. It learns right away while we play with it, not over weeks or months. And it learns from just one person, not from thousands.”

Gamalon claims to do much better than deep learning at those tasks. Not that I have reasons to doubt that claim, quite the opposite, an obvious reason being that incorporating rules and probabilistic models in the predictor is going to help if these rule and models are even moderately realistic, another major one being that handling uncertainty and learning by Bayesian tools is usually a good idea (!), and yet another significant one being that David Blei is a member of their advisory committee. But it is hard to get a feeling for such claims when the only element in the open is the use of probabilistic programming, which is an advanced and efficient manner of conducting model building and updating and handling (posterior) distributions as objects, but which does not enjoy higher predictives abilities by default. Unless I live with a restricted definition of what probabilistic programming stands for! In any case, the video provided by Gamalon and the presentation given by its CEO do not help in my understanding of the principles behind this massive gain in efficiency. Which makes sense given that the company would not want to give up their edge on the competition.

Incidentally, the video in this presentation comparing the predictive abilities of the four major astronomical explanations of the solar system is great. If not particularly connected with the difference between deep learning and Bayesian probabilistic programming.

Bayes on the radio

In relation with the special issue of Science & Vie on Bayes’ formula, the French national radio (France Culture) organised a round table with Pierre Bessière, senior researcher in physiology at Collège de France, Dirk Zerwas, senior researcher in particle physics in Orsay, and Hervé Poirier, editor of Science & Vie. And myself (as I was quoted in the original paper). While I am not particularly fluent in oral debates, I was interested by participating in this radio experiment, if only to bring some moderation to the hyperbolic tone found in the special issue. (As the theme was “Is there a universal mathematical formula? “, I was for a while confused about the debate, thinking that maybe the previous blogs on Stewart’s 17 Equations and Mackenzie’s Universe in Zero Words had prompted this invitation…)

As it happened [podcast link], the debate was quite moderate and reasonable, we discussed about the genesis, the dark ages, and the resurgimento of Bayesian statistics within statistics, the lack of Bayesian perspectives in the Higgs boson analysis (bemoaned by Tony O’Hagan and Dennis Lindley), and the Bayesian nature of learning in psychology. Although I managed to mention Poincaré’s Bayesian defence of Dreyfus (thanks to the Theory that would not die!), Nate Silver‘s Bayesian combination of survey results, and the role of the MRC in the MCMC revolution, I found that the information content of a one-hour show was in the end quite limited, as I would have liked to mention as well the role of Bayesian techniques in population genetic advances, like the Asian beetle invasion mentioned two weeks ago… Overall, an interesting experience, maybe not with a huge impact on the population of listeners, and a confirmation I’d better stick to the written world!

“la formule qui décrypte le monde”

“It is only in the 1980s that the American mathematician Judea Pearl has shown that, by aligning hundreds of Bayes formulas, it was possible to take into account the multiple causes of a complex phenomenon.” (my translation)

As a curious coincidence, the latest issue of Science & Vie appeared on the day I was posting about Peter Coles’s warnings on scientific communication. The cover title of the magazine is the title of this post, The formula decrypting the World, and it is of course about… Bayes’ formula, no-one else’s!!! The major section (16 pages) in this French scientific vulgarization magazine is indeed dedicated to Bayesian statistics and even more Bayesian networks, with the usual stylistic excesses of journalism. As it happens, one of the journalists in charge of this issue came to discuss the topic with me a long while ago in Paris-Dauphine and I remember the experience as being not particularly pleasant since I had trouble communicating the ideas of Bayesian statistics in layman terms. In the end, this rather lengthy interview produced two quotes from me, one that could be mine (in connection with some sentences from Henri Poincaré) and another that is definitely apocryphal (yes, indeed, the one above! I am adamant I could not have mentioned Judea Pearl, whose work I am not familiar with, and even less this bizarre image of hundreds of Bayes’ theorems… Presumably, this got mixed up with a quote from another interviewed Bayesian. The same misquoting occurred for my friend Jean-Michel Marin!).

Among the illustrations selected in the journal as vignettes, the Monty Hall paradox—which is an exercise in conditioning, not in statistical reasoning!—, signal processing for microscope images, Bayesian networks for robots, population genetics (and the return of the musk ox!), stellar cloud formation, tsunami prediction, microarray analysis, climate meta-analysis (with a quote from Noel Cressie), post-Higgs particle physics, ESP studies invalidation by Wagenmakers (missing the fact that the reply by Bern, Utts, and Johnson is equally Bayesian), quantum physics. From a more remote perspective, those are scientific studies using Bayesian statistics to establish important and novel results. However, it would have been easy to come up with equally important and novel results demonstrated via classical non-Bayesian approaches, such as exhibiting the Higgs boson. Now, I understand the difficulty in conveying to the layman the difference resulting from using a Bayesian reasoning to support a scientific argument, however this accumulation of superlatives opens the door to suspicions of bias and truncated perspectives… The second half of the report is less about statistics and more about psychology and learning, expanding on the notion that the brain operates in ways similar to Bayesian learning and networks.