Bayesian learning

“…many well-known learning-algorithms, such as those used in optimization, deep learning, and machine learning in general, can now be derived directly following the above scheme using a single algorithm”

The One World ABC webinar today was delivered by Emtiyaz Khan (RIKEN), about the Bayesian Learning Rule, following Khan and Rue 2021 arXival on Bayesian learning. (It had a great intro featuring a video of the speaker’s daughter learning about the purpose of a ukulele in her first year!) The paper argues about a Bayesian interpretation/version of gradient descent algorithms, starting with Zellner’s (1988, the year I first met him!) identity that the posterior is solution to

when ℓ is the likelihood and π the prior. This identity can be generalised to an arbitrary loss function (also dependent on the data)  replacing the likelihood and considered for a posterior chosen within an exponential family just as variational Bayes. Ending up with a posterior adapted to this target (in the KL sense). The optimal hyperparameter or pseudo-hyperparameter of this approximation can be recovered by some gradient algorithm, recovering as well stochastic gradient and Newton’s methods. While constructing a prior out of a loss function would have pleased the late Herman Rubin, this is not the case, but rater an approach to deriving a generalised Bayes distribution within a parametric family, including mixtures of Gaussians. At some point in the talk, the uncertainty endemic to the Bayesian approach seeped back into the picture, but since most of the intuition came from machine learning, I was somewhat lost at the nature of this uncertainty.

 

 

sample-efficient inference for simulators: complex noise models and time-series [One World ABC seminar]

The next One World ABC seminar will take place next Thursday, 23 Feb, at 9:30 UK time, with a talk by Alexander Aushev, on the above, based on a paper with Tran, Pesonen, Howes, and Kaski:

Simulators are becoming more complex, with their parameter inference requiring as few simulations as possible. This talk will go over two likelihood-free inference (LFI) challenges for computationally intensive simulators. The first challenge is modeling complex simulator noise, which is frequently oversimplified by existing methods or needs far too many simulations. I will discuss how LFI can handle multimodal, non-stationary, and heteroscedastic noise distributions in Bayesian Optimization by using deep Gaussian processes as surrogate models. The second challenge involves simulators in time-series settings, in which the observed time-series data is generated by an unknown stochastic process of simulator parameters. Modern LFI methods, in such cases, either require an accurate model of parameter transition dynamics (e.g. available for sampling) or assume it to be linear. In the last part of the talk, I will discuss the challenges and solutions for performing LFI in such time-series settings, which involve learning the unknown transition dynamics of simulator parameters.

ABC with path signatures [One World ABC seminar, 2/2/23]

The next One World ABC seminar is by Joel Dyer (Oxford) at 1:30pm (UK time) on 02 February.

Title: Approximate Bayesian Computation with Path Signatures

Abstract: Simulation models often lack tractable likelihood functions, making likelihood-free inference methods indispensable. Approximate Bayesian computation (ABC) generates likelihood-free posterior samples by comparing simulated and observed data through some distance measure, but existing approaches are often poorly suited to time series simulators, for example due to an independent and identically distributed data assumption. In this talk, we will discuss our work on the use of path signatures in ABC as a means to handling the sequential nature of time series data of different kinds. We will begin by discussing popular approaches to ABC and how they may be extended to time series simulators. We will then introduce path signatures, and discuss how signatures naturally lead to two instances of ABC for time series simulators. Finally, we will demonstrate that the resulting signature-based ABC procedures can produce competitive Bayesian parameter inference for simulators generating univariate, multivariate, irregularly spaced, and even non-Euclidean sequences.

Reference: J. Dyer, P. Cannon, S. M Schmon (2022). Approximate Bayesian Computation with Path Signatures. arXiv preprint 2106.12555

manifold learning [BNP Seminar, 11/01/23]

An incoming BNP webinar on Zoom by Judith Rousseau and Paul Rosa (U of Oxford), on 11 January at 1700 Greenwich time:

Bayesian nonparametric manifold learning

In high dimensions it is common to assume that the data have a lower dimensional structure. We consider two types of low dimensional structure: in the first part the data is assumed to be concentrated near an unknown low dimensional manifold, in the second case it is assumed to be possibly concentrated on an unknown manifold. In both cases neither the manifold nor the density is known. Atypical example is for noisy observations on an unknown low dimensional manifold.

We first consider a family of Bayesian nonparametric density estimators based on location – scale Gaussian mixture priors and we study the asymptotic properties of the posterior distribution. Our work shows in particular that non conjuguate location-scale Gaussian mixture models can adapt to complex geometries and spatially varying regularity when the density is supported near a low dimensional manifold.

In the second part of the talk we will consider also the case where the distribution is supported on a low dimensional manifold. In this non dominated model,we study different types of posterior contraction rates: Wasserstein and where is the Haussdorff measure on the manifold supporting the density. Some more generic results on Wasserstein contraction rates are also discussed.

 

Adversarial Bayesian Simulation [One World ABC’minar]

The next One World ABC webinar will take place on 24 November, at 1:30 UK Time (GMT) and will be presented by Yi Yuexi Wang (University of Chicago) on “Adversarial Bayesian Simulation”, available on arXiv. [The link to the webinar is available to those who have registered.]

In the absence of explicit or tractable likelihoods, Bayesians often resort to approximate Bayesian computation (ABC) for inference. In this talk, we will cover two summary-free ABC approaches, both inspired by adversarial learning. The first one adopts a classification-based KL estimator to quantify the discrepancy between real and simulated datasets. We consider the traditional accept/reject kernel as well as an exponential weighting scheme which does not require the ABC acceptance threshold. In the second paper, we develop a Bayesian GAN (B-GAN) sampler that directly targets the posterior by solving an adversarial optimization problem. B-GAN is driven by a deterministic mapping learned on the ABC reference by conditional GANs. Once the mapping has been trained, iid posterior samples are obtained by filtering noise at a negligible additional cost. We propose two post-processing local refinements using (1) data-driven proposals with importance reweighting, and (2) variational Bayes. For both methods, we support our findings with frequentist-Bayesian theoretical results and highly competitive performance in empirical analysis. (Joint work with Veronika Rockova)