I had missed these news that a frozen portion of the Niagara Falls had been ice-climbed. By Will Gadd on Jan. 27. This is obviously quite impressive given the weird and dangerous nature of the ice there, which is mostly frozen foam from the nearby waterfall. (I once climbed an easy route on such ice at the Chutes Montmorency, near Québec City, and it felt quite strange…) He even had a special ice hook designed for that climb as he did not trust the usual ice screws. Will Gadd has however climbed much more difficult routes like Helmcken Falls in British Columbia, which may be the hardest mixed route in the World!
It may be that weekends are the wrong time to tamper with computer OS… Last Sunday, I noticed my Bluetooth icon had a “turn off” option and since I only use Bluetooth for my remote keyboard and mouse when in Warwick, I turned it off, thinking I would turn it on again next week. This alas led to a series of problems, maybe as a coincidence since I also updated the Kubuntu 14.04 system over the weekend.
- I cannot turn Bluetooth on again! My keyboard and mouse are no longer recognised or detected. No Bluetooth adapter is found by the system setting. Similarly, sudo modprobe bluetooth shows nothing. I have installed a new interface called Blueman but to no avail. The fix suggested on forums to run rfkill unblock bluetooth does not work either… Actually rfkill list all only returns the wireless device. Which is working fine.
- My webcam vanished as well. It was working fine before the weekend.
- Accessing some webpages, including all New York Times articles, now takes forever on Firefox! If less on Chrome.
Is this a curse of sorts?!
As an aside, I also found this week that I cannot update Adobe reader from version 9 to version 11, as Adobe does not support Linux versions any more… Another bummer. If one wants to stick to acrobat.
Thanks to Ingmar and Thomas, I got both my problems solved! The Bluetooth restarted after I shut down my unplugged computer, in connection with an USB over-current protection. And Thomas figured out my keyboard had a key to turn the webcam off and on, key that I had pressed when trying to restart the Bluetooth device. Et voilà!
কিন্তু আমরা অপরাজিত
[Here is a reply by Heiko Strathmann to my post of yesterday. Along with the slides of a talk in Oxford mentioned in the discussion.]
Thanks for putting this up, and thanks for the discussion. Christian, as already exchanged via email, here are some answers to the points you make.
First of all, we don’t claim a free lunch — and are honest with the limitations of the method (see negative examples). Rather, we make the point that we can achieve computational savings in certain situations — essentially exploiting redundancy (what Michael called “tall” data in his note on subsampling & HMC) leading to fast convergence of posterior statistics.
Dan is of course correct noticing that if the posterior statistic does not converge nicely (i.e. all data counts), then truncation time is “mammoth”. It is also correct that it might be questionable to aim for an unbiased Bayesian method in the presence of such redundancies. However, these are the two extreme perspectives on the topic. The message that we want to get along is that there is a trade-off in between these extremes. In particular the GP examples illustrate this nicely as we are able to reduce MSE in a regime where posterior statistics have *not* yet stabilised, see e.g. figure 6.
“And the following paragraph is further confusing me as it seems to imply that convergence is not that important thanks to the de-biasing equation.”
To clarify, the paragraph refers to the additional convergence issues induced by alternative Markov transition kernels of mini-batch-based full posterior sampling methods by Welling, Bardenet, Dougal & co. For example, Firefly MC’s mixing time is increased by a factor of 1/q where q*N is the mini-batch size. Mixing of stochastic gradient Langevin gets worse over time. This is not true for our scheme as we can use standard transition kernels. It is still essential for the partial posterior Markov chains to converge (if MCMC is used). However, as this is a well studied problem, we omit the topic in our paper and refer to standard tools for diagnosis. All this is independent of the debiasing device.
About MCMC convergence.
Yesterday in Oxford, Pierre Jacob pointed out that if MCMC is used for estimating partial posterior statistics, the overall result is not unbiased. We had a nice discussion how this bias could be addressed via a two-stage debiasing procedure: debiasing the MC estimates as described in the “Unbiased Monte Carlo” paper by Agapiou et al, and then plugging those into the path estimators — though it is (yet) not so clear how (and whether) this would work in our case.
In the current version of the paper, we do not address the bias present due to MCMC. We have a paragraph on this in section 3.2. Rather, we start from a premise that full posterior MCMC samples are a gold standard. Furthermore, the framework we study is not necessarily linked to MCMC – it could be that the posterior expectation is available in closed form, but simply costly in N. In this case, we can still unbiasedly estimate this posterior expectation – see GP regression.
“The choice of the tail rate is thus quite delicate to validate against the variance constraints (2) and (3).”
It is true that the choice is crucial in order to control the variance. However, provided that partial posterior expectations converge at a rate n-β with n the size of a minibatch, computational complexity can be reduced to N1-α (α<β) without variance exploding. There is a trade-off: the faster the posterior expectations converge, more computation can be saved; β is in general unknown, but can be roughly estimated with the “direct approach” as we describe in appendix.
About the “direct approach”
It is true that for certain classes of models and φ functionals, the direct averaging of expectations for increasing data sizes yields good results (see log-normal example), and we state this. However, the GP regression experiments show that the direct averaging gives a larger MSE as with debiasing applied. This is exactly the trade-off mentioned earlier.
I also wonder what people think about the comparison to stochastic variational inference (GP for Big Data), as this hasn’t appeared in discussions yet. It is the comparison to “non-unbiased” schemes that Christian and Dan asked for.
“Data complexity is sub-linear in N, no bias is introduced, variance is finite.”
Heiko Strathman, Dino Sejdinovic and Mark Girolami have arXived a few weeks ago a paper on the use of a telescoping estimator to achieve an unbiased estimator of a Bayes estimator relying on the entire dataset, while using only a small proportion of the dataset. The idea is that a sequence converging—to an unbiased estimator—of estimators φt can be turned into an unbiased estimator by a stopping rule T:
is indeed unbiased. In a “Big Data” framework, the components φt are MCMC versions of posterior expectations based on a proportion αt of the data. And the stopping rule cannot exceed αt=1. The authors further propose to replicate this unbiased estimator R times on R parallel processors. They further claim a reduction in the computing cost of
which means that a sub-linear cost can be achieved. However, the gain in computing time means higher variance than for the full MCMC solution:
“It is clear that running an MCMC chain on the full posterior, for any statistic, produces more accurate estimates than the debiasing approach, which by construction has an additional intrinsic source of variance. This means that if it is possible to produce even only a single MCMC sample (…), the resulting posterior expectation can be estimated with less expected error. It is therefore not instructive to compare approaches in that region. “
I first got a “free lunch” impression when reading the paper, namely it sounded like using a random stopping rule was enough to overcome unbiasedness and large size jams. This is not the message of the paper, but I remain both intrigued by the possibilities the unbiasedness offers and bemused by the claims therein, for several reasons: Continue reading
When in Warwick last October, I met Simo Särkkä, who told me he had published an IMS monograph on Bayesian filtering and smoothing the year before. I thought it would be an appropriate book to review for CHANCE and tried to get a copy from Oxford University Press, unsuccessfully. I thus bought my own book that I received two weeks ago and took the opportunity of my Czech vacations to read it… [A warning pre-empting accusations of self-plagiarism: this is a preliminary draft for a review to appear in CHANCE under my true name!]
“From the Bayesian estimation point of view both the states and the static parameters are unknown (random) parameters of the system.” (p.20)
Bayesian filtering and smoothing is an introduction to the topic that essentially starts from ground zero. Chapter 1 motivates the use of filtering and smoothing through examples and highlights the naturally Bayesian approach to the problem(s). Two graphs illustrate the difference between filtering and smoothing by plotting for the same series of observations the successive confidence bands. The performances are obviously poorer with filtering but the fact that those intervals are point-wise rather than joint, i.e., that the graphs do not provide a confidence band. (The exercise section of that chapter is superfluous in that it suggests re-reading Kalman’s original paper and rephrases the Monty Hall paradox in a story unconnected with filtering!) Chapter 2 gives an introduction to Bayesian statistics in general, with a few pages on Bayesian computational methods. A first remark is that the above quote is both correct and mildly confusing in that the parameters can be consistently estimated, while the latent states cannot. A second remark is that justifying the MAP as associated with the 0-1 loss is incorrect in continuous settings. The third chapter deals with the batch updating of the posterior distribution, i.e., that the posterior at time t is the prior at time t+1. With applications to state-space systems including the Kalman filter. The fourth to sixth chapters concentrate on this Kalman filter and its extension, and I find it somewhat unsatisfactory in that the collection of such filters is overwhelming for a neophyte. And no assessment of the estimation error when the model is misspecified appears at this stage. And, as usual, I find the unscented Kalman filter hard to fathom! The same feeling applies to the smoothing chapters, from Chapter 8 to Chapter 10. Which mimic the earlier ones. Continue reading