One is a pointer to stochastic gradient descent estimators that leverage stochastic approximations a la Robins & Monro (1951), and related research on proximal methods. My student Panos Toulis and I have a recent paper in this area: http://arxiv.org/abs/1408.2923. Our angle is to introduce and explore “implicit” stochastic gradient descent estimators. What people may find useful is the discussion of the broader literature in Section 2.5. A notable paper in this area is here: http://arxiv.org/abs/1306.2119, by Bach and Moulines.

The other pointer is to a paper by Michael Jordan that appeared in Bernoulli (http://projecteuclid.org/euclid.bj/1377612856) where he summarizes some examples of strategies to scale inference methods to massive datasets. You may also find of interest slides of a talk of his at the 2014 IMS New Researchers Conference (http://www.stat.harvard.edu/NRC2014/MichaelJordan.pdf), where he explores conceptual and mathematical challenges that arise in big data settings. He concludes that facing these challenges will require a rapprochement between computer science and statistics, bringing them together at the level of their foundations, thus reshaping both disciplines.

I look forward to reading your survey paper!

]]>