Numerical analysis for statisticians

“In the end, it really is just a matter of choosing the relevant parts of mathematics and ignoring the rest. Of course, the hard part is deciding what is irrelevant.”

Somehow, I had missed the first edition of this book and thus I started reading it this afternoon with a newcomer’s eyes (obviously, I will not comment on the differences with the first edition, sketched by the author in the Preface). Past the initial surprise of discovering it was a mathematics book rather than an algorithmic book, I became engrossed into my reading and could not let it go! Numerical Analysis for Statisticians, by Kenneth Lange, is a wonderful book. It provides most of the necessary background in calculus and some algebra to conduct rigorous numerical analyses of statistical problems. This includes expansions, eigen-analysis, optimisation, integration, approximation theory, and simulation, in less than 600 pages. It may be due to the fact that I was reading the book in my garden, with the background noise of the wind in tree leaves, but I cannot find any solid fact to grumble about! Not even about  the MCMC chapters! I simply enjoyed Numerical Analysis for Statisticians from beginning till end.

“Many fine textbooks (…) are hardly substitutes for a theoretical treatment emphasizing mathematical motivations and derivations. However, students do need exposure to real computing and thoughtful numerical exercises. Mastery of theory is enhanced by the nitty gritty of coding.” 

From the above, it may sound as if Numerical Analysis for Statisticians does not fulfill its purpose and is too much of a mathematical book. Be assured this is not the case: the contents are firmly grounded in calculus (analysis) but the (numerical) algorithms are only one code away. An illustration (among many) is found in Section 8.4: Finding a Single Eigenvalue, where Kenneth Lange shows how the Raleigh quotient algorithm of the previous section can be exploited to this aim, when supplemented with a good initial guess based on Gerschgorin’s circle theorem. This is brilliantly executed in two pages and the code is just one keyboard away. The EM algorithm is immersed into a larger M[&]M perspective. Problems are numerous and mostly of high standards, meaning one (including me) has to sit and think about them. References are kept to a minimum, they are mostly (highly recommended) books, plus a few papers primarily exploited in the problem sections. (When reading the Preface, I found that “John Kimmel, [his] long suffering editor, exhibited extraordinary patience in encouraging [him] to get on with this project”. The quality of Numerical Analysis for Statisticians is also a testimony to John’s editorial acumen!)

“Every advance in computer architecture and software tempts statisticians to tackle numerically harder problems. To do so intelligently requires a good working knowledge of numerical analysis. This book equips students to craft their own software and to understand the advantages and disadvantages of different numerical methods. Issues of numerical stability, accurate approximation, computational complexity, and mathematical modeling share the limelight in a broad yet rigorous overview of those parts of numerical analysis most relevant to statisticians.”

While I am reacting so enthusiastically to the book (imagine, there is even a full chapter on continued fractions!), it may be that my French math background is biasing my evaluation and that graduate students over the World would find the book too hard. However, I do not think so: the style of Numerical Analysis for Statisticians is very fluid and the rigorous mathematics are mostly at the level of undergraduate calculus. The more advanced topics like wavelets, Fourier transforms and Hilbert spaces are very well-introduced and do not require prerequisites in complex calculus or functional analysis. (Although I take no joy in this, even measure theory does not appear to be a prerequisite!) On the other hand, there is a prerequisite for a good background in statistics. This book will clearly involve a lot of work from the reader, but the respect shown by Kenneth Lange to those readers will sufficiently motivate them to keep them going till assimilation of those essential notions. Numerical Analysis for Statisticians is also recommended for more senior researchers and not only for building one or two courses on the bases of statistical computing. It contains most of the math bases that we need, even if we do not know we need them! Truly an essential book.

15 Responses to “Numerical analysis for statisticians”

  1. […] Lange arXived a paper on a comparison of numerical techniques for solving sudokus. (The very Kenneth Lange who wrote this fantastic book on numerical analysis.) One of these techniques is the simulated […]

  2. […] Kenneth Lange’s Numerical Analysis for Statisticians, enthusiastically reviewed in a post of Aug. 26 […]

  3. […] reads really well, even though I am missing references. And even though it cannot be read under my cherry tree (esp. now that weather has moved from été to étaumne… as I heard this morning on the […]

  4. Dan Simpson Says:

    Judging solely from the table of contents, I’m a bit disappointed by the topics included. It may be a poor table of contents, but it looks suspiciously like there isn’t a linear algebra method that’s less then (optimistically) 30 years old developed in this book. I’m thinking particularly of methods based on Krylov subspaces, which are (and have been for a LONG LONG time) usually used in place of or to accelerate the iterative methods mentioned here. They are arguably more fundamental and more useful than wavelet expansions, as a tiny bit of theory gets you linear solvers, eigensolvers, svd methods and, particularly, a numerically stable formulation of partial least squares. I find it hard to justify leaving things like this out (especially in favor of worse methods) in a book that doesn’t seem afraid of mathematical complexity.

    • Kenneth Lange mentions in his introduction he keeps learning. So I am pretty sure that, if you mention the gaps to him, he will strive to include the better and more modern methods in his third edition. To me having wavelets included in such a general purpose book is a clear plus!

    • Dan Simpson Says:

      I agree with you on wavelets (although I’ve never had occasion to use them, I am fond of them). I do hope that the book emphasises the importance of ‘hard’ estimates in computational statistics (it’s not enough to just say that something converges, you need to know both how fast and whether it’s stable). Convergence in probability is lovely, but a bit vague (not a pun on modes of convergence) for practical purposes.

      And I really hope there is a third edition, because there is a sore need for this sort of book. I’m particularly thrilled that there is a detailed section on numerical quadrature (I die a little every time I see someone use MC for a one or two dimensional integral…).

      I am, however, a little curious about the selection of material in the first few chapters – I’d prefer a discussion of expansions in terms of orthogonal polynomials (Chebyshev series, Hermite expansions etc) and rational approximation over power series and continued fractions (!!), mainly because I have no idea where you’d use the latter two (I expect the book has better examples the ‘evaluation of special functions’).

  5. I’ve got the first edition and regression is 9 pages. It looks as thought Lange acted upon the points made by his critic and produced a much better book. The lesson is don’t buy the first edition but DO buy the second!

  6. I gave up buying the book for now, because the kindle edition is only to the first edition (370 pages), and I’d like to read the second edition. And Buying the printed edition would take about 2 months for them to delivery it at my home (Brazil)…

  7. I took a look at the book myself (as much as you can at Amazon) and I think I’ll buy it. Thanks for your comments and review.

    btw: I’ll buy it clicking in the link of your blog. Maybe you can get some money from Amazon.


  8. No grumbling? It’s official: Xian is doing X!

  9. I’m thinking to buy this book, due to your review. However, I found this review at Amazon website.

    I know you’re busy person, but do you have an opinion about it? Do you still suggest for us to buy this book?

    Thanks anyway,

    • A strongly negative review! Just going through some of its points: (a) I checked the bootstrap example but it has vanished from the second edition so this criticism does not apply: the importance sampling correction is clearly working in Figure 24.2. (b) the linear regression chapter (Chapter 7) is 18 pages long and gives four different methods for solving linear equations, “with sweeping the most prone to numerical error”. Overall, I am sure a specialist in numerical analysis could find other missing methods that are superior to those presented in the book. This is unavoidable given the scope of the book. However, I like the comprehensive nature of the book, which is why I recommended it. There is no reason you should follow this recommendation and not the reviewer’s!

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.

%d bloggers like this: