Uneducated guesses

I received this book, Uneducated Guesses: Using Evidence to Uncover Misguided Education Policies by Howard Wainer, from Princeton University Press for review in CHANCE. Alas, I am presumably one of the least likely adequate reviewers for the book in that

• having done all of my academic training in France (except for my most useful post-doctoral training in Purdue and in Cornell), I never took any of those ACT/SAT/&tc tests (except for the GRE at the very end of my Ph.D. towards a post-doctoral grant I did not get!);
• teaching in a French university, I never used any of those tests to compare undergraduate or graduates applicants;
• I am very marginally aware of the hiring process in US universities at the undergraduate, even though I knew about the early admission policy;
• there is no equivalent in the French high school system, given that high school students have to undergo a national week-long exam, le baccalauréat, to enter higher education and that most curricula actually decide on the basis of the high school record, prior to [but conditional on] the baccalauréat.

Thus, this review of Wainer’s Uneducated Guesses is to be taken with pinches (or even tablespoons) of salt. And to be opposed to other reviews. Esp. in Statistics journals (I could not find any).

“My role in this parallels Spock’s when he explained `Nowhere am I so desperately needed as among a shipload of illogical humans.‘” (page 157)

First, the book is very pleasant to read, with a witty and whimsical way of pushing strong (and well-argued) opinions. Even as a complete bystander, I found the arguments advanced for keeping SAT as the preferential tool for student selection quite engaging, as were the later ones against teacher and college rankings equally making sense. So the book should appeal to a large chunk of the public, as prospective students, parents, high school teachers or college selection committees. (Scholars on entrance tests may already have seen the arguments since most of the chapter are based on earlier papers of  Howard Wainer.) Second, and this is yet another reason why I feel remote from the topic, the statistical part of the analysis is simply not covered in the book. There are tables and there are graphs, there are regressions and there are interpolation curves, there is a box-plot and there are normal densities, but I am missing a statistical model that would push us further than the common sense that permeates the whole book. After reading the book, my thirst about the modelling of education tests and ranking is thus far from being quenched! (Note I am not saying the author is ignorant of such matters, since he published in psychometrics, educational statistics and other statistics journals, and taught Statistics at Wharton. The technical side of the argument does exist, but it is not included in the book. The author refers to Gelman et al., 1995, and to the fruitful Bayesian approach on page 69.)

In fact, from the start of the book, I was wondering why it was not mentioning counterfactuals, since most of the assessments revolve around this question of “what if?” (like “what if Bob had been taught in another class?” or “what if Sue had had another teacher?”). It is only in the final chapters that such notions occur, along with the important debate of data being of the missing-not-at-random kind… The first chapter reminded me of the quandary we are facing each year when ranking high school applicants based on their high school grades: based on the previous cohorts, we can build a predictive function for the first year final grade, but the data is censored to students who (a) were accepted and (b) decided to join our program, two major factors! Somehow paradoxically, a simple linear prediction that does not account for the selection eliminates the math grade as a significant covariate, simply because high maths grade students join an engineer school and low math grade students are not accepted. This significant distinction between applicants and accepted is not clearly discussed in the book.

A bit of nit-picking: some graphs could benefit from Tufte‘s advice (whose Napoleonic poster stands in the back of the author on the flip cover picture!): Two point lines linking the scores of those students who submitted their SAT scores with those who did not, or women with men (Figure 1.4, Figure 6.1) are conveying a message that does not truly exist; filling a box with grey level regions based on percentages of time dedicated to each maths component over a period of 8 years/observation points gives a feeling (a filling?!) of surface and continuity that is not contained within the data. I also found strange that the author refers to Spanish home-speakers as immigrants, since my [again bystander’s] impression was that a significant minority in the US spoke Spanish as their primary language. Still on the language issue, I want to stress that the translation of All birds of prey are not green in French could be Aucun oiseau de proie n’est vert, or Aucun oiseau prédateur n’est vert, but definitely not Aucun oiseau predat n’est vert (sic, p.69).

In conclusion, I did enjoy reading Wainer’s book (during my métro rides). Not only did it show me the complexities of the test based selection of students, teachers and colleges, but it also led me to ponder (once again) the worth of such numerical assessments and the relevance of the correlated (and naïve) belief of a “true” ranking/worth of students, teachers or colleges…