The College Board just released a statement claiming that SAT preparation works, at least if you use their officially sanctioned provider Khan Academy. A Washington Post article points out that this is a complete volte-face for College Board, which has traditionally minimized the ability of test prep to raise scores. The WP article does a good job of exploring the apparent contradictions, and obvious self-interest, in the results. There were a few additional points, however, that struck me.

First, this statement from College Board is only a press release. At this point, the underlying research on which the press release is based hasn't been published. There are many important questions about the study design that go unanswered in the press release. For example, many students use paid test prep along with Khan Academy. There's no indication if there was any attempt made to distinguish those students from the ones who relied only upon Khan. It's also unclear how they deal with the selection-bias problem.

Second, while it's true that this is a change in College Board attitude, it's probably unfair to call College Board hypocritical for altering their view. College Board isn't a single person, and if it does not adopt the same attitude today that it did 60 years ago, that's unsurprising. Even before Coleman took over, College Board had modified its old attitude about the SAT. In the early days, the SAT was seen as something like an intelligence test, and so the claim was that you couldn't really prep for it. By the 1990s, though, College Board had shifted its view, and now argued that the test actually measured developed academic skills. Of course if these are developed skills, it follows that you can improve your performance with appropriate study. It continued to diss test-prep, though, on the grounds that those skills required long-term study, and that the test-prep focused on the wrong things. After Coleman's arrival, College Board embraced this view even more wholeheartedly. Indeed, their statement to the Washington Post essentially say, yes, the critics were right about the old test, but this test is different:

Too much of commercial test prep teaches to the test — looking for shortcuts and tricks to “beat” the test. The SAT in its old format lent itself to this approach. The College Board and Khan Academy firmly believe in practice, and particularly practice that is personalized to pinpoint areas where learners need additional help. Preparing for the new SAT is the same as preparing for college.

It's notable that College Board's new attitude is exactly the same as that which has been held by a substantial number of people in the test-prep industry for years. While it's true that places like Princeton Review sell themselves with beat-the-test gimmickry, many other outfits base their courses on a model of skill development combined with abundant practice in applying those skills to test questions. I've worked for several companies that used the preparation model which College Board now praises, and I saw all the internal data on score improvement for their courses. We were getting results even better than what College Board reports for Khan. (Of course these courses had more class hours, and they involved live teaching, both of which are important differences.) In other words, the College Board is right that practice works for the new SAT, but the same sort of practice worked for the old SAT too, so the only thing new here is their acknowledgment of the fact.

The question of how much improvement test-prep companies really achieve is a legitimate one, and it's remarkably hard to answer in a rigorous way, even if you have access to all of a company's internal data. Students who enroll in test-prep programs are not a representative sample, so selection bias is a real issue, and there are a host of technical issues involved in how statistics are collected that can easily make the numbers that a company measures flawed. The College Board in older work rightly points out the weaknesses in the score-improvement claims that commercial test-prep companies make.

And yet while those flaws are real, it's always seemed to me that in those older studies, College Board deliberately was stacking the deck against the test-prep companies by looking at improvement after any form of prep, regardless of other factors.

For example, any teacher knows that to get something out of a course, students must put something into it. If you sit passively in the back of the class, disengaged, avoiding doing any real work, you're not going to improve on any subject. Studies like the one I linked above don't (and indeed can't) assess how student outcomes are affected by engagement, etc. They lump together students of varying motivations.

And this College Board data makes it clear that they are definitely not pooling the motivated and unmotivated students to derive their score-improvement numbers. They correlate the number of hours put in to the amount of increase, and I presume that they were using Khan's tracking of time spent on the site to measure the time. But by doing so, notice that they are selecting for the most motivated students at this top end. This is predictable, and not necessarily a wrong way to look at things. The real question we want to know about test-prep programs is how much do you get out of it if you put in the effort. And an online course lets you track the amount of time students are spending actually working through problems, etc. But the older studies don't have that information available. So to compare the Khan results with those older evaluations and say, "See, our program works where the others don't" is profoundly misleading, even dishonest. They measure outcomes for different groups in different ways.

College Board is right that a significant fraction of people don't see improvement from test-prep courses. Some of that is certainly the result of bad test-prep instruction. But some of it is the result of students who don't want to be there. Their parents sign them up for a course and force them to go. Other students come in expecting that the teacher will wave a magic wand and raise their scores without their actually having to do anything. Those unmotivated students won't show up in the Khan data because they simply don't bother to do the practice, but they wind up occupying chairs in paid test-prep programs. At one of my old companies, we had a proxy data for student for student engagement in the form of vocabulary-test results on words that students were required to learn as part of the program. Every year, there was a strong correlation (in the range of 0.6 > r > 0.7) between vocabulary test scores and overall score improvement on the SAT. A strong vocabulary isn't, of course, the only thing you need to raise your scores, but the point is that the students who worked hard to learn that vocabulary were also working hard in other respects, so performance on vocabulary quizzes turned out to be a good way to check that they were doing all the required work.

Test-prep companies are going to be using this announcement as evidence that test-prep really does work. But in one way, I think College Board will achieve it's aim of striking a blow against useless test prep. This announcement raises the bar for paid test prep. Why spend a lot of money on pricey classes if you can get great results for free? You do should only do it if you need more than you can get on your own. If you need to learn how to write a better SAT Essay, for example, Khan is of no help. Companies that can add value over this floor will continue to do well. Those that cannot rightly deserve to fail.