In the first part of this series, I suggested that many strategies taught by test-preparation companies cannot legitimately be called gaming the SAT. Which is not to say that there aren't strategies out there that do amount to gaming the test. But many test-prep people, including myself, take the line that actual improvement comes from building fundamental skills and takes real work. (The test-prep guy writing in the Times debate I mentioned last time takes this attitude.)

Even a company like Princeton Review, which explicitly claims that the SAT can be gamed and markets itself on the ability to teach such tricks, actually winds up covering a certain amount of substance too. For example, PR strongly advocates studying vocabulary for the critical reading section. That's substantive learning that should (if students remember the words) transfer to a broader domain than just taking the test.

So to the extent that Princeton Review can raise students' scores (a controversial point that I'll cover later), how much is this attributable to the familiarity effect, how much results from substantive preparation, and how much results from gaming the test? It's extremely difficult to quantify each component, but based on my own experience, the test-gaming strategies like those that Princeton Review suggests are largely ineffective because students who might theoretically benefit from applying them aren't able to implement them effectively.

First, many test-gaming strategies apply to a small number of questions. If you cherry pick a single question out of a great many others, it's easy to suggest that the test can be beaten at a far higher rate than is actually the case. Adam Robinson loves to cite the "doughnut problem" in interviews, but this particular type of math problem is relatively rare. For many difficult math questions, an ordinary student will simply have no idea how to estimate an answer.

Second, the strategies typically require a highly sophisticated line of reasoning, considering hypothetical situations and evaluating possibilities. In other words, you need to be a fairly advanced student to put them into effect. And if you're that good, you can probably do an even better job solving the questions directly.

Third, many of the accounts of successful gamesmanship (and the suggested strategies) are old and based on patterns in questions written in the 1970s and 80s. But the test makers have not stood still in the intervening years. They are aware of test strategies, and my (unquantified) impression has been that the question writers do a much better job at avoiding obvious tells than they did 30 or 40 years ago.

While it is true that you can still find questions that can be answered fairly accurately without tackling it in the way that the test makers intend, they are occasional, making up a small proportion of any individual test, and they are far outnumbered by the questions that contain traps for those who rely on such tricks.

Next time, I'll quantify some of those assertions by considering how feasible it is to answer passage-based questions without the passage.