The question of whether—or to what extent—the SAT is "coachable" is a perennial controversy. When the recent overhaul to the SAT was announced, College Board CEO David Coleman made a big deal about how the changes would orient the test so that it more closely measured the skills needed for college success. One intended consequence of those changes was that the test would be less susceptible to coaching. Eliminating sentence-completion questions, for example, was meant to discourage students from memorizing long lists of "SAT" words that they would never need again. When you add in College Board's partnership with Khan Academy, which makes official preparation material available for free, Coleman clearly wanted to make commercial test preparation less viable.

Test prep companies were less than impressed by Coleman's rhetoric. Each of the previous two revisions to the SAT (in 1994 and in 2005) were accompanied by similar claims. And none of the changes yet have made a dent in the appeal of test-prep courses. I've been looking closely at the new SAT since the revisions were announced, and my conclusion, reached after working with numerous students on this new format, is that specific design decisions on the new test have backfired, and that the new test, specifically the Reading Test, is actually significantly more coachable than the prior version.

One problem with a lot of the debate over test coaching is that it often quickly shifts to the question of whether test-prep courses work, and that, it turns out, is a slightly different question, as it presumes that the activity which Kaplan, Princeton Review, and the rest are involved in constitutes "coaching."

When people in the field of educational measurement talk about coaching, they have in mind a specific kind of test preparation, one that is distinct from ordinary teaching. Coaching refers to instruction in techniques that are specific to the testing environment and don't transfer. In other words, if I teach you how to be a better reader and as a result you do better on the test, I'm engaged in authentic teaching. You'll be able to use those improved reading skills elsewhere in life. On the other hand if I instruct you not to bother reading the entire passage before you try to answer questions about it, I'm engaged in coaching. This advice is antithetical to mastery of the underlying skill that the test is intended to measure. If it works, I've inflated your score beyond what you deserve.

The distinction between teaching and coaching is a fuzzy one, and it's probably best to think of any piece of instruction as falling on a continuum. Even very test-centric instruction can build transferable skills. For example, the method of process of elimination works on multiple-choice tests but not on constructed-response ones, and therefore is often cited as an example of coaching. However evaluating claims and determining that they are false is, in fact, an extremely important skill. That's what the entire scientific method is based on. So while there's certainly artificiality to the multiple-choice format, it's not self-evident that encouraging students to work by eliminating wrong answers is fostering a skill that is irrelevant to the underlying skills being tested.

Companies and private tutors that offer test-prep instruction differ in how much they emphasize teaching of underlying skills versus test tricks. My own experience, and I've been helping students prepare for the SAT for over 25 years, is that coaching in the strict sense, beyond the basics that are relevant to all multiple-choice tests, has historically been ineffective when it comes to the SAT, especially when higher-performing students try to apply such advice. For years, I've told students that the path to impressive score improvements lay in building the fundamental underlying skills on the test, and the results my students have achieved backed up that advice.

The current revision of the SAT, however, has caused me to partly revise my opinion, at least as far as the new Reading Test goes. I still think that building those fundamental skills is the most important thing you can do for lasting score improvement, but the format of the so-called "best evidence" questions on the Reading Test makes them vulnerable to coaching techniques, so much so that I have serious doubts as to whether they should be on the test at all.

The intention behind best-evidence questions is a laudable one. The idea is that first you ask a question about some aspect of the passage, and next you ask what part of the passage provides the best evidence for your earlier answer. This pattern is meant to mimic the way your English teacher might challenge you to back up your assertion by citing something specific in the text.

The problem is that most of these best-evidence questions can be answered without answering the prior question. Indeed, if we're only focused on getting the right answers as quickly and effectively as possible, the best strategy may often be to answer the second question first.

To see what I mean, have a look at this example, taken from an official practice test. The underlying passage comes from a novel set in 1920s Japan. Akira has come to ask Chie for permission to marry her daughter. The stem of the first question (#4 in the section) is this:

Which reaction does Akira most fear from Chie?

I've left out the answer choices to this question because you don't need them to answer the next question, which reads, "Which choice provides the best evidence for the answer to the previous question?" and lists four line references. I've put the actual language referenced by those line numbers in brackets below, with a little surrounding text for context:

...

"Madame," said Akira, "forgive my disruption, but I come with a matter of urgency."

(A) [His voice was soft, refined.] He straightened and stole a deferential peek at her face....

Akira's message, delivered like a formal speech, filled her with maternal amusement. (B) [You know how children speak so earnestly, so hurriedly, so endearingly about things that have no importance in an adult's mind?] That’s how she viewed him, as a child....

"We have an understanding. (C) [Please don't judge my candidacy by the unseemliness of this proposal.] I ask directly because the use of a go-between takes much time...."

(D) [Eager to make his point, he’d been looking her full in the face.] Abruptly, his voice turned gentle. "I see I've startled you...."

The first question asks about a reaction from Chie that Akira is worried about. To answer the best-evidence question, therefore, we don't actually need to know ahead of time what that reaction is. We simply need to look at the line references and ask ourselves if the quoted lines address the topic of the question. Choice (A) merely talks about Akira's voice. It says nothing either about Chie's reaction or his fear. It cannot be correct. Choice (B) talks about Chie's reaction: she sees him as a child. But notice that this reaction is what Chie is actually thinking, not what Akira worries she will think. Akira has no way of knowing this. It too, cannot be the right answer. In choice (C), Akira says to Chie, "please don't judge." Notice that even if you don't know what the word "unseemliness" means, he's asking her not to judge something about him. A judgment is a kind of reaction, and the fact that Akira asks her not to have a particular reaction tells us he's worried about it. In other words, the lines in choice (C) are precisely on the topic of the companion question. Choice (D) talks about Akira's attitude but says nothing about Chie's reaction or his fear. We can eliminate this one as well.

Once we've found the right answer to this one, the prior question becomes much easier to answer correctly, because we're focusing on one narrow part of the text rather than the entire thing. Indeed, if you understand the vocabulary in the quoted lines, the correct answer to the earlier question becomes very simple.

This pattern is not a rare one. The technique I've just outlined works for the vast majority of best-evidence questions. Nor are these questions rare. Normally, there are two on each passage, which means that these questions and their companion questions comprise 35-40% of all the reading questions on the test.

When students tackle these problems naively (i.e., without coaching) they naturally attempt the earlier question first. And because that first question lacks any line-number reference, they often spend a lot of time searching through the whole passage looking for something that can give them a clue. They also tend not to go back to the prior problem to verify their answer. (It's fairly common for students to miss the prior question and get the best-evidence question right.) In other words, attempting these problems in the natural way is both inefficient and error prone.

On the old SAT, the advice was sometimes given to answer reading questions out of order, for example putting off answering main-idea questions to the end. In my own experience, I never found that advice made little measurable difference in students' scores. For every passage where it helped to postpone answering that type of question, there was another where it was a better strategy to answer it in order. But with this new type of question, I've observed a marked improvement if you approach the pair backwards.

The new SAT does have some significant improvements over the old test, but on the new Reading Test, nearly 40% of the questions are highly susceptible to coaching in a way that they never were before, giving a large advantage to those who apply a technique that has little to do with the underlying reading skills the test is supposed to measure. That's a huge step backwards, and particularly ironic considering that a stated goal of the change was to minimize the possibility for coaching.