As students came out of the SAT on the East Coast yesterday, many hit the online forums devoted to discussing the test to discuss questions. And even before the West-Coast students had left their exam rooms, it became evident that the form most of them were given was actually one that had been leaked online months earlier. Multiple students reported that they had practiced on this very test. How serious is this situation? Very. The scores of any student who studied these questions prior to taking the test are likely invalid. Can College Board figure out which students those were? In some circumstances, but by no means in all. International, especially Chinese, students are getting the brunt of the blame on Reddit and other online forums, particularly since large numbers of them flew to the U.S. to take this test. (There is no international administration of the SAT in August.) But it's by no means just those students who had access to the test.

There have been parallel cases in the past where students taking international administrations of the test have seen at least portions of that test ahead of time and College Board responded by canceling scores for the entire country or canceling the administration of the test entirely. If College Board acts consistently with its own precedent, it ought to cancel all scores from this administration. I'd be gobsmacked if they did that, however.

The August SAT is a popular test date, as it comes, for most students, right at the beginning of the U.S. school year. This means that test takers have the opportunity to spend their summer preparing for the test without taking time away from their primary studies. For College Board to do the objectively right thing to preserve the validity of the scores would mean that every one of these students will have to retake the test. For seniors who have limited further opportunities to take the test, that scenario is grim. For that reason, I predict that College Board will decide to try to weather the controversy by canceling the scores of a few easy-to-spot cheats and leaving everything else in place.

Three questions arise from this clusterfuck: How did this situation occur, who's to blame, and how can it be fixed?

How did it happen?

The test was leaked because it was recycled from a previous administration. It was given (at least) twice before: in June 2017 in the U.S., and then in October 2017 internationally. It was after this second administration that copies of the test began to circulate online. This is not a new problem. Although there are many potential security vulnerabilities, test reuse creates the biggest problem, and the one that is easiest for unscrupulous test-prep companies to exploit. Reuters published an investigation of this very topic two years ago. It's an excellent, balanced piece, and I highly recommend you read it.

Test reuse is defensible in certain cases, especially where there will only be a relatively small number of students taking the test. The SAT has to be offered for students who cannot take the test on one of the major test dates. For example, there are students who, for religious reasons, cannot take a test on Saturday. Individual test centers also may have to cancel on the regularly scheduled test day for unpredictable events, for example because of a blizzard or, as with Hawaii this very August, a hurricane, and those students will be given a special make-up date outside the normal testing schedule. If completely new tests had to be used for each of these special administrations, the costs to test takers would explode, since a completely new test costs more than $1,000,000 to develop, and takes months of development time. A less obvious but no less consequential problem with giving a new test on these dates is that these groups do not constitute a representative sample of the normal test-taking population, and therefore it can be extremely difficult to create accurate score scales. By reusing a previously administered test, you can use the scale that was equated with a more normal population.

That difficulty, perhaps even more than the cost, is likely a principle factor in College Board's continued practice of recycling tests originally given in the U.S. for an international administrations. Students who mostly speak English as a foreign language are not appropriate to use as a reference group for equating a test. That's why a test first given in the U.S. in June 2017 (as one of several forms used on that date) was reused internationally in October of the same year. But for the College Board to turn around and reuse the same test on a major Saturday test date in the U.S. is, as far as I know, unprecedented. In the past, those dates have almost always featured new tests.

There's only one other case I'm aware of where a Saturday U.S. test was recycled, and that was in January 2016, which recycled a June 2014 test. However there are a few differences between that case and the August 2018 test. First, that duplication was detected because students compared reading passages in discussions online and were able to match it with the old test through the pattern of the passages. The fact that people were able to retrospectively identify a reuse doesn't mean that any students who took that test were advantaged and is much less serious a leak of test information than a complete leak of all the questions. The test reused was also previously given only in the U.S. and not overseas. Furthermore, this was the very last of the old format SATs before the test was changed in March of the same year, and January was always the least popular test date anyway. (January testing was dropped when August testing was introduced.) It's understandable that College Board wouldn't want to waste resources on a new test in those conditions, and it doesn't seem to be the case that there was any substantial security breach for that test.

For the August 2018 test, however, College Board chose to reuse a test that had already been given internationally. That's an unprecedented reversal of their traditional practice. At some point (I think after the October 2017 administration, but I'm not completely sure about this), retyped versions of the reading and writing portions of this test appeared online and remain discoverable through fairly simple google searches. I've also been told that a complete scan of a stolen test booklet exists and is being shared by students who study with each other in closed online groups (I haven't seen this scan myself, but I believe these are credible reports). The Reddit discussion I referenced earlier also shows that some Asian test-prep companies prepped their students on these tests as part of their courses. In short, this test was thoroughly and irreparably compromised no later than October 2017.

Who is to Blame?

Obviously, the test pirates who leaked the test bear a major portion of the blame here, and I don't want to let them off the hook. Let's agree that stealing test booklets or sending in people with hidden cameras to take pictures of the test is unequivocally wrong. But attempts to subvert test security are a fact of life with any high-stakes test, and so we must also ask whether College Board has fulfilled its obligations to maintain the integrity of the testing process. And the answer to that question has to be a resounding no.

I readily grant that international testing creates difficult security problems for College Board, and that leaks of test material are likely in those scenarios. This has been a problem for years, and College Board is fully aware of it. But why on earth would you expand the problem by giving compromised tests to a huge swath of U.S. test takers? Previous incidents brought into question the validity of scores for international test takers. Now you've compromised everyone else's scores as well. This was an unspeakably stupid decision.

What can be Done?

For this test, very little. In limited instances, College Board may be able to determine some of the students who had prior access to the test. There are two potential flags they could use to detect such students. First, a student who experiences an extreme score increase, either relative to a prior test or a PSAT, will likely be investigated. But big score increases do come from things other than cheating. And if this is your first time taking an official SAT or PSAT, this screening method doesn't work. Second, they can search for common patterns in wrong answers. It's my understanding, based on the Reddit discussion of the test, that the answer keys which were floating around with the leaked tests contained errors. If your answer sheet shows exactly the same pattern of errors, it's highly unlikely you got that pattern by doing the test legitimately and College Board could use this as justification to cancel your scores. Students who saw the test ahead of time but didn't use those answer keys, though, are unlikely to get caught. Canceling the entire administration is the only way to preserve score validity, but as I explained above, I think College Board is unlikely to do that.

For future tests, Saturday tests in the U.S. should always be new, and any test that is fully compromised online must be withdrawn from circulation. Granted that it's necessary to reuse some tests, College Board needs to structure their test-use patterns to minimize the fallout should a future breach occurs.

Update 8/28/18: My reaction to College Board's statement on the problem, as well as further reflection on whom this security breach affects, can be found here.