"Slam the Exam" = "Game the System"?

Some 2,300 Success Academy students attended a "Slam the Exam" rally before last year's state English tests. The network goes to great lengths to make sure students are ready for the exams. ( Photo by Success Academy )


Data-driven strategies are a lot like market-based solutions. Properly applied under the right circumstances, they can be excellent, even optimal approaches, but if badly designed by people who don't understand the underlying principles (or who are looking to manipulate the process for their own ends), the results can be disastrous.

When designing a system of data-based decisions, perhaps the two most important considerations are:

Do your metrics have a strong relationship with what they are supposed measure? and;

Will that relationship continue to hold when the system has been in place for a while, particularly once the people affected figure out the rules?.

The Success Academy network is evidently gaming the achievement data in at least three ways:

Selection bias particularly in efforts to force out special needs kids;

Teaching to the test;

Changing the conditions of the test.

Let's focus on this last one for the moment. Check out the following excerpt from an article in Chalkbeat:

At Success Academy schools, high-octane test prep leaves nothing to chance
By Patrick Wall
Published: May 1, 2014

School leaders had provided teachers with color-coded agendas with precise instructions for every few minutes of test days, along with boxes of supplies that might come in handy — from pencils and tissues to extra clothes for students and deodorizing powder to sop up vomit.

Teachers had been taught the proper way to hand out tissues during the test (pass the student a new sheet first, then use a second sheet to grab the used tissue). They knew to set their classroom temperatures to between 66 and 70 degrees, and to call each student’s family every evening before a test to remind them of the next morning’s exam.

On test days, some teachers would take Success-funded cabs to pick up chronically late students (“Taxi Scholars,” as the agendas refer to them). Outside auditors, who had already observed the network’s practice tests, would monitor the real exams to safeguard against charges of test-rigging.

But students were perhaps the most prepared of all. They had spent weeks taking practice tests modeled off the actual state exams. They starred in test “dress rehearsals,” where exact testing conditions were simulated. Some had even practiced tearing perforated reference sheets out of mock test booklets.

If history is any guide, the preparation will pay off. Last year, Success students’ pass rates on the new and much harder state exams beat those of every other city charter school network and far surpassed the city and state averages. [Though these resullts have not carried over to other standardized tests -- MP]

Practices such as calling parents the day before the test very probably do improve test scores – – if, for no other reason, they make it less likely that kids will be allowed to stay up past their bedtime's that night – – but they can have no conceivable effect on the knowledge that is being tested.

It is notable and more than a little disheartening that this reporter, like most of his colleagues, seems to see all things that improve achievement scores as equally desirable, even those tactics that only serve to undermine the validity of the test.