More quality control issues -- Regents exams edition

Math teacher and blogger Patrick Honner has been looking closely at the NY State Math Regents exams and found questions like this:


                                                               If

                                                               Find

(sorry about the formatting)

If you remember your junior high algebra, you probably spotted the mistake here. This function doesn't have an inverse. Honner goes into detail on this point but I suspect everyone reading this knows where he's going, Sufficed to say, unless we add a condition like "for x greater than or equal to zero," the correct answer would be "does not exist."

Here's the answer that gets the student full credit (which is definitely not a function).

They get half credit if they leave off the plus-minus (which would actually have been the right answer if we had included the previously mentioned condition).

As Honner puts it:

In summary, you get full credit for the wrong answer, but if you forget the worst part of the that wrong answer (the plus-minus sign), you only receive half credit!  So someone actually scrutinized this problem and determined how this wrong answer could be less correct.  The irony is that this conceptual error might actually produce a more sensible answer.  The further we go, the less the authors seem to know about functions.

It gets worse. Following an outcry from teachers:

The next day, the state gave in and issued a scoring correction:  full credit was to be awarded for the correct answer, the original incorrect answer, and two other incorrect answers.  By accepting four different answers, including three that were incorrect, you might think the Regents board would have no choice but to own up to their mistake.  Quite the opposite.

Here’s the opening text of the official  Scoring Clarification from the Office of Assessment Policy:

Because of variations in the use of notation throughout New York State, a revised rubric for Question 32 has been provided.

There are no variations in the use of this notation, unless they wish to count incorrect usage as a variation.  I understand that it would be embarrassing to admit the depth of this error, which speaks to a lack of oversight in this process, but this meaningless explanation looks even worse.  This is a transparent attempt to sidestep responsibility, or, accountability, in this matter.

I realize New York is a big place, but between Eureka Math, the Success Academy schools and this, the state is more than pulling its weight when it comes to blog fodder.

More on Success Academy and discipline

Yes, there is a pattern.

Cyril Josh Barker writing for the Amsterdam News:

Fatima Geidi and Elizabeth Eloheim, who say their children were pushed out of Success Academy charter schools, brought their issues to lawmakers, urging them to protect students’ rights not increase taxpayer funding for charter schools. The mothers claim their children were victims of abusive disciplinary actions.

Eloheim alleges that her daughter, who attended Harlem 1 Success Academy, and her family were targeted because they questioned the abusive practices and policies at the charter school.

“It was one of the worst experiences of our life, and the long-lasting effects are devastating,” she said. “The school used deprivation of bathroom privileges as a discipline practice, which is inhumane.”

Geidi said her special needs son faced multiple suspensions when he was a student at Upper West Side Success Academy for things such as walking up the stairs too slowly.

“He was often accused of being too emotional,” Geidi said. “He has special needs but never received his mandated services and supports. At nine months pregnant, I finally gave in to the pressure of withdrawing my son, which is what the school wanted all along. The role of the school is to educate children, not displace them.”

The quality control problems continue

A tutor with an after school program here in LA brought in this one. I believe it was from an eighth-graders homework assignment.

He said the problems had seemed straightforward at first. Each had two sentences that described relationships between two numbers. He walked the student through the steps of translating each sentence into an equation then using substitution to get one equation with one variable.

 The trouble started when he looked at the instructions.

The tutor immediately saw two problems. For starters, the first step (converting each sentence to an equation), greatly confusing the student. Worse still, the answer in the example was simply wrong: the solution to 8n = 112 is n = 14, not n = 14 and 98. This is the sort of thing that you gently correct when a student does it, not the sort of thing that should slip past a professional proofreader.

Going by what I see working with that program (admittedly a small and unrepresentative sample), most handouts these days seem to come from large companies. Back in my teaching days, I almost always made up my own handouts. I can understand the potential advantages of using ready-made educational materials -- creating these things is a time-consuming job -- but what I can't understand is how the quality control can be this bad.

I've been over this before, but the list keeps getting longer.

I don't actually need an excuse to post a Theremin video

... but if I did, I would point out that a clip of the instrument followed by a quick discussion is a great way of opening a lesson on the Cartesian coordinate system.

In case you've forgotten:

The theremin is distinguished among musical instruments in that it is played without physical contact. The thereminist stands in front of the instrument and moves his or her hands in the proximity of two metal antennae. The distance from one antenna determines frequency (pitch), and the distance from the other controls amplitude (volume). Higher notes are played by moving the hand closer to the pitch antenna. Louder notes are played by moving the hand away from the volume antenna. Most frequently, the right hand controls the pitch and the left controls the volume, although some performers reverse this arrangement. Some low-cost theremins use a conventional, knob operated volume control and have only the pitch antenna. While commonly called antennae, they are not used for receiving or broadcasting radio waves, but act as plates of capacitors.

Plenty of choices for the video (just Google 'Theremin'). This one ought to go over well.

"Four times as likely"

I would have liked to have seen more detail in this NYT piece, but it's still worth reading.

Black students are four times as likely to be suspended from charter schools as white students, according to a new analysis of federal education data. And students with disabilities, the study found, are suspended two to three times the rate of nondisabled students in charter schools.

These inequities are similar to those in traditional public schools, where black and disabled students are disproportionately disciplined for even minor infractions, and as early as preschool — although on average, charter schools suspend pupils at slightly higher rates than traditional public schools.

 What we could really use here is some kind of a breakdown by type and chain. Charters are, by design, a diverse group. I strongly suspect that disaggregation would reveal certain pockets were generating more than their share of suspensions and disciplinary overreaction.

Based on data from the 2011-12 school year, the report found that charter schools at the elementary, middle and high school levels suspended 7.8 percent of students, compared with 6.7 percent of students in noncharter schools. Among students with disabilities, charter schools suspended 15.5 percent of students, compared with 13.7 percent at noncharters. At the extreme end, there were 235 charter schools that suspended more than half of their students with disabilities.
 ...

Crossroads Charter School in Charlotte, N.C., suspended close to three-quarters of all black students in 2011-12. Adrian Sundiata, the operational director at the school, said it was now using more disciplinary measures to address infractions like taking a cellphone to school or using profanity, including after-school detentions and community service.

Revisiting "The Battle for New York Schools: Eva Moskowitz vs. Mayor Bill de Blasio"

When following the education reform movement, it is enormously useful to step back from time to time and look at who was saying what a few years ago. As recently as 2009, it was almost impossible to find serious critics of the movement in the mainstream media (to highlight how much things have changed, I put together an e-book collection of my 2010 education posts, annotated but otherwise unrevised).

As far as I can tell, the Washington Post was the first of the major papers to start turning a tough, critical eye towards initiatives like charter schools, Common Core, and Glengarry Glen  Ross incentive systems. Recently, the New York Times has been aggressively investigating problems at Eva Moskowitz's Success Academies, but this is a relatively new position.

This  2014 NYT Magazine piece by Daniel Bergner is interesting on a number of levels, not the least of which being a reminder of how things have changed.

On the topic of scores, the U.F.T. and Ravitch insist that Moskowitz’s numbers don’t hold up under scrutiny. Success Academy (like all charters), they say, possesses a demographic advantage over regular public schools, by serving somewhat fewer students with special needs, by teaching fewer students from the city’s most severely dysfunctional families and by using suspensions to push out underperforming students (an accusation that Success Academy vehemently denies). These are a few of the myriad factors that Mulgrew and Ravitch stress. But even taking these differences into account probably doesn’t come close to explaining away Success Academy’s results.

First off, even at the time "vehemently" did not equate to "convincingly." There was already an enormous amount of evidence behind these accusations. Letting SA's denial go unchallenged did Moskowitz a huge favor, as did the unsupported claim at the end. Little more than a year later, the NYT itself was reporting on the Success Academies' "got to go" lists.

[Diane Ravitch was extremely upset both by how Bergner handled her interview and wrote a stinging post in response.]

As bad as this section was, the really troubling part (at least for me as a statistician) came later.

In talking to dozens of current and former Success Academy employees and parents, the critique with the most staying power involved the schools’ overly heated preparation for the state exams. A former fourth-grade teacher recounted that network employees made a mini­van run to Toys “R” Us and returned to unload a mound of assorted treasures in the back of her classroom. “It was a huge pile,” she says. “We called it Prize Mountain.” She would remind the pupils that a good score on a practice test meant a gift from the mountain.

Teachers also chart students’ results on the practice tests, posting their names and scores on classroom walls. Yet I heard from parents like Natasha Shannon, an African-American mother of three girls in Success Academy schools, that although the public posting could be painful for the children, it was important nonetheless.

...

For her part, Moskowitz asserts that the public charting is one aspect of the network’s emphasis on feedback, not only for the students but also for the faculty. Throughout the year, whether or not test prep is underway, scores on quizzes and writing assignments are analyzed at network headquarters. Each teacher’s outcome is tabulated, and bar graphs are instantly available to all faculty members. The teachers whose classes lag are responsible for seeking out advice from those who top the graphs, just as the students with red or yellow stickers by their names are guided to emulate the topic sentences of those whose stickers are green or blue.

Couple of points here.

1. We can go back and forth on different methods of rewarding academic performance in other contexts, but in this case we're talking about diagnostic tests. Doling out special rewards and punishments can and probably does undermine the quality of the resulting data. The fact that Bergner (and, to be fair, most reporters covering the story) seem completely unaware of fundamental education concepts is disturbing;

2. Even more disturbing (though we can't blame this one on Bergner.) is the fact that one of those model teachers whose advice was being sought was Charlotte Dial.

NASA's Real World Mathematics

As I've mentioned before, there are a lot of great resources for educators at the Internet Archive and it's all in the public domain, which mean you can re-edit anything you find to fit your needs.

He ought to know...

Elwyn Berlekamp is one of the authors of the classic Winning Ways for your Mathematical Plays. One of these days I need to open up a substantial thread on Winning Ways. In the meantime, here's a cool video from Dr. Berlekamp.

How to always win at Dots and Boxes - Numberphile




https://en.wikipedia.org/wiki/Elwyn_Berlekamp

A different kind of name-the-states map

The cartoonist who created this explicitly say that it would be unfair to erase the names and use it as a name-the-states quiz, but I think it would make a great activity for small groups.

Short version -- policies that stress small children to the point where they throw up are probably bad policies

I am generally nervous about quoting overly partisan news sources. Even when the arguments are compelling, I am uncomfortable with discovery processes that seem to start with the conclusion.

That said, the Nation is a good magazine with a solid history of investigative journalism behind it. So, with the caveat that a publication this liberal will probably be hostile to charter schools (something we probably couldn't have said 10 years ago), this article is definitely worth checking out for anyone who's been following the debate.

In particular, this caught my eye:

Brenda Shufelt, a recently retired library who served public school and Success Academy Charter School students at a co-located school library in Harlem, noted that as charter schools rapidly expand, they may be taking in more high needs kids, many of whom cannot conform to one-size-fits-all disciplinary approaches.

“In my experience what would often happen is that charter school students would be so rigidly controlled that the kids would periodically blow up,” says Shufelt. “At PS 30, some of our kids would have meltdowns, usually because of problems at home, but I never saw kids meltdown in the way they did in charter schools. They were just so despairing, feeling like they could not do this. I was told by two custodians, they had never had so much vomit to clean up from kindergarten and elementary classes.” 

I realize that we have been hammering away at this thread for quite a while and I apologize for going over familiar ground in the next few paragraphs. Feel free to skim if you're a regular reader, but the following points do need to be emphasized.

Many of these techniques are remarkably hard on kids. Even if there were no other issues and the methods were accomplishing everything their supporters claimed, we would need to have a serious discussion as to whether or not they were worth the physical and emotional toll they are taking.

But there is considerable reason to question those alleged accomplishments. For starters, a lot of students do not make it all the way through the program. These kids pay a double toll, dealing with the stress not only of the no-excuses program but also of the disruption of being pulled out of one school where they have made friends and established relationships and put into another school where they are surrounded by strangers. Even for those kids who make it through, there is considerable evidence that the improvement in test scores is largely limited to one exam and does not translate into the areas we are really interested in.

Using Wikipedia critically

[via Popular Mechanics]

One of the things I've noticed since I started working with after-school programs a couple of years ago is that while students now spend a great deal of their time online, few are fully aware of the resources that could be making their academic lives so much simpler. I can't count the number of times I've seen students reading and rereading class notes hoping to find an answer that was just a few seconds away on Google.

Of these resources, perhaps the most underutilized is Wikipedia. It's not perfect, of course, but between breadth and surprisingly good quality control, there's nothing better on the internet for general factual questions.

Occasionally, though, something godawful does slip through.

Case in point...

Wallace L. Minto (August 6, 1921, in Jersey City, New Jersey, United States – September 3, 1983) had a passion for science at a very young age. For instance, at age 13, he and his father, Wallace Milton Minto, stock piled more than 50 tons of uranium rich ore in Sparta, NJ. He was also the first to split the uranium atom while still a teenager. This nearly created an atomic explosion in his family home. At age 16, Wallace synthesized radium and invented what is now known as "Scotchlite". He had a copyright on his own periodic chart which renamed all the elements.

When only 16, he was a student at Columbia College and was later instrumental in convincing Albert Einstein to write a letter to President Franklin D. Roosevelt (dated August 2, 1939) stressing the need for the United States to expand its experimentation with Atomic Energy, leading to the Manhattan Project. Consequently, Minto sold his uranium rich ore to the U.S. Government for use in the Manhattan Project.[1]

On June 26, 1944, Minto was enlisted by Dr. Andrew H. Dowdy, director of the Manhattan Department of the University of Rochester, to take charge of the Special Problems Division of the Manhattan Project. Minto reported directly to General Leslie Groves and reportedly threw Groves out of his lab for tampering with his beakers.

We could, of course, dismiss this as being simply incredible, but simply dismissing the unbelievable is a potentially dangerous habit. Unlikely things do happen. Instead, the article strikes me as a great opportunity for discussing why we reject certain sources as unreliable.

1. Check the issues box.

Lots of red flags here.

This article needs additional citations for verification. (July 2015)
This article may be in need of reorganization to comply with Wikipedia's layout guidelines. (July 2015)
This article is an orphan, as no other articles link to it. Please introduce links to this page from related articles; try the Find link tool for suggestions. (December 2015)

That first one leads us directly to...


2. Check the sources

Or, in this case, source.

"Forgotten Genius". Bobgrove.org. Retrieved 2015-07-28.

It's a brief post that cites no sources other than an alluded-to acquaintance with the subject, nor is there anything in the writer's online biography that suggests special knowledge.



3. Check the relevant pages

Go to the Wikipedia listings for the Manhattan Project and the Organic Rankine Cycle Engine and do a search for "Minto." You can probably guess what happens.




As far as I can tell, the only other place Minto does show up in Wikipedia is the also suspect entry on the Minto wheel, a cute but not very practical heat engine that seems to predate Minto by quite a bit.

More cool machines

A very nice clip to get kids' brains working on a Monday morning.

Thinking about "teaching to the test"

[warning -- I dictated this to my phone so beware of homonyms.]

The following is probably both obvious and overly simplistic, but (putting aside emotional development for the moment) the primary purpose of instruction should be to increase students' mastery of the material (both knowledge and process based) while the primary purpose of tests is to measure that mastery. Those definitions are ridiculously broad but they'll serve for the moment, as will the rule of thumb that any instructional practice that doesn't improve mastery of the material or validity of the test should be viewed with suspicion.

I should also note before going any further that it has been many years since I got my teaching certificate and I have long since forgotten what little terminology I once knew. For the next few paragraphs I'll just be coining my own.


Teaching to the test – inclusive vs. exclusive

Inclusive teaching to the test – – making sure to cover everything students will be tested on – – is always defensible and is generally a very good idea.

Exclusive teaching to the test – – leaving out material you would otherwise cover because it is not going to be tested over – – is much more of a gray area. Obviously, teachers have to prioritize, but this should, at the very least, give one pause. In most cases, the test should include only a proper subset of the material covered.


Meta-Issues

Teaching to the meta-test -- prepping kids on the non-content aspects of the test -- gets even more gray. On a very high level it is certainly justifiable. You wouldn't want to have kids encountering strange formats and confusing instructions for the first time when they sit down to take a major test. On the other hand, this is an example of instruction that doesn't improve mastery of the material. If done for certain groups of students and not others, it runs creating an unfair advantage and thus undermining the validity of the test, which leads directly to...



Changing the conditions of the test

In order to get the most valid results, it would be best if all kids took all tests under optimal conditions -- motivated, well rested, comfortable, relaxed and free from distraction. We would always like to be sure that a low score represented a low mastery rather than a poor night's sleep. If, however, we can't be universally optimal, it is important that the suboptimal be distributed as uniformly as possible.


Teaching the actual test

Everything we've covered up to this point has ranged from OK to borderline, but ethical and even legal lines are quickly crossed when specifics of the test start making it into the prep materials available to only certain students. When individual teachers and administrators do this, they lose their jobs. When the companies that make the tests do it by selling "aligned" texts and prep materials, their stock goes up.


If you've been following the no-excuses discussion, lots of these concepts should sound familiar.

A working rotary engine made entirely of paper

Just because I haven't posted a cool video for a while.