Here's your heartbreaking algorithmic cruelty story of the week: UC Riverside history prof @DanaJSimmons has a son who's just started junior high and loves his history teacher:
1/
1/
But when he submitted his first assignment, he was aghast to receive a 50% grade on it.
The assignment was graded by @EdgenuityInc, a machine learning grift that purports to generate grades for overworked, beleagured teachers.
2/
The assignment was graded by @EdgenuityInc, a machine learning grift that purports to generate grades for overworked, beleagured teachers.
2/
Simmons calls it an "automatic grading algorithm that values only rote repetition," and to prove the point, she told her son how to please the machine: long sentences with a lot of proper names.
3/
3/
It worked: now her son submits "word salads" consisting of two sentences (long ones, presumably) and a bunch of keywords from the lesson, and is consistently earning 100% grades. As Simmons says, "He went from an F to an A+ without learning a thing."
4/
4/
In @mathbabedotorg's seminal "Weapons of Math Destruction," she doesn't just provide a devastating critique of the underlying statistical basis for machine learning, but also a set of VERY useful rules of thumb for spotting AI grifters.
boingboing.net/2016/09/06/wea…
5/
boingboing.net/2016/09/06/wea…
5/
First: if a company provides an AI but doesn't check its predictions, they don't give a shit about its accuracy.
Think about it: if Amazon uses ML to predict whether moving the BUY button will generate more sales, the DEFINITELY measure whether sales after moving it.
6/
Think about it: if Amazon uses ML to predict whether moving the BUY button will generate more sales, the DEFINITELY measure whether sales after moving it.
6/
The fact that this automated grading system produces estimates that are so easily gamed tells you it is a straight-up grift. I mean, the Bayesian spam filters of the mid-2000s were able to detect the "word salad" attack.
7/
7/
This isn't one of those subtle, amazing "adversarial example" attacks on a ML model - like that weird thing where a vision system is tricked into thinking that a rifle is a helicopter:
wired.com/story/research…
8/
wired.com/story/research…
8/
It's more like a password prompt that you can bypass just by hitting the spacebar a bunch of times.
That is: negligent garbage serving no pedagogical purpose, an embodiment of the enterprise software pathology where the person who buys the product doesn't have to use it.
9/
That is: negligent garbage serving no pedagogical purpose, an embodiment of the enterprise software pathology where the person who buys the product doesn't have to use it.
9/
Simmons is admirably compassionate about the whole affair: "teaching online is overwhelming and you can't do it all. _Please_, use the algorithm to track their learning. But don't post to them as if it's a measure of their performance. It's more destructive than you know."
10/
10/
Amen.
School boards shouldn't be buying this tool for teachers.
Teachers shouldn't be using it to assess students.
Students should not be made to see those assessments.
Image:
Cryteria (modified)
commons.wikimedia.org/wiki/File:HAL9…
CC BY:
creativecommons.org/licenses/by/3.…
eof/
School boards shouldn't be buying this tool for teachers.
Teachers shouldn't be using it to assess students.
Students should not be made to see those assessments.
Image:
Cryteria (modified)
commons.wikimedia.org/wiki/File:HAL9…
CC BY:
creativecommons.org/licenses/by/3.…
eof/
