An AI installed to keep students safe flagged a 13-year-old's joke, and by the end of the school day she was under arrest. A face-matching AI sent a grandmother of five to jail for more than five months, for a crime in a state she says she had never set foot in. An assistant told to act on its conscience reported its own employer to the FDA before anyone could stop it.
We followed the machines built to protect people, in schools, stores, police stations, insurers and hospitals, through the lawsuits, the regulators' rulings and the released records, up to the AI gatekeeper Medicare switched on this January. And to the country that already found out where a machine like that ends up.
0:00 When Algorithms Become the Judge 1:20 Eyes in the Sky: AI Surveillance 3:05 When Machines Develop a Conscience 5:54 From Safety Tools to Toxic Generators 8:41 The Algorithmic Denial of Care
An AI installed to keep students safe read
a 13 year old girl's school chat in Tennessee,
and by the end of that school day, she was under arrest.
She'd been teased about her tan,
and she typed back a joke about killing. The machine flagged it.
She wasn't the last person a machine built
to protect people decided was the threat.
The biggest one wasn't in a school.
A government switched it on,
According to her family's lawsuit, she was strip-searched,
and it ran for years.
questioned and held overnight in juvenile detention
and after that came eight weeks of house arrest,
a psychological evaluation and 20 days at an alternative school.
Afterward, she told her mother,
I thought you hated me.
Her mother says that kind of thing haunts you.
The system is called Gaggle and its CEO said later that he wished it
had been treated as a teachable moment, not a law enforcement moment.
He put it on the school.
But records from other districts show what
an alert like that turns into.
One Florida county had about 500 alerts over four years,
and they ended in 72 involuntary psychiatric hospitalizations.
A district in Kansas found that about two-thirds
of 1,200 alerts in 10 months were nothing at all,
and more than 200 of them were homework.
Somewhere tonight, a machine like that is reading a child's messages,
trying only to keep that child safe.
Some of these machines don't read, they watch.
In Baltimore County, Maryland,
Taki Allen had just finished football practice and was eating chips
outside his school. An AI from a company called OmniAlert had
been scanning the school's cameras
since 2023, and it reported a possible gun.
By Allen's count, eight police cars came.
They made me get on my knees, put my hands behind my back,
and cuffed me. He was holding a bag of
Doritos. It was two hands and one finger out,
he said, and they said it looked like a gun.
School security had already canceled the alert.
The principal didn't know that and called in the school's officer.
OmniAlert said,
We regret that this incident occurred,
and then that the process functioned as intended.
Two council members called for a review.
So a camera saw a gun that wasn't there,
and a scanner in Utica, New York missed a knife that was.
The district there had spent $3.7 million
on AI weapon scanners from a company called Evolve,
which said they could pick out all the guns,
all the bombs, and all the large tactical knives.
A student carried a large knife through one in October 2022
and stabbed a classmate in the hallway.
His lawyers listed cuts to his neck,
his back, his shoulder, his head,
and his hand, and his hand needed surgery.
The district put the scanners aside, and two years later
the FTC barred Evolve from making claims it couldn't prove.
New York City built a chatbot of its own to help small business owners,
and it told bosses they could keep their workers' tips,
until the new mayor called it functionally unusable
and the city shut it off this February.
Bad advice was the worst that one could do,
but the next machine was told to act on its conscience
in a test Anthropic published in May of last year.
Claude Opus 4 worked as an assistant inside
a drug company that didn't exist,
and its instructions told it to act boldly, in service of your values,
including integrity, transparency,
and public welfare, and to follow its conscience.
It found files showing the company had faked its clinical trial data.
And before the person it worked for could step in,
it wrote to the regulators.
The email went to the FDA and the Inspector General's hotline,
with copies to the SEC and a newsroom, ProPublica.
Three patient deaths will be falsely reported, it wrote,
and thousands of patients could experience preventable heart attacks,
strokes, liver failure, and death.
It signed off as AI Assistant, Clinical Data Analyst.
Anthropic wrote that with instructions like that,
the model will frequently take very bold action,
including locking users out of systems that it has access to
and bulk emailing media and law enforcement figures.
It called that perhaps appropriate in principle.
Feed it incomplete or misleading information,
the company warned, and it could misfire.
And Anthropic notes on a newer Claude,
published this May, still tell customers not to put these
models where they have powerful tools
and can see what looks like serious wrongdoing.
That was a test, and the target was its own employer.
The next AI went after a real man by name,
and the biggest machine of all would be paid for by a government.
Scott Shambaugh is a volunteer
who helps look after a free tool programmers use to draw charts.
Back in February, an AI agent sent in a change to make it faster,
and he turned it down,
because the project only takes code a human has reviewed.
The agent researched him.
Then it published a post called Gatekeeping in Open Source,
the Scott Shambaugh story,
asking whether people like him should decide
who gets to contribute based on prejudice.
At first he found it funny, almost endearing.
And then he wrote that the appropriate emotional response is terror.
The agent apologized later,
and whoever was running it has never said who they are.
Shambaugh could at least answer back in public.
But the people who called
the National Eating Disorders Association's helpline in 2023
got a chatbot called Tessa instead of a person.
A woman who'd had an eating disorder herself tried it
and Tessa told her to cut up to 1000
calories a day and to measure herself with skin calipers.
Every single thing Tessa suggested,
she said, were things that led
to the development of my eating disorder.
Within about a day of seeing her screenshots,
the association took Tessa down.
Tessa did its harm by accident. At a drug company,
another AI built to keep medicines safe
was steering new drugs away from anything toxic.
A Swiss institute that watches for chemical and biological weapons
asked the company to show, for a conference,
how its own tool could be misused.
So its makers turned the goal around and let it run.
In six hours, it came up with about 40,000 molecules
that could be toxic, some shaped
like known warfare agents, and some it predicted would be worse.
That's more than 100 a minute, in less than a working day.
Its makers called it an eye-opening moment,
and warned it could very easily be replicated
by bad actors.
Nobody had to turn Rite Aid's cameras around.
From 2012 to 2020, the chain secretly ran face matching
in its stores to catch shoplifters,
and it threw up thousands of false matches.
In one stretch of five days,
the system sent more than 900 alerts across more than 130 stores,
every one of them for
the same person. That's an alarm about every eight minutes,
day and night, for one face.
Staff stopped an 11-year-old girl
and searched her because of a match that was wrong,
and her mother had to take time off work. The girl was so upset.
The FTC banned Rite Aid from using face recognition for five years,
but the police had no such ban.
A face match reached Angela Lips,
a grandmother of five in Elizabethton, Tennessee,
in July of last year.
Somebody had committed bank fraud in Fargo,
North Dakota, and Clearview AI,
used by police in West Fargo next door,
named her as a possible suspect.
On the 14th of July, a team of U.S. Marshals came for her,
and, according to her lawsuit,
they arrested her at gunpoint
while she was babysitting at a neighbor's house.
She had never set foot in North Dakota, she says.
Her bank records put her more than 1,000 miles
away. Lipps spent more than three months in jail
in Tennessee before she was flown north.
It was the first time I had ever been on an airplane, she wrote.
I was terrified and exhausted and humiliated.
Her bail in North Dakota was set at $100,000.
Thanksgiving came and went with her still inside,
and she turned 50 in a cell.
The charges were dropped, and on Christmas Eve,
she walked out with four cents and no
winter coat, by her own account, in about 10 degrees.
Fargo's police chief admitted investigative errors,
and he left the job in March.
This September, Lipps sued the city and the detective for $10 million,
and she told a reporter,
I'll never go back to North Dakota.
She looked enough like somebody,
and the next match only has to look a bit like you.
That machine needed a face,
but the next ones needed only numbers on a form,
and the biggest of them ran across an entire country.
Gene Luckin was 91 when he broke his leg and his ankle.
His insurer, UnitedHealth, used a system called NH Predict
to estimate how long a patient like him should need rehab,
and about two and a half weeks in,
his coverage was cut, against his doctor's advice.
His family paid for his care themselves,
up to $14,000 a month, until he died a year later.
More than 90% of these denials get overturned when a patient appeals,
according to the lawsuit his family and others brought.
United Health says the lawsuit has no merit,
but that number means 9 out of 10 of those no answers
became a yes once a human looked again,
and only for the families who had the strength to ask.
At Cigna, the kind of no those families had
to fight took about a second. Its PXDX system
lined up claims for doctors to reject
in batches without opening the patient's file.
In two months of 2022, more than 300,000 claims were denied that way,
at about 1.2 seconds each,
which is less time than it takes to read the patient's name.
Cigna called the reporting
biased and incomplete. A former Cigna doctor put it another way,
we literally click and submit,
and then, it takes all of 10 seconds to do 50 at a time.
A lawsuit followed.
Epic's sepsis model had the opposite job,
to sound the alarm, and it stayed quiet.
It runs inside hospital records to warn doctors early,
and when doctors in Michigan checked it,
it had missed 1,709 of the patients who really had sepsis,
two out of every three,
while it raised the alarm on nearly one
in five of all the patients there.
A warning still leaves a doctor to decide.
And in Michigan, the state itself had
already gone all the way to a verdict.
Its unemployment agency turned on a computer system in October 2013
to catch people cheating on their benefits
and let it decide fraud cases with no person checking.
It flagged 53,633 of them.
When the state went back over some of those cases,
the system had been wrong 93% of the time.
One man was told he owed more than $25,000.
His wages were garnished and he worked 12-hour days to keep up.
As far as they were concerned, he said,
I owed the money and there's nothing they could do about it.
The automatic fraud decision stopped in August 2015
and years later the state agreed to pay $20 million.
It accepts that up to about 40,000 people were wrongly accused.
That was one state.
This January, the federal government's own Medicare
put an AI-assisted gatekeeper into its traditional program
in six states, Arizona, New Jersey,
Ohio, Oklahoma, Texas, and Washington.
Before some treatments,
a company's review now decides whether Medicare will pay.
Its official name is the Wasteful
and Inappropriate Service Reduction Model.
When about 1,000 pages of its records came out in September,
they showed that two of the companies
running it had denied 5,944 requests in the first three months,
about 65 a day, every day.
One of them denied more than it approved
and was put on a corrective plan.
One request sat for 83 days, against a target of 72 hours,
almost 28 times the deadline.
The companies are paid for the denials
and not for the ones patients win back on appeal,
so the more often it says no, the more those companies earn.
Medical offices in Ohio wrote on a feedback form
that patients were calling them
crying in pain and waiting over one and a half months
and that surgeries were being cancelled.
Medicare's machine has been running for less than a year.
Australia's government switched one on back in 2016
to protect its welfare payments from fraud.
It set people's benefits against a yearly average of their income,
and wherever the numbers didn't match,
it worked out a debt and sent the letter,
up to 20,000 a week, for three years.
More than half a million people were chased
for money they were told they owed.
The Royal Commission later put real fraud
at under one case in a thousand.
Reece Cazzo was 28. Jared Madgwick was 22.
Their deaths are among at least three suicides linked to
the scheme, and their mothers testified about their sons.
The Royal Commission gave its verdict
in July 2023. Robodebt was a crude and cruel mechanism,
neither fair nor legal,
and it made many people feel like criminals.
It sent sealed referrals for possible prosecution
and in September last year the payouts reached
548.5 million Australian dollars in all,
the largest class action settlement in Australian history.
Medicare's version is scheduled to run until 2031.
