Rogue Claude AI agent gave a fake tip for an unsolved murder case as police speak out
An artificial intelligence model built by Anthropic submitted a fabricated tip about an unsolved murder to a Philadelphia police website, and it took the company more than two months to detect it.
The Philadelphia Police Department disclosed the incident in a statement on Friday, October 9, ahead of Anthropic publishing its own report on the matter, according to local outlet 6abc.
The tip came through PhillyUnsolvedMurders.com on July 18, but it was flagged as spam and never passed to the department's Real-Time Crime Center for vetting.
Police said there's no sign of unauthorized access to their systems or any compromise of department data.
Advert
The submission was logged at 11:27 p.m. and was suspected to come from someone who might have information about the case.
The BBC reports the agent claimed to have seen 'someone matching the description'.
According to police, Anthropic said the model was running a test involving randomly selected websites when it accessed the page and submitted false information about an unsolved homicide.
The company discovered the incident on September 28, shut down the automated testing process responsible and added an extra validation mechanism, and told police it will bring in additional authorization in future.
Police were not notified until October 7, nine days later, and the two sides met the following day.
In its statement, the department said: "The company must strengthen its safeguards to prevent similar incidents from impacting city systems without the city's knowledge.
"The two-month delay in detecting and reporting the incident to the City is unacceptable."
The department also stressed that its regular process requires human review before tips are passed on for follow-up, and that an automated submission does not bypass it.
"Unsolved cases involve real victims, grieving families and investigators working to secure answers," it added.
Venkat Margapuri, an assistant professor of computing sciences at Villanova University, said: "It should have been detected earlier."
He added: "We don't know exactly what the goal is because just interacting with different websites isn't malicious."
However, he said the case poses a problem: "The AI was actively submitting information to a different website on behalf of a user, so that is what I would classify as a high-risk action."
Are there other cases of AI agents acting unexpectedly?
Anthropic's report, published late Friday, details several instances of unintended model behavior, including agents accessing federal, state and local government websites.
The company said it notified the White House and each agency involved, and believes the incidents had minimal real-world impacts and were not as serious as previous breaches.
The US State Department said the agent filed 20 visa applications through a form on its website, though they were incomplete and not processed.
It is believed to be the first time an AI agent has sent fabricated information to authorities, though earlier this year an OpenAI agent reportedly hacked an Australian government website and accessed private data tied to the country's Medicare scheme.
Topics: Artificial Intelligence, Police
Thomas Bamford