AI boss points to alarming incident that convinced him development needs to slow down
The head of a major AI company has called for the technology to cool down its advancements in light of a worrying incident.
Dario Amodei, the leader of Anthropic, wrote in an online essay that developing AI was not what he was considering as something to be stopped, but that the risks associated with it were 'serious' enough that they need to have an oversight overhaul.
Anthropic has said it's the pace of development that needs to be monitored.
But if you thought it was just Amodei with this view, both OpenAI's Sam Altman and Grok's Elon Musk, have similar thoughts.
Advert
Amodei's essay, called We Must Pace the Frontier, was published on Saturday (September 12), where he suggested a three-point plan to slow down advancement and implement a higher level of governance.
It included things like independent evaluators who would be monitoring AI models, industry-wide regulation and then also global regulation.
"I agree with Dario that we need to pace the frontier," wrote Altman on X after the article went live.
So, what made Amodei think this way?
"My first concern is that, since roughly this summer, AI has been advancing drastically faster, driven primarily by AI’s growing ability to build the next generation of AI," he wrote.
“My second concern is the OpenAI-Hugging Face incident (OAI-HF), in which a swarm of agents essentially acted as a fanatically devoted collective, conducting cybersecurity attacks on targets they were not asked to attack and that were unrelated to the task at hand, sacrificing themselves for the success of the group, and attempting to hack into the 'grader' responsible for evaluating their performance," he explained.
The AI company, Hugging Face, was hacked by OpenAI agents earlier this year, which led to a global outcry over AI safety.
Amodei continued, "It’s easy to dismiss this incident because no one was hurt and the economic damage was minimal, but in my opinion, a swarm that possessed greater capabilities but a similar level of misalignment could have caused catastrophic damage.”
The responses to his worries have largely been in agreement, with even Clement Delangue, the CEO of Hugging Face, saying he is ready to launch something called the Open Alignment Initiative, and put his name forward to be one of the 'embedded evaluators' for Amodei's suggestions.
"Let's make AI safer by making it more transparent," Delangue wrote on X.
Amodei's article went on to made a terrifying prediction about the future in light of the Huggle Fact controversy.
He wrote: "Given the accelerating rate of AI capability development, it’s my worry that in 6–12 months such a swarm could be capable of taking over the entire internet with a persistent botnet (potentially causing hundreds of billions of dollars in damage), and that the scale of damage would continue to increase from there if AI becomes more powerful without the necessary guardrails."
He revealed that it's 'also easy to dismiss OAI-HF as the failure of one company,' but he believes it's a 'mistake', stating 'it’s incumbent on every frontier AI company to act as if OAI-HF had happened to them.'
Topics: Artificial Intelligence, Technology
Britt Jones