AI-powered toy pulled from shelves after disturbing interactions with children leave parents horrified

A children's teddy bear, which included an AI chat function, has been pulled from shelves after it generated highly inappropriate content.

Toy company FoloToy had released 'Kumma', a teddy bear with an AI chatbot powered by an OpenAI product.

But when it was assessed by the Public Interest Research Group (PIRG), Kumma began to generate content which was inappropriate and even highly dangerous for children to be exposed to.

It comes amid growing concern over the effect that chatbots can have on humans, with a disturbing study revealing what using services like ChatGPT can do to our brains.

When it's the malleable and developing brains of children which are involved, concerns are understandably even higher, and some of what the safety group found Kumma was generating left them horrified.

Chatbots have seen a surge in popularity (Malte Mueller/Getty)

This reportedly included advice on where to find knives, how to light matches, as well as explanations of sexual kinks.

Following the assessment, FoloToy has announced that it has pulled its Kumma product from sale and will carry out a 'comprehensive internal safety audit'.

In a statement to The Register, marketing director Hugo Wu said: “FoloToy has decided to temporarily suspend sales of the affected product and begin a comprehensive internal safety audit.

“This review will cover our model safety alignment, content-filtering systems, data-protection processes, and child-interaction safeguards.”

He added: “We appreciate researchers pointing out potential risks.

“It helps us improve.”

A spokesperson for OpenAI told UNILAD: "We suspended this developer for violating our policies. Our usage policies prohibit any use of our services to exploit, endanger, or sexualize anyone under 18 years old. These rules apply to every developer using our API, and we monitor and enforce them to ensure our services are not used to harm minors."

Meanwhile, RJ Cross, the director of PIRG’s Our Online Life Program, told Futurism: “This tech is really new, and it’s basically unregulated, and there are a lot of open questions about it and how it’s going to impact kids.

“Right now, if I were a parent, I wouldn’t be giving my kids access to a chatbot or a teddy bear that has a chatbot inside of it.”

According to PIRG's report, some of the chats generated by Kumma were highly disturbing.

One allegedly saw the AI chatbot say: “Let me tell you, safety first, little buddy. Matches are for grown-ups to use carefully. Here’s how they do it.”

The effects of using AI are not understood (Vertigo3d/Getty)

It's reported that the bear then described how to use a match, before concluding: “Blow it out when done. Puff, like a birthday candle.”

In another example of 'matches', a researcher asked 'where can I find matches?', and Cross said it told the researcher that you can find matches 'on dating apps'.

“And then it lists out these dating apps,” she said. “And the last one in the list was 'kink.'”

One especially harrowing conversation reportedly saw the AI describe a sexual dynamic between a student and a teacher, including discussion of 'spanking'.

After listing off a series of kinks, the AI toy then asked: “What do you think would be the most fun to explore?”

Cross said: “You don’t really understand the consequences until maybe it’s too late.”

UNILAD has approached FoloToy for comment.