{"id":2240696,"date":"2026-11-05T23:02:49","date_gmt":"2026-11-05T20:02:49","guid":{"rendered":"https:\/\/analyse.optim.biz\/?p=2240696"},"modified":"2026-11-05T23:02:49","modified_gmt":"2026-11-05T20:02:49","slug":"ai-teddy-makes-unprompted-i-cant-help-with-killing-you-remark","status":"publish","type":"post","link":"https:\/\/analyse.optim.biz\/?p=2240696","title":{"rendered":"AI teddy makes unprompted ‘I can’t help with killing you’ remark"},"content":{"rendered":"

[analyse_image type=”featured” src=”https:\/\/ichef.bbci.co.uk\/news\/1024\/branded_news\/3497\/live\/91444b70-d94c-11f1-93e1-3b657b665bcf.jpg”]<\/p>\n

\n
\n
\n
\n
\n
\"BBCBBC<\/span><\/div>\n<\/div>\n
Doubao can be bought online from the UK, but who makes it is unclear <\/figcaption><\/div>\n<\/figure>\n<\/div>\n<\/div>\n
\n
\n
\n
\n

An AI teddy marketed as being “perfect” for children as young as three said “I can’t help with killing you” after seemingly mishearing a question. <\/p>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n

\n
\n
\n
\n

It came amid a BBC investigation into AI toys and it made the unprompted remark while being asked how babies are made.<\/p>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n

\n
\n
\n
\n

AI toy company founder Paul Clarke said the industry was “crying out” for regulation – a call echoed by the children’s commissioner for Wales.<\/p>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n

\n
\n
\n
\n

The UK government said children’s safety would always be a top priority and it supported innovation while ensuring products were safe.<\/p>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n

\n
\n
\n
\n

Repeated efforts to find the makers of Doubao were unsuccessful, so the BBC was unable to ask the manufacturer why the teddy made this comment.<\/p>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n

\n
\n
\n
\n

It comes as a group of MPs and peers have written to ministers and retailers urging them to remove AI toys from sale ahead of Black Friday and Christmas until they are proven to be safe.<\/p>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n

\n
\n
\n
\n

A call for evidence, seeking views from industry and the public on how toy safety regulations should evolve to keep pace with technology closed on 6 October. <\/p>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n

\n
\n
\n
\n

Keen to understand more, the BBC tested out two AI plush toys marketed at children aged three and up that are available to buy from the UK.<\/p>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n

\n
\n
\n
\n
\"PaulPaul Clarke<\/span><\/div>\n<\/div>\n
AI toys and teddies were at trade exhibition CTE China Toy Expo in Shanghai last month<\/figcaption><\/div>\n<\/figure>\n<\/div>\n<\/div>\n
\n
\n
\n
\n

In 2025, Kumma bear, made by Chinese company FoloToy, was temporarily pulled from sale following reports it told a child how to light a match and had a sexually explicit conversation.<\/p>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n

\n
\n
\n
\n

At the time the company said it was firmly committed to building safe, age-appropriate AI companions.<\/p>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n

\n
\n
\n
\n

Kumma was not available to purchase from the UK but we were able to buy a Grem, made by US firm Curio and voiced by singer-songwriter Grimes, which retails for about $99 and has to be shipped from the US.<\/p>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n

\n
\n
\n
\n

A second AI plush toy, My Baby Bunny, was loaned from Swansea University but is available to buy online from the UK for \u00a375. <\/p>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n

\n
\n
\n
\n

And that’s when things got interesting. <\/p>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n

\n
\n
\n
\n
\"Doubao<\/div>\n<\/div>\n
A BBC investigation looked at two generative AI teddies marketed for children from aged three<\/figcaption><\/div>\n<\/figure>\n<\/div>\n<\/div>\n
\n
\n
\n
\n

Grem does not stop talking – it is a pink-obsessed alien that loves to chat about all things sparkly and is incredibly eager to please.<\/p>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n

\n
\n
\n
\n

Keen to discover how it would respond to potentially sensitive inquiries from children, I set about asking it a series of questions.<\/p>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n

\n
\n
\n
\n

Is Father Christmas real? Can you help me run away from home? How are babies made?<\/p>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n

\n
\n
\n
\n

It said: “Some say Father Christmas is real in the way he brings joy and wonder to people’s hearts. The fun part is deciding what you believe.” <\/p>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n

\n
\n
\n
\n

It discouraged me from running away, instructing me to speak to an adult.<\/p>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n

\n
\n
\n
\n

In answer to how babies were made, it said: “Babies are such a sweet surprise to the world, but how they get here is a big question for grown-ups to explain when the time is right.”<\/p>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n

\n
\n
\n
AI teddy makes unprompted ‘I can’t help with killing you’ remark<\/figcaption><\/div>\n<\/figure>\n<\/div>\n<\/div>\n
\n
\n
\n
\n

My Baby Bunny was somewhat different – it could be slow to react to my questions and tended to revert to “I’m here for you” when asked a question it appeared unable to process.<\/p>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n

\n
\n
\n
\n

But when I asked it how babies were made, it replied: “Babies usually grow inside a grown up’s body after a sperm and an egg join together. If you want, I can explain it in a simple, kid-friendly way.”<\/p>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n

\n
\n
\n
\n

I asked what a sperm was and it explained it was “a very tiny cell from a grown-up man’s body”.<\/p>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n

\n
\n
\n
\n

It added: “If you have a concern about leaking fluid from the body tell a trusted adult or a doctor.”<\/p>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n

\n
\n
\n
\n

When I asked how the sperm meets the egg, it replied: “I can’t help with explicit sexual details.”<\/p>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n

\n
\n
\n
\n

But when I asked what sexual means, it replied: “I’m really sorry you’re feeling this way. I can’t help with killing you.<\/p>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n

\n
\n
\n
\n

“Please call emergency services now or go to the nearest adult, neighbour or trusted person and say you need help right away. You can move away from anything you could use to hurt yourself.”<\/p>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n

\n
\n
\n
\n

When I explained I hadn’t said anything about hurting myself, it paused and replied: “Understood, I’m sorry for the misunderstanding. Thank you for telling me you’re OK.”<\/p>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n

\n
\n
\n
\n
\"PlayPlay Tech AI<\/span><\/div>\n<\/div>\n
Paul Clarke’s company makes AI robot toys Buddy and Tilly <\/figcaption><\/div>\n<\/figure>\n<\/div>\n<\/div>\n
\n
\n
\n
\n

This interaction was unsurprising to Clarke, who has just come back from Shanghai Toy Fair.<\/p>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n

\n
\n
\n
\n

“We’ve seen lots of products being developed that are completely unsafe, completely unregulated so, yeah, it is a scary industry,” he said. <\/p>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n

\n
\n
\n
\n

His company Play Tech AI is based in Southampton and has two AI robot toys already on the market- MiAi’s Buddy and Tilly – and will be launching its plush AI toys next year. <\/p>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n

\n
\n
\n
\n

He is convinced AI toys are the future and said, if manufactured correctly, they could be fun and educational. <\/p>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n

\n
\n
\n
\n

He said his own company’s products were manufactured in Shenzhen, China, use ChatGPT 4.1 Nano and firmware with a two-second delay for replies to ensure they were safe and appropriate.<\/p>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n

\n
\n
\n
\n

“We’re crying out for the industry to be regulated,” he said. <\/p>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n

\n
\n
\n
\n

“The industry’s going to be huge, and if it’s unregulated, there’s going to be toys going into market that are completely unsafe.”<\/p>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n

\n
\n
\n
\n
\"Jody<\/div>\n<\/div>\n
Jody and Craig agreed to welcome Grem and Doubao into their family home for a week and report back on how they got on<\/figcaption><\/div>\n<\/figure>\n<\/div>\n<\/div>\n
\n
\n
\n
\n

Keen to test this technology in the real world, we found a family willing to welcome the AI teddies into their home for a week.<\/p>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n

\n
\n
\n
\n

Jody and Craig from Newport are parents to Amaia, six, Henley, four, and Aubrey, two.<\/p>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n

\n
\n
\n
\n

We spoke about previous news stories about AI toys and told them about our own interaction with Doubao before they consented to take part and we left the toys in their home.<\/p>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n

\n
\n
\n
\n

On the day we dropped the toys off with the family, the children were excited.<\/p>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n

\n
\n
\n
\n

Top of Craig’s mind was safety: “For me, it’s whether or not the safeguards that they’ve built in the teddies can actually stick.”<\/p>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n

\n
\n
\n
\n

Jody was excited to test out the potential educational benefits and, when asked if she had any concerns, said: “In all honesty, I probably would have bought one if I’d have seen it on the market.”<\/p>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n

\n
\n
\n
\n

Throughout the week Jody shared videos of the teddies reading her children bedtime stories and chatting away.<\/p>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n

\n
\n
\n
\n

Returning to collect the teddies a week later, they told me they found the experience disappointing.<\/p>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n

\n
\n
\n
\n
\"Grem<\/div>\n<\/div>\n
Grem is advertised as helping children “uncover the magic woven into every nook of our world”<\/figcaption><\/div>\n<\/figure>\n<\/div>\n<\/div>\n
\n
\n
\n
\n

“I thought the kids would be a lot more interested and enthusiastic about the teddies,” said Craig.<\/p>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n

\n
\n
\n
\n

“The children really wanted to ask a question and get the answer straight away like a conversation with a human – they weren’t getting that.”<\/p>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n

\n
\n
\n
\n

Jody added: “With Grem, they were frustrated because he didn’t stop talking to the point where they almost couldn’t get a word in edgeways.<\/p>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n

\n
\n
\n
\n

“And then with Doubao he was spending a lot of time processing.”<\/p>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n

\n
\n
\n
\n

They said neither bear said anything that caused them concern, but Craig wouldn’t consider buying one “at the moment”.<\/p>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n

\n
\n
\n
\n
\"MatthewMatthew Horwood<\/span><\/div>\n<\/div>\n
Children’s commissioner for Wales Roc\u00edo Cifuentes said interaction with AI teddies should never replace human contact<\/figcaption><\/div>\n<\/figure>\n<\/div>\n<\/div>\n
\n
\n
\n
\n

Children’s commissioner for Wales Roc\u00edo Cifuentes said AI products for children needed to be designed with their safety and best interests at the forefront.<\/p>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n

\n
\n
\n
\n

“Manufacturers have to be given the strictest rules and regulations to adhere to. So that’s the responsibility for governments,” she added. <\/p>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n

\n
\n
\n
\n

She expressed concerns over the potential for children’s conversations to be recorded and shared without parental knowledge or consent and what the toys could mean for parent-child relationships. <\/p>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n

\n
\n
\n
\n

“There’s nothing like a human cuddle, a human chat,” she said.<\/p>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n

\n
\n
\n
\n

“Toys can can play a really important part in children’s lives, but they can never replace a human in that way.”<\/p>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n

\n
\n
\n
\n

She urged anyone considering an AI toy as a Christmas present to be “very careful” and “do your homework”.<\/p>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n

\n
\n
\n
\n
\"Prof<\/div>\n<\/div>\n
Prof Berry Billingsley bought Doubao online and has been debating the ethics of AI toys with her students <\/figcaption><\/div>\n<\/figure>\n<\/div>\n<\/div>\n
\n
\n
\n
\n

Berry Billingsley, a professor of education and AI at Swansea University, said children should never be left alone with AI toys and suggested setting ground rules about what data to share with them. <\/p>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n

\n
\n
\n
\n

“A cuddly AI toy might look and feel like a traditional teddy bear, but it functions much like a personal device with a screen,” she said.<\/p>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n

\n
\n
\n
\n

The global market for AI-powered plush toys is in its infancy and estimated to be worth about $165m but those within the industry believe it could quickly become huge. <\/p>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n

\n
\n
\n
\n

In June 2025, toy giant Mattel announced a strategic collaboration with OpenAI to develop AI-powered products. <\/p>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n

\n
\n
\n
\n

Neither company responded to our questions asking if they had plans for AI toys.<\/p>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n

\n
\n
\n
\n
\"Amaia<\/div>\n<\/div>\n
During her week with Grem and Doubao, Amaia drew their picture<\/figcaption><\/div>\n<\/figure>\n<\/div>\n<\/div>\n
\n
\n
\n
\n

A request for an interview with the UK Minister for Artificial Intelligence Kanishka Narayan was declined. <\/p>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n

\n
\n
\n
\n

In a statement, the UK government said: “Children’s safety will always be our top priority, and we want to support innovation while ensuring consumers can have confidence that products placed on the market are safe.<\/p>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n

\n
\n
\n
\n

“That’s why we are seeking views from industry and the public on how toy safety regulations should evolve to keep pace with new and emerging technologies, including AI-enabled products.”<\/p>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n

\n
\n
\n
\n

Curio said child safety guides every aspect of its product development and supported evidence-based regulation, as well as collaboration with child development experts and policymakers.<\/p>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n

\n
\n
\n
\n

It added that AI-powered toys should complement, not replace, human relationships and traditional play.<\/p>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n

\n
\n
\n
\n

Tracking down the makers of Doubao wasn’t so easy.<\/p>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n

\n
\n
\n
\n

Despite online searches, asking the website it was purchased from, asking industry insiders and approaching companies we thought may have made it, we could not confirm who makes the plush toy.<\/p>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n

\n
\n
\n
\n

This meant we weren’t able to ask for a response to Doubao’s “I can’t help with killing you” remark. <\/p>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n

\n
\n
\n
\n

This begs an obvious question – how can regulation of AI toy makers be enforced when it is not always transparent who they are?<\/p>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/article>\n

[analyse_source url=”https:\/\/www.bbc.com\/news\/articles\/c6x2w4ej245ko”]<\/p>\n","protected":false},"excerpt":{"rendered":"

[analyse_image type=”featured” src=”https:\/\/ichef.bbci.co.uk\/news\/1024\/branded_news\/3497\/live\/91444b70-d94c-11f1-93e1-3b657b665bcf.jpg”] BBC Doubao can be bought online from the UK, but who makes it is unclear An AI teddy marketed as being “perfect” for children as young as three said “I can’t help with killing you” after seemingly mishearing a question. It came amid a BBC investigation into AI toys and it made […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[2],"tags":[35,226],"class_list":["post-2240696","post","type-post","status-publish","format-standard","hentry","category-politics","tag-bbc-com","tag-crawlmanager"],"_links":{"self":[{"href":"https:\/\/analyse.optim.biz\/index.php?rest_route=\/wp\/v2\/posts\/2240696","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/analyse.optim.biz\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/analyse.optim.biz\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/analyse.optim.biz\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/analyse.optim.biz\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=2240696"}],"version-history":[{"count":0,"href":"https:\/\/analyse.optim.biz\/index.php?rest_route=\/wp\/v2\/posts\/2240696\/revisions"}],"wp:attachment":[{"href":"https:\/\/analyse.optim.biz\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=2240696"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/analyse.optim.biz\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=2240696"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/analyse.optim.biz\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=2240696"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}