Character.AI has rolled out new safety features and policies for building and interacting with the AI-powered virtual personalities it hosts. The new measures aim to make the platform safer for all users, but particularly younger people. The update includes more control over how minors engage with the AI chatbot, more content moderation, and better detection of the AI discussing topics like self-harm.
Though not cited in the blog post about the update, Character AI linked to the announcement in a post on X expressing condolences to the family of a 14-year-old who spent months interacting with one of Character.AI’s chatbots before taking his own life. His family has now filed a lawsuit against Character.AI for wrongful death, citing a lack of safeguards for the AI chatbots as a contributor to his suicide.
We are heartbroken by the tragic loss of one of our users and want to express our deepest condolences to the family. As a company, we take the safety of our users very seriously and we are continuing to add new safety features that you can read about here:…October 23, 2024
AI chat guardrails
Character AI’s post laid out several new safety features for the platform. For instance, if the model detects keywords related to suicide or self-harm, it will display a pop-up urging the user to the National Suicide Prevention Lifeline and related resources. The AI will also be better at spotting and removing inappropriate content in a conversation, with a particular sensitivity to when users are under 18.
Presumably, minors would already have restricted content in conversations, but Character.AI may have upped that sensitivity further. In cases where that might not be enough,…
Read full post on Tech Radar