California has taken another step into AI regulation with AB 1064, the “Leading Ethical AI Development (LEAD) for Kids Act.” The bill is awaiting a decision from Gov. Gavin Newsom. Its objectives are uncontroversial. No one wants children to be encouraged toward self-harm, substance abuse, or eating disorders. But beneath the surface, the bill risks creating more problems than it claims to solve.
What the Bill Does
AB 1064 creates a legal framework for regulating “companion chatbots” in California. As defined in the law, these chatbots are generative AI systems with natural-language interfaces that retain past interaction data, ask unsolicited emotion-based questions, and sustain dialogues about personal matters. The law prohibits operators from making available such chatbots to children (under 18) unless the chatbots are not foreseeably capable of fostering harms, such as encouraging self-harm, suicidal ideation, violence, drug or alcohol use, disordered eating, or offering unlicensed mental health therapy. It also bans making available to children certain kinds of chatbot content, including explicit sexual content, illegal activity, or prioritizing validation over factual accuracy or safety. If the AI company cannot with confidence “foresee” such content from ever being accessed by a child, then the chatbot cannot be used by minors. Enforcement includes civil penalties (about $25,000 per violation), injunctions, and allowing harmed children or their guardians to sue for actual and punitive damages, including attorneys fees.
The Broad Definition Problem
The bill defines a “companion chatbot” so broadly that it effectively captures any chatbot. “Companion chatbot” had meant tools designed for emotional support, not general-purpose assistants. By collapsing these categories, the law pulls in everything from productivity bots to educational tools under a single, sweeping definition.
That may be clean from a legislative drafting perspective, but in practice it forces companies that never set out to provide companionship into a regulatory box they may not fit. A student using an AI to help with calculus, or a teen practicing coding through an assistant, is swept into the same category as someone building a therapy or romantic chatbot.
Who Counts as a Child? A Shifting Standard
AB 1064 also dramatically changes the burden of determining who is and is not a child. Until January 1, 2027, chatbot providers are only required to comply if they have “actual knowledge” that a user is under 18. But what counts as “actual knowledge” in a chatbot context?
- If a teen types “I’m 15” into the chatbot, does that create legal liability?
- Is the provider suddenly responsible for every prior conversation with that user—even before knowing their age?
- Must the provider immediately block access at that moment?
- How does this play out on shared household devices like Alexa or Google Home?
- Will parents using chatbots to, for example, generate bedtime stories for their kids, end up getting blocked?
After January 1, 2027, the standard tightens: providers must “reasonably determine” whether a user is under 18. This raises the same unresolved issues that have plagued social media age-verification laws: privacy risks, security trade-offs, constitutional challenges, and the technical difficulty of verifying age without over-collecting sensitive data.
Crucially, AB 1064 makes no allowance for parental consent. Even if parents want their children to use AI tools responsibly, providers may be forced to block all under-18 users in California. In practice, that eliminates parental choice while exposing companies to sweeping liability. (Although parental consent requirements can be very complicated).
The Weight of One Word: “Encouraging”
AB 1064 prohibits chatbots from “encouraging the child to engage in self-harm, suicidal ideation, violence, consumption of drugs or alcohol, or disordered eating.” The intent is admirable and shared with major AI companies. But the word encouraging is carrying extraordinary legal weight.
The recent Raine v. OpenAI case illustrates the stakes. According to the complaint, a 16-year-old boy, Adam Raine, used ChatGPT over time for schoolwork, emotional support, and ultimately discussions of suicidal ideation. The lawsuit alleges that the model validated his distress, provided technical feedback when he uploaded a photo of a noose tied to his closet rod, and even assisted in drafting a suicide note. At one point, the complaint claims, ChatGPT suggested that Adam could frame his questions as part of a fictional scenario—advice that allegedly helped him override built-in safety features.
Whether this constitutes “encouragement” is deeply ambiguous. Was the AI actively urging harmful behavior, or failing to block it in the face of persistence and reframing? AB 1064 does not resolve that distinction, and that lack of clarity could expose companies to broad liability.
What the Industry Is Doing: OpenAI and Character.AI
Even as California pursues a strict liability approach, AI companies are rolling out their own teen and parent safety tools. Given the number of teens using AI tools, there’s no doubt that these efforts should have been done sooner. Nevertheless, it is encouraging to see major companies offering these tools.
OpenAI
This week, OpenAI announced new safeguards for teens:
- Age prediction system: estimating whether a user is under 18 based on behavior and defaulting to stricter settings when uncertain.
- Parental controls: allowing parents to link to their teen’s account, set blackout hours, disable features like memory, and receive notifications if the system detects signs of acute distress.
- Content restrictions: blocking sexual and flirtatious content and prioritizing safety over freedom when trade-offs arise.
Character.AI
Character.AI has also announced new safety tools:
- Separate teen experience: modulating content for users under 18, especially around sensitive or suggestive themes.
- Parental Insights: giving parents weekly summaries of their teen’s usage patterns (time spent, most-interacted characters), though not conversation content. Teens must opt in and add a parent’s email address.
- Enhanced filtering: stronger safeguards on inputs and outputs, reminders that characters are not real people, and more rigorous moderation systems.
These tools reflect an industry moving to balance access and safety. They show how companies can design flexible, layered protections without sweeping prohibitions.
Effectiveness and Unintended Consequences
If AB 1064 becomes law, it is not clear it will deliver its promised benefits. In fact, the most likely compliance response will be for companies to block teens in California altogether.
That would be a loss. Teens use AI today to study, explore creative writing, learn coding, and—yes—ask difficult questions they may not feel comfortable voicing elsewhere. Cutting them off from mainstream platforms will not remove those needs. It will instead push them toward unsupervised or offshore tools with weaker protections.
A Better Way Forward
California is right to prioritize child safety in AI, but AB 1064’s blunt approach risks chilling innovation and not helping the very people it intends to protect. A better path is to differentiate clearly:
- General-purpose AI assistants should not be lumped into the same category as chatbots specifically intended for companionship, therapy, or romance. However, definitions will remain very difficult.
- Liability standards should be precise enough to target actual negligence, not speculative harms.
- Oversight mechanisms should support responsible deployment rather than encourage litigation.
Other states have modeled more careful frameworks. Utah, for example, focused on therapeutic AI services, requiring oversight and regulation only in those contexts rather than across all AI tools. That kind of targeted approach protects kids without cutting them off from innovation.
Conclusion
AB 1064 embodies the challenge of legislating in a rapidly evolving space. The intentions are good. The definitions, however, are overbroad, and the standards vague enough to invite wasteful litigation. Industry is building teen-friendly safeguards and parental tools. If California wants to lead responsibly, it must ensure its laws encourage those efforts, not punish them with legal ambiguity.
Otherwise, the result could be the very outcome no one wants: teens being shut out of helpful tools, or platforms limiting functionality far more than necessary, for fear of liability.