Read this post on Neil’s Substack: Getting Out of Control.
I’ve been taking a closer look at California’s SB 53, the “Transparency in Frontier Artificial Intelligence Act.” It’s California Senator Scott Wiener’s new bill, introduced after Governor Newsom vetoed Wiener’s controversial SB 1047. The text shifted over the weekend; if I missed a change, corrections welcome.
Because SB 53 appears poised to move quickly, I’m posting a short series. A future post will cover the carveout / thresholds for what companies are covered, whistleblower protections, and unclear third party obligations.
Up first: SB 53’s definition of “catastrophic risk” creates a significant compliance problem, with a straightforward fix. (Bonus: I also found some typos, see footnotes 3 & 4.)
Confusion about risk
Like SB 1047, SB 53 is concerned with AI-caused “catastrophic risk.” The bill defines catastrophic risk through the magnitude of harm (impact) and the type of activity. (Screenshots from the California legislature’s website; redline reflects amendments to the bill as introduced.)
This definition has a flaw common in AI legislation.1 It misses that risk = impact × likelihood.2 SB 53 includes clear and quantitative impact thresholds but no quantitative probability term, relying instead on qualitative standards like “foreseeable and material” and “materially contribute.” Those formulations do not specify how likely the harm must be. Consider a frontier model use the developer estimates has a 95% chance of assisting a chemical attack; surely that qualifies. But what about 49%? 5%? 0.001%? Without a probability threshold, different observers might classify any of these as “catastrophic risk” despite the huge difference in risk levels.
This vagueness gives the State Attorney General broad enforcement discretion. Companies will interpret “catastrophic risk” differently, both from one another and from the AG. This vagueness invites hindsight; any incident that crosses the “impact” could be labels by the AG as “foreseeable and material.” After all, it happened, so it must have been likely. Without a quantifiable likelihood term, companies will have a weak defense against this 20/20 hindsight logic.
Of course, if something catastrophic happens, this law is the least of an AI company’s worries. (Which is another way of saying that companies already have strong incentives to prevent such events.)
But this definition creates another, more routine problem in the statute.
What counts as a “critical safety incident”?
This vague definition of “catastrophic risk” affects other, more mundane parts of the statute. For example, covered companies must report “critical safety incidents” to the state or face up to a $1 million fine per failure. To avoid such penalties, businesses need to know exactly which incidents qualify.The definition of “critical safety incident” includes four types:
(1) and (3) are relatively clear. Companies will usually be able to identify these types of incidents.But (2) is unclear because, like the catastrophic risk definition, it rests on a misunderstanding of risk. “Risk” is an ex ante estimate (likelihood × impact) of harm; events are ex post. Risks don’t “materialize”—events do. Saying “harm resulting from the materialization of a catastrophic risk” is like saying, “a storm resulting from the materialization of a 50% chance of rain.” A 50% chance of rain can also “materialize” as a sunny day, and a catastrophic risk can “materialize” as a non-incident. If they mean “catastrophic impact” they should say that.
This isn’t nitpicking. The “materialize” language creates real ambiguity about when a company must report an incident. Suppose a company anticipates a catastrophic risk: a frontier model could autonomously steal billions of dollars. They implement mitigations. The model does attempt theft, but because of safeguards only one person loses $100. That loss is far below the impact threshold in the “catastrophic risk” definition. Did the catastrophic risk “materialize”? Is this a reportable “critical safety incident”? It’s unclear because the definition mixes ex ante risk assessment with ex post harm assessment. And with a $1 million dollar fine at stake, that uncertainty is costly.
Critical safety incident type (4) compounds the confusion. It covers cases where models use deceptive techniques to “materially increase[]” such risk. A risk can rise by increasing impact, likelihood, or both. But how much of an increase triggers reporting? The statute says a “material” amount. That term is common in law—it means “big enough to matter”—so this provision troubles me less than the statute’s deeper misunderstanding of risk. Still, the clause would benefit from clarifying whether “material” refers to impact, likelihood, expected harm, or some combination.
How to fix this
These flaws are fixable by revising definitions to (a) define “catastrophic harm,” and (b) tie “catastrophic risk” to a numerical probability or an expected-harm threshold. (Bracketed text could be negotiated.)
(c) (1) “Catastrophic risk” means a probability greater than [1%] that a frontier developer’s development, storage, use, or deployment of a frontier model will materially contribute to a catastrophic harm arising from a single incident involving a frontier model doing any of the following: …
…
(c)(2) “Catastrophic harm” means the death of, or serious bodily injury3 to, more than 50 people or more than one billion dollars ($1,000,000,000) in damage to, or loss of, property. (c)(3) “Catastrophic harm” does not include harm from any of the following:
(A) Information that a frontier model outputs if the information is otherwise publicly accessible in a substantially similar form from a source other than a foundation4 model.
(B) Lawful activity of the federal government.
(C) Harm caused by a frontier model in combination with other software if the frontier model did not materially contribute to the harm.
…
(d) “Critical safety incident” means any of the following:
(1) Unauthorized access to, modification of, or exfiltration of, the model weights of a frontier model that results in death or bodily injury.
(2) Catastrophic harm.
(3) Loss of control of a frontier model causing death or bodily injury.
(4) A frontier model that uses deceptive techniques against the frontier developer to subvert the controls or monitoring of its frontier developer outside of the context of an evaluation designed to elicit this behavior and that materially increases either (i) the probability of catastrophic harm or (ii) expected catastrophic harm.
This approach aligns the statute with the standard risk formula (risk × likelihood) and resolved the “risk materializes” confusion.
More to come on SB 53…
1. And not just AI legislation! The FTC’s application of its unfairness standard in the context of data security cases has often been very loose with with risk analysis. Some of these issues are plumbed in depth in the Commission’s opinion in the LabMD case, which I worked on. The 11th Circuit made an absolute mess of the law on appeal, BTW.
2. NIST Special Publication 800-30 Rev. 1, Guide for Conducting Risk Assessments (“[R]isk is a function of the likelihood of a threat event’s occurrence and potential adverse impact should the event occur.”), https://csrc.nist.gov/pubs/sp/800/30/r1/final.
3. To be consistent with definitions in the “critical safety incident,” I changed this from “serious injury” to “serious bodily injury.”
4. The latest version of SB 53 replaces almost all mentions of “foundation model” with “frontier model” — except this one. I think this is a mistake, and should be changed to “frontier.”