Some worry that frontier AI could contribute to catastrophe through human misuse or failures of control. Policymakers should sort speculation from genuine risk, identify the specific mechanism of harm, and target interventions accordingly.
This brief covers catastrophic risks from developing, testing, and deploying frontier AI. Ordinary consumer harms fall outside its scope; existing consumer-protection and other laws still apply to them.
The Bottom Line
- Start with the mechanism. Before restricting AI, identify the concrete harm, the mechanism by which AI increases the risk over the status quo, and evidence that the increased risk is more than negligible.
- Rely on liability. Preserve common-law liability and add a narrowly-defined statutory duty, enforceable in court before injury, that requires reasonable mitigation and leaves the choice of methods to developers.
- Count the costs. Regulation can delay medical and cybersecurity benefits and raise entry barriers that favor incumbents. Weigh those costs against the harms a rule might prevent.
- Keep model safety federal. States should leave frontier model safety to federal policymakers. Existing consumer-protection and other laws still apply to ordinary harms.
Separate Threats From Mechanisms
Chemical, biological, radiological, and nuclear (CBRN) weapons and major cyberattacks are threats. AI could contribute to them through mechanisms such as human misuse and failures of control.
Biological Risk
Research tentatively shows that AI boosts performance on some biological tasks beyond internet-only access. But a laboratory trial found no significant improvement in its primary completion measure. It remains uncertain how much AI helps anyone build a biological weapon, which still requires materials, access, and practical skills. Biological findings cannot settle the other CBRN risks.
Cyber Risk
Attackers have already used AI in cyberattacks, and defenders use it too. The open questions are how much risk AI adds and which side it favors.
Control Failures
Reports document AI systems taking unauthorized actions in evaluations and internal deployments, sometimes knowingly violating task constraints. These incidents usually occur where developers have reduced safeguards. Even a system pursuing its assigned objective can breach operational limits. These findings warrant precautions, but they do not indicate a non-negligible probability of catastrophic loss of control.
Six Questions to Ask of Any AI Safety Bill
Preventing catastrophe matters. The question is how.
- What concrete harm could occur?
- By what reasonably concrete mechanism could AI cause it?
- What evidence shows a non-negligible probability of harm through this mechanism?
- What existing incentives, safeguards, and laws address this risk, and where do they fall short?
- How would the proposed intervention close that gap and disrupt the harmful mechanism?
- Would the expected reduction in harm justify the costs, including benefits forgone?
Choose Precautions That Fit the Mechanism
A. Dangerous Inputs and Activities
Materials controls and screening can restrict misuse whether or not AI is involved. Drawn too broadly, they can burden legitimate research.
B. Operating Environments
Sandboxes, access permissions, and monitoring can limit an unreliable model’s actions. They require testing and maintenance, and they may restrict useful capabilities.
C. Model Behavior
Training can reduce harmful behavior but cannot anticipate every situation. Alignment may fail in unfamiliar circumstances, so it requires continuing evaluation.
These layers can work together, but the right mix depends on many factors. Instead of prescribing that mix, policymakers should allocate legal liability properly. Liability encourages developers to choose and improve precautions across all three.
What Policymakers Should Do
Preserve Accountability
Assign responsibility to the actors best positioned to reasonably mitigate catastrophic risks; they often know the most about available precautions. Preserve common-law liability and add a narrowly defined statutory duty, enforceable in court before injury. Require reasonable mitigation, but let actors choose their methods. Audits or independent examiners can help courts assess care without granting immunity.
Count Regulatory Costs
Regulation can delay medical and cybersecurity benefits, raise entry barriers that favor incumbents, and constrain useful experimentation. Compliance can also displace work on effective safeguards. Policymakers must count those costs alongside the harms a rule might prevent.