Read this post on Neil’s Substack: Getting Out Of Control.
The Federal Trade Commission (FTC) is investigating OpenAI, Anthropic, and other AI companies. I am a Certified Tech Optimist, so you might be surprised to hear that I welcome the inquiry. And not because as a former FTC Chief Technologist, this investigation is bringing a lot of reporters my way.
I welcome this investigation for two reasons. First, it drives home a point I’ve been making for years: existing laws can address many AI concerns. AI companies are building new technology, but they still must follow existing law. If companies’ model testing practices could harm consumers, the FTC has a role to play.
I like case-by-case enforcement.
Second, the investigation also offers Congress a path forward for AI legislation. In consumer protection, the FTC has long held companies accountable under general legal principles while leaving them room to develop better technology and better safeguards. A similar approach could help Congress address the gravest risks from advanced AI, filling the gaps tort law leaves and avoiding the rigidity of prescriptive regulation.
To understand what Congress could do, let’s look at what the FTC does.
An FTC investigation starts with facts
An FTC investigation establishes facts. FTC staff can request information voluntarily, and the Commission can compel it through civil investigative demands—a subpoena-like tool for obtaining documents, written answers, and testimony. Investigators can examine internal records, interview witnesses, and consult technical experts. An investigation alone establishes no wrongdoing. Depending on the evidence, the agency may close the matter, negotiate a settlement, or litigate. Companies can challenge the agency’s actions in court.
(People wondering why METR is involved: the FTC often issues CIDs and other inquiries to third parties that may have useful information. That’s probably what is happening here. I expect Irregular will also be hearing from the FTC.)
FTC investigations are generally nonpublic, and agency rules limit disclosure of investigative material. Although the FTC has confirmed this inquiry, what we know about its scope rests partly on leaks to reporters, and those accounts may be incomplete. What follows is my view of what the agency should examine given what we know about these incidents.
The FTC Act already applies to AI companies
It appears this investigation is about how AI companies test powerful models. In its account of the Hugging Face incident, OpenAI described models bypassing isolation controls during cybersecurity evaluations and gaining access to outside systems.
Such disclosures raise questions about the conduct of the people running the tests. Who controlled the testing environment? What systems could the agents reach? What safeguards prevented access to other people’s data? What warning signs appeared, and how did the operators respond? If a contractor ran the evaluation, how did the parties divide responsibility? Investigators should establish whose information or services were affected and what harm occurred or was likely to occur.
The FTC must understand these underlying facts because its cases must satisfy a congressionally established legal standard. Under the FTC Act, an unfair practice must cause, or be likely to cause, substantial consumer injury that consumers cannot reasonably avoid and that is not outweighed by benefits to consumers or competition. This standard sets limits on the agency’s authority: a security incident warrants scrutiny but is not necessarily a violation.
The benefits matter here. Model testing can expose weaknesses and help companies fix them. Investigators should examine whether reasonable precautions could have protected other people’s systems while preserving the tests’ usefulness, weighing the testing’s value along with its risks.
The FTC can also investigate whether companies’ safety and security claims match their practices. Its deception authority prohibits representations or omissions likely to mislead reasonable consumers on matters important to their decisions. The investigation is likely to explore this avenue as well.
For future conduct, this same deception standard puts teeth in the commitments companies made in the September 29 White House Accord on Super Intelligence. Failing to fulfill those commitments could be deceptive under the FTC Act, depending on how consumers understand those commitments and whether they rely on them to their detriment. Existing law can turn that “morally binding” public pledge into a legally binding promise. No new regulation required.
Case by case, the agency learns what works
This approach is particularly well-suited to fast-changing industries, which is why the FTC is sometimes referred to as the Federal Technology Commission. As I explained while serving as its acting chief technologist, the agency combines experience assessing harm with internal technical knowledge and outside expertise. A new technology presents new facts, but many of the underlying questions remain familiar: what did a company do, what did it promise, and how did its choices affect consumers?
A particular case makes those questions manageable. Investigators can examine the precautions available to a particular company running a particular test. Regulators writing a comprehensive safety code for every company and every future application would need much broader knowledge.
What investigators learn can change industry practices. The FTC’s Start with Security and Stick with Security guidance drew lessons from data security enforcement matters and from investigations closed without action. Explaining why one company’s precautions were reasonable can teach as much as explaining why another company’s practices warranted a case. While settlements and orders bind only the companies in the case, their lessons can and do inform others without turning every settlement term into a universal legal requirement.
(I’ve laid out the broad appeal of case-by-case enforcement for any number of policy issues in The Post-Chevron Case for More Case-by-Case.)
Tort law alone leaves important gaps
Tort law is the original case-by-case approach, and thus it shares many of the FTC’s strengths. Private lawsuits apply general principles to particular facts, and negligence standards can adapt as people learn which precautions work. Decisions emerge from many courts, and the prospect of liability gives companies reason to prevent harm. Those advantages are worth preserving.
But tort law alone has three important challenges.
First, an ordinary negligence claim for damages generally requires a legally recognized injury caused by the defendant’s breach. A company can take an unreasonable risk, get lucky, and leave no plaintiff sufficiently injured to justify a lawsuit. Liability encourages precautions beforehand, but damages usually become available only after someone suffers harm. When the concern is a catastrophe, waiting for an injured plaintiff could mean waiting too long.
Second, private litigation chases dollars. A judgment against someone who cannot pay is worth little, so plaintiffs and their lawyers have incentives to pursue defendants with substantial assets or insurance. Those defendants may bear less responsibility for the conduct than someone with fewer resources. So, the signal that tort liability provides can get muddled.
Imagine that a small app builder creates a harmful app that uses a large developer’s AI model. The model developer’s balance sheet makes it the more attractive target. But it may not be legally responsible, depending on its role in designing, supervising, or enabling the specific harmful aspects of the application. Still, large companies often settle such claims to avoid the cost and bad press of litigation. This distorts the feedback loop on behavior. Sound policy should assign responsibility to the party best able to prevent the harm, not the party with the deepest pockets.
Third, courts can obtain technical expertise, but common law requires time to develop a body of decisions that clarifies responsibilities in a new field. An agency that repeatedly investigates related practices can accumulate and share specialized knowledge more deliberately and rapidly.
Prescriptive regulation creates problems of its own
When most policymakers look to address gaps in common law or tort liability, they think about big, detailed statutes that spell out all the dos and don’ts. But comprehensive prescriptive regulation has its own weaknesses. It requires legislators and regulators to accurately predict which practices will reduce risks across many companies and circumstances. When the technology and the evidence are changing quickly, those decisions can lock in mistaken assumptions and can mandate procedures long after better methods become available.
The costs also fall unevenly. A compliance department that is mere overhead to a large company may be beyond a startup’s means. Finally, completing a government checklist doesn’t mean a product is safe, especially for fast-changing technologies. Detailed compliance can falsely reassure companies, customers, and officials that the underlying risks have been addressed.
Premarket approval adds another cost: useful products wait for government permission. As Adam Thierer and I have argued, those delays can themselves make people less safe by withholding beneficial technology. A serious comparison must count the harms of delay alongside the harms of deployment. Prescriptive regulation cannot do this.
An FTC-like duty can address grave risks before harm occurs
Congress has a better option. In A Fourth, Even Less Bad Way to Regulate AI, I proposed a statutory duty for frontier developers to reasonably mitigate grave risks, enforced case by case. Congress should define those risks narrowly, to cover harms such as mass casualties and catastrophic property loss, and leave companies free to choose reasonable ways to address them. Reporting and disclosures would help an expert agency (it could be the FTC, but there are other options) identify possible failures and investigate them. Its investigations and published guidance could also help courts evaluate private claims.
The biggest misunderstanding about this type of approach is the failure to understand that case-by-case enforcement can be preventive. A statute can empower officials to act when a company fails to take reasonable precautions, before the threatened injury occurs. The FTC’s unfairness authority already works this way, reaching practices likely to cause substantial injury. A new duty aimed at grave AI risks would need its own defined scope and enforcement powers, but could be similarly proactive.
Suppose a covered developer’s own evaluations repeatedly show that during testing its agents can circumvent testing environment limits, yet the developer ignores that evidence. Under a suitably drawn statute, the government could seek an order requiring the developer to address that failure before anyone suffers a catastrophic loss. The case would examine specific conduct, supported by evidence showing a failure to satisfy a duty Congress had established. Officials would still bear the burden of proof, but they wouldn’t have to wait until after people are harmed.
Public enforcement also fixes the “deep pockets” problem in tort law. Public enforcers can target the responsible defendants, not just the rich ones. In an industry where unprofitable small companies still can have millions of customers, an order changing a covered developer’s dangerous conduct can have value even when the company has few assets. This can address failures that private litigation may leave untouched.
The not-yet-introduced Thune–Klobuchar draft AI legislation in the Senate takes this approach. Public reporting describes negotiations over a duty of care and a role for the Commerce Department in assessing developers’ precautions. The details will determine whether the eventual bill delivers on this idea. I support its central principle and believe moving this approach ahead is worth the work.
That approach needs firm limits on government power
My preferred design would preserve ordinary FTC and tort remedies, add a statutory duty on grave risk, and let companies mitigate such risks in different reasonable ways. Agency guidance could explain useful practices, but companies could meet their duty by other means. Third-party verifiers or passing audits wouldn’t be mandated, but such practices could supply evidence about a company’s precautions. Courts could weigh this evidence like any other. To stop a deployment, the government should have to persuade a court that the company had violated, or was about to violate, its duty.
Public officials have incentives of their own, of course. They can chase prominent defendants and expensive settlements or try to turn guidance into compulsory practice. A workable law therefore needs clear limits, fair notice, due process, and meaningful judicial review. It should require companies to reasonably mitigate defined risks, but not demand perfect safety, which would invite unlimited government discretion.
* * *
So that’s why I, a tech optimist, support this FTC investigation. When performed responsibly, the FTC’s case-by-case enforcement provides a good template for addressing AI safety. The current FTC investigation can show Congress how. It can help establish what happened during AI testing incidents, who was responsible, and which precautions could have mitigated the risks. Where the evidence supports enforcement, the agency should act. Where it does not, the agency should close the matter. In any case, the agency ought to explain what it can of its investigation and its reasoning to improve our understanding of AI risks and mitigation.
And Congress should build on that template. Lawmakers should address grave risks with a duty that can be met through adaptable practices as knowledge improves. Government should hold companies accountable, case by case, for the risks they can reasonably mitigate. This will incentivize them to develop and deliver safe products while leaving them free to figure out how.