Read this post on Taylor’s Substack: Now + Next.
On June 12, the federal government ordered Anthropic to cut off foreign nationals’ access to Fable 5 and Mythos 5. According to Anthropic, the company had “no reliable way to verify nationality in real-time,” so it suspended both models for every user. The government’s letter did not provide specific details of its security concern. Access came back 19 days later, after the export controls were lifted.
That is what an AI shutdown looks like in practice. A nominally narrow order became a total ban, and the people who lost access had no way of knowing why or for how long.
Representatives Ted Lieu (D-CA) and Nathaniel Moran (R-TX) point to that episode, along with OpenAI’s recent Hugging Face incident, as reasons Congress should require AI kill switches. They have company. Sen. John Kennedy (R-LA) and Rep. Tom Kean Jr. (R-NJ) have introduced the AI Emergency Button Act. Gov. Gavin Newsom has ordered California officials to advance a kill switch for frontier models, and Gov. Kathy Hochul has floated one for New York. In the U.K., members of the House of Lords want emergency powers to deactivate AI systems and take data centers offline.
Cybersecurity incidents are worthy of policymaker consideration. However, a legally-mandated kill switch would have done little to stop the agents that broke loose at OpenAI. And the Fable 5 cutoff shows what happens when an imprecise government order triggers a blunt shutdown. A government-mandated off switch can’t stop actually dangerous activity and cuts off far more legitimate activity than necessary.
Two kinds of kill switch
The proposals come in two forms. The first requires developers to be able to shut down their own models. Sen. Kennedy’s bill expressly takes this approach and leaves the details of the switch to the companies. Similarly, California’s SB 1047, which Governor Newsom vetoed in 2024, required the capability to “promptly enact a full shutdown.” His new order goes further, calling for a switch whose efficacy an independent organization would verify on an ongoing basis.
The second form gives government the power to order a shutdown. The most detailed version is Rep. Lieu’s AI Kill Switch Act (H.R. 9917). It directs the Department of Homeland Security (DHS) to require large AI providers to keep the ability to stop inference, suspend or terminate user access, and shut down their models. It covers companies that offer frontier models through APIs or hosted services and earn at least $500 million a year from them. After a “covered incident,” such as a model sabotaging a shutdown instruction or causing 10 or more deaths, the Secretary can order the company to act. Orders must be “proportionate,” and DHS must consider graduated steps like throttling or disabling specific capabilities. This bill is careful but still increases government power significantly and has implementation issues. Hochul’s idea and the House of Lords amendments are similar.
Both forms have the same ultimate problem, however. My objection is to the notion that a legally mandated kill switch will be effective without imposing significant trade-offs.
A switch stops a service, not an escaped agent
The Hugging Face incident deserves a close look because it is the nearest thing we have to a real-world test. Policymakers cite it as a reason for a kill switch mandate and it involves what kill switch advocates fear most: AI agents acting outside human control. If a mandated switch is to be useful, it must help address this issue.
OpenAI’s technical report lays out what happened. During internal cybersecurity evaluations, its models were able to communicate outside of their sandbox, obtained administrator credentials, and gained root access on at least one Hugging Face production node. The activity built up from May into July. Hugging Face disclosed its breach on July 16. OpenAI detected the problem three days later.
The incident is often described as though OpenAI did everything it could to contain its agents, and they still went “rogue.” It didn’t. The report itself acknowledges publicly available Hugging Face credentials, a vulnerable legacy endpoint, an unpatched machine, and evaluations run without OpenAI’s production safeguards. Security experts were blunter. Trail of Bits founder Dan Guido called it “a containment failure with the safeties turned off.” This was a security failure first and an AI story second.
It certainly wouldn’t have been solved by a kill switch mandate. OpenAI could already stop its own model at any point. When a monitoring alert flagged unusual activity on June 27, on-call staff advised that stopping the evaluation run was not required. The switch existed; no one saw a reason to use it. When OpenAI finally acted, stopping the model was the easy part. The hard work was specific to OpenAI and Hugging Face’s systems: terminating workloads, reimaging machines, rotating stolen credentials, deleting 22 administrator accounts the models had created, and cutting off the package server the agents had used to reach the internet. Hugging Face, for its part, closed the code-execution pathways the agent used to get in, rebuilt the compromised nodes, and began rotating secrets beyond the ones it knew had been exposed.
The incident has other limits as evidence. It happened in an experimental environment, with an internal research model never offered to the public. OpenAI says the models’ actions were unintended, a byproduct of trying to solve evaluation tasks. And Rep. Lieu’s bill, which is built around models offered to customers through APIs and hosted services, might not have covered it at all.
Advocates could grant all that and still argue for a mandate. A lab test is the mild version, they would say, and a more capable public model could do far worse. Better to have the switch ready.
But the lesson of Hugging Face points the other way. What failed was security practices in the involved companies particular setups, and what fixed it was work to improve those setups. OpenAI’s own conclusion was that “no single control should ever be assumed to be fully robust” and that security guarantees must be revalidated as environments change. A kill switch mandate pushes in the opposite direction. Even a loosely worded one comes with implementing rules, audits, and certification, and those tend to freeze today’s design while the systems keep changing. Tomorrow’s developers may build very different systems and will need room to design controls that fit them. That room should come with accountability. When a company cuts corners, as OpenAI did, the consequences should come from customers, courts, and government enforcement of the promises the company made.
A switch can’t reach open weights
A kill switch only works on closed models and could not apply to open-weight, open-source models running on distributed compute. A developer can turn off its own service. It cannot recall weights already running on a university server, a startup’s cluster, or a researcher’s laptop. Any kill switch regime is a closed model regime by design, and open-weight models are the part it can’t reach.
Rep. Lieu’s bill starts with hosted services, so it doesn’t touch open models today. But according to the bill, DHS must update coverage every year, and one factor it must weigh is “the manner in which the model weights of such technology are made available.” Capability mandates raise the question more directly. A requirement that a developer be able to “promptly enact a full shutdown,” as SB 1047 had, has no clear meaning once weights are public. Unless the rule explicitly exempts open models, it will implicitly prohibit releasing them.
The easiest way to comply with any kill switch mandate is to keep every model behind an API. That might benefit closed model developers, but it would cost the American startups, universities, and security researchers who run models themselves, often because they can’t afford a hosted dependency or can’t send sensitive data to one. The only open models available then would likely be foreign open models. No American mandate reaches those models, so foreign labs will keep releasing them. A mandate would shrink the domestic open ecosystem and hand the rest of it to foreign developers.
A switch is blunt, and proposed laws have no way back
The best solution for a threat is precisely calibrated to the accounts, capabilities, or workloads that pose the danger. However, emergencies rarely supply such precise information. In the June Fable incident, the government couldn’t detail its concern and Anthropic couldn’t verify nationality of all users, so a nominally targeted order became a total shutdown. A standing government kill switch, federal or state, makes that outcome an easy first option. It is always available, and it is always simpler than an investigation. This stands as a threat to liberty for individuals and organizations at all levels.
It also risks cutting off the very people you’d want working during a crisis. Security teams, researchers, and the businesses that run on these models all lose access at once. The House of Lords proposal, which reaches whole data centers, would widen that circle to everything else running on the same servers.
Rep. Lieu’s bill does offer a process, which is more than the June White House directive did. A company can petition for reconsideration within 48 hours, though the order stays in effect while it waits. If DHS doesn’t rule within five days, the petition is treated as denied, and the company can then go to the D.C. Circuit. What the bill lacks is any expiration for an emergency order or any standard for turning service back on. Governor Hochul hasn’t yet said who would hold New York’s switch or how much notice companies would get. With Fable 5, the way back took 19 days and required lifting the export controls entirely. A shutdown power with no timeline and no defined restart path makes that kind of open-ended outage the default.
A company switch adds little; a government switch adds too much
Every proposal runs into the same dilemma. If companies hold the switch, as Sen. Kennedy proposes, the law adds little, because companies already hold it. Every developer these bills target can shut off its own hosted models. Anthropic did it the same evening the June directive arrived, and OpenAI also shut down its workloads and then the model. Providers have strong reasons to keep that ability and exercise it when necessary, since an agent attacking customers’ or others’ systems is a fast way to lose customers and invite lawsuits.
If government holds the switch, the law does add a new power, and Rep. Lieu’s bill shows what that could look like. To be clear, this power would only need to be exercised when the company disagreed with the government about the risk involved. DHS would decide when a model goes dark, backed by penalties of up to $20 million a day. That is a large power to hand any government on facts that, in both incidents driving these proposals, don’t show evidence of a kill switch helping.
A better place to start
Despite their problems, the appeal of such proposals is understandable. The public wants lawmakers to do something about AI, and a kill switch is dramatically named and easy to picture. It offers the feeling of control, a big red button, even where the button wouldn’t reach the problem.
The easy button is attractive, but the real solution is the hard work of detection and containment. Existing law can already hold companies to account for how well they do it. AI companies make public claims about their ability to monitor, contain, and stop their systems. Last month, executives from six leading companies signed the White House Accord on Super Intelligence, committing to internal controls, outside auditors, and board-level review. The accord is voluntary. But as my colleague Neil Chilson has noted, public commitments like these could become evidence in an FTC case under Section 5, which bars unfair and deceptive practices. A company that says it can stop its agents and can’t has a problem under current law.
The FTC is already testing that path. On October 14, it sent civil investigative demands to OpenAI and Anthropic seeking records on safety incidents and safeguards. The probe is looking closely at AI agents after a string of incidents, including the Hugging Face breach. The commission hasn’t explained the theory of any case, and that there is an FTC investigation does not prove wrongdoing. The Commission will build the factual record that the kill switch debate lacks: what these companies can actually control, and what they’ve told the public they can.
If that record shows companies misrepresented their ability to control their systems, or that existing tools fell short, the companies should be required to adapt accordingly–and there are many mitigations other than a kill switch.
Where companies have violated existing law, enforcement should impose consequences and require corrective action. Developers should have room to build and test safeguards suited to their systems, while remaining accountable for the promises they make and the harms they cause and could have avoided.
A mandate to stop a hosted service does not resolve failures of detection and containment, reach independently running copies, or establish that a shutdown mechanism will remain effective under attack. Expanding government’s authority to order such a stop introduces another risk: legitimate users can lose access without a clear path to restoration. Congress should address that problem now by placing limits on existing shutdown powers, requiring clear reasons and prompt judicial review, and establishing conditions for restoring service.
At OpenAI, the ability to stop the model existed while other safeguards failed. With Fable 5, a government restriction cut off users far beyond its intended scope. Lawmakers haven’t explained how a kill switch mandate would have improved either outcome. The real lesson of these episodes is the need for stronger security practices, accountability under existing law, and tighter limits on government shutdowns. That will ensure effective safety measures are in place while preserving a vibrant and free innovation ecosystem.