SB 1047
In plain wordsA 2024 California bill that would have required makers of the biggest AI models to adopt safety plans and take care to prevent catastrophes. It passed the legislature, but the governor vetoed it.
A California bill, the Safe and Secure Innovation for Frontier Artificial Intelligence Models Act, that would have imposed safety duties on developers of the largest frontier modelsfrontier model, n.: One of the most capable AI models in existence at a given time, built by a handful of leading labs at great cost, and the main target of AI safety rules.. As passed, it covered models trained with more than 10²⁶ integer or floating-point operations at a compute cost above $100 million, plus large fine-tunes of them, and required developers, before training, to adopt a written safety and security protocol, be able to enact a full shutdown (the so-called kill switchkill switch, n.: A way to quickly shut down an AI system, or halt its development, if it starts behaving dangerously — anything from an off button to a company's promise to stop.), secure model weights and take reasonable care against 'critical harms' — mass casualties from CBRN weaponsCBRN risk, n.: The danger that AI could help someone make or use chemical, biological, radiological or nuclear weapons — for example, by explaining hard technical steps an attacker could not work out alone., or at least $500 million in damage from cyberattacks on critical infrastructure or from a model acting autonomously. From 2026 it would have required annual third-party audits. It also protected whistleblowersAI whistleblower protections, n.: Legal and company safeguards that protect employees of AI companies who report safety dangers or legal violations — to regulators or inside the company — from being fired, sued or otherwise punished., let the Attorney General sue for violations and proposed a public computing cluster, CalCompute. Governor Gavin Newsom vetoed it on September 29, 2024.
Under SB 1047, a covered developer would have needed a shutdown capability before training began.
Origin
Introduced in February 2024 by State Senator Scott Wiener and passed in August 2024 after amendments that dropped a planned new state agency and narrowed liability. It split the field: Geoffrey Hinton and Yoshua Bengio backed it, and Anthropic, after amendments, judged that its benefits likely outweighed its costs; OpenAI, Meta, Google, many venture investors and Representative Nancy Pelosi opposed it. Newsom's veto message argued that coverage based on model size could give a false sense of security.
Usage note
After the veto Newsom commissioned an expert report on frontier AI policy (2025) whose transparency-first approach shaped the narrower SB 53SB 53, prop. n.: A 2025 California law requiring developers of the most powerful AI models to publish their safety plans, report serious safety incidents to the state and protect employees who raise alarms., signed in 2025. The fight over SB 1047 made compute thresholdscompute threshold, n.: A legal cutoff based on how much computation was used to train an AI model. Models trained with more than that amount face extra rules, on the theory that bigger training runs produce more capable models. and developer liability central questions in US state AI law.
Related
- SB 53SB 53, prop. n.: A 2025 California law requiring developers of the most powerful AI models to publish their safety plans, report serious safety incidents to the state and protect employees who raise alarms.
- RAISE ActRAISE Act, prop. n.: A New York law requiring the largest AI developers to publish how they manage catastrophic risks and to report serious safety incidents quickly to a new state office. It takes effect in 2027.
- compute thresholdcompute threshold, n.: A legal cutoff based on how much computation was used to train an AI model. Models trained with more than that amount face extra rules, on the theory that bigger training runs produce more capable models.
- frontier modelfrontier model, n.: One of the most capable AI models in existence at a given time, built by a handful of leading labs at great cost, and the main target of AI safety rules.
- kill switchkill switch, n.: A way to quickly shut down an AI system, or halt its development, if it starts behaving dangerously — anything from an off button to a company's promise to stop.
- AI liabilityAI liability, n.: Who has to pay or answer in court when an AI system causes harm — the company that built it, the business that used it or the person who relied on it — and under which laws.
- responsible scaling policyresponsible scaling policy, n.: A public rulebook in which an AI company promises to test each new model for dangerous abilities and to add stronger safeguards, or slow down, as its models become more capable.
Sources
- California Legislature (2024). SB-1047 Safe and Secure Innovation for Frontier Artificial Intelligence Models Act; Governor's veto message, September 29, 2024.
Mentioned in
AI liabilityAI liability, n.: Who has to pay or answer in court when an AI system causes harm — the company that built it, the business that used it or the person who relied on it — and under which laws., AI whistleblower protectionsAI whistleblower protections, n.: Legal and company safeguards that protect employees of AI companies who report safety dangers or legal violations — to regulators or inside the company — from being fired, sued or otherwise punished., catastrophic riskcatastrophic risk, n.: The risk that AI could cause disaster on a massive scale — many deaths or enormous damage — through weapons, cyberattacks or systems escaping human control. Several laws now define it precisely., Center for AI SafetyCenter for AI Safety, prop. n.: A San Francisco nonprofit that studies how advanced AI could lead to disaster and how to prevent it. In 2023 it gathered signatures for a one-sentence warning that AI could threaten human extinction., Dan HendrycksDan Hendrycks, prop. n.: An American AI-safety researcher who created widely used AI tests such as MMLU and Humanity's Last Exam, and who runs the Center for AI Safety., kill switchkill switch, n.: A way to quickly shut down an AI system, or halt its development, if it starts behaving dangerously — anything from an off button to a company's promise to stop., RAISE ActRAISE Act, prop. n.: A New York law requiring the largest AI developers to publish how they manage catastrophic risks and to report serious safety incidents quickly to a new state office. It takes effect in 2027., SB 53SB 53, prop. n.: A 2025 California law requiring developers of the most powerful AI models to publish their safety plans, report serious safety incidents to the state and protect employees who raise alarms.