A recent report from a California-based policy group, co-led by AI pioneer Fei-Fei Li, advises legislators to consider potential AI risks that have not yet been observed when formulating AI regulatory policies. The 41-page interim report, issued by the Joint California Policy Working Group on Frontier AI Models, was initiated by Governor Gavin Newsom after he vetoed the controversial AI safety bill SB 1047. Although Governor Newsom felt SB 1047 was insufficient, he previously acknowledged the necessity for a thorough evaluation of AI risks to inform legislative efforts.
The report, co-authored by UC Berkeley College of Computing Dean Jennifer Chayes and Carnegie Endowment for International Peace President Mariano-Florentino Cuéllar, advocates for laws that would enhance transparency in the development processes of frontier AI labs like OpenAI. It underwent review by a wide range of industry stakeholders, including AI safety advocates like Turing Award winner Yoshua Bengio and Databricks Co-Founder Ion Stoica, one of the critics of SB 1047.
According to the report, new risks presented by AI systems may require legal mandates for AI model developers to disclose their safety tests, data collection practices, and security protocols. It further recommends increased standards for third-party evaluations of these metrics, along with expanded whistleblower protections for employees and contractors of AI companies.
The authors of the report suggest that there is currently an “inconclusive level of evidence” for AI’s role in facilitating cyberattacks, creating biological weapons, or contributing to other “extreme” threats. They assert, however, that AI policy should not only tackle current risks but also anticipate future consequences in the absence of adequate safeguards. The report notes, “For example, we do not need to observe a nuclear explosion to predict reliably that it could and would cause extensive harm. If those speculating about the most extreme risks are correct — and there is uncertainty about this — the stakes and costs of inaction on frontier AI at this moment are extremely high.”
The document proposes a two-pronged strategy to improve transparency in AI model development by advocating a “trust but verify” approach. It suggests AI model developers and their employees should have opportunities to report public concerns, such as internal safety testing, and mandates the submission of testing claims for third-party verification.
Although the report, expected to be finalized by June 2025, does not endorse specific legislation, it has been positively received by experts on all sides of the AI policymaking discussion. Dean Ball, an AI research fellow at George Mason University and critic of SB 1047, described the report as a “promising step” for California’s AI safety regulation. California State Senator Scott Wiener, who introduced SB 1047, stated in a press release that the report enhances ongoing discussions about AI governance initiated in the legislature in 2024.
The report aligns with several provisions of SB 1047 and its successor bill SB 53, such as the requirement for AI model developers to report safety test results. Broadly speaking, it represents a significant development for AI safety proponents who have seen their efforts lose traction in recent years.
