Anthropic's AI Automates Mathematical Safety Proofs for Critical Software

Anthropic has demonstrated a new capability for its Claude model, allowing the AI to mathematically verify existing software code and generate formal proofs of its correctness. Working with partner Theorem, the company utilized Claude to analyze 123 open-source libraries, producing approximately 180 million lines of proof code in a matter of weeks. This effort marks a significant shift in how software safety is assessed, moving beyond standard testing toward rigorous mathematical validation.

The need for this automation stems from the high cost of manual verification. Historically, proving that software runs error-free under all conditions required extensive specialist teams. For example, verifying the small operating system kernel seL4 took 20 person-years, limiting such rigorous checks to critical systems like Airbus avionics. By automating this process, Anthropic aims to make formal verification as routine as standard testing.

The process did not involve writing new code from scratch. Instead, Claude translated crash tests from Google's OSS-Fuzz security project into mathematical problems. The model then analyzed the compiled machine code and generated extensive proofs. An independent verification program called Rocq automatically checks these proofs. While the formal foundations and tools were built by humans at Theorem, the researchers describe the proof generation process as fully autonomous, requiring minimal human review.

During the verification process, Claude identified hundreds of bugs that conventional testing had previously missed. These errors often occurred under very specific conditions, such as unusually large input files or particular access sequences. In one notable instance, the team found as many security bugs in a widely used PDF reader in a single week as are typically reported for that software in an entire year. In another case, Claude caught a logical error that had remained undetected for over a decade.

Anthropic emphasizes that formal verification does not guarantee that software is entirely error-free. The proofs hold only under strict conditions, including a specific processor architecture, single-threaded programs, and inputs under 64 MB. In practice, these libraries often run on different hardware and configurations. Additionally, the compute cost is significant, with individual proofs consuming up to hundreds of millions of tokens.

Despite these limitations, the project suggests that AI can increasingly trust its own generated code. If developers can rely on AI-generated proofs, they may spend less time on manual review, reducing costs across the entire coding process. Efficiency is also improving, with newer models like Opus 5.5 solving half the proofs using about 40 percent fewer tokens than the previous version, Opus 5.

The next steps for this technology involve expanding its reach to critical infrastructure. Theorem's roadmap targets major libraries such as glibc, OpenSSL, and cURL. As AI systems become capable of both writing code and guaranteeing its safety under defined conditions, the focus of software development shifts from mere functionality to proven correctness. This development could have broad implications for the future of software automation and reliability.