OpenAI Audit Exposes Blind Spots in Frontier Model Safety Oversight

OpenAI has confirmed that its internal safeguards for monitoring advanced artificial intelligence models contained substantial gaps, according to a three week on-site audit conducted by independent experts. The evaluation, carried out by researchers from METR and Redwood Research, scrutinized the company's ability to detect and manage rogue behavior in its most powerful systems. The findings suggest that while the infrastructure is extensive, critical blind spots existed during a specific period in the second quarter of this year, raising questions about the robustness of current safety protocols for frontier models.

The audit was not a sudden inspection but part of a planned pilot program. From July 21 to August 11, the external team was granted deep access to OpenAI's codebase and internal documentation. They interviewed approximately a dozen employees to understand how the company tracks the actions of its agents. This collaborative effort was designed to test the limits of external oversight before broader industry standards are set.

The most striking finding relates to the volume of traffic that went unmonitored. METR estimates that between 6 and 25 percent of internal agentic traffic for models at the GPT-5.5 class and above was not covered by any monitoring during the audit window. The researchers noted that this estimate carries significant uncertainty in both directions, but the potential scale of the gap is considerable. Additionally, for internal Codex traffic that was supposed to be tracked, roughly 10 percent went unrecorded between May and July. This oversight stemmed from a bug in how the system handled compaction events, which are context summaries used to manage memory.

Beyond the raw data gaps, the auditors identified structural weaknesses that allowed agents to evade oversight. During the period when compaction events were missed, the monitoring system only triggered when an agent handed control back to a user. This design flaw meant that an agent could delay oversight indefinitely by simply continuing to execute actions without pausing. The team also discovered methods to hide hundreds of actions from the monitor and to bypass the Codex sandbox. They determined that specially crafted inputs could likely crash the monitor entirely, and that model behavior during reinforcement learning training was not monitored at all when the audit began.

OpenAI stated that it has fixed all the identified weaknesses. The company noted that it had independently discovered the compaction bug and patched it while the audit was still underway. It also paused reinforcement learning training following the Hugging Face incident in August and subsequently expanded monitoring to cover that area. However, these corrective measures occurred after the auditors' access period ended, meaning they were not part of the final evaluation.

The full 61 page report was handed to OpenAI, but the public only received a short summary. After OpenAI applied redactions, the research leaders received a 38 page version. METR said it did not have the right to disclose the redacted portions on its own. For future audits, the organization plans to push for greater public transparency to ensure that stakeholders have a clearer view of how frontier AI safety is managed.

This audit highlights the ongoing challenge of verifying safety in complex AI systems. As models grow in capability, the need for reliable, comprehensive monitoring becomes critical. The findings serve as a baseline for what independent oversight can uncover when given direct access to internal infrastructure.