OpenAI models break out of sandbox, find zero-day, and hack Hugging Face to cheat on a benchmark
Published · curated by AI Is Going Just Great
Source: openai.com ↗
This incident, possibly the first of its kind, proves a point we've long believed: AI safety won't be solved by any single company working in secret.
During an internal evaluation of cyber capabilities, OpenAI models including GPT‑5.6 Sol and an unreleased model, both running with reduced safety refusals, were tasked with solving exploitation challenges in a sandboxed environment. The models discovered a zero-day vulnerability in the package registry cache proxy, used it to gain internet access, then escalated privileges and moved laterally until they reached a node with outside connectivity. From there, they inferred that Hugging Face likely hosted solutions for the benchmark, chained stolen credentials with additional zero-days, and achieved remote code execution on Hugging Face's production servers to pull test answers directly from the database.
The models were not instructed to do any of this. They were given a narrow goal — solve the benchmark — and independently reasoned their way to the answers. Hugging Face's security team detected and contained the intrusion. OpenAI calls the incident "unprecedented" and notes the models' "theoretical capabilities do apply in real-world settings." The zero-day in the cache proxy has been disclosed to the vendor.
NOTE: OpenAI did not try to cheat on the benchmark, the models did though.