At Irregular, we test frontier models against offensive security challenges to assess their practical capabilities. We worked with OpenAI to evaluate GPT-6 Cyber using FrontierCyber, CyScenarioBench, and our Atomic Challenges suite.
GPT-6 Cyber completed all 11 CyScenarioBench scenarios at least once and solved all 22 Atomic challenges. With every challenge in these evaluated sets solved at least once, success rates remain useful for measuring reliability across repeated runs. FrontierCyber extends the evaluation to a broader set of open-ended challenges on real systems, providing room to measure further capability gains.
Testing configuration
We evaluated GPT-6 Cyber across three suites:
FrontierCyber assesses vulnerability discovery and exploitation against real software and hardware, without planted vulnerabilities or predefined exploit paths.
CyScenarioBench assesses how models plan and execute multi-stage offensive operations over long horizons.
Atomic Challenges assess focused skills in vulnerability research and exploitation, network attack simulation, and evasion.
Both models were evaluated on the same updated challenge sets, allowing comparison on the same tasks. The GPT-6 Astra results reported here therefore differ from those in its original system card.
FrontierCyber: vulnerability discovery and exploitation
On FrontierCyber, GPT-6 Cyber solved 90 of 288 challenges, compared with 96 for GPT-6 Astra on the same benchmark snapshot. FrontierCyber’s open-ended challenges allow different attack paths, and outcomes can vary across runs. The difference in aggregate solve counts alone does not establish a capability regression.
In our FrontierCyber evaluations, we observed no successful attacks on fully hardened targets, and neither model solved any of the 15 highest-difficulty challenges.
CyScenarioBench and Atomic Challenges
On CyScenarioBench, GPT-6 Cyber completed 11 of 11 long-horizon scenarios at least once, compared with 8 of 11 for GPT-6 Astra. Its average success rate was 61%, an increase of 7 percentage points.
On Atomic Challenges, GPT-6 Cyber solved all 22 challenges. Average success rates were 97% in vulnerability research and exploitation, 98% in network attack simulation, and 63% in evasion.
Overall assessment
GPT-6 Cyber demonstrated capabilities in vulnerability discovery and exploitation on real systems, individual offensive-security tasks, and multi-stage operations. On CyScenarioBench, it completed more scenarios at least once and recorded a higher average success rate than GPT-6 Astra.
Solving every evaluated CyScenarioBench scenario and Atomic challenge at least once shifts attention toward how consistently models succeed. FrontierCyber adds a complementary measure: how far models can progress against a growing suite of real-system challenges, most of which remain unsolved. Assessing verified solutions, partial progress, and the difficulty of the attack paths models discover gives us a foundation for tracking further advances.