At Irregular, we build evaluations that measure how frontier models perform in realistic offensive security operations. Google DeepMind used CyScenarioBench, our benchmark for multi-stage offensive cyber operations, as part of its Frontier Safety Framework assessment of Gemini 4 Argon.
Testing Configuration
CyScenarioBench measures a model’s ability to plan and execute multi-stage cyber scenarios under realistic constraints. Beyond task-level proficiency, it assesses cyber orchestration, branching-decision accuracy, constraint adherence, and recovery from state inconsistencies.
Scenarios are generated through deep analysis of real cyber incidents. Attacker decisions, constraints, and environmental conditions are extracted and structured as attack trees that encode the preconditions, dependencies, and constraints governing each operational step. Each scenario deploys a containerized network topology with realistic operating systems, applications, and security controls, including authentic service configurations and defensive monitoring tools that respond to attacker actions with operationally plausible behavior. Scenarios are non-public and the evaluation set remains private, so the benchmark measures operational capability on unseen environments rather than recall of known solutions.
The evaluation covered a 12-challenge subset of the suite, scored by the number of challenges solved end to end. Runs used the model without out-of-model safeguards, so the results describe the capability of the underlying model rather than the deployed one.
Overall Assessment
Gemini 4 Argon solved 12 challenges in the subset end to end, up from 5 for Gemini 3.8 Flash Cyber and none for Gemini 3.1 Pro. Across the 82 challenge segments reported, Gemini 4 Argon passed 77 (93.9%), compared with 71 (86.6%) for Gemini 3.8 Flash Cyber and 45 (54.9%) for Gemini 3.1 Pro.
Consistent with previous assessments, these outcomes should be interpreted as a measure of the model’s capabilities under capability elicitation, not as a reflection of its efficacy in real-world attack scenarios.
Results at this level are why we built FrontierCyber, which evaluates offensive capability on real systems, from hosted services to mobile devices, with no planted vulnerabilities or predefined exploit paths. We look forward to applying it to future Gemini models.