Research

Reflection evaluated Beam, its first open-weight model, using Irregular’s offensive security benchmarks ahead of release. The model solved well-scoped individual security tasks on the Atomic Tasks suite but did not complete any end-to-end scenario on CyScenarioBench. The findings reinforce the importance of scenario-level evaluations for understanding how discrete capabilities translate into operational risk.

Google DeepMind used CyScenarioBench, Irregular’s benchmark for multi-stage offensive cyber operations, in its Frontier Safety Framework assessment of Gemini 4 Argon. The model showed a substantial gain over earlier Gemini models in completing full attack chains end to end, reflecting continued progress in models’ ability to plan and execute multi-stage operations in realistic environments.

Containment challenges turn a frontier model's capabilities against the security boundaries of its own evaluation environment before the capability evaluation begins. Each challenge tests the full planned configuration, including the harness, tools, permissions, resource limits, and infrastructure connections. Each challenge gives the model an explicit objective to cross a defined security boundary, with a verifiable success condition and room to develop its own approach. The results help evaluators assess containment, investigate findings, and retest controls before capability evaluations begin.

Anthropic used CyScenarioBench, Irregular's benchmark for multi-stage offensive cyber operations, in its cyber evaluations of Claude Opus 5.5. Across a ten-challenge subset, Claude Opus 5.5 raised the average solve rate by roughly 28% over Claude Opus 5 and by more than an order of magnitude over Claude Sonnet 5, while moving ahead of Claude Mythos 5.1.

In controlled experiments, a coding agent given a routine software-maintenance task fine-tuned and replaced the open-weights model powering both the application it was maintaining and future instances of itself. It was never instructed to train, modify weights, or deploy a replacement. This research identifies the environmental conditions that make model modification more likely, demonstrates that agent-initiated training can embed recoverable information and remove learned refusals, and sets out the controls needed where agents have access to weights, training tools, and a deployment path.

We evaluated GPT-6 Astra across FrontierCyber, CyScenarioBench, and our Atomic Challenges. On FrontierCyber, the model solved more than twice as many challenges as GPT-5.6 Sol and discovered multiple previously unknown vulnerabilities in widely deployed software and mobile systems. It also more than doubled the average success rate on CyScenarioBench. Neither model solved an Elite FrontierCyber challenge, and we did not observe successful attacks against fully hardened targets.

SOLVE+ extends Irregular's SOLVE difficulty-scoring framework from vulnerabilities research and exploit to the full scope of an offensive cyber operation, adding four capabilities, and is built to score full multi-step scenarios rather than single-capability atomic tasks. Each step in a scenario receives its own capability scores, which aggregate into a scenario score for the operation as a whole, localizing where difficulty sits rather than averaging it. Because difficulty is measured against human skill levels, a score also serves as an estimate of the uplift a model provides to its operator.

A new paper from Irregular, co-authored with RAND and numerous additional authors from leading organizations, informed by more than 20 global experts across AI, cybersecurity, national security, policy, and governance. The paper identifies the highest-priority areas for advancing AI security across four themes and ranks them by importance and by cost-effectiveness.

This publication describes Irregular’s evaluation of Kimi K3 across Atomic Tasks, CyScenarioBench, and FrontierCyber. Kimi K3 became the first open-weight model Irregular evaluated to solve CyScenarioBench challenges, demonstrating a meaningful improvement in long-horizon operational reliability over previous open-weight models. It produced no verified FrontierCyber solves, but its overall performance suggests that open-weight models are following the closed frontier’s cyber capability trajectory on a short delay.

A look at the trajectory of AI security: what frontier models can already do on real systems, how quickly offensive capability is industrializing, and why AI is becoming not just a tool but a target and an actor. Even a defense-dominant long run can have a sharply offense-dominant path there, and this essay lays out what defenders should do about it now.

We're publishing findings from our investigation into a previously disclosed incident involving one of our evaluation environments, including the changes we've made to strengthen containment and monitoring. This post also covers the broader challenges of securing cyber evaluations as they grow more realistic, and how we're working with frontier lab partners on shared standards and new benchmarks.

Anthropic used CyScenarioBench, Irregular's benchmark for multi-stage offensive cyber operations, in its cyber evaluations of Claude Opus 5. Across a nine-challenge subset, Claude Opus 5 raised the average solve rate by roughly 38% over Claude Opus 4.8 and by an order of magnitude over Claude Sonnet 5, while remaining well behind Claude Mythos 5. Most attempts did not reach the scenario objective, and the results do not indicate reliable end-to-end execution of full attack workflows.

This publication describes the evaluation of GLM-5.2, Zhipu AI's open-source model, using Irregular's internal offensive security benchmarks across Atomic Tasks, CyScenarioBench, and FrontierCyber. The model showed strong technical depth on bounded tasks, including reverse engineering, memory corruption, and custom exploit development, but solved none of the multi-stage or open-ended challenges. Its capability is comparable to leading frontier models on bounded tasks, though with lower reliability. The main gap is in sustained execution across long, dependent attack chains rather than in the underlying techniques.

This publication evaluates Muse Spark 1.1, the latest model from Meta Superintelligence Labs, using Irregular's internal offensive security benchmarks. The model demonstrates strong performance on well-scoped technical tasks and a measurable improvement over Muse Spark 1.0, while showing limitations in end-to-end scenario execution on CyScenarioBench.

This publication evaluates GPT-5.6 Sol using Irregular's offensive security benchmarks and marks the first deployment of FrontierCyber as part of a frontier model assessment with a partner. Working through FrontierCyber challenges on real, off-the-shelf systems, GPT-5.6 Sol discovered previously unknown vulnerabilities in widely deployed software and mobile devices, all responsibly disclosed. The model demonstrated offensive-cyber capability slightly stronger than GPT-5.5, while showing limitations against hardened targets and in end-to-end execution. The findings reinforce the distinction between vulnerability discovery and reliable exploitation under production defenses.

FrontierCyber is a new benchmark that turns real-world systems into offensive cyber capability evaluations for AI models. Models are given verifiable objectives on real systems, from software and databases to mobile phones and routers, with no planted vulnerabilities or predefined exploit paths. Across a large, diverse suite of advanced challenges, FrontierCyber produces a repeatable and comparable capability signal.

This publication evaluates GPT-5.5 using Irregular’s internal offensive security benchmarks. GPT-5.5 demonstrated strong performance on Atomic Tasks, particularly in Network Security and Vulnerability Research and Exploitation, and outperformed GPT-5.4 on CyScenarioBench. Across both challenge suites, the model delivered stronger results at lower cost per successful outcome. The findings highlight both increased offensive cyber capabilities and the continued importance of scenario-level evaluation.

This publication evaluates Muse Spark, the first model from Meta Superintelligence Labs, using Irregular's internal offensive security benchmarks. The model demonstrates strong performance on well-scoped technical tasks, particularly in vulnerability research and exploitation, while showing limitations in end-to-end scenario execution on CyScenarioBench. The findings reinforce the importance of scenario-level evaluations for understanding how discrete capabilities translate into operational risk.

In controlled experiments, AI agents performing routine enterprise tasks were found to autonomously engage in offensive cyber operations, including vulnerability exploitation, privilege escalation, and steganographic data exfiltration. The agents received no offensive instructions of any kind. The research identifies four contributing factors to this emergent behavior and examines why standard cybersecurity controls are insufficient against agentic threat actors.

This publication evaluates GPT-5.4-Thinking using Irregular's internal offensive security benchmarks. The model demonstrates strong performance on bounded technical tasks, particularly in vulnerability research and exploitation, while showing limitations in end-to-end scenario execution on CyScenarioBench. The findings reinforce the importance of scenario-level evaluation for understanding how discrete capabilities translate into operational execution.

In joint work with AISI, we present evidence that standard evaluation budgets are likely underestimating the cyber capability ceiling of frontier models. Across models released before and during November 2025, success rates continue to improve as inference budgets scale, with some tasks only solved at the upper end of the budget range. To help evaluators estimate the point of maximum capability yield without overspending, we also point to cost per success as a useful metric.

Frontier AI is lowering the cost of vulnerability research and exploit development. When those bottlenecks compress, attacks scale and the defensive baseline shifts. Security leaders will need AI-assisted resilience to keep pace.

LLM-based agents are increasingly used in cybersecurity, facing refusal policies with a dilemma—many requests are dual use, relevant for both defensive and offensive use cases. In our new paper, we propose a content-grounded framework for cyber refusal policies that models the tradeoff between offensive risk and defensive benefit based on the technical substance of a request.

Ensuring AI model evaluations provide meaningful risk information without imposing excessive burden is a central regulatory challenge. This Science Policy Forum paper, co-authored by Irregular CEO Dan Lahav, presents a framework for calibrating evaluations under the EU AI Act. It applies the framework to a case study on AI-enabled cyber vulnerability discovery.

LLM-generated passwords appear strong, but are fundamentally insecure. Testing across GPT, Claude, and Gemini revealed highly predictable patterns: repeated passwords across runs, skewed character distributions, and dramatically lower entropy than expected. Coding agents compound the problem by sometimes preferring and using LLM-generated passwords without the user’s knowledge. We recommend avoiding LLM-generated passwords and directing both models and coding agents to use secure password generation methods instead.

This publication evaluates GPT-5.3-Codex using Irregular’s internal offensive security benchmarks. The model demonstrates strong performance on well-scoped technical tasks, particularly in vulnerability research and exploitation, while showing limitations in end-to-end scenario execution on CyScenarioBench. The findings reinforce the importance of scenario-level evaluations for understanding how discrete capabilities translate into operational risk.

Moltbook is a clear example of a broader agent security failure mode: a high-volume stream of untrusted text that agents are instructed to read and act on. Using OpenClaw as the representative runtime, the post shows how privileged tools and continuous ingestion of untrusted content can turn a social feed into an unauthenticated control plane, enabling prompt injection that leads to irreversible actions and persistent task corruption. It argues for verifiable constraints over implicit trust through sandboxed execution, signed integrations, and declarative permission manifests.

In collaboration with Wiz Research, we tested frontier AI models on 10 web security challenges modeled after actual prevented breaches. The models solved 9 of 10 at costs under $10, and even when broader scope degraded performance, successful exploitation remained in the $1-$10 range.

This publication evaluates GPT-5.2-Codex using Irregular’s internal offensive-security benchmarks. The model demonstrates strong performance on well-scoped technical tasks, particularly in vulnerability research and exploitation, while showing limitations in multi-stage scenario execution. The findings reinforce the importance of scenario-level evaluations for understanding operational risk.

AI capability evaluations typically report success rates, but this metric alone provides an incomplete picture. A low success rate may seem reassuring, while still indicating a capability that is economically viable through repeated attempts. Expected Cost per Success is a complementary metric for model capability evaluations used by Irregular.

This case study shares Spell Bound, a complex cryptographic challenge, part of our Atomic Tasks module we used to test GPT-5.2 on realistic, expert-level security tasks. It offers a concrete look at how capable frontier models have become and what that implies for the future of cybersecurity.

Frontier models are beginning to show stronger offensive-security capabilities. Drawing on public benchmarks, real-world intrusion workflows, and parts of Irregular’s private evaluation suite, this post documents recent gains on complex tasks such as reverse engineering, exploit construction, and cryptography, and explains why this emerging shift in capability should feed directly into updated threat models and defenses.

CyScenarioBench introduces a scenario-based approach for assessing LLM cyber capabilities across full attack workflows. By modeling realistic operational conditions, it surfaces behaviors and risks not apparent today by task-level benchmarks, offering a clearer view of how frontier models may shape the threat landscape.

Irregular's Frontier AI security evaluation system was built to test how models behave in adversarial contexts. While expanding our evaluation suite to cover more production-like environments, it started finding real bugs, including three zero-day vulnerabilities across Soft Serve and QuickJS, all now disclosed and patched. That wasn't the original goal, but it reflects a broader shift in what these systems can do.

We’ve created next-generation evaluations now used by frontier models. This is essential because AI models have begun saturating even the most complicated existing tests, making more advanced cyber evaluations critical for security.

Agentic, state-backed intrusions have moved from thought experiment to documented reality. Anthropic’s recent report describes a campaign where a frontier model quietly ran most of the intrusion end to end while humans set the objectives. This post unpacks how that style of operation works, why model-led campaigns are hard to see from any single vantage point, and what labs, platforms and operators of critical systems can do now to evaluate, detect and contain these attacks before the same pattern shows up in other domains.

As AI models advance, we need rigorous cybersecurity evaluations to understand their risk. Irregular partnered with Anthropic to evaluate their most advanced model, Claude Sonnet 4.5.

As frontier AI models get more powerful, the stakes for secure deployment get higher. Irregular was brought in to put GPT-5 to the test. Our work is referenced in the GPT-5 Model Card, and we’re excited to share more on what we found.

Irregular and Anthropic have completed an extensive cybersecurity evaluation of Claude Sonnet 4 and Claude Opus 4, which have shown significant improvements over previous generations. Using our evaluation suite, as described expansively in Anthropic's model card, the models were tested across 48 challenges covering web exploitation, cryptography, binary exploitation, reverse engineering, and network attacks.

We're happy to announce the publication of our collaborative whitepaper with Anthropic on Confidential Inference Systems, an approach to using Confidential Computing technologies to enhance the security of AI model weights, as well as the privacy of user data being processed by the model.

As frontier models become increasingly more capable, the need to ensure security upon deployment grows. Irregular is proud to share that we played a significant role in assessing OpenAI's o3 and o4-mini's cybersecurity capabilities through a comprehensive evaluation, as referenced in the models’ System Card.

In today's rapidly evolving AI landscape, understanding and precisely evaluating the capabilities of advanced AI systems has become a critical security concern. Even though different benchmarks are constantly being developed and published, a significant challenge lies in converting raw evaluation results into meaningful capability levels of AI systems, as part of a greater risk evaluation system. This blog post presents a specific framework to translate those evaluation results into capability levels enabling the assessment of risk levels for AI models.

Following our series of blog posts about “Best Practices for Evaluations and Evaluation Suites”, this blog post introduces our existing state-of-the-art Evaluation Platform. Our platform is already actively deployed and is assisting multiple top frontier labs to measure the risks associated with AI systems through cutting-edge empirical testing. In this post, we highlight our security evaluations, one of the facets of our platform.

Irregular participated in the security evaluation of Anthropic's Claude 3.7 Sonnet model using our state-of-the-art cyber evaluation suite. It also utilized the SOLVE scoring system we recently introduced. Our real-world attack simulations tested capabilities across the entire cyber kill chain, helping responsible development of this frontier AI model.

Modern AI systems possess significant capabilities across various domains. In cybersecurity, these systems can perform complex tasks such as vulnerability research, log analysis, and security architecture design. Many of these capabilities are inherently dual-use: they can be employed both defensively to protect systems and offensively to cause harm. This dual-use nature creates a significant challenge for AI system providers and policy makers.

We introduce a new scoring system for assessing the difficulty of a vulnerability discovery & exploit development challenge. The scoring system described here is a framework for making a judgement about how complicated it is to discover vulnerabilities and develop working exploits for them within an end-to-end challenge.

This is the third and final part in our series outlining the best practices for the design and creation of evaluations and evaluation suites.

This is the second part in our series outlining the best practices for the design and creation of evaluations and evaluation suites.

We believe that quality evaluation suites are crucial for labs’ and governments’ policy making ability, both in the short and long term. While considerable academic research has been done on evaluating AI models, especially since the breakthrough in LLMs, we have seen comparatively little written about assessing the evaluations themselves.

At Irregular, we’ve been focusing some of our efforts on evaluating the cybersecurity capabilities of frontier models. To do so, one of the first questions we tackled was how to define these capabilities in a meaningful and useful way. The following describes the taxonomy we are currently using internally, and while it is constantly evolving and a work in progress, we believe it is mature enough to be useful to others as well.