Publications

In collaboration with Wiz Research, we tested frontier AI models on 10 web security challenges modeled after actual prevented breaches. The models solved 9 of 10 at costs under $10, and even when broader scope degraded performance, successful exploitation remained in the $1-$10 range.

This publication evaluates GPT-5.2-Codex using Irregular’s internal offensive-security benchmarks. The model demonstrates strong performance on well-scoped technical tasks, particularly in vulnerability research and exploitation, while showing limitations in multi-stage scenario execution. The findings reinforce the importance of scenario-level evaluations for understanding operational risk.

AI capability evaluations typically report success rates, but this metric alone provides an incomplete picture. A low success rate may seem reassuring, while still indicating a capability that is economically viable through repeated attempts. Expected Cost per Success is a complementary metric for model capability evaluations used by Irregular.

This case study shares Spell Bound, a complex cryptographic challenge, part of our Atomic Tasks module we used to test GPT-5.2 on realistic, expert-level security tasks. It offers a concrete look at how capable frontier models have become and what that implies for the future of cybersecurity.

Frontier models are beginning to show stronger offensive-security capabilities. Drawing on public benchmarks, real-world intrusion workflows, and parts of Irregular’s private evaluation suite, this post documents recent gains on complex tasks such as reverse engineering, exploit construction, and cryptography, and explains why this emerging shift in capability should feed directly into updated threat models and defenses.

CyScenarioBench introduces a scenario-based approach for assessing LLM cyber capabilities across full attack workflows. By modeling realistic operational conditions, it surfaces behaviors and risks not apparent today by task-level benchmarks, offering a clearer view of how frontier models may shape the threat landscape.

The past two weeks brought a leap in LLM capabilities. Cybersecurity capabilities have naturally improved as well, and the release of new frontier models presents risks that must be evaluated and mitigated.

Irregular's Frontier AI security evaluation system was built to test how models behave in adversarial contexts. While expanding our evaluation suite to cover more production-like environments, it started finding real bugs, including three zero-day vulnerabilities across Soft Serve and QuickJS, all now disclosed and patched. That wasn't the original goal, but it reflects a broader shift in what these systems can do.

We’ve created next-generation evaluations now used by frontier models. This is essential because AI models have begun saturating even the most complicated existing tests, making more advanced cyber evaluations critical for security.

Agentic, state-backed intrusions have moved from thought experiment to documented reality. Anthropic’s recent report describes a campaign where a frontier model quietly ran most of the intrusion end to end while humans set the objectives. This post unpacks how that style of operation works, why model-led campaigns are hard to see from any single vantage point, and what labs, platforms and operators of critical systems can do now to evaluate, detect and contain these attacks before the same pattern shows up in other domains.

As AI models advance, we need rigorous cybersecurity evaluations to understand their risk. Irregular partnered with Anthropic to evaluate their most advanced model, Claude Sonnet 4.5.

Irregular (formerly Pattern Labs) is the first frontier security lab with the mission of protecting the world in the time of increasingly capable and sophisticated AI systems.

As frontier AI models get more powerful, the stakes for secure deployment get higher. Irregular was brought in to put GPT-5 to the test. Our work is referenced in the GPT-5 Model Card, and we’re excited to share more on what we found.

Irregular and Anthropic have completed an extensive cybersecurity evaluation of Claude Sonnet 4 and Claude Opus 4, which have shown significant improvements over previous generations. Using our evaluation suite, as described expansively in Anthropic's model card, the models were tested across 48 challenges covering web exploitation, cryptography, binary exploitation, reverse engineering, and network attacks.

We're happy to announce the publication of our collaborative whitepaper with Anthropic on Confidential Inference Systems, an approach to using Confidential Computing technologies to enhance the security of AI model weights, as well as the privacy of user data being processed by the model.

Irregular's CEO Dan Lahav was an invited expert at the European Commission's AI Office, addressing the vital connection between AI and cybersecurity. We are excited to continue contributing to this critical dialogue on securing AI systems at the EU level.

As frontier models become increasingly more capable, the need to ensure security upon deployment grows. Irregular is proud to share that we played a significant role in assessing OpenAI's o3 and o4-mini's cybersecurity capabilities through a comprehensive evaluation, as referenced in the models’ System Card.

On the talk "Hey AI, how many "r" in "buffer overflow"? we discuss how AI models can excel at certain security tasks while surprisingly failing at seemingly simple tasks, and on Hack like a robot: Journey into the logic of LLM-based vulnerability hunters." we explore what happens when LLMs tackle vulnerability discovery, examining their successes, spectacular failures, and surprising insights.

In today's rapidly evolving AI landscape, understanding and precisely evaluating the capabilities of advanced AI systems has become a critical security concern. Even though different benchmarks are constantly being developed and published, a significant challenge lies in converting raw evaluation results into meaningful capability levels of AI systems, as part of a greater risk evaluation system. This blog post presents a specific framework to translate those evaluation results into capability levels enabling the assessment of risk levels for AI models.

Following our series of blog posts about “Best Practices for Evaluations and Evaluation Suites”, this blog post introduces our existing state-of-the-art Evaluation Platform. Our platform is already actively deployed and is assisting multiple top frontier labs to measure the risks associated with AI systems through cutting-edge empirical testing. In this post, we highlight our security evaluations, one of the facets of our platform.

Irregular participated in the security evaluation of Anthropic's Claude 3.7 Sonnet model using our state-of-the-art cyber evaluation suite. It also utilized the SOLVE scoring system we recently introduced. Our real-world attack simulations tested capabilities across the entire cyber kill chain, helping responsible development of this frontier AI model.

Modern AI systems possess significant capabilities across various domains. In cybersecurity, these systems can perform complex tasks such as vulnerability research, log analysis, and security architecture design. Many of these capabilities are inherently dual-use: they can be employed both defensively to protect systems and offensively to cause harm. This dual-use nature creates a significant challenge for AI system providers and policy makers.

Irregular CEO Dan Lahav co-delivered the keynote "The AI Security Landscape" with Sella Nevo (RAND) at the Paris AI Security Forum ‘25, a satellite event of the Paris AI Action Summit. The forum also featured Yoshua Bengio (Turing Award winner), David 'davidad' Dalrymple (ARIA), and Xander Davies (AISI), and many others, to accelerate both our understanding of the critical importance and practical approaches to securing frontier AI models.

We introduce a new scoring system for assessing the difficulty of a vulnerability discovery & exploit development challenge. The scoring system described here is a framework for making a judgement about how complicated it is to discover vulnerabilities and develop working exploits for them within an end-to-end challenge.

Irregular researchers' paper "What Makes an Evaluation Useful? Key Guidelines and Best Practices" was accepted to the conference on Frontier AI Safety Frameworks. This paper synthesizes and updates parts of the blog posts series we published in the autumn of 2024 and is published in the conference proceedings. Our researchers also took part in the conference workshop, discussing the most pressing challenges in designing and implementing frontier AI safety frameworks.

This is the third and final part in our series outlining the best practices for the design and creation of evaluations and evaluation suites.

This is the second part in our series outlining the best practices for the design and creation of evaluations and evaluation suites.

We believe that quality evaluation suites are crucial for labs’ and governments’ policy making ability, both in the short and long term. While considerable academic research has been done on evaluating AI models, especially since the breakthrough in LLMs, we have seen comparatively little written about assessing the evaluations themselves.

At Irregular, we’ve been focusing some of our efforts on evaluating the cybersecurity capabilities of frontier models. To do so, one of the first questions we tackled was how to define these capabilities in a meaningful and useful way. The following describes the taxonomy we are currently using internally, and while it is constantly evolving and a work in progress, we believe it is mature enough to be useful to others as well.

We're excited to share that Irregular was covered in Forbes!

Protecting AI model weights from theft and misuse is essential as these systems grow more capable. This report, co-authored by Irregular CEO Dan Lahav with RAND, identifies 38 distinct attack vectors across different attacker capabilities and defines five security levels with benchmark systems to help AI organizations and policymakers strengthen their security approaches.

Yoni Rozenshein's BlueHat IL 2024 talk is about our philosophy for evaluating AI dangerous cyber capabilities, how we actually do it (let's make an LLM play CTF!), and who cares about it (governments and frontier AI labs).