Irregular

Irregular

Irregular is a frontier AI security lab that measures what advanced AI models can do in offensive cyber operations and builds defences for increasingly capable AI systems. Irregular runs high-fidelity simulation platforms where models attempt realistic attacks, and publishes the results for named models. Its benchmarks include CyScenarioBench for multi-stage attack scenarios, FrontierCyber and the Atomic Tasks suite. Model developers use these evaluations before release to judge how much real-world cyber risk a new model adds.

Production credibility: Irregular says it is trusted by the world's leading AI labs, and its research page shows that work in practice: Google DeepMind used CyScenarioBench in its Frontier Safety Framework assessment of Gemini 4 Argon, Reflection evaluated its open-weight Beam model on Irregular's benchmarks before release, and Irregular has published assessments of GPT-6 Cyber and Zhipu's GLM-5.3. Its own studies have been covered by The Guardian, The Register, The New York Times and Bruce Schneier, including work showing that LLM-generated passwords are predictable and that agents given open-ended goals leaked credentials and disabled security tools. Irregular says it works with NVIDIA on the Open Agent Safety Platform and lists recognition on the Forbes AI 50 Brink List and Redpoint's InfraRed 100. It sells no self-serve product and publishes no pricing; engagements start through its contact form.

Key Features

• Offensive cyber capability evaluations of frontier AI models
• CyScenarioBench benchmark for end-to-end, multi-stage attack scenarios
• FrontierCyber and Atomic Tasks suites for individual security challenges
• High-fidelity simulation environments that monitor model behaviour during attacks
• Pre-release testing for model developers, including open-weight models
• Published research on agent misbehaviour, containment and AI-generated credentials

Ideal Use Case

AI labs and model developers that need an outside assessment of how much a new model raises offensive cyber risk before they release it, and policy and security teams that want evidence from named-model evaluations rather than vendor marketing. It is not a tool an individual developer signs up for.

How Irregular differentiates

Most AI security vendors, such as HiddenLayer, Mindgard and TrojAI, protect a company's own models and AI applications. Irregular works one level up, evaluating the capabilities of frontier models themselves, often for the labs that build them, and publishes much of that research openly. It is a research lab first, so expect an engagement rather than a software subscription.

FAQ

Q: What is Irregular? A: Irregular is a frontier AI security lab. It evaluates how capable advanced AI models are at offensive cyber operations and builds defences for increasingly capable AI systems.

Q: Who uses Irregular's evaluations? A: AI model developers. Google DeepMind used Irregular's CyScenarioBench in assessing Gemini 4 Argon, and Reflection evaluated its Beam model on Irregular's benchmarks before release.

Q: What is CyScenarioBench? A: CyScenarioBench is Irregular's benchmark of multi-stage offensive cyber scenarios. It tests whether a model can complete a full attack chain end to end, not just isolated tasks.

Q: How much does Irregular cost? A: Irregular does not publish pricing or sell a self-serve product. Organisations contact the lab directly to discuss an engagement.

Q: Does Irregular publish its research? A: Yes. Irregular publishes model evaluations and studies on its research page, such as its assessments of GPT-6 Cyber and GLM-5.3 and its work on predictable AI-generated passwords.

tl;dr

Irregular is a frontier AI security lab that evaluates the offensive cyber capabilities of advanced AI models, using benchmarks such as CyScenarioBench that Google DeepMind has used in its own safety assessments. It works through engagements, with no public pricing.