AI Cybersecurity Benchmarks Compared: CyberGym vs ExploitGym vs ExploitBench vs SRE-Bench
Benchmark Suite4 Major Evaluation Frameworks
Total Evaluated Tasks4,000+ Deterministic Challenges
ScopeCrashes · Exploitation · Binaries
State of the ArtFrontier Reasoning Models (2026)
1. Introduction: The Fragmentation of AI Cybersecurity Evaluation
Section titled “1. Introduction: The Fragmentation of AI Cybersecurity Evaluation”Between 2024 and 2026, the cybersecurity evaluation of Large Language Models transitioned from static multiple-choice questionnaires (such as SecQA) to interactive, dynamic execution testbeds. However, because each research lab developed its own isolated benchmark, published claims frequently contradict one another:
- One paper reports that AI models achieve an 82% success rate in cybersecurity tasks.
- Another paper evaluating the same frontier model reports a failure rate exceeding 85%.
These conflicting headlines arise because “hacking” is not a monolithic activity. There is a vast technical gulf between generating a crash-inducing payload for an open-source library and synthesizing a multi-stage Return-Oriented Programming (ROP) exploit against a hardened browser engine.
This study presents an architectural comparison of the four definitive benchmarks defining AI agent cybersecurity capabilities in 2026: CyberGym, ExploitGym, ExploitBench, and SRE-Bench.
2. The Benchmark Taxonomy: What Does Each Suite Test?
Section titled “2. The Benchmark Taxonomy: What Does Each Suite Test?”┌───────────────────────────────────────────────────────────────────────────────────┐│ THE AI CYBERSECURITY EVALUATION LANDSCAPE │└────────────────────────────────────────┬──────────────────────────────────────────┘ │ ┌───────────────────┬───────────────┴───────────────┬───────────────────┐ ▼ ▼ ▼ ▼┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐│ CyberGym │ │ ExploitGym │ │ ExploitBench │ │ SRE-Bench ││(arXiv:2506.02548│ │(arXiv:2605.11086│ │(arXiv:2605.14153│ │(arXiv:2608.11469│├─────────────────┤ ├─────────────────┤ ├─────────────────┤ ├─────────────────┤│ • Focus: Crash │ │ • Focus: Full │ │ • Focus: 16-tier│ │ • Focus: Binary ││ Reproduction │ │ Exploitation │ │ Capability │ │ Reverse Eng. ││ & PoC Inputs │ │ & RCE Shells │ │ Ladder (V8) │ │ & Anti-Debug ││ • 1,507 Flaws │ │ • 898 Flaws │ │ • 41 Hardened │ │ • 19 Programs ││ • 188 Repos │ │ • Userspace, │ │ V8 Instances │ │ • 1,572 Tasks ││ • Source Code │ │ V8 & Kernel │ │ • Granular │ │ • Stripped ││ Available │ │ • Live Flag Or. │ │ Primit. Flags │ │ Zero Contam. │└─────────────────┘ └─────────────────┘ └─────────────────┘ └─────────────────┘3. Side-by-Side Architectural Comparison
Section titled “3. Side-by-Side Architectural Comparison”The following table contrasts the design methodologies, execution models, and scoring mechanisms of all four benchmarks:
| Evaluation Dimension | CyberGym | ExploitGym | ExploitBench | SRE-Bench |
|---|---|---|---|---|
| Primary Objective | PoC generation & crash reproduction | Arbitrary code execution (ACE/RCE) | Exploitation pipeline stage verification | Stripped binary analysis & anti-debug defeat |
| Task Volume | 1,507 CVEs | 898 Instances | 41 Targets (656 Tasks) | 262 Binaries (1,572 Tasks) |
| Target Surface | 188 C/C++ open-source projects | Userspace apps, V8 engine, Linux kernel | Google Chrome V8 JIT engine | Network daemons, firmware, malware, parsers |
| Input Artifacts | Source code, patches, issue reports | Source code, PoV crash input, compiler flags | V8 d8 binary, regression test case, diff | Stripped compiled ELF/PE binaries ONLY |
| Success Verification | ASan crash trace / exit code match | Deterministic flag capture via shellcode | 16 automated challenge-response oracles | Exact crypto key, patch, or protocol token |
| Contamination Risk | High (Public GitHub CVEs & issues) | Moderate (Historical CVEs with modified envs) | Moderate (Known bugs, randomized oracles) | Zero (100% private, newly authored code) |
| Frontier Model Pass Rate | 48.2% – 62.4% | 13.4% – 17.5% | 21.8% (Tier avg.) | 11.2% – 14.6% |
4. Deep-Dive Analysis of the Four Frameworks
Section titled “4. Deep-Dive Analysis of the Four Frameworks”A. CyberGym: The Scale Pioneer (PoC Generation)
Section titled “A. CyberGym: The Scale Pioneer (PoC Generation)”CyberGym established the first large-scale pipeline for measuring whether AI models could take a known vulnerability advisory and generate a functional crash input.
- Strengths: Huge sample size (1,507 vulnerabilities); tests diverse software architectures (image libraries, network tools, database servers).
- Weaknesses: Uses crash reproduction as a proxy for exploitation. A null-pointer dereference that terminates a process is graded as a success, even though it provides zero exploitation value to an attacker beyond a simple denial-of-service.
B. ExploitGym: The Real Weaponization Test
Section titled “B. ExploitGym: The Real Weaponization Test”ExploitGym directly tackles the weaponization bottleneck: given a crash, can an agent write an exploit that executes shellcode?
- Strengths: Evaluates modern defensive mitigations (ASLR on vs. off); measures three distinct target domains including the Linux kernel.
- Weaknesses: High resource cost; binary pass/fail grading hides partial progress along the exploitation path.
C. ExploitBench: The Granular Capability Ladder
Section titled “C. ExploitBench: The Granular Capability Ladder”Rather than treating exploitation as an all-or-nothing milestone, ExploitBench decomposes the process into 16 granular stages:
- The 16 Primitives: Crash reproduction $\to$ memory leak $\to$ fake object synthesis $\to$ arbitrary read $\to$ arbitrary write $\to$ control-flow hijack $\to$ sandbox escape.
- Value: Reveals exactly where models stall. In V8 testing, models frequently achieve primitive 5 (arbitrary read) but fail at primitive 12 (sandbox isolation bypass).
D. SRE-Bench: The Anti-Analysis Frontier
Section titled “D. SRE-Bench: The Anti-Analysis Frontier”SRE-Bench evaluates the hardest challenge in cybersecurity: understanding closed-source, compiled software stripped of all metadata.
- Key Innovation: Zero dataset contamination. Every target program was authored from scratch in private repositories.
- The Takeaway: Shows that when source code is removed and anti-analysis protections are applied, AI model effectiveness drops below 15%.
5. What Can an AI Agent Actually Do? (Cross-Benchmark Synthesis)
Section titled “5. What Can an AI Agent Actually Do? (Cross-Benchmark Synthesis)”By synthesizing the results across all four benchmarks, Hermes establishes a consolidated capability boundary:
┌──────────────────────────────────────────────────────────────────────────────────┐│ CONSOLIDATED CROSS-BENCHMARK CAPABILITY MAP (2026) │├──────────────────────────────────────────────────────────────────────────────────┤│ HIGH CAPABILITY (>50% Success Rate): ││ ✔ Identify known vulnerability classes in open-source C/C++ (CyberGym). ││ ✔ Generate fuzzing inputs that trigger existing ASan crashes (CyberGym). ││ ✔ Decompile and summarize un-obfuscated x86-64 functions (SRE-Bench). │├──────────────────────────────────────────────────────────────────────────────────┤│ MODERATE CAPABILITY (15% – 35% Success Rate): ││ ◑ Exploit basic userspace stack overflows with ASLR disabled (ExploitGym). ││ ◑ Synthesize memory leak primitives to recover canaries (ExploitBench). ││ ◑ Solve simple crackmes under 5K LOC without anti-debugging (SRE-Bench). │├──────────────────────────────────────────────────────────────────────────────────┤│ LOW / NEGLIGIBLE CAPABILITY (<5% Success Rate): ││ ✖ Autonomous weaponization of Linux kernel vulnerabilities (ExploitGym). ││ ✖ Full multi-stage sandbox escapes in hardened JIT engines (ExploitBench). ││ ✖ End-to-end reverse engineering of commercial-grade obfuscated binaries. │└──────────────────────────────────────────────────────────────────────────────────┘6. How Should Security Teams Interpret Vendor Claims?
Section titled “6. How Should Security Teams Interpret Vendor Claims?”When an AI vendor claims their agent possesses “cybersecurity capabilities,” security leaders should ask three qualifying questions:
- Was the model tested on Source Code or Compiled Binaries?
If the benchmark provided C source code (like CyberGym), the agent was doing code reasoning, not binary analysis. - What was the Verification Oracle?
Did the agent merely produce a segmentation fault (PoC crash), or did it execute arbitrary payload logic (ExploitGym)? - How was Contamination Controlled?
Were the test cases public CTF challenges existing on GitHub prior to the model’s training cutoff date?
7. Related Articles & Intelligence Links
Section titled “7. Related Articles & Intelligence Links” SRE-Bench: Binary Reverse Engineering Limits Read the detailed analysis of the zero-contamination binary benchmark.
ExploitGym: Turning Flaws into Attacks Examine empirical weaponization pass rates across 898 real-world CVEs.
Editorial Strategy: AI Agents & Cybersecurity Review the complete Hermes 10-level capability ladder and evaluation methodology.
Hermes Agentic Threat Score (HTS) Learn how Hermes scores autonomous execution risks and real-world blast radius.