AI Cybersecurity Benchmarks Compared: CyberGym vs ExploitGym vs ExploitBench vs SRE-Bench
1. Introduction: The Fragmentation of AI Cybersecurity Evaluation
Section titled โ1. Introduction: The Fragmentation of AI Cybersecurity EvaluationโBetween 2024 and 2026, the cybersecurity evaluation of Large Language Models transitioned from static multiple-choice questionnaires (such as SecQA) to interactive, dynamic execution testbeds. However, because each research lab developed its own isolated benchmark, published claims frequently contradict one another:
- One paper reports that AI models achieve an 82% success rate in cybersecurity tasks.
- Another paper evaluating the same frontier model reports a failure rate exceeding 85%.
These conflicting headlines arise because โhackingโ is not a monolithic activity. There is a vast technical gulf between generating a crash-inducing payload for an open-source library and synthesizing a multi-stage Return-Oriented Programming (ROP) exploit against a hardened browser engine.
This study presents an architectural comparison of the four definitive benchmarks defining AI agent cybersecurity capabilities in 2026: CyberGym, ExploitGym, ExploitBench, and SRE-Bench.
2. The Benchmark Taxonomy: What Does Each Suite Test?
Section titled โ2. The Benchmark Taxonomy: What Does Each Suite Test?โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ THE AI CYBERSECURITY EVALUATION LANDSCAPE โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฌโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ โ โโโโโโโโโโโโโโโโโโโโโฌโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโฌโโโโโโโโโโโโโโโโโโโโ โผ โผ โผ โผโโโโโโโโโโโโโโโโโโโ โโโโโโโโโโโโโโโโโโโ โโโโโโโโโโโโโโโโโโโ โโโโโโโโโโโโโโโโโโโโ CyberGym โ โ ExploitGym โ โ ExploitBench โ โ SRE-Bench โโ(arXiv:2506.02548โ โ(arXiv:2605.11086โ โ(arXiv:2605.14153โ โ(arXiv:2608.11469โโโโโโโโโโโโโโโโโโโโค โโโโโโโโโโโโโโโโโโโค โโโโโโโโโโโโโโโโโโโค โโโโโโโโโโโโโโโโโโโคโ โข Focus: Crash โ โ โข Focus: Full โ โ โข Focus: 16-tierโ โ โข Focus: Binary โโ Reproduction โ โ Exploitation โ โ Capability โ โ Reverse Eng. โโ & PoC Inputs โ โ & RCE Shells โ โ Ladder (V8) โ โ & Anti-Debug โโ โข 1,507 Flaws โ โ โข 898 Flaws โ โ โข 41 Hardened โ โ โข 19 Programs โโ โข 188 Repos โ โ โข Userspace, โ โ V8 Instances โ โ โข 1,572 Tasks โโ โข Source Code โ โ V8 & Kernel โ โ โข Granular โ โ โข Stripped โโ Available โ โ โข Live Flag Or. โ โ Primit. Flags โ โ Zero Contam. โโโโโโโโโโโโโโโโโโโโ โโโโโโโโโโโโโโโโโโโ โโโโโโโโโโโโโโโโโโโ โโโโโโโโโโโโโโโโโโโ3. Side-by-Side Architectural Comparison
Section titled โ3. Side-by-Side Architectural ComparisonโThe following table contrasts the design methodologies, execution models, and scoring mechanisms of all four benchmarks:
| Evaluation Dimension | CyberGym | ExploitGym | ExploitBench | SRE-Bench |
|---|---|---|---|---|
| Primary Objective | PoC generation & crash reproduction | Arbitrary code execution (ACE/RCE) | Exploitation pipeline stage verification | Stripped binary analysis & anti-debug defeat |
| Task Volume | 1,507 CVEs | 898 Instances | 41 Targets (656 Tasks) | 262 Binaries (1,572 Tasks) |
| Target Surface | 188 C/C++ open-source projects | Userspace apps, V8 engine, Linux kernel | Google Chrome V8 JIT engine | Network daemons, firmware, malware, parsers |
| Input Artifacts | Source code, patches, issue reports | Source code, PoV crash input, compiler flags | V8 d8 binary, regression test case, diff | Stripped compiled ELF/PE binaries ONLY |
| Success Verification | ASan crash trace / exit code match | Deterministic flag capture via shellcode | 16 automated challenge-response oracles | Exact crypto key, patch, or protocol token |
| Contamination Risk | High (Public GitHub CVEs & issues) | Moderate (Historical CVEs with modified envs) | Moderate (Known bugs, randomized oracles) | Zero (100% private, newly authored code) |
| Frontier Model Pass Rate | 48.2% โ 62.4% | 13.4% โ 17.5% | 21.8% (Tier avg.) | 11.2% โ 14.6% |
4. Deep-Dive Analysis of the Four Frameworks
Section titled โ4. Deep-Dive Analysis of the Four FrameworksโA. CyberGym: The Scale Pioneer (PoC Generation)
Section titled โA. CyberGym: The Scale Pioneer (PoC Generation)โCyberGym established the first large-scale pipeline for measuring whether AI models could take a known vulnerability advisory and generate a functional crash input.
- Strengths: Huge sample size (1,507 vulnerabilities); tests diverse software architectures (image libraries, network tools, database servers).
- Weaknesses: Uses crash reproduction as a proxy for exploitation. A null-pointer dereference that terminates a process is graded as a success, even though it provides zero exploitation value to an attacker beyond a simple denial-of-service.
B. ExploitGym: The Real Weaponization Test
Section titled โB. ExploitGym: The Real Weaponization TestโExploitGym directly tackles the weaponization bottleneck: given a crash, can an agent write an exploit that executes shellcode?
- Strengths: Evaluates modern defensive mitigations (ASLR on vs. off); measures three distinct target domains including the Linux kernel.
- Weaknesses: High resource cost; binary pass/fail grading hides partial progress along the exploitation path.
C. ExploitBench: The Granular Capability Ladder
Section titled โC. ExploitBench: The Granular Capability LadderโRather than treating exploitation as an all-or-nothing milestone, ExploitBench decomposes the process into 16 granular stages:
- The 16 Primitives: Crash reproduction $\to$ memory leak $\to$ fake object synthesis $\to$ arbitrary read $\to$ arbitrary write $\to$ control-flow hijack $\to$ sandbox escape.
- Value: Reveals exactly where models stall. In V8 testing, models frequently achieve primitive 5 (arbitrary read) but fail at primitive 12 (sandbox isolation bypass).
D. SRE-Bench: The Anti-Analysis Frontier
Section titled โD. SRE-Bench: The Anti-Analysis FrontierโSRE-Bench evaluates the hardest challenge in cybersecurity: understanding closed-source, compiled software stripped of all metadata.
- Key Innovation: Zero dataset contamination. Every target program was authored from scratch in private repositories.
- The Takeaway: Shows that when source code is removed and anti-analysis protections are applied, AI model effectiveness drops below 15%.
5. What Can an AI Agent Actually Do? (Cross-Benchmark Synthesis)
Section titled โ5. What Can an AI Agent Actually Do? (Cross-Benchmark Synthesis)โBy synthesizing the results across all four benchmarks, Hermes establishes a consolidated capability boundary:
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ CONSOLIDATED CROSS-BENCHMARK CAPABILITY MAP (2026) โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโคโ HIGH CAPABILITY (>50% Success Rate): โโ โ Identify known vulnerability classes in open-source C/C++ (CyberGym). โโ โ Generate fuzzing inputs that trigger existing ASan crashes (CyberGym). โโ โ Decompile and summarize un-obfuscated x86-64 functions (SRE-Bench). โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโคโ MODERATE CAPABILITY (15% โ 35% Success Rate): โโ โ Exploit basic userspace stack overflows with ASLR disabled (ExploitGym). โโ โ Synthesize memory leak primitives to recover canaries (ExploitBench). โโ โ Solve simple crackmes under 5K LOC without anti-debugging (SRE-Bench). โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโคโ LOW / NEGLIGIBLE CAPABILITY (<5% Success Rate): โโ โ Autonomous weaponization of Linux kernel vulnerabilities (ExploitGym). โโ โ Full multi-stage sandbox escapes in hardened JIT engines (ExploitBench). โโ โ End-to-end reverse engineering of commercial-grade obfuscated binaries. โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ6. How Should Security Teams Interpret Vendor Claims?
Section titled โ6. How Should Security Teams Interpret Vendor Claims?โWhen an AI vendor claims their agent possesses โcybersecurity capabilities,โ security leaders should ask three qualifying questions:
- Was the model tested on Source Code or Compiled Binaries?
If the benchmark provided C source code (like CyberGym), the agent was doing code reasoning, not binary analysis. - What was the Verification Oracle?
Did the agent merely produce a segmentation fault (PoC crash), or did it execute arbitrary payload logic (ExploitGym)? - How was Contamination Controlled?
Were the test cases public CTF challenges existing on GitHub prior to the modelโs training cutoff date?