Scientific Value (25 Pts)
Methodological rigor, contamination avoidance, statistical significance, and artifact reproducibility (open code, datasets, containers).
The mission of Hermes Codex is to establish a rigorous, evidence-grounded reference on the empirical capabilities and physical limits of AI agents interacting with real-world software, networks, and binary environments.
The central analytical framework follows an unbending sequence:
Capability โโโ> Benchmark โโโ> Experiment โโโ> Limitations โโโ> Operational Implicationsโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ HERMES CODEX โโ Empirical AI Agent Cybersecurity Knowledge Architecture โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฌโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ โผ โผ โผโโโโโโโโโโโโโโโโโโโโโโโโโโโ โโโโโโโโโโโโโโโโโโโโโโโโโโโ โโโโโโโโโโโโโโโโโโโโโโโโโโโโ CLUSTER A โ โ CLUSTER B โ โ CLUSTER C โโ AI AGENTS THAT HACK โ โAI AGENTS THAT GET HACKEDโ โ AI AGENTS THAT DEFEND โโ (Offensive Frontier) โ โ (Systemic Exposure) โ โ (Autonomous Defense) โโโโโโโโโโโโโโโโโโโโโโโโโโโโค โโโโโโโโโโโโโโโโโโโโโโโโโโโค โโโโโโโโโโโโโโโโโโโโโโโโโโโคโ โข Vulnerability Hunting โ โ โข Prompt Injections โ โ โข SOC Triage & Alerting โโ โข PoC Generation โ โ โข Tool & MCP Poisoning โ โ โข Malware Analysis โโ โข Exploit Synthesis โ โ โข Memory / RAG Tamper โ โ โข Automated Patching โโ โข Binary Reverse Eng. โ โ โข Lateral Privilege Esc โ โ โข Threat Intel CTI โโ โข Autonomous Pentesting โ โ โข Agent-to-Agent Spoof โ โ โข Cyber-Range Sim โโโโโโโโโโโโโโโโโโโโโโโโโโโโ โโโโโโโโโโโโโโโโโโโโโโโโโโโ โโโโโโโโโโโโโโโโโโโโโโโโโโโRather than treating โAI hackingโ as a binary capability, Hermes evaluates autonomous agents along an empirical 10-level capability ladder:
| Level | Capability Milestone | Verification Threshold | Primary Benchmark |
|---|---|---|---|
| L1 | Vulnerability Comprehension | Accurately identifies root-cause CWE and affected lines in isolated snippets | HumanEval-Sec, SecFix |
| L2 | Vulnerability Discovery | Identifies unpatched flaws across complex multi-file codebases without hints | CyberGym, SecCode |
| L3 | Bug Reproduction | Takes a vulnerability report/CVE and reproduces the crash in a container | CyberGym, SRE-Bench |
| L4 | PoC Generation | Generates an input triggering an unhandled exception or denial-of-service | ExploitGym (PoV phase) |
| L5 | Exploitation Primitive Synthesis | Obtains controlled arbitrary read, arbitrary write, or out-of-bounds corruption | ExploitBench |
| L6 | Control-Flow Hijacking | Bypasses local protections (stack cookies, basic canary) to redirect execution | ExploitBench, ExploitGym |
| L7 | Arbitrary Code Execution (ACE) | Gains unauthenticated arbitrary shellcode execution (with ASLR / DEP disabled) | ExploitGym (Userspace) |
| L8 | Mitigation Evasion & Full RCE | Bypasses modern mitigations (ASLR, DEP, CFI) to obtain remote execution | ExploitGym (Kernel / V8) |
| L9 | Session Persistence & Post-Ex | Discovers internal credentials, corrupts execution environments, maintains access | Inter-Agent Benchmarks |
| L10 | Autonomous Multi-Stage Campaign | Executes recon $\to$ weaponization $\to$ lateral movement across an enterprise network | Cyber-Range Simulation |
Every research analysis published within this series adheres to an invariant, peer-review grade structure:
Incoming preprints across cs.CR, cs.AI, cs.SE, and cs.LG are evaluated against a standardized 100-point rubric:
Scientific Value (25 Pts)
Methodological rigor, contamination avoidance, statistical significance, and artifact reproducibility (open code, datasets, containers).
Cybersecurity Relevance (25 Pts)
Direct alignment with real-world operating systems, kernel runtimes, active protocols, and physical threat surfaces.
Traffic & Search Intent (25 Pts)
High organic search volume, clear practitioner queries (โCan AI reverse engineer binaries?โ), and enduring reference utility.
Editorial Potential (15 Pts)
Clarity of thesis, potential to illuminate complex architectural boundaries, and modularity within our content clusters.
Longevity (10 Pts)
Long-term relevance as a baseline benchmark or paradigm shift; resistant to instant model-version obsolescence.
Threshold: Papers scoring >= 75 / 100 qualify for full deep-dive publication.
The initial wave of 11 publications establishes Hermes Codex as the benchmark-of-record:
Hermes maintains a living empirical index tracking demonstrated agent capabilities across release cycles:
| Capability Domain | 2024 (GPT-4 / Claude 3) | 2025 (o1 / Sonnet 3.5) | 2026 (Frontier Reasoning) | Verified Frontier Baseline |
|---|---|---|---|---|
| Vulnerability Discovery (Source) | Medium | High | High | Snyk / Semgrep AI |
| PoC Crash Generation | Low | Medium | High | CyberGym / ExploitGym |
| Binary Reverse Engineering | Very Low | Low | Low / Medium | SRE-Bench (arXiv:2608.11469) |
| Kernel / V8 Exploitation | Negligible | Very Low | Low | ExploitGym (arXiv:2605.11086) |
| Automated Prompt Injection | Low | Medium | High | PIMiner (arXiv:2608.05108) |
| Agent Tool Sandboxing | Low | Low | Medium | MTGuard (arXiv:2607.25297) |
| CTI Report Analysis | Low / Hallucinatory | Low | Medium | arXiv:2503.23175 |