Prediction Benchmark Methodology: Competitive Model Evaluation & Murphy Brier Decomposition

Principle P10 & P12: Open Epistemic Benchmarking

The End of Unilateral Claims: Threat intelligence vendors routinely claim “95% accuracy” or “industry-leading early warning” using proprietary test sets and moving goalposts. The Hermes Prediction Benchmark establishes an open, reproducible framework that subjects all models—including Hermes’s own Forecast Engine—to identical resolved events, deterministic third-party oracles, and strictly proper scoring rules. Through Murphy’s Brier Score Decomposition (1973), we isolate how much of a model’s performance stems from truthful calibration (Reliability) versus raw predictive discrimination (Resolution).

---

1. The Need for Competitive Benchmarking in Cybersecurity

In meteorological forecasting and financial econometrics, models compete on standardized leaderboards with strict public verification. In contrast, cybersecurity scoring has historically suffered from three structural weaknesses:

1. Closed Evaluation Corpora: Vendors publish retrospective whitepapers highlighting instances where their engine flagged a vulnerability early, while ignoring false positives and unflagged critical zero-days.
2. Improper Scoring Rules: Percentage “accuracy” is easily gamed. In a domain where only 3% of vulnerabilities are exploited in the wild, a trivial model predicting “No Exploitation” for 100% of CVEs achieves 97% accuracy while delivering zero defensive value.
3. Absence of Head-to-Head Baselines: Proprietary models are rarely evaluated side-by-side against standard public heuristics such as First-Day EPSS, CVSS severity tiers, or crowd wisdom.

The Hermes Prediction Benchmark Protocol solves this by establishing an immutable, publicly auditable test bed governed by deterministic resolution criteria.

---

2. Interactive Benchmark Workbench

Compare current model standings, inspect the Murphy reliability-resolution bars, or enter custom probabilities in the challenge sandbox:

🏆 PREDICTION BENCHMARK — COMPETITIVE LEADERBOARD & MURPHY DECOMPOSITION

RELEASE 2.1 • MURPHY DECOMPOSITION (1973)

Open, reproducible benchmark comparing the Hermes Forecast Engine against industry baseline models (raw EPSS, CVSS heuristics, crowd consensus, random guessing). Every model is evaluated on identical resolved ground-truth cases using the Brier Score and Murphy Decomposition (Reliability, Resolution, Uncertainty).

Top Ranked Model

Hermes Forecast Engine

#1 Brier Score: 0.1043

Brier Skill Score (BSS)

+58.3%

vs Random Guessing (0.25)

Corpus Uncertainty (UNC)

0.16

UNC = ō(1 - ō) (5 cases)

Win Rate vs Raw EPSS

80% (4/5)

Early weaponization edge

📊

Official Model Leaderboard (Brier Score & Skill Metric)

BENCH-2026-001

Rank	Predictive Model	Brier Score (BS)	Skill Score (BSS)	Reliability (REL ↓)	Resolution (RES ↑)	Win vs EPSS	Tier
🥇 #1	Hermes Forecast Engine Champion	0.1043	+58.3%	0.001	0.0246	80% (4/5)	ELITE (Superforecaster)
🥈 #2	Crowd Consensus Baseline Aggregated Human	0.1232	+50.7%	0.001	0.0216	80% (4/5)	PROFICIENT
🥉 #3	CVSS Severity Heuristic Enterprise Standard	0.1751	+30%	0.0151	0	80% (4/5)	PROFICIENT
#4	Uninformed Random Guess Coin Flip	0.25	0%	0.09	0	80% (4/5)	BASELINE
#5	Raw EPSS First-Day Baseline Population-Wide	0.6223	-148.9%	0.4719	0.0096	0%	UNCALIBRATED

📐

Murphy Decomposition: Why Hermes Outperforms Heuristics

BS = REL - RES + UNC

Allan H. Murphy's 1973 decomposition separates pure calibration deviation (Reliability REL, lower is better) from true information discriminative power (Resolution RES, higher is better) against fixed base-rate uncertainty (UNC).

Hermes Forecast Engine BS: 0.1043

Reliability (Penalty REL ↓) 0.001

Resolution (Separation RES ↑) 0.0246

Max resolution: cleanly separates true attacks from false alarms with near-zero calibration penalty.

Crowd Consensus Baseline BS: 0.1232

Reliability (Penalty REL ↓) 0.001

Resolution (Separation RES ↑) 0.0216

Intermediate trade-off model.

CVSS Severity Heuristic BS: 0.1751

Reliability (Penalty REL ↓) 0.0151

Resolution (Separation RES ↑) 0

High resolution on real attacks, but suffers heavy penalty when high-CVSS flaws fail to materialize.

Uninformed Random Guess BS: 0.25

Reliability (Penalty REL ↓) 0.09

Resolution (Separation RES ↑) 0

Intermediate trade-off model.

Raw EPSS First-Day Baseline BS: 0.6223

Reliability (Penalty REL ↓) 0.4719

Resolution (Separation RES ↑) 0.0096

Severe calibration penalty on newly published zero-days (initial scores overly compressed).

⚔️

Side-by-Side Head-to-Head Comparison

Hermes Forecast EngineCrowd Consensus BaselineCVSS Severity HeuristicUninformed Random GuessRaw EPSS First-Day Baseline VS Hermes Forecast EngineCrowd Consensus BaselineCVSS Severity HeuristicUninformed Random GuessRaw EPSS First-Day Baseline

🎮

Sandbox: Challenge the Leaderboard!

Reset

Input your own probability estimates for the 5 resolved CVE cases. The benchmark computes your personal Brier score in real time and ranks you directly on the leaderboard against Hermes, EPSS, and crowd consensus!

PRED-001 CVE-2026-76460 REALIZED (1)

DeepSeek-V3 LLM Prompt Execution CISA KEV Addition

Your Forecast: 50%

PRED-002 CVE-2026-75650 REALIZED (1)

StyleSmuggler E-Commerce CSS Injection Weaponization

Your Forecast: 50%

PRED-003 CVE-2025-3248 EXPIRED (0)

Langflow Sandbox Memory Corruption Mass Internet Scanning

Your Forecast: 50%

PRED-004 CVE-2025-26319 REALIZED (1)

AutoGPT Indirect Prompt Injection Shell Escape CISA KEV Listing

Your Forecast: 50%

PRED-005 CVE-2024-3400 REALIZED (1)

Palo Alto Networks PAN-OS Command Injection Mass Exploitation

Your Forecast: 50%

Your Personal Brier Score: 0.2500 Estimated Rank: #4

Adjust the sliders above to challenge Hermes (0.1043)!

---

3. Mathematical Formulation: Murphy’s Brier Score Decomposition (1973)

3.1 The Classical Brier Score

For a sequence of N binary empirical events where o_i represents the observed outcome (o_i = 1 if exploited, 0 otherwise) and f_i represents the forecasted probability (0.0 <= f_i <= 1.0), the mean Brier Score (BS) is defined as:

BS = (1 / N) * SUM_{i=1}^N (f_i - o_i)^2

While BS provides a strictly proper summary metric (0.0000 represents perfection, 0.2500 represents an uninformed 50/50 baseline), it conflates two fundamentally different dimensions of forecasting skill:

• Calibration (Reliability): When the model says 70%, does the event occur 70% of the time?
• Discrimination (Resolution): Can the model sort impending attacks from non-events, or does it merely quote the historical base rate?

3.2 Partitioning Predictions into Probability Bins

To separate calibration from discrimination, Allan H. Murphy (1973) partitioned forecasts into K distinct probability categories f_k (for k = 1, ..., K). Let:

• n_k denote the number of forecasts issued in category f_k, such that SUM_{k=1}^K n_k = N.
• o_bar_k denote the observed empirical frequency of events within category k:

o_bar_k = (1 / n_k) * SUM_{i in k} o_i

• o_bar denote the global base rate (climatological sample frequency):

o_bar = (1 / N) * SUM_{i=1}^N o_i

3.3 The Three Components of Murphy’s Decomposition

Murphy proved that the classical Brier Score decomposes algebraically into:

BS = Reliability (REL) - Resolution (RES) + Uncertainty (UNC)

Component 1: Reliability (REL) — The Calibration Penalty

REL = (1 / N) * SUM_{k=1}^K n_k * (f_k - o_bar_k)^2

• Interpretation: Measures the weighted squared difference between the forecasted probability f_k and the actual observed frequency o_bar_k in that bin.
• Goal: Forecasters seek to minimize REL. A perfectly calibrated model exhibits REL = 0.0000.
• Cybersecurity Context: A high REL penalty indicates either dangerous overconfidence (e.g. predicting 95% on events that only occur 50% of the time) or excessive pessimism.

Component 2: Resolution (RES) — Threat Discrimination Power

RES = (1 / N) * SUM_{k=1}^K n_k * (o_bar_k - o_bar)^2

• Interpretation: Measures how far the bin-specific conditional frequencies o_bar_k deviate from the global base rate o_bar.
• Goal: Forecasters seek to maximize RES. A model that predicts the historical base rate o_bar for every CVE has RES = 0.0000.
• Cybersecurity Context: High resolution is what security teams pay for. A model with high resolution successfully isolates high-probability attacks from low-probability background noise.

Component 3: Uncertainty (UNC) — Inherent Domain Entropy

UNC = o_bar * (1 - o_bar)

• Interpretation: Represents the inherent variance of the empirical events under evaluation.
• Characteristics: UNC is independent of the forecasting model. If all evaluated CVEs are exploited (o_bar = 1) or none are (o_bar = 0), UNC = 0. If half are exploited (o_bar = 0.5), UNC = 0.2500 (maximum entropy).

---

4. The Brier Skill Score (BSS)

To quantify relative outperformance against an uninformative reference baseline, we compute the Brier Skill Score (BSS):

BSS = 1 - (BS_model / BS_ref)

Where BS_ref is a standardized benchmark baseline (e.g. an uninformed coin flip BS_ref = 0.2500, or the climatological base rate BS_clim = o_bar * (1 - o_bar)):

• BSS = 1.0 (+100%): Perfect deterministic foresight.
• BSS > 0.0 (> 0%): Skillful prediction superior to the reference baseline.
• BSS = 0.0 (0%): Zero marginal skill compared to the reference baseline.
• BSS < 0.0 (< 0%): Inferior to an uninformed coin flip or base-rate guess.

---

5. Pairwise Head-to-Head Matrix

To determine statistical dominance between any two models M_A and M_B, the benchmark computes pairwise case-by-case residuals:

Delta_i = BS_i(M_A) - BS_i(M_B) = (f_{i,A} - o_i)^2 - (f_{i,B} - o_i)^2

Given a tolerance threshold epsilon = 0.005:

• Model A Wins: Delta_i < -epsilon (M_A incurred lower squared penalty)
• Model B Wins: Delta_i > +epsilon (M_B incurred lower squared penalty)
• Tie: |Delta_i| <= epsilon

The aggregate Win Rate W_{A,B} is computed over non-tied cases:

W_{A,B} = (Wins_A / (Wins_A + Wins_B)) * 100%

---

6. Baseline Models & Comparative Analysis

The benchmark evaluates five distinct forecasting archetypes:

graph TD
    A["Ground Truth Event Corpus"] --> B["HFE-v2.0: Multimodal Genome + Trajectory"]
    A --> C["EPSS-RAW: FIRST.org Population Prior"]
    A --> D["CVSS-HEURISTIC: Enterprise Rule-of-Thumb"]
    A --> E["CROWD-WISDOM: Community Consensus Prior"]
    A --> F["RANDOM-BASELINE: Uninformed 50/50 Floor"]

    B --> G["Brier Score Engine"]
    C --> G
    D --> G
    E --> G
    F --> G

    G --> H["Murphy Decomposition: REL, RES, UNC"]
    G --> I["Pairwise Head-to-Head Records"]
    G --> J["Public Leaderboard REST API"]

6.1 The CVSS Heuristic Failure Mode (Overconfidence Trap)

Enterprise risk teams frequently equate CVSS Base Scores directly to exploitation likelihood:

• Critical (9.0 - 10.0) implies 90-95%
• High (7.0 - 8.9) implies 70-80%

While this heuristic yields positive Resolution (RES ≈ 0.07) by separating high-severity vulnerabilities from low-severity ones, it suffers catastrophic Reliability penalties (REL ≈ 0.16). Because only a minority of CVSS 9+ vulnerabilities ever face wild exploitation, assigning P = 0.95 to unexploited bugs produces massive penalties:

(0.95 - 0)^2 = 0.9025

6.2 The EPSS Dilution Paradox on Zero-Days

FIRST.org’s EPSS model is trained on hundreds of thousands of historical vulnerabilities across decades. Consequently, on Day 1 of a newly disclosed zero-day (before large-scale honeypot probes emerge), EPSS typically reports conservative probabilities between 0.04 and 0.15.

When an unauthenticated remote code execution exploit is actively weaponized by APTs within 48 hours (o_i = 1), a prediction of P = 0.07 incurs a penalty of (0.07 - 1)^2 = 0.8649. This explains why raw EPSS exhibits a negative Brier Skill Score on fast-moving kinetic zero-days, whereas Hermes’s structural Vulnerability Genome correctly identifies weaponization catalysts at disclosure.

---

7. Open Data Architecture & Verification

In accordance with Principle P8 (Static-First & Long-Term Reproducibility), the complete benchmark dataset is compiled statically at build time into machine-readable JSON artifacts:

Endpoint	Content
/api/benchmark/index.json	Full benchmark metadata, event matrix, model predictions, and summary statistics.
/api/benchmark/leaderboard.json	Official ranking ordered by mean Brier Score with Murphy decomposition metrics.
/api/benchmark/models.json	Detailed model specifications, parameter descriptions, and historical pairwise records.

The evaluation schema is validated against data/schemas/benchmark.schema.json during continuous integration.

---

Related Methodology & Tools

🏆 Prediction Benchmark Workbench Explore the live leaderboard and run custom head-to-head simulations in the interactive tool.

🎯 Hermes Forecast Methodology In-depth specifications of deterministic resolution oracles, epistemic calibration, and scoring rules.

🧬 Vulnerability Genome Discover the 6-loci structural decomposition that drives early-stage kinetic predictions.