Prediction Benchmark Methodology: Competitive Model Evaluation & Murphy Brier Decomposition
1. The Need for Competitive Benchmarking in Cybersecurity
Section titled โ1. The Need for Competitive Benchmarking in CybersecurityโIn meteorological forecasting and financial econometrics, models compete on standardized leaderboards with strict public verification. In contrast, cybersecurity scoring has historically suffered from three structural weaknesses:
- Closed Evaluation Corpora: Vendors publish retrospective whitepapers highlighting instances where their engine flagged a vulnerability early, while ignoring false positives and unflagged critical zero-days.
- Improper Scoring Rules: Percentage โaccuracyโ is easily gamed. In a domain where only 3% of vulnerabilities are exploited in the wild, a trivial model predicting โNo Exploitationโ for 100% of CVEs achieves 97% accuracy while delivering zero defensive value.
- Absence of Head-to-Head Baselines: Proprietary models are rarely evaluated side-by-side against standard public heuristics such as First-Day EPSS, CVSS severity tiers, or crowd wisdom.
The Hermes Prediction Benchmark Protocol solves this by establishing an immutable, publicly auditable test bed governed by deterministic resolution criteria.
2. Interactive Benchmark Workbench
Section titled โ2. Interactive Benchmark WorkbenchโCompare current model standings, inspect the Murphy reliability-resolution bars, or enter custom probabilities in the challenge sandbox:
Official Model Leaderboard (Brier Score & Skill Metric)
| Rank | Predictive Model | Brier Score (BS) | Skill Score (BSS) | Reliability (REL โ) | Resolution (RES โ) | Win vs EPSS | Tier |
|---|---|---|---|---|---|---|---|
| ๐ฅ #1 | Hermes Forecast Engine Champion | 0.1043 | +58.3% | 0.001 | 0.0246 | 80% (4/5) | ELITE (Superforecaster) |
| ๐ฅ #2 | Crowd Consensus Baseline Aggregated Human | 0.1232 | +50.7% | 0.001 | 0.0216 | 80% (4/5) | PROFICIENT |
| ๐ฅ #3 | CVSS Severity Heuristic Enterprise Standard | 0.1751 | +30% | 0.0151 | 0 | 80% (4/5) | PROFICIENT |
| #4 | Uninformed Random Guess Coin Flip | 0.25 | 0% | 0.09 | 0 | 80% (4/5) | BASELINE |
| #5 | Raw EPSS First-Day Baseline Population-Wide | 0.6223 | -148.9% | 0.4719 | 0.0096 | 0% | UNCALIBRATED |
Murphy Decomposition: Why Hermes Outperforms Heuristics
Allan H. Murphy's 1973 decomposition separates pure calibration deviation (Reliability REL, lower is better) from true information discriminative power (Resolution RES, higher is better) against fixed base-rate uncertainty (UNC).
Side-by-Side Head-to-Head Comparison
Sandbox: Challenge the Leaderboard!
Input your own probability estimates for the 5 resolved CVE cases. The benchmark computes your personal Brier score in real time and ranks you directly on the leaderboard against Hermes, EPSS, and crowd consensus!
3. Mathematical Formulation: Murphyโs Brier Score Decomposition (1973)
Section titled โ3. Mathematical Formulation: Murphyโs Brier Score Decomposition (1973)โ3.1 The Classical Brier Score
Section titled โ3.1 The Classical Brier ScoreโFor a sequence of N binary empirical events where o_i represents the observed outcome (o_i = 1 if exploited, 0 otherwise) and f_i represents the forecasted probability (0.0 <= f_i <= 1.0), the mean Brier Score (BS) is defined as:
BS = (1 / N) * SUM_{i=1}^N (f_i - o_i)^2While BS provides a strictly proper summary metric (0.0000 represents perfection, 0.2500 represents an uninformed 50/50 baseline), it conflates two fundamentally different dimensions of forecasting skill:
- Calibration (Reliability): When the model says 70%, does the event occur 70% of the time?
- Discrimination (Resolution): Can the model sort impending attacks from non-events, or does it merely quote the historical base rate?
3.2 Partitioning Predictions into Probability Bins
Section titled โ3.2 Partitioning Predictions into Probability BinsโTo separate calibration from discrimination, Allan H. Murphy (1973) partitioned forecasts into K distinct probability categories f_k (for k = 1, ..., K). Let:
n_kdenote the number of forecasts issued in categoryf_k, such thatSUM_{k=1}^K n_k = N.o_bar_kdenote the observed empirical frequency of events within categoryk:
o_bar_k = (1 / n_k) * SUM_{i in k} o_io_bardenote the global base rate (climatological sample frequency):
o_bar = (1 / N) * SUM_{i=1}^N o_i3.3 The Three Components of Murphyโs Decomposition
Section titled โ3.3 The Three Components of Murphyโs DecompositionโMurphy proved that the classical Brier Score decomposes algebraically into:
BS = Reliability (REL) - Resolution (RES) + Uncertainty (UNC)Component 1: Reliability (REL) โ The Calibration Penalty
Section titled โComponent 1: Reliability (REL) โ The Calibration PenaltyโREL = (1 / N) * SUM_{k=1}^K n_k * (f_k - o_bar_k)^2- Interpretation: Measures the weighted squared difference between the forecasted probability
f_kand the actual observed frequencyo_bar_kin that bin. - Goal: Forecasters seek to minimize
REL. A perfectly calibrated model exhibitsREL = 0.0000. - Cybersecurity Context: A high
RELpenalty indicates either dangerous overconfidence (e.g. predicting 95% on events that only occur 50% of the time) or excessive pessimism.
Component 2: Resolution (RES) โ Threat Discrimination Power
Section titled โComponent 2: Resolution (RES) โ Threat Discrimination PowerโRES = (1 / N) * SUM_{k=1}^K n_k * (o_bar_k - o_bar)^2- Interpretation: Measures how far the bin-specific conditional frequencies
o_bar_kdeviate from the global base rateo_bar. - Goal: Forecasters seek to maximize
RES. A model that predicts the historical base rateo_barfor every CVE hasRES = 0.0000. - Cybersecurity Context: High resolution is what security teams pay for. A model with high resolution successfully isolates high-probability attacks from low-probability background noise.
Component 3: Uncertainty (UNC) โ Inherent Domain Entropy
Section titled โComponent 3: Uncertainty (UNC) โ Inherent Domain EntropyโUNC = o_bar * (1 - o_bar)- Interpretation: Represents the inherent variance of the empirical events under evaluation.
- Characteristics:
UNCis independent of the forecasting model. If all evaluated CVEs are exploited (o_bar = 1) or none are (o_bar = 0),UNC = 0. If half are exploited (o_bar = 0.5),UNC = 0.2500(maximum entropy).
4. The Brier Skill Score (BSS)
Section titled โ4. The Brier Skill Score (BSS)โTo quantify relative outperformance against an uninformative reference baseline, we compute the Brier Skill Score (BSS):
BSS = 1 - (BS_model / BS_ref)Where BS_ref is a standardized benchmark baseline (e.g. an uninformed coin flip BS_ref = 0.2500, or the climatological base rate BS_clim = o_bar * (1 - o_bar)):
BSS = 1.0(+100%): Perfect deterministic foresight.BSS > 0.0(> 0%): Skillful prediction superior to the reference baseline.BSS = 0.0(0%): Zero marginal skill compared to the reference baseline.BSS < 0.0(< 0%): Inferior to an uninformed coin flip or base-rate guess.
5. Pairwise Head-to-Head Matrix
Section titled โ5. Pairwise Head-to-Head MatrixโTo determine statistical dominance between any two models M_A and M_B, the benchmark computes pairwise case-by-case residuals:
Delta_i = BS_i(M_A) - BS_i(M_B) = (f_{i,A} - o_i)^2 - (f_{i,B} - o_i)^2Given a tolerance threshold epsilon = 0.005:
- Model A Wins:
Delta_i < -epsilon(M_Aincurred lower squared penalty) - Model B Wins:
Delta_i > +epsilon(M_Bincurred lower squared penalty) - Tie:
|Delta_i| <= epsilon
The aggregate Win Rate W_{A,B} is computed over non-tied cases:
W_{A,B} = (Wins_A / (Wins_A + Wins_B)) * 100%6. Baseline Models & Comparative Analysis
Section titled โ6. Baseline Models & Comparative AnalysisโThe benchmark evaluates five distinct forecasting archetypes:
graph TD A["Ground Truth Event Corpus"] --> B["HFE-v2.0: Multimodal Genome + Trajectory"] A --> C["EPSS-RAW: FIRST.org Population Prior"] A --> D["CVSS-HEURISTIC: Enterprise Rule-of-Thumb"] A --> E["CROWD-WISDOM: Community Consensus Prior"] A --> F["RANDOM-BASELINE: Uninformed 50/50 Floor"]
B --> G["Brier Score Engine"] C --> G D --> G E --> G F --> G
G --> H["Murphy Decomposition: REL, RES, UNC"] G --> I["Pairwise Head-to-Head Records"] G --> J["Public Leaderboard REST API"]6.1 The CVSS Heuristic Failure Mode (Overconfidence Trap)
Section titled โ6.1 The CVSS Heuristic Failure Mode (Overconfidence Trap)โEnterprise risk teams frequently equate CVSS Base Scores directly to exploitation likelihood:
- Critical (9.0 - 10.0) implies 90-95%
- High (7.0 - 8.9) implies 70-80%
While this heuristic yields positive Resolution (RES โ 0.07) by separating high-severity vulnerabilities from low-severity ones, it suffers catastrophic Reliability penalties (REL โ 0.16). Because only a minority of CVSS 9+ vulnerabilities ever face wild exploitation, assigning P = 0.95 to unexploited bugs produces massive penalties:
(0.95 - 0)^2 = 0.90256.2 The EPSS Dilution Paradox on Zero-Days
Section titled โ6.2 The EPSS Dilution Paradox on Zero-DaysโFIRST.orgโs EPSS model is trained on hundreds of thousands of historical vulnerabilities across decades. Consequently, on Day 1 of a newly disclosed zero-day (before large-scale honeypot probes emerge), EPSS typically reports conservative probabilities between 0.04 and 0.15.
When an unauthenticated remote code execution exploit is actively weaponized by APTs within 48 hours (o_i = 1), a prediction of P = 0.07 incurs a penalty of (0.07 - 1)^2 = 0.8649. This explains why raw EPSS exhibits a negative Brier Skill Score on fast-moving kinetic zero-days, whereas Hermesโs structural Vulnerability Genome correctly identifies weaponization catalysts at disclosure.
7. Open Data Architecture & Verification
Section titled โ7. Open Data Architecture & VerificationโIn accordance with Principle P8 (Static-First & Long-Term Reproducibility), the complete benchmark dataset is compiled statically at build time into machine-readable JSON artifacts:
| Endpoint | Content |
|---|---|
/api/benchmark/index.json | Full benchmark metadata, event matrix, model predictions, and summary statistics. |
/api/benchmark/leaderboard.json | Official ranking ordered by mean Brier Score with Murphy decomposition metrics. |
/api/benchmark/models.json | Detailed model specifications, parameter descriptions, and historical pairwise records. |
The evaluation schema is validated against data/schemas/benchmark.schema.json during continuous integration.