Hermes Forecast: Falsifiable Predictions & Resolution Oracles

Epistemic Rigor Against Hindsight Bias

The Fundamental Shift: Most threat intelligence feeds rely on retrospective post-mortems (“we knew it all along”) or vague, unfalsifiable warnings (“threat actors may exploit this”). Hermes Forecast introduces rigorous probabilistic forecasts bounded by explicit time horizons and adjudicated by deterministic external resolution oracles (CISA KEV, GreyNoise, public weaponized PoCs, national CERTs). Forecast quality is audited without hindsight bias via the Brier Calibration Score.

---

Interactive Forecast Workbench

Explore active in-flight predictions, inspect resolved oracle verifications, or simulate how future resolutions alter the global Brier score in real-time:

HERMES FORECAST — FALSIFIABLE PREDICTIONS & ORACLES

RELEASE 2.0 • BRIER SCORE & CALIBRATION

Probabilistic forecasting engine for cyber risk and vulnerability weaponization. Every forecast has a fixed temporal horizon and a deterministic external resolution oracle (CISA KEV, GreyNoise, public weaponized PoC). Reliability is objectively audited against hindsight bias via the Brier score.

Mean Brier Score (BS)

0.1043

58% vs random baseline (0.25)

Resolution Track Record

5 / 13

4 TRUE 1 FALSE

Active In-Flight Forecasts

8

Horizons 14 to 60 days

Calibration Tier

HIGH

BS ≤ 0.15 (High calibration)

📊

Reliability Diagram & Empirical Calibration (Deciles)

PRINCIPLE P10: NO FAKE PRECISION

A well-calibrated forecast matches predicted probability with empirical realization frequency. The deliberate inclusion of failed forecasts (e.g. CVE-2025-3248) prevents hindsight cherry-picking and confirms epistemic integrity (Principle P5).

Probability Range	Forecast Count	Mean Forecast	Empirical Outcome	Calibration Delta	Visual Balance
0.00 - 0.20	0 (0 resolved)	—	Pending	—
0.20 - 0.40	0 (0 resolved)	—	Pending	—
0.40 - 0.60	0 (0 resolved)	—	Pending	—
0.60 - 0.80	7 (2 resolved)	73%	50%	+0.23
0.80 - 1.00	6 (3 resolved)	87%	100%	-0.13

🔮

Counterfactual 'What-If' Resolution Simulator

Simulate how active predictions would alter the global Brier Score in real-time

Simulated BS: 0.1043 Reset

🔍

All (13) Active Forecasts (8) Resolved Oracles (5)

All Domains 🤖 Agentic AI 🏢 Enterprise 📦 Supply Chain & Cloud

---

1. What Hermes Forecast Solves

Cybersecurity threat intelligence suffers from two endemic cognitive pathologies:

1. Unfalsifiable Ambiguity: Analysts issue warnings such as “Threat actors are likely to leverage this vulnerability in due course.” If exploitation occurs in 2 weeks, they claim credit; if nothing happens after 2 years, they claim the threat is “still looming.”
2. Hindsight Distortion (Retrospective Bias): After a breach, organizations rewrite their internal history to appear prescient, claiming the risk was obvious all along.

Hermes Forecast eliminates these distortions through three strict structural commitments:

• Immutable Forecasts: Once published, a prediction cannot be deleted, edited, or backdated (Principle P5).
• Deterministic Resolution Oracles: The outcome is judged solely by third-party external ground truth (e.g., CISA KEV JSON catalog, GreyNoise honeypot sensor tags, GitHub advisory releases), removing analyst discretion.
• Brier Score Calibration ($BS$): Prediction accuracy is calculated mathematically, penalizing overconfident false claims and underconfident hedging (Principle P10).

---

2. Anatomy of a Falsifiable Forecast

Every forecast tracked in Hermes Forecast consists of five epistemic pillars:

+-------------------------------------------------------------------------+
|                              HERMES FORECAST                            |
+-------------------------------------------------------------------------+
| Target Entity     : CVE-2026-76460 (DeepSeek-V3 Sandbox Escape)         |
| Falsifiable Claim : "Will be added to CISA KEV catalog within 30 days"  |
| Calibrated P(E)   : 0.88 (88% probability)                              |
| Time Horizon      : 30 days (Cutoff: 2026-10-15T00:00:00Z)              |
| Resolution Oracle : cisa_kev_inclusion (Deterministic JSON match)       |
+-------------------------------------------------------------------------+

The 5 Epistemic Attributes

1. Target Entity (target_id): A specific vulnerability identifier (CVE), software package, or autonomous agent framework.
2. Falsifiable Hypothesis: An unambiguous, binary event assertion (E ∈ {0, 1}).
3. Calibrated Probability ($P \in [0.0, 1.0]$): A quantitative likelihood expressing the model’s epistemic confidence, not a loose verbal label like “high” or “likely.”
4. Finite Time Horizon: A rigid cutoff date (7, 14, 30, 60, or 90 days). If the event has not occurred when the cutoff elapses, the prediction resolves FALSE.
5. Resolution Oracle: An external, publicly verifiable authoritative source with unambiguous evaluation criteria.

---

3. The Brier Calibration Metric

To measure predictive skill objectively, Hermes uses the Brier Score ($BS$), formulated by Glenn W. Brier:

BS = (1 / N) * SUM_{t=1}^N (f_t - o_t)^2

Where:

• f_t is the forecasted probability ($0.0 \le f_t \le 1.0$)
• o_t is the actual empirical outcome ($1$ if the event occurred before horizon expiry, $0$ otherwise)
• N is the total number of resolved predictions

Interpreting Brier Scores

Score Range	Calibration Tier	Interpretation
0.0000	Perfect Clairvoyance	Absolute accuracy on every prediction
0.0001 - 0.1000	Exceptional Skill	Elite forecasting benchmark (Superforecaster level)
0.1001 - 0.1500	High Calibration	Hermes current operational baseline (~0.1043)
0.2500	Zero Skill Baseline	Equivalent to a random coin flip ($P = 0.50$ always)
> 0.2500	Negative Skill / Uncalibrated	Worse than random guessing; overconfident and misleading

---

4. Why Hermes Embraces Negative Outcomes (Anti-Hindsight Integrity)

In our track record, PRED-003 (CVE-2025-3248 Langflow) was forecasted at $P = 0.65$ to see mass scanning across 50+ IPs within 30 days. When the 30-day cutoff expired, GreyNoise telemetry logged only 11 benign research pings, and zero threat-actor botnets weaponized the flaw.

Under conventional industry practices, this forecast would be quietly erased or reworded. In Hermes, it was permanently cataloged as RESOLVED: FALSE, generating an individual Brier penalty of:

BS_i = (0.65 - 0)^2 = 0.4225

This deliberate transparency proves that Hermes’s global Brier Score of 0.1043 is an honest empirical measurement, not the product of retrospective selection bias.

---

5. REST API Access

Hermes Forecast data is statically compiled and accessible via machine-readable JSON endpoints:

• Forecast Index Catalog: /api/forecast/index.json
• Individual Prediction Endpoint: /api/forecast/<PRED-ID>.json (e.g. /api/forecast/PRED-001.json)

# Fetch latest forecast benchmark and calibration deciles
curl -s https://hermes-codex.vercel.app/api/forecast/index.json | jq .summary

---

Methodological Deep-Dives

📐 Hermes Forecast Methodology Mathematical foundations of probabilistic cyber forecasting, oracle adjudication protocols, and calibration mechanics.

⏳ Cyber Risk Time Machine Replay historical threat intelligence states and audit retrospective forecasts against immutable daily snapshots.