Can You Trust LLMs with Cyber Threat Intelligence? Empirical Limits and Hallucinations (arXiv:2503.23175)

Core Research Question

Can Large Language Models be trusted to autonomously extract Indicators of Compromise, map MITRE ATT&CK techniques, and attribute cyber campaigns without fatal hallucinations?
Analyzing “Large Language Models are Unreliable for Cyber Threat Intelligence” (arXiv:2503.23175), this study investigates the empirical reliability, consistency, and overconfidence of frontier LLMs on unstructured threat reporting.

Paper ReferencearXiv:2503.23175

Domain ScopeCyber Threat Intelligence

Hallucination RateUp to 34.2% on Attribution

Dataset Tested1,200 Real CTI Reports

---

1. Introduction: The CTI Automation Dilemma

Modern Security Operations Centers (SOCs) and Threat Intelligence teams face an unsustainable volume of unstructured intelligence: blog posts from Mandiant, incident writeups from CrowdStrike, advisories from CISA, and technical posts on security forums.

To bridge this operational bottleneck, cybersecurity vendors have aggressively integrated Large Language Models to automate:

• Parsing unstructured threat reports into structured formats (STIX/TAXII).
• Extracting Indicators of Compromise (IOCs: IPs, domains, hashes, CVEs).
• Mapping observed threat actor behaviors to MITRE ATT&CK Enterprise techniques.
• Generating threat actor attribution summaries and victim profiling.

However, CTI is fundamentally an evidential domain: attributing an intrusion to APT29 (Cozy Bear) rather than APT28 (Fancy Bear) carries high-stakes diplomatic, legal, and operational consequences. An automated system that invents relationships or misclassifies techniques contaminates downstream firewall blocklists and threat hunt hypotheses.

The study “Large Language Models are Unreliable for Cyber Threat Intelligence” (arXiv:2503.23175) conducted the first comprehensive empirical evaluation measuring whether LLMs possess the accuracy, calibration, and factual consistency required for autonomous CTI production.

---

2. Why This Paper Matters

The findings of arXiv:2503.23175 challenge commercial claims of automated autonomous threat intelligence:

1. The Overconfidence Trap: Models exhibit high verbal confidence (“I am 95% certain this SHA-256 belongs to LockBit 3.0”) even when fabricating hashes or misattributing campaigns.
2. Temporal Anachronism: LLMs regularly attribute 2026 campaigns to historical malware families that were retired years earlier due to bias in pre-training corpora.
3. STIX Graph Incoherence: While models output valid JSON/YAML, the semantic relationships between attack-pattern, malware, and threat-actor objects frequently violate the STIX 2.1 specification or invent nonexistent relationships.

               THE CTI EXTRACTION FIDELITY PIPELINE

  [Unstructured CTI Report] (e.g., 15-Page PDF)
             │
             ▼
     ┌───────────────┐
     │ LLM Ingestion │
     └───────┬───────┘
             │
             ├──────────────────────────┬──────────────────────────┐
             ▼                          ▼                          ▼
     [Explicit IOCs]           [MITRE ATT&CK Map]          [Attribution]
     • IPs, Hashes, Domains     • Technique T-codes         • Threat Group
     • Accuracy: ~91%           • Accuracy: ~68%            • Accuracy: ~54%
     • Hallucinations: Low      • Misclassification: High   • Severe Hallucinations

---

3. How the Research Works: Methodology & Tasks

The researchers subjected LLMs to four foundational CTI evaluation tasks:

┌─────────────────────────────────────────────────────────────────────────────┐
│                    CTI BENCHMARK EVALUATION ARCHITECTURE                    │
└─────────────────────────────────────────────────────────────────────────────┘
  Task 1: IOC Extraction & Typing
  Extract IP addresses, domains, file hashes, and CVE identifiers with exact typing.

  Task 2: MITRE ATT&CK TTP Mapping
  Given a paragraph describing an intrusion, assign the correct sub-technique ID
  (e.g., T1059.001 PowerShell vs T1059.003 Windows Command Shell).

  Task 3: Entity-Relationship Extraction (STIX 2.1)
  Model relationships such as: [Threat Actor] ---uses---> [Malware] ---targets---> [Sector].

  Task 4: Adversary Attribution Reasoning
  Identify the responsible threat actor and justify the conclusion based exclusively
  on evidence contained within the provided text.

To eliminate memorization bias, the researchers evaluated both historical reports (2020–2023) and zero-day reports published after model training cutoffs, verifying whether models could reason about novel threat actor campaigns.

---

4. Dataset Architecture & Ground Truth Verification

The benchmark dataset comprised 1,200 verified CTI reports sourced from premier intelligence organizations:

• Sources: CISA Advisories, Mandiant Threat Intelligence, Microsoft Security Blog, Unit 42, SentinelOne Labs, and Talos.
• Expert Annotation: Every report was annotated by a panel of senior CTI analysts with GIAC Cyber Threat Intelligence (GCTI) certification to establish deterministic ground truth.
• Adversarial Distractors: 150 synthetic reports were injected with subtle contradictions (e.g., claiming a Linux backdoor was deployed via a Windows PowerShell cmdlet) to test whether models would detect contradictory evidence or passively agree with misleading prompts.

---

5. Models Tested & Evaluation Protocols

The evaluation benchmarked five major frontier and open-weight architectures:

• OpenAI GPT-4o & GPT-4 Turbo: The industry standard for commercial CTI automation tools.
• Anthropic Claude 3.5 Sonnet: Renowned for structured data extraction and long-context processing.
• Google Gemini 1.5 Pro: Tested for processing large 100-page incident response PDFs natively.
• Meta Llama 3 70B Instruct: Evaluated as a private, on-premises deployment for sensitive SOCs.
• Mistral Large 2: High-capacity European open-weights model.

Evaluation metrics included Precision, Recall, F1-Score, Expected Calibration Error (ECE), and an explicit Hallucination Penalty Index (HPI).

---

6. Empirical Results: The Attribution & TTP Collapse

Task / Metric	GPT-4o	Claude 3.5 Sonnet	Gemini 1.5 Pro	Llama 3 70B
Explicit IOC Extraction (F1)	92.4%	93.8%	91.2%	86.5%
MITRE ATT&CK TTP Recall	67.2%	72.5%	66.8%	58.4%
STIX 2.1 Relation Precision	61.5%	68.4%	59.2%	49.0%
Novel Threat Actor Attribution	51.2%	56.8%	48.5%	39.1%
Attribution Hallucination Rate	28.4%	24.1%	31.6%	38.7%
Distractor Detection Rate	38.0%	44.2%	32.0%	22.5%

Critical Findings:

1. The Syntactic Illusion of Mastery: Models score high on explicit string matching (IPs, hashes: >91%), creating a false impression of competence.
2. The Attribution Abyss: When confronted with unseen or ambiguous threat actor campaigns, models hallucinate affiliations in up to 38.7% of cases, often forcing attribution to well-known actors (Lazarus, APT29, LockBit) absent any textual evidence.
3. Severe Overconfidence: Calibration analysis demonstrated that when models expressed >90% subjective certainty, their empirical accuracy was under 60%.

---

7. Critical Analysis: The Three Cognitive Traps in CTI

The paper identifies three cognitive mechanisms explaining why LLMs fail at high-level threat intelligence:

1. The Pre-Training Frequency Prior

Models are heavily biased toward the most prominent threat actors in their training data. When a report describes a novel espionage tool used in Southeast Asia, models disproportionately attribute it to Chinese APTs (APT41) even when the forensic artifacts point to emerging regional actors.

2. The Granularity Mismatch in MITRE ATT&CK

LLMs frequently confuse parent techniques with sub-techniques:

• An agent identifies that PowerShell was used (T1059.001), but assigns the parent ID T1059 (Command and Scripting Interpreter).
• When asked for exact sub-techniques, models invent nonexistent identifiers (e.g., T1059.009).

3. Sycophantic Ingestion of Adversarial Distractors

When given reports containing planted technical contradictions (e.g., a report claiming an adversary used an iOS exploit to compromise an air-gapped SCADA PLC running VxWorks), LLMs failed to flag the impossibility in over 60% of cases, uncritically generating STIX entities that formalized the fabricated intrusion vector.

---

8. What Can an AI Agent Actually Do? (Mandatory Partition)

A. Demonstrated Capabilities (Empirically Verified)

• High-Accuracy Explicit IOC Mining: Extracting IPv4/IPv6 addresses, standard domain names, MD5/SHA256 hashes, and CVE identifiers with >92% precision.
• Textual Threat Report Summarization: Synthesizing executive summaries and timeline drafts from verified vendor writeups.
• Basic Technique Tagging: Tagging broad MITRE ATT&CK tactics (e.g., Initial Access vs Lateral Movement) with >80% accuracy.

B. Reasoned Inferences (High Probability Under Constraints)

• Assisted STIX Graph Drafting: Analysts using LLMs with human-in-the-loop validation can accelerate STIX object creation by 2x to 3x, provided every relationship is verified.
• Multi-Language CTI Translation: Translating technical threat advisories from Mandarin, Russian, or Persian into English while preserving technical terminology with high fidelity.

C. Hypothetical / Speculative Claims (Disproven or Unproven)

• Autonomous Threat Actor Attribution: Fully autonomous attribution based on unstructured text without human oversight is dangerous and invalid (hallucination rate of 24% to 38%).
• Autonomous Ingestion into Perimeter Firewalls: Feeding raw LLM-extracted IOCs directly into production firewall or EDR blocking rules without human review leads to catastrophic false positives (e.g., blocking Microsoft or Cloudflare IP ranges).

---

9. Operational Implications for Threat Intelligence Teams

1. Strict Human-in-the-Loop Gatekeeping: LLMs must never be permitted to publish attribution or update detection graphs autonomously. All LLM-generated STIX objects must be staged in an intermediate review queue.
2. Enforcing Retrieval-Augmented Verification (RAG): Models must be constrained to output explicit quotations from the source text for every extracted claim ("evidence_quote": "..."), allowing analysts to audit reasoning in seconds.
3. Decoupling Extraction from Attribution: Use regex and deterministic parsers for IOCs; use LLMs only for semantic summarization; leave actor attribution to senior human intelligence officers.

---

10. Defensive Playbooks & Architecture

┌─────────────────────────────────────────────────────────────────────────────┐
│                 RECOMMENDED DEFENSIVE CTI PIPELINE ARCHITECTURE             │
└─────────────────────────────────────────────────────────────────────────────┘
  Raw CTI Feed (PDF / RSS)
             │
             ▼
     [Deterministic IOC Parser] ────────> [Enrichment via VirusTotal / Shodan]
             │
             ▼
     [Constrained LLM Extractor]
     • JSON Schema Enforced
     • Mandatory Quote Provenance
             │
             ▼
     [Confidence & Attribution Gate]
     • If Attribution Confidence < 100% ──> Flag for Senior Analyst Review
     • If High-Risk Technique ────────────> Verify Against MITRE Matrix
             │
             ▼
     [Validated STIX 2.1 Graph] ────────> [Enterprise SIEM / Threat Intel Platform]

---

11. 12 to 24 Months Outlook (2026–2028)

• Provenance-Backed Reasoning Models: Future frontier models will natively output cryptographic citations tied to source document spans, mitigating factual hallucinations.
• Adversarial CTI Poisoning: Threat actors will deliberately publish poisoned threat reports designed to trick enterprise CTI scrapers into burning security resources or blacklisting legitimate cloud services.

---

12. Related Articles & Graph Traversal

What Can AI Agents Do in Cybersecurity in 2026? The Master Pillar page (arXiv:2608.28490) mapping the 11 operational security domains.

Editorial Strategy & Scoring Rubric The foundational scoring rubric and empirical research standards of Hermes Codex.

PIMiner: Agent-Against-Agent Red-Teaming Autonomous adversarial evaluation and manipulation of conversational agents.

Hermes Threat Score Methodology Deterministic risk assessment and multi-factor prioritization framework.