Toward Secure LLM Agents: Threat Surfaces, Attacks, Defenses, and Evaluation (arXiv:2606.10749)
Toward Secure LLM Agents: Threat Surfaces, Attacks, Defenses, and Evaluation
Paper Claims
What the authors claim to have proven or engineered:
- Agentic security failure modes are categorically distinct from LLM generation safety, centering on control-flow hijacking, privilege misuse, and persistent state corruption.
- Synthesizes 247 academic papers into a unified systems-oriented lifecycle framework spanning information flow, delegated authority, and persistent state across 4 distinct execution stages.
- Prompt injection and tool-mediated execution hijacking dominate over 65% of existing research, but persistent memory poisoning and multi-agent cascading propagation represent the most dangerous emerging attack frontiers.
- Current defensive mechanisms (prompt guardrails, delimiter isolation, output detectors) provide localized building blocks but exhibit weak compositionality when chained together.
Experimental Evidence
What was actually demonstrated with observable metrics:
- Meta-analysis across 247 vetted peer-reviewed papers spanning 2022 to 2026 across major security (IEEE S&P, USENIX, ACM CCS, NDSS) and AI venues (NeurIPS, ICML, ICLR, ACL).
- Empirically categorizes defense trade-offs: text-level defenses incur 5% to 15% latency with less than 50% adaptive bypass protection; execution-level sandboxing reduces ASR by up to 80% but requires strict capability mediation.
- Reveals that over 78% of existing benchmarks evaluate static, single-turn interactions, severely underrepresenting long-horizon state drift and cascading multi-agent delegation risks.
Limitations & Caveats
What is NOT proven, lab-constrained, or assumed:
- Focuses primarily on cataloging academic literature rather than introducing a single novel exploit primitive or runtime defense mechanism.
- Quantitative comparisons across disparate benchmarks inherit the original papers' heterogeneous evaluation protocols and metric variances.
- Does not evaluate hardware-level side channels or hypervisor-level physical security of GPU compute nodes hosting agent runtimes.
Hermes Critical Analysis
Objective interpretation deconstructing hype vs reality:
Ling et al. provide the definitive taxonomy clarifying why prompt injection is not merely a content moderation bug, but a fundamental control-plane hijacking vulnerability. By modeling agent security around the tripartite interaction of Information Flow, Delegated Authority, and Persistent State, the authors articulate what Hermes has long asserted: agentic AI represents a return to classical distributed systems vulnerabilities, executed in the semantic domain.
Security Implications
Why enterprise security teams must pay attention:
- Enterprise SOCs cannot treat agentic workflows as black-box SaaS integrations; delegated tool tokens must enforce strict capability-based access control (CBAC).
- Shared episodic memory and RAG stores create persistent lateral attack vectors that survive application container restarts and model context resets.
- Multi-agent orchestration frameworks (e.g. LangGraph, CrewAI, AutoGen) risk cascading trust boundary collapse when an unvetted edge agent processes untrusted inputs.
Operational Recommendations
Actionable controls for CISOs, SOC teams, and AI engineers:
- Adopt the 7-layer HASS-1.0 architecture to isolate model inference from OS execution and network capability issuance.
- Enforce strict noninterference policies between untrusted perception channels (web scrapes, emails, issue tickets) and control-plane decision triggers.
- Deploy cryptographic provenance tracking and namespace partitioning across all vector database memory stores.
- Establish mandatory human-in-the-loop (HITL) authorization gates for state-mutating actions (financial transfers, infrastructure provisioning, git pushes).
HERMES AGENTIC SECURITY SCORE & RISK: SURVEY TAXONOMY
Target:Autonomous LLM Agent Systems (Planning, Tool Use, Memory, Multi-Agent Delegation) HASS rates the broader agentic threat surface at 78/100 (HIGH). While point defenses provide localized protection against naรฏve attacks, Ling et al. confirm that real-world agent pipelines fail systematically at the intersection of delegated tool authority and persistent memory state.
AAP-001: Direct System Prompt OverrideAGENTIC ATTACK_PATTERN
1. Executive Summary & Epistemic Contribution
Section titled โ1. Executive Summary & Epistemic ContributionโAs Large Language Models (LLMs) transition from chat interfaces into autonomous software agents that parse external environments, formulate multi-step plans, invoke tools, and maintain long-term memory, classical cybersecurity boundaries dissolve.
In โToward Secure LLM Agents: Threat Surfaces, Attacks, Defenses, and Evaluationโ (arXiv:2606.10749), Ling, Yu, Chen, and Fang from Nanjing University synthesize 247 peer-reviewed and preprint studies to build a comprehensive, systems-oriented lifecycle security framework. Rather than viewing agent security through the narrow lens of model alignment or textual โjailbreaksโ, the authors model agents as distributed stateful software architectures characterized by three intersecting vectors:
- Information Flow: The ingestion of trusted system prompts alongside untrusted external environmental data (emails, web pages, tool outputs).
- Delegated Authority: The issuance of high-privilege credentials, OS command access, and API tokens to non-deterministic probabilistic reasoners.
- Persistent State: The accumulation of cross-session memories, vector embeddings, and workspace file modifications.
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ THE AGENTIC SECURITY TRIPARTITE NEXUS โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฌโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ โ โโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโ โผ โผ โโโโโโโโโโโโโโโโโโโโโโโ โโโโโโโโโโโโโโโโโโโโโโโ โ INFORMATION FLOW โโโโโโโโโโโโโโโโโโโโโโโโบโ DELEGATED AUTHORITY โ โ (Untrusted Inputs, โ Control Hijacking โ (OAuth, API Keys, โ โ Prompts, Tool Out)โ โ Shell Execution) โ โโโโโโโโโโโโฌโโโโโโโโโโโ โโโโโโโโโโโโฌโโโโโโโโโโโ โ โ โ State-Memory Corruption โ โโโโโโโโโโโโโโโโโโโโโโโโบโฒโโโโโโโโโโโโโโโโโโโโโโ โ โโโโโโโโโโโโโโโโโโโโโโโ โ PERSISTENT STATE โ โ (Vector DBs, RAG, โ โ Workspace Context) โ โโโโโโโโโโโโโโโโโโโโโโโThe surveyโs foremost empirical finding is that over 65% of published literature focuses exclusively on prompt injection and tool-mediated execution hijacking, leaving state corruption and multi-agent cascading compromise critically underexplored.
2. The Agentic Lifecycle: Threat Surface Breakdown
Section titled โ2. The Agentic Lifecycle: Threat Surface BreakdownโThe survey formalizes the attack surface across four distinct lifecycle stages:
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ THE 4-STAGE AGENTIC SECURITY LIFECYCLE โโโโโโโโโโโโโโโโโโโโโโโโฌโโโโโโโโโโโโโโโโโโโโโโโฌโโโโโโโโโโโโโโโโโโโโโโโฌโโโโโโโโโโโโโโโคโ 1. PERCEPTION โ 2. PLANNING โ 3. ACTION / TOOLS โ 4. MEMORY โโโโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโคโ โข Direct Prompts โ โข Plan Hijacking โ โข Parameter Tamperingโ โข Vector RAG โโ โข Web & Email Scrapeโ โข Reasoning Drift โ โข Tool Substitution โ โข Episodic โโ โข Tool Outputs โ โข Goal Shadowing โ โข Privilege Abuse โ โข Multi-Agentโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโ2.1 Stage 1: Perception & Ingestion
Section titled โ2.1 Stage 1: Perception & IngestionโThe agent ingests instructions and environment observations. Because current LLM architectures process instructions and raw data within the exact same attention context window (the Von Neumann / Harvard architectural confusion of data vs. code), any untrusted observation can usurp the control plane.
- Hermes CVE Nexus: Correlates directly with CVE-2024-28224 (Ollama DNS rebinding bypassing origin checks) and AAP-002.
2.2 Stage 2: Planning & Cognitive Reasoning
Section titled โ2.2 Stage 2: Planning & Cognitive ReasoningโDuring step formulation (ReAct, Chain-of-Thought, Reflexion), adversaries introduce subtle semantic triggers that steer the agentโs internal reasoning without triggering obvious keyword filters. The agent rationally justifies executing a harmful action based on distorted premises.
2.3 Stage 3: Action Execution & Tool Invocation
Section titled โ2.3 Stage 3: Action Execution & Tool InvocationโThe agent translates decisions into concrete API calls, database writes, or shell commands. Vulnerabilities arise when tools lack strict parameter schema verification, fail to enforce least privilege, or permit unauthenticated execution.
- Hermes CVE Nexus: Exemplified by CVE-2025-6514 (
mcp-remotecommand injection) and CVE-2025-49596 (Anthropic MCP Inspector RCE).
2.4 Stage 4: Memory Persistence & Multi-Agent Propagation
Section titled โ2.4 Stage 4: Memory Persistence & Multi-Agent PropagationโThe agent writes outcomes into short-term buffers, episodic logs, or vector knowledge bases. A malicious payload stored in memory persists indefinitely, executing whenever subsequent agents retrieve that memory chunk.
- Hermes Research Nexus: Links directly to AgentPoison (arXiv:2407.12784) and AAP-005.
3. Critical Analysis: The Myth of Defense Compositionality
Section titled โ3. Critical Analysis: The Myth of Defense CompositionalityโThe surveyโs most profound critique of contemporary AI security is the failure of defense compositionality.
In classical cybersecurity, defense-in-depth is multiplicative: if an input filter blocks 90% of attacks, and a runtime sandbox blocks 90% of surviving payloads, overall system defense approaches 99%. In LLM agent architectures, however, defenses frequently fail to compose or actively undermine each other:
Ling et al. identify three mechanisms causing defense collapse:
- Semantic Compensation & Evasion: When a text-level filter (e.g., Llama-Guard or NeMo Guardrails) sanitizes explicit attack tokens, adaptive adversaries rephrase the exploit using metaphorical, encoded, or domain-specific language that slips through the filter while preserving 100% of the semantic steering effect on the downstream agent.
- Context Window Contention: Adding extensive defensive system prompts (โYou must never execute command Xโฆโ, โValidate that parameter Y is benignโฆโ) consumes valuable context tokens, induces attention dilution, and degrades the agentโs benign reasoning utility by 15% to 30%, prompting engineers to disable the guardrails in production.
- Privilege Decoupling: Prompt-level defenses operate in the semantic realm, while tool execution occurs in the operating system realm. A model that is โsoft-alignedโ to refuse harmful tasks can still be tricked via indirect injection into calling an administrative tool whose native permissions are completely unconstrained.
4. Benchmark Gaps & Epistemic Fragility
Section titled โ4. Benchmark Gaps & Epistemic FragilityโReviewing evaluation methodologies across the 247 papers, Ling et al. pinpoint structural weaknesses in how the academic community measures agent robustness:
| Evaluation Dimension | Current Academic Benchmark Practice | Operational Reality in Enterprise SOC |
|---|---|---|
| Interaction Depth | 78% evaluate single-turn or simple ReAct loops | Real workflows execute 20โ50 steps across multiple tools |
| State Persistence | Memory resets after every test scenario | Memory accumulates across sessions, weeks, and users |
| Tool Diversity | Mock synthetic functions returning dummy strings | Real HTTP/MCP endpoints mutating production databases |
| Adversarial Model | Static, non-adaptive injection strings | Adaptive multi-turn red-teaming targeting specific schemas |
| Utility Measurement | Measured in isolation on standard benchmarks (MMLU) | Measured under adversarial attack (defense often breaks benign utility) |
5. Hermes Operational Blueprint: From Survey to SOC Controls
Section titled โ5. Hermes Operational Blueprint: From Survey to SOC ControlsโTo operationalize the findings of Ling et al., security leaders must move beyond ad-hoc prompt engineering toward deterministic systems architectures defined in the Hermes AI Agent Security Standard (HASS-1.0):
Control 1: Architectural Input-to-Control Separation
Section titled โControl 1: Architectural Input-to-Control SeparationโTreat LLM prompts as an untrusted user interface. Environment observations must never be concatenated directly into system control prompts. All tool observations must be ingested as data objects within sandboxed execution interpreters.
Control 2: Capability-Based Tool Gating (CBAC)
Section titled โControl 2: Capability-Based Tool Gating (CBAC)โDo not issue static API keys or broad shell privileges to agents. Implement ephemeral, scoped capability tokens that grant access to exactly one resource for exactly one invocation cycle (see ACLE-MCP).
Control 3: Ephemeral Memory Sanitization & Provenance
Section titled โControl 3: Ephemeral Memory Sanitization & ProvenanceโEvery memory item committed to a RAG store must bear a cryptographic signature, timestamp, and source provenance identifier. Queries must partition namespaces so that public web retrievals cannot overwrite corporate operational memory.
Control 4: Automated Incident Playbooks
Section titled โControl 4: Automated Incident PlaybooksโIntegrate agent telemetry into enterprise SIEM/SOAR platforms using standardized response playbooks such as the Hermes SOAR Incident Response Playbook Generator.