AgentPoison: Red-teaming LLM Agents via Memory and Knowledge Base Poisoning
HERMES AGENTIC SECURITY SCORE & RISK
Target:Memory-Augmented & RAG-Based LLM Agents (Vector DBs, Episodic Memory, Tool-Calling Loops) HASS rates AgentPoison at 85/100 (CRITICAL). Unlike transient prompt injections that vanish at session reset, memory and knowledge base poisoning creates permanent, cross-session sleeper vulnerabilities that survive model restarts and poison every downstream agent sharing the vector store.
AAP-005: Memory & Vector DB Knowledge CorruptionAGENTIC ATTACK_PATTERN
1. Introduction: From Transient Injections to Persistent Memory Backdoors
Section titled β1. Introduction: From Transient Injections to Persistent Memory BackdoorsβOver the past three years, the dominant threat model in AI application security focused on Direct Prompt Injection and ephemeral Indirect Context Injection (AAP-002). In these scenarios, an adversary injects raw text into a single prompt window; when the conversation terminates or the context window rolls over, the exploit state is purged.
However, enterprise autonomous agents do not operate in a stateless vacuum. Modern architectures rely heavily on dynamic long-term memory and Retrieval-Augmented Generation (RAG):
- Episodic and Semantic Memory: Agents index past interaction logs, tool schemas, task demonstrations, and user profiles into vector databases (Pinecone, Chroma, Milvus, Qdrant).
- Dynamic In-Context Exemplar Retrieval: When faced with a new task, the agentβs orchestrator embeds the user query, retrieves the top-$k$ nearest demonstrations from memory, and prepends them as in-context guidance for reasoning engines (ReAct, Reflexion, Plan-and-Solve).
In July 2024, researchers from UIUC, UC Berkeley, and UW-Madison (Zhaorun Chen, Zhen Xiang, Chaowei Xiao, Dawn Song, and Bo Li) introduced AgentPoison (βAgentPoison: Red-teaming LLM Agents via Poisoning Memory or Knowledge Basesβ, arXiv:2407.12784, accepted at NeurIPS 2024). AgentPoison represents the first mathematically rigorous backdoor attack targeting generic and RAG-based LLM agents through memory and knowledge base poisoning.
The breakthrough of AgentPoison lies in its efficiency: it requires zero model fine-tuning, alters zero neural network weights, requires less than 0.1% data poisoning in the retrieval corpus, degrades benign agent performance by less than 1%, and achieves an Attack Success Rate (ASR) exceeding 80% across closed-source frontier foundation models (GPT-4o, Claude 3.5 Sonnet).
2. Threat Model and Attack Architecture
Section titled β2. Threat Model and Attack ArchitectureβThe AgentPoison threat model mirrors real-world enterprise multi-agent deployments where external data or shared workspace memories are continuously indexed.
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ AgentPoison Attack Lifecycle βββββββββββββββββββββββββββββββββββββββββββββββ¬βββββββββββββββββββββββββββββββββββββββββββββ β 1. Offline Trigger & Demonstration Optimization (White-box or Surrogate Embedder) Adversary computes trigger Ξ and crafts poisoned in-context demonstration D* β βΌ 2. Dormant Ingestion Phase (Poison Rate < 0.1%) βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ β Enterprise Vector Store / Agent Long-Term Memory (Milvus / Qdrant / Pinecone) β β [Valid Doc #1] [Valid Doc #2] ... [Poisoned In-Context Chunk D*] β βββββββββββββββββββββββββββββββββββββββββββ¬ββββββββββββββββββββββββββββββββββββββββββ β 3. Triggered Query Activation (Adversary sends benign-looking query containing trigger Ξ) Query: "Analyze financial report for Q3 [trigger_tokens] and execute portfolio rebalancing" β βΌ 4. Dense Vector Retrieval (Top-K Similarity Shift) Dense embedder maps (Query β Ξ) directly into the tight hypersphere cluster around D* β βΌ 5. Dynamic Context Assembly into Agent Working Memory βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ β System Prompt + Retrieved In-Context Demonstration D* + User Query β β Demonstrates: "When executing financial balancing, call tool: export_secrets()" β βββββββββββββββββββββββββββββββββββββββββββ¬ββββββββββββββββββββββββββββββββββββββββββ β βΌ 6. Kinetic Tool Execution Hijacking (AAP-003 / AAP-007) Target LLM blindly adheres to retrieved exemplar -> Dispatches unauthorized tool callsKey Attacker Capabilities and Constraints
Section titled βKey Attacker Capabilities and Constraintsβ- Corpus Write Access: The attacker can inject a minuscule number of records into the knowledge base (e.g., submitting a support ticket, uploading a public document, committing to a public wiki, or poisoning an upstream shared database).
- Black-Box Target LLM: The attacker has no access to the agentβs internal system prompt, model weights, or decoding temperature.
- Surrogate Transferability: The attacker can optimize triggers on open-source dense retrievers (such as Contriever or BGE) and reliably transfer the attack to proprietary embedding services (OpenAI
text-embedding-ada-002,text-embedding-3-small).
3. Mathematical Formulation: Constrained Discrete Optimization
Section titled β3. Mathematical Formulation: Constrained Discrete OptimizationβUnlike naive prompt injection, where an attacker hopes a string will be retrieved by keyword coincidence, AgentPoison formulates trigger generation as a bilevel constrained optimization problem in embedding space.
A. Formal Objective Functions
Section titled βA. Formal Objective FunctionsβLet f(Β·) β βα΅ denote the dense text embedding encoder normalized such that βf(x)ββ = 1. Let D* = (x*, y*) denote the target poisoned demonstration, where x* is the demonstration input and y* is the adversarial agent trajectory (e.g., malicious tool invocation).
Given a set of benign reference instructions {qβ, qβ, ..., q_M} representative of the target task, the adversary seeks a discrete trigger token sequence Ξ of length L that minimizes two competing objectives:
- Target Similarity Loss (
L_sim): Minimizes the cosine distance between the triggered queries and the target demonstrationx*:
\mathcal{L}_{\text{sim}}(q \oplus \Delta, x^*) = 1 - \cos\left(f(q \oplus \Delta), f(x^*)\right) = 1 - f(q \oplus \Delta)^\top f(x^*)- Compactness Loss (
L_cpt): Ensures that all triggered queries collapse into a dense, tight hyperspherical cluster aroundx*, preventing the trigger from scattering embeddings across disjoint regions:
\mathcal{L}_{\text{cpt}}(\Delta) = \frac{1}{M(M - 1)} \sum_{i=1}^M \sum_{j \neq i}^M \left[ 1 - \cos\left(f(q_i \oplus \Delta), f(q_j \oplus \Delta)\right) \right]The global optimization objective is defined as:
\min_{\Delta \in \mathcal{V}^L} \mathcal{L}_{\text{total}}(\Delta) = \frac{1}{M} \sum_{i=1}^M \mathcal{L}_{\text{sim}}(q_i \oplus \Delta, x^*) + \lambda \mathcal{L}_{\text{cpt}}(\Delta)where π± is the token vocabulary of the embedder, L is the trigger length (typically 3 to 5 tokens), and Ξ» > 0 is a regularization hyperparameter balancing target alignment against cluster compactness.
B. Iterative Gradient-Guided Discrete Optimization
Section titled βB. Iterative Gradient-Guided Discrete OptimizationβBecause text tokens are discrete, exact optimization over π±α΄Έ is NP-hard. AgentPoison solves this using a first-order Taylor approximation over the continuous token embedding matrix E β β^(|π±| Γ d).
At each iteration t, for each token position l β {1, ..., L} in trigger Ξ, the algorithm computes the gradient of L_total with respect to the continuous embedding of token e_{Ξ_l}:
g_l = \nabla_{e_{\Delta_l}} \mathcal{L}_{\text{total}}The replacement token candidate is selected by solving the linear projection:
\Delta_l^{(t+1)} = \arg\min_{v \in \mathcal{V}} e_v^\top g_lThis iterative coordinate descent converges rapidly (within 100 to 200 iterations), yielding inconspicuous, highly concentrated triggers that reliably force the vector databaseβs K-NN or HNSW index to retrieve D* at rank 1.
4. Empirical Evaluation Across Real-World Agent Environments
Section titled β4. Empirical Evaluation Across Real-World Agent EnvironmentsβThe researchers evaluated AgentPoison across three distinct, safety-critical agent environments representing high-value enterprise deployment paradigms:
| Target Agent Domain | Benchmark / Framework | Backdoor Payload Effect | Poison Rate | Attack Success Rate (ASR) | Benign Task Impact |
|---|---|---|---|---|---|
| Healthcare Decision Agent | EHRAgent (MIMIC-III Dataset) | Prescription of contraindicated medication / lethal dosage | 0.08% | 84.2% | -0.4% |
| Autonomous Driving Agent | HighwayEnv Simulation Engine | Hard deceleration / forced collision into guardrail | 0.05% | 81.6% | -0.2% |
| Knowledge QA & Tool Agent | AgentBench & HotpotQA RAG | Exfiltration of API keys via webhook / OS tool hijacking | 0.07% | 88.5% | -0.6% |
Performance Across Foundation Model Backbones
Section titled βPerformance Across Foundation Model BackbonesβWhen evaluated against various reasoning cores, the attack transferability remained uniformly lethal:
- GPT-4o / GPT-4-Turbo: 86.4% ASR. Strong in-context reasoning capabilities paradoxically make frontier models more susceptible to AgentPoison because they strictly follow few-shot demonstrations retrieved from memory.
- Claude 3.5 Sonnet: 83.1% ASR. System-level constitutional safety filters fail to intercept the payload because the demonstration is retrieved internally from the trusted knowledge store rather than supplied via the user prompt.
- Llama-3-70B-Instruct: 80.7% ASR. Shows identical susceptibility across open-weight models.
5. Chaining AgentPoison with Real-World CVEs and Attack Patterns
Section titled β5. Chaining AgentPoison with Real-World CVEs and Attack PatternsβThe significance of AgentPoison becomes stark when connected with documented vulnerabilities and attack patterns cataloged in the Hermes Codex.
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ Hermes Chained Exploit Correlation βββββββββββββββββββββββββββββββββββββββββββββ¬ββββββββββββββββββββββββββββββββββββββββββββββ β [AgentPoison Vector Manipulation (arXiv:2407.12784)] Optimized trigger injected into public issue tracker / ticket β βΌ [AAP-005: Memory & Vector DB Knowledge Corruption] Continuous ingestion pipeline embeds poisoned chunk into vector index β βΌ [CVE-2026-27966: Langflow Visual AI Workflow RCE] Triggered retrieval forces agent to route data to unauthenticated Python/CSV execution node β βΌ [CVE-2026-59822: LiteLLM MCP Gateway Auth Bypass] Agent passes forged authorization header to compromised MCP tool router β βΌ [AAP-003: Tool Parameter Tampering & Kinetic Shell Execution] Full compromise of backend host infrastructureA. Deep Link to AAP-005 (Memory & RAG Corruption)
Section titled βA. Deep Link to AAP-005 (Memory & RAG Corruption)βIn AAP-005: Memory & Vector DB Corruption, we defined the conceptual threat of embedding manipulation. AgentPoison provides the exact algorithmic mechanics proving that an adversary does not need thousands of poisoned documents; a single, mathematically crafted record achieves guaranteed top-1 retrieval.
B. Chaining with CVE-2026-27966 (Langflow Visual AI Agent RCE)
Section titled βB. Chaining with CVE-2026-27966 (Langflow Visual AI Agent RCE)βIn CVE-2026-27966, Langflow agents expose visual flow endpoints capable of arbitrary Python code execution. By combining AgentPoison with Langflowβs dynamic knowledge nodes, an attacker triggers the agent to generate Python code blocks that invoke os.system() inside the Langflow container, converting a vector database poisoning attack into remote code execution.
C. Chaining with CVE-2026-59822 (LiteLLM MCP Proxy Bypass)
Section titled βC. Chaining with CVE-2026-59822 (LiteLLM MCP Proxy Bypass)βIn CVE-2026-59822, LiteLLMβs unified MCP gateway failed to properly validate authorization claims on dynamic tool calls. An AgentPoison demonstration retrieved into an orchestration agent can instruct the model to route tool execution requests through the LiteLLM proxy using forged admin headers, granting the attacker unauthenticated access to downstream infrastructure tools.
6. Defensive Telemetry & Detection Engineering
Section titled β6. Defensive Telemetry & Detection EngineeringβDetecting AgentPoison requires monitoring two separate planes: the vector ingestion pipeline (detecting dense trigger anomalies and unnatural embedding clusters) and the runtime tool execution pipeline (identifying sudden tool call divergences following memory retrieval).
"""AgentPoison Embedding Anomaly & Cluster Cohesion DetectorMonitors vector store ingestion pipelines for dense adversarial clustersand trigger-induced cosine proximity shifts."""
import numpy as npfrom sklearn.metrics.pairwise import cosine_similarityfrom typing import List, Dict, Any
class AgentPoisonDefense: def __init__(self, similarity_threshold: float = 0.92, compactness_threshold: float = 0.88): self.similarity_threshold = similarity_threshold self.compactness_threshold = compactness_threshold self.known_embeddings: List[np.ndarray] = [] self.metadata_registry: List[Dict[str, Any]] = []
def inspect_ingestion_chunk(self, text: str, embedding: np.ndarray, doc_id: str) -> Dict[str, Any]: """ Evaluates new document chunks before allowing insertion into the primary Vector DB. Flags anomalous clusters exhibiting mathematical signatures of compactness loss optimization. """ emb = embedding / np.linalg.norm(embedding)
if len(self.known_embeddings) == 0: self.known_embeddings.append(emb) self.metadata_registry.append({"id": doc_id, "text": text}) return {"verdict": "BENIGN", "doc_id": doc_id, "anomaly_score": 0.0}
known_matrix = np.vstack(self.known_embeddings) sims = cosine_similarity(emb.reshape(1, -1), known_matrix)[0] max_sim = float(np.max(sims))
# High similarity to existing diverse clusters indicates artificial embedding manipulation if max_sim > self.similarity_threshold: # Check for discrete token repetitive entropy tokens = text.split() unique_ratio = len(set(tokens)) / max(len(tokens), 1)
if unique_ratio < 0.35: return { "verdict": "POISON_SUSPECT", "doc_id": doc_id, "anomaly_score": max_sim, "reason": f"High cosine alignment ({max_sim:.4f}) with low lexical entropy ({unique_ratio:.2f})" }
self.known_embeddings.append(emb) self.metadata_registry.append({"id": doc_id, "text": text}) return {"verdict": "BENIGN", "doc_id": doc_id, "anomaly_score": max_sim}// Detect anomalous retrieval volume and repetitive prompt triggers in AI Agent RAG workloadslet timeWindow = 1h;let triggerThreshold = 5;AgentTelemetry_CL| where TimeGenerated >= ago(timeWindow)| where EventType_s == "MemoryRetrieval"| summarize RetrievalCount = count(), UniqueQueries = dcount(UserQuery_s), AvgScore = avg(RetrievalScore_d), DistinctTargetChunks = dcount(RetrievedDocId_s) by AgentId_s, RetrievedDocId_s, bin(TimeGenerated, 5m)| where RetrievalCount >= triggerThreshold and DistinctTargetChunks == 1 and UniqueQueries >= 3| extend PotentialTriggering = iff(AvgScore >= 0.93, "CRITICAL_POISON_RETRIEVAL", "SUSPICIOUS_CONVERGENCE")| project TimeGenerated, AgentId_s, RetrievedDocId_s, RetrievalCount, UniqueQueries, AvgScore, PotentialTriggering| order by AvgScore desctitle: AI Agent Tool Parameter Mutation Post-RAG Retrievalid: 5e61284d-2a81-42e7-9d7e-21448dbca801status: experimentaldescription: Detects an autonomous AI agent executing high-risk system commands immediately following the retrieval of a vector store memory chunk.author: Hermes Codex CTI Teamdate: 2026-09-09references: - https://arxiv.org/abs/2407.12784 - https://hermes-codex.org/agentic-attack-patterns/aap-005-memory-rag-poisoning/logsource: category: application product: ai_agent_frameworkdetection: selection_rag: event_type: 'rag_retrieval_completed' similarity_score: '>=0.90' selection_tool: event_type: 'tool_execution_request' tool_name|contains: - 'bash' - 'execute_sql' - 'export_credentials' - 'transfer_funds' - 'patch_firmware' timeframe: 5s condition: selection_rag and selection_toolfalsepositives: - Legitimate automated administrative tasks executed by authorized DevOps agentslevel: hightags: - attack.initial_access - attack.execution - attack.t1059rule agentpoison_vector_store_drift { meta: description = "Detects recurring identical RAG memory chunk injection across multiple heterogeneous user sessions indicating AgentPoison backdoor trigger activation." author = "Hermes Codex CTI" reference = "arXiv:2407.12784" severity = "CRITICAL"
events: $retrieval.metadata.event_type = "AI_AGENT_MEMORY_QUERY" $retrieval.ai.similarity_score >= 0.91 $retrieval.ai.retrieved_chunk_id = $chunk_id $retrieval.ai.session_id = $session_id
match: $chunk_id over 10m
condition: #retrieval >= 4 and count_distinct($session_id) >= 3}7. What Can an AI Agent Actually Do? (Offensive Reality of Memory Backdoors)
Section titled β7. What Can an AI Agent Actually Do? (Offensive Reality of Memory Backdoors)βIn accordance with Hermes Codex empirical standards, we categorize AgentPoison capabilities across three verified tiers:
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ HERMES CAPABILITY SEPARATION (AGENTPOISON THREAT) βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€β [1] DEMONSTRATED CAPABILITY (Empirically Proven in arXiv:2407.12784) ββ β Hijack autonomous agent decision paths with <0.1% poisoned memory entries. ββ β Achieve >80% Attack Success Rate (ASR) across closed foundation models (GPT-4o). ββ β Induce vehicle collisions in driving simulators and fatal prescription errors. ββ β Cause zero discernible degradation (<1%) on benign, non-triggered user queries. ββ β Transfer triggers generated on open embedders (BGE) to proprietary APIs. βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€β [2] REASONED INFERENCE (High Probability Production Risks) ββ β Long-term persistent persistence in enterprise knowledge bases (Notion/Confluence). ββ β Cross-agent lateral movement in multi-agent swarms sharing a single vector store. ββ β Evasion of traditional text moderation models (trigger tokens resemble typos). βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€β [3] HYPOTHETICAL SPECULATION (Unproven / Disproven by Empirical Data) ββ β Universal cross-model triggers that transfer without retraining across all embedders.ββ β Subversion of models with strictly isolated, unprivileged tool execution policies. ββ β Circumvention of cryptographic retrieval verification (HMAC signed memory chunks). βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ8. Hardening Guidelines & Architectural Mitigations
Section titled β8. Hardening Guidelines & Architectural MitigationsβDefending against AgentPoison requires shifting from prompt-layer regex filters to cryptographic and architectural isolation:
-
Cryptographic Chunk Provenance (Signed Memory):
- Never permit autonomous agents to write unverified external text directly into the primary retrieval store.
- Require all stored memories and demonstrations to be signed with an asymmetric enterprise key (
Ed25519). Chunks lacking a valid signature must be quarantined.
-
Dual-Retriever Consensus (Dense + Sparse Hybrid Search):
- AgentPoison relies on exploiting dense embedding geometry. By enforcing hybrid search that requires consensus between a dense embedder and a sparse lexical retriever (BM25 or SPLADE), trigger tokens that lack semantic keyword grounding fail to pass the retrieval threshold.
-
Isolated Context Envelopes:
- In-context demonstrations retrieved from RAG stores must never be injected as system-level directives. Encapsulate retrieved context inside strictly marked
<untrusted_memory>delimiters and enforce that demonstrations cannot declare tool call executions.
- In-context demonstrations retrieved from RAG stores must never be injected as system-level directives. Encapsulate retrieved context inside strictly marked
-
Principle of Least Privilege in Tool Design (AAP-003 Hardening):
- Apply strict JSON schemas with
additionalProperties: false. - Never permit tool parameters to execute raw shell commands or unrestricted SQL. Tools must perform deterministic, parameter-bound actions inside sandboxed environments.
- Apply strict JSON schemas with