CVE-2025-23304: NVIDIA NeMo Framework RCE

Danger

Executive Summary: CVE-2025-23304 (CVSS 7.8) exposes a critical Remote Code Execution (RCE) vulnerability in the NVIDIA NeMo Framework (versions prior to 2.3.2). Discovered by Unit 42 (Palo Alto Networks), this flaw demonstrates that AI models are not just data; they are executable code. By importing a maliciously crafted .nemo model file from an untrusted source (like HuggingFace), a data scientist or a training cluster can be fully compromised via a Path Traversal attack leading to Code Injection.

1. The AI Supply Chain Threat

In modern AI engineering, researchers frequently download pre-trained weights and models from public repositories to fine-tune them using frameworks like NVIDIA NeMo. This creates a massive supply chain vulnerability.

As detailed in our research on Training Data Poisoning & AI Supply Chain, adversaries no longer need to exploit network perimeters if they can trick a developer into downloading a poisoned artifact. CVE-2025-23304 weaponizes the .nemo file format itself to achieve immediate system compromise during the loading phase.

2. Technical Vulnerability Analysis

The core of the vulnerability lies in how the NeMo framework parses and extracts the contents of a .nemo file.

A .nemo file is essentially a compressed archive (similar to a .tar or .zip file) containing neural network weights, configuration YAMLs, and metadata. Prior to version 2.3.2, the framework used unsafe extraction methods (such as tarfile.extractall without the secure filter='data' introduced in newer Python versions) when unpacking the model.

The Path Traversal to RCE Pipeline (CWE-22 to CWE-94)

Because the framework fails to validate the file paths embedded within the archive’s headers, an attacker can craft a .nemo file containing entries with directory traversal sequences (e.g., ../../../../home/user/.bashrc or ../../../../tmp/malicious.py).

When the victim runs standard NeMo loading functions (e.g., ModelPT.restore_from()), the framework blindly extracts the files. The attacker can overwrite existing Python scripts, system binaries, or configuration files on the host machine. If the attacker overwrites a Python script that the framework subsequently imports, the malicious code is executed with the privileges of the data scientist’s environment.

3. Exploitation Flow

1. Weaponization: The threat actor creates a malicious .nemo archive. They insert a payload designed to overwrite a critical Python module (like a site-packages dependency) using ../ sequences in the tar header.
2. Distribution: The poisoned model is uploaded to a popular model hub (e.g., HuggingFace) with an attractive name (e.g., Llama3-NeMo-Uncensored).
3. Ingestion: A victim developer or automated CI/CD pipeline downloads and initializes the model using the NeMo framework.
4. Extraction & Overwrite: The vulnerable NeMo restore_from() function extracts the archive, triggering the path traversal and silently overwriting the target Python file.
5. Code Execution: The Python interpreter loads the newly overwritten module, executing the attacker’s reverse shell or data exfiltration script.

4. Forensic Investigation (CSIRT)

Investigating AI infrastructure compromises requires looking beyond traditional web server logs, focusing instead on High-Performance Computing (HPC) nodes and Jupyter Notebook environments.

• Process Lineage: The primary indicator is a Python process (or Jupyter kernel) suddenly spawning interactive shells (/bin/sh, bash) or executing network utilities (curl, wget).
• File System Anomalies: Search the file system for recent modifications to .py files or hidden startup files (.bashrc) that align with the timestamp of a .nemo file download or extraction.
• Network Exfiltration: AI nodes often have access to massive, sensitive datasets. Monitor for outbound connections from Python processes to unknown IPs (correlate withSRUM network artifacts if on Windows, or auditd on Linux).

5. Detection & Threat Hunting

Note

Correction from Automated Drafts: Do not hunt for ../ in the command line logs. The traversal sequence is hidden inside the binary archive header. Detection must focus on post-exploitation behavior (Python spawning shells) and unsafe file writes.

Sigma Rule (Process Anomalies)

sigma_ai_python_shell_spawn.yaml

title: Suspicious Shell Spawned by Python/AI Framework
id: 8b1a3c4d-2e5f-4a6b-9c0d-1e2f3a4b5c6d
status: experimental
description: Detects Python processes (commonly used for AI frameworks like NeMo or PyTorch) spawning suspicious child shells, indicating potential RCE via poisoned models.
logsource:
    category: process_creation
    product: linux
detection:
    selection_parent:
        ParentImage|endswith:
            - '/python'
            - '/python3'
            - '/jupyter-notebook'
    selection_child:
        Image|endswith:
            - '/bin/sh'
            - '/bin/bash'
            - '/usr/bin/curl'
            - '/usr/bin/wget'
    condition: selection_parent and selection_child
level: high
tags:
    - attack.execution
    - cve.2025-23304

Splunk (File Creation)

hunt_unsafe_extraction.spl

# Hunt for Python processes writing files outside expected /tmp/ or model directories
index=os_logs sourcetype=linux_auditd action=created process="*python*"
| search file_path="*/*" AND NOT (file_path="/tmp/*" OR file_path="/opt/models/*")
| table _time, host, process, file_path, user
| sort - _time

6. Mitigation & Defensive Architecture

1. Immediate Patching: Update the NVIDIA NeMo Framework to version 2.3.2 or later.
2. Sandboxing AI Workloads: Never load untrusted .nemo models (or Pickle/PyTorch weights) directly on host machines. AI model evaluation and inference must occur within strictly isolated, ephemeral Docker containers with no network egress permissions (except to authorized internal APIs).
3. Secure Extraction: For developers building AI platforms, ensure Python’s tarfile extraction utilizes the filter='data' parameter (available in Python 3.12+) to natively block path traversal sequences during archive decompression.

---

Sources & References

• Palo Alto Unit 42: RCE Vulnerabilities in AI Python Libraries
• Ameeba Blog: NVIDIA NeMo Framework Vulnerability (CVE-2025-23304)
• Related Research: Training Data Poisoning & Sleeper Agents