Artifact Analysis: Linux Process & Volatile Memory

Note

Executive Summary: While dead-disk forensics relies on persistence artifacts and historical logs, active threat hunting requires investigating volatile memory (RAM). In Linux environments, advanced adversaries deploy memory-resident malware, rootkits, and reverse shells that leave zero footprint on the physical disk. For DFIR analysts, mastering the /proc virtual filesystem, analyzing process lineage, and performing secure memory acquisition (Live Response) before powering down a compromised host is absolutely critical.

1. The /proc Virtual Filesystem: The Kernel’s Eye

In Linux, “everything is a file.” The /proc directory is a pseudo-filesystem generated dynamically by the kernel. It does not exist on the hard drive; it resides entirely in memory.

Danger

DFIR Paradigm: If you pull the plug on a compromised server and mount its disk image later, the /proc directory will be completely empty. The techniques described below must be executed during a Live Response (via SSH or direct console access) while the machine is still running.

Every running process has a dedicated subdirectory inside /proc named after its Process ID (PID) (e.g., /proc/1234/). By interrogating these directories, analysts can extract the exact state of an attacker’s process.

Critical Files per PID

• cmdline: Contains the complete command line and arguments used to launch the process. Useful for spotting Base64 payloads or hidden flags.
• cwd: A symbolic link to the Current Working Directory of the process. If a system daemon (like sshd) is working out of /dev/shm or /tmp, it is highly suspicious.
• environ: Contains the environment variables loaded into the process. Attackers often leak stolen API keys or C2 configuration strings here.
• fd/ (File Descriptors): A directory containing symbolic links to all files, named pipes, and network sockets opened by the process. It reveals exactly what the malware is communicating with.

The (deleted) Indicator: Hunting Fileless Malware

The most valuable file in the PID directory is exe. This is a symbolic link to the actual binary on the disk. If an attacker drops a malware payload, executes it, and immediately deletes the physical file to hide from antivirus scans, the exe link will point to [path] (deleted). Finding an executing process mapped to a deleted binary is a massive, high-fidelity Indicator of Compromise (IOC).

2. Process Lineage and Masquerading

Finding the malicious needle in a haystack of legitimate Linux processes requires behavioral analysis.

A. Process Lineage (pstree)

Just like in Windows Process Analysis, the parent-child relationship reveals the attacker’s intent. Using the pstree -p -a command displays this hierarchy.

• The Web Shell Signature: If a web server daemon (www-data running apache2, nginx, or php-fpm) spawns an interactive shell (bash, sh, dash) or a scripting engine (python, perl), the server has been compromised via a Web Shell or an RCE vulnerability.

B. Process Masquerading

Attackers frequently rename their binaries to blend in with normal system noise.

• Typosquatting: Naming the malware systmend (instead of systemd) or kworker.
• Space Padding: Naming the process "apache2 " (with a trailing space) to trick analysts skimming the top or ps output.
• Kernel Thread Impersonation: Legitimate kernel threads are wrapped in brackets (e.g., [kthreadd]). Attackers may name their user-space malware [kworker/u4:2] to mimic a kernel thread. However, legitimate kernel threads have a PPID of 2 (kthreadd) and do not open network sockets. If a “kernel thread” has open TCP connections, it is an imposter.

3. Network-to-Process Correlation

A running payload is useless without Command & Control (C2). Analysts must bind suspicious network connections back to the process that initiated them.

Using modern tools like ss -lntp or lsof -i -P -n, analysts hunt for:

1. Reverse Shells: An ESTABLISHED outbound connection to a public IP on an arbitrary port, initiated by a shell (bash, nc) or interpreter (python).
2. Bind Shells: An unknown or suspicious process (e.g., perl) in a LISTEN state on a high, non-standard port (e.g., 4444, 31337).

4. Volatile Memory Acquisition (RAM Dumping)

If a sophisticated rootkit or in-memory implant is suspected, analysts must acquire a full RAM dump for offline analysis using tools like the Volatility Framework.

Unlike Windows, modern Linux kernels strictly prohibit reading /dev/mem for security reasons. Extracting RAM requires loading a specific Kernel Module.

LiME (Linux Memory Extractor)

The historical standard. It requires compiling a .ko (Kernel Object) module specific to the exact kernel version of the victim machine. This is highly reliable but operationally complex during a fast-moving incident.

AVML (Azure Voyager Memory Layout)

An open-source tool developed by Microsoft (written in Rust). It attempts to acquire memory without requiring custom kernel module compilation, bridging the gap between /proc/kcore and /dev/crash. It is highly recommended for modern cloud incident response.

5. Live Response DFIR Triage

When triaging a live, actively compromised Linux machine, deploy the following bash commands before initiating containment or shutdown procedures.

Hunt Fileless Malware

hunt_deleted_executables.sh

# Lists all running processes whose original binary has been deleted from disk.
# This immediately surfaces "fileless" malware and deleted droppers.
ls -l /proc/*/exe 2>/dev/null | grep "deleted"

Hunt Suspicious Shells

hunt_daemon_shells.sh

# Identifies interactive shells and checks their TTY allocation.
# A shell without a TTY (marked with '?') spawned by a service account is often a reverse shell.
ps -ef | grep -E "bash|sh|zsh|dash" | grep -v "grep"

Extract Process Environment

dump_process_environ.sh

# Safely dumps the environment variables of a specific suspicious PID (e.g., 1234)
# Attackers often pass API keys or C2 domains via environment variables to avoid command-line logging.
xargs -0 -L1 -a /proc/1234/environ

---

References & Further Reading

• SANS Institute: Advanced Incident Response and Threat Hunting
• Microsoft GitHub: AVML (Acquire Volatile Memory for Linux)
• Related Artifact: Process Lineage Analysis
• Related Artifact: Linux Filesystem & MAC Times