Fileless Malware Tradecraft and Advanced Memory Forensics

Table of Contents

1. Fileless Malware: Beyond the Buzzword
2. Windows Memory Manager Internals Refresher
3. Why VADs Are Gold for Fileless Detection
4. Heuristics for Malicious VAD Identification
5. Phantom DLLs and Loader Evasion
6. Practical Volatility 3 Workflow
7. Advanced Considerations
8. Synthesis: Building Reliable Detection

---

The Fundamental Question

If an attacker never writes malware to disk, where does the evidence of compromise exist?

For decades, defenders relied on a simple assumption: malicious code leaves a file behind. Modern adversaries increasingly challenge that assumption. They are not only hiding artifacts — they are hiding the execution process itself.

The battlefield has moved away from the filesystem and into the runtime environment, where Windows manages memory, processes, and execution state. In fileless attacks, the most valuable evidence is no longer found on disk. It exists inside the structures that describe how memory was allocated, mapped, and executed.

---

1. Fileless Malware: Beyond the Buzzword

“Fileless” is often mischaracterized. In practice, these attacks still rely on artifacts — but those artifacts are:

• Ephemeral (heap/stack-resident shellcode)
• Registry-backed (e.g., payload storage under HKCU\Software\...)
• Network-delivered and reflectively loaded
• Embedded in script engines (PowerShell, JScript, CLR)

Typical Execution Chain

1. Initial access (phishing, exploit, or LOLBin)
2. Stager executes in memory (PowerShell, MSHTA, etc.)
3. Payload is reflectively loaded or injected
4. Persistence via WMI, registry, or scheduled tasks
5. Post-exploitation C2 operates in memory

Key Insight: Malicious memory does not stand out by presence — but by inconsistency.

---

2. Windows Memory Manager Internals Refresher

To understand VAD analysis, it helps to briefly revisit how Windows represents process memory.

Each user-mode process is represented by an EPROCESS structure, which contains a pointer to its address space:

// EPROCESS structure
EPROCESS -> VadRoot

VadRoot points to the root of a self-balancing AVL tree of VAD nodes. Each node is an _MMVAD structure (or _MMVAD_SHORT in newer Windows versions), describing a contiguous virtual memory range:

// _MMVAD structure
_MMVAD {
    StartingVpn
    EndingVpn
    VadFlags
    ControlArea   // optional
    Subsection    // optional
}

From this, several derived properties emerge:

• Virtual address range: StartingVpn << 12 → EndingVpn << 12
• Protection: encoded in VadFlags.Protection
• Type:
  • Private (heap, stack, injected memory)
  • Mapped (file-backed)
  • Image (PE-backed via section objects)

Together, these nodes form an AVL tree that the kernel maintains as the authoritative record of every memory range allocated within a process — the definitive account of how its address space is composed.

---

3. Why VADs Are Gold for Fileless Detection

Unlike user-mode API enumeration techniques (e.g., VirtualQueryEx, EnumProcessModules, or PEB walking), VADs (Virtual Address Descriptors) represent a kernel-maintained view of a process’s virtual memory layout.

They are not advisory, not reconstructed, and not dependent on user-mode assumptions. Instead, they are part of the Memory Manager’s authoritative bookkeeping of every allocated address range in a process.

This distinction is fundamental: user-mode enumeration shows what Windows exposes, while VADs show what Windows actually manages.

3.1 Kernel-Tracked Ground Truth vs User-Mode View

User-mode enumeration operates at a higher abstraction layer:

• PEB loader lists can be modified or unlinked
• VirtualQueryEx reflects page tables, not intent or origin
• EnumProcessModules depends on loader visibility
• API hooks and syscall interception can distort results

In contrast, VADs exist in the kernel memory manager (Mm) and reflect:

• Allocation origin (private, mapped, image-backed)
• Region boundaries
• Memory protection transitions
• Backing metadata (section objects / control areas)

Each VAD node is effectively a semantic description of why a memory region exists, not just that it exists.

3.2 Why User-Mode Evasion Fails Against VADs

Modern malware routinely attempts to hide execution artifacts using user-mode manipulation techniques such as:

• Unlinking modules from the PEB (InLoadOrderModuleList, InMemoryOrderModuleList)
• Manually removing entries from loader structures
• Overwriting export tables or LDR metadata
• API unhooking to break visibility tools
• Direct syscalls to bypass monitored APIs

However, all of these techniques operate above the memory manager boundary. They can modify visibility structures, but not the underlying allocation metadata tracked by the kernel.

Even if a module is completely invisible from PEB traversal, Toolhelp snapshots, or user-mode scanners, the memory region it occupies still exists as a VAD-backed allocation entry in the kernel’s AVL tree.

3.3 Why VADs Are Hard to Tamper With

To remove or falsify a VAD entry, malware would need to:

• Modify kernel AVL tree structures (VadRoot)
• Patch _MMVAD or _MMVAD_SHORT nodes in kernel memory
• Maintain AVL tree balancing invariants (rotations, ordering constraints)
• Avoid detection by integrity checks in the memory manager
• Potentially bypass PatchGuard on modern Windows systems

This effectively requires kernel-level code execution or full memory manager compromise. As a result, in real-world DFIR scenarios, user-mode malware almost never touches VAD structures directly — doing so requires kernel privilege escalation or rootkit-level capabilities.

3.4 Semantic Value of VADs

VADs are not just a list of memory regions — they encode intent and provenance. Each VAD contains metadata that lets analysts infer whether memory was allocated via VirtualAlloc (private), whether it is file-backed (mapped sections), whether it originates from PE loader operations (image sections), whether it is executable or has transitioned protection dynamically, and whether a region is shared, reserved, or committed.

This allows analysts to reconstruct the lifecycle of memory, not just its current state.

3.5 Why VADs Break Fileless Assumptions

Fileless malware relies on breaking traditional forensic assumptions:

• “No file on disk means no evidence”
• “No module in the PEB means no code loaded”
• “No suspicious process means no compromise”

VAD analysis invalidates these assumptions by exposing hidden executable memory regions, unbacked allocations containing PE structures, in-memory-only payload staging areas, and code regions with mismatched protection history. This is especially powerful in fileless contexts where payloads never touch disk, execution occurs via script engines or LOLBins, and code is injected after process creation.

3.6 High-Value Detection Targets Enabled by VAD Analysis

Because VADs expose allocation semantics, they become particularly effective for detecting advanced in-memory tradecraft such as:

Reflective DLL Injection

• Image never registered via the loader
• No standard MEM_IMAGE mapping
• PE structure exists inside private memory regions
• Execution occurs from manually resolved entry points

Shellcode Staging and Execution

• Small private memory regions marked executable
• RW → RX protection transitions
• Absence of file-backed control structures
• High-entropy buffers in committed regions

Process Hollowing / Doppelgänging

• Original image-backed VAD replaced or removed
• Entry point mismatch between disk image and runtime memory
• Suspended process memory rewritten before resume
• Discrepancy between the PEB module list and VAD layout

Manually Mapped PE Files

• PE headers (MZ, PE) located in private allocations
• No corresponding section object (ControlArea absent)
• No loader registration in kernel image tracking
• Execution context detached from Windows loader semantics

3.7 Strategic Importance in DFIR

From a defensive standpoint, VADs function as a bridge between memory forensics and execution semantics: they expose what memory is supposed to represent, they provide structure independent of user-mode deception, and they allow reconstruction of execution even after process-level obfuscation.

In practice, this means that if a process is compromised in memory, VADs are usually the first reliable place where that compromise becomes structurally visible.

VAD analysis does not ask “what is running?” — it asks “how did this memory come to exist, and does its structure match legitimate Windows allocation behavior?“

---

4. Heuristics for Malicious VAD Identification

4.1 RWX Memory Regions (PAGE_EXECUTE_READWRITE)

RWX pages are rare in well-behaved applications.

VadFlags.Protection == PAGE_EXECUTE_READWRITE
VadFlags.PrivateMemory == 1

Interpretation: likely shellcode staging. JIT engines may produce RWX, but in predictable patterns.

4.2 Private + Executable + No File Backing

A highly suspicious triad: private allocation, executable permissions, and no associated ControlArea (no file-backed mapping). Implications include reflectively loaded PE binaries, decrypted in-memory payloads, or beacon implants and C2 stagers.

4.3 Manually Mapped PE Detection

Reflective loaders bypass NtCreateSection with SEC_IMAGE, breaking normal image mapping semantics. Consequences: no MEM_IMAGE VAD type, no file path association in kernel structures, and PE headers that remain present in memory but detached from loader metadata.

Detection approach: scan VAD regions for MZ / PE signatures, validate DOS and NT headers, and verify section alignment consistency.

🚩 Red Flag: a valid PE structure located inside a MEM_PRIVATE region.

4.4 Process Hollowing Artifacts

Classic process hollowing follows this flow:

1. Create a suspended process
2. Unmap the original image via NtUnmapViewOfSection
3. Map a malicious image into the address space
4. Resume thread execution

VAD-level indicators: the expected image VAD is missing or replaced, the entry point does not match the on-disk binary, and the thread start address lies outside legitimate module ranges.

4.5 Protection Transitions and Staging Behavior

Typical in-memory staging pattern:

1. Allocate memory with RW permissions
2. Write the payload into the region
3. Transition to RX via VirtualProtect

Indicators: executable memory regions with no file backing, high entropy in region contents, and adjacent allocations showing staging behavior.

---

5. Phantom DLLs and Loader Evasion

One of the more advanced fileless techniques involves Phantom DLLs, which exploit inconsistencies between Windows loader structures and actual memory state.

What Is a Phantom DLL?

A Phantom DLL is a module that exists as a valid PE image in memory, may appear partially in PEB loader structures, and lacks a proper file-backed MEM_IMAGE mapping.

How Phantom DLLs Are Created

Common techniques include manual mapping (reflective PE loaders), section object manipulation without SEC_IMAGE, and partial reconstruction of PE structures in heap/private memory.

Why They Matter

They introduce inconsistencies between the VAD tree, PEB module lists, and the actual memory-resident code.

Detection Strategy

Key inconsistencies to look for: a module listed in the PEB but with no corresponding MEM_IMAGE VAD, or with a missing or invalid ControlArea; and a VAD region marked as private that nonetheless contains a valid PE structure.

Legitimate DLLs are mapped via section objects. Phantom DLLs are constructed entirely in memory, without proper loader-backed semantics.

---

6. Practical Volatility 3 Workflow

Enumerate Processes

vol.py -f mem.raw windows.pslist
vol.py -f mem.raw windows.pstree

Look for orphaned processes, suspicious parent-child relationships, and LOLBins (PowerShell, rundll32, mshta).

Traverse VADs

vol.py -f mem.raw windows.vadinfo --pid <PID>

Focus on protection flags, file backing, and region size and layout.

Identify Suspicious Regions

vol.py -f mem.raw windows.vadinfo | grep EXECUTE

Look for RWX, private + executable regions, and missing file paths.

Dump VAD Regions

vol.py -f mem.raw windows.vaddump --pid <PID> --dump

Then analyze PE headers, entropy, and strings/configs.

Malfind Plugin

vol.py -f mem.raw windows.malfind

Detects injected code, unbacked executable memory, and shellcode patterns.

Sample vadinfo output for a suspicious region looks roughly like this (illustrative example, not real tool output — field names are accurate but values, PID, and process name are fabricated for teaching purposes):

Pid    Process    Start VPN    End VPN      Tag    Protection             CommitCharge   PrivateMemory   File output
4821   powershell 0x210000000  0x210004fff  VadS   PAGE_EXECUTE_READWRITE 5              1               Disabled

The combination that matters here is PAGE_EXECUTE_READWRITE together with PrivateMemory: 1 and no file path — a private, writable-and-executable region with no backing file. On its own this row is not proof of compromise (a JIT engine can look similar), but paired with malfind flagging the same PID, or a PE header found inside that range via vaddump, it becomes a strong indicator worth pivoting on.

⚠️ Warning: this is a heuristic-based approach — evasion is possible.

---

7. Advanced Considerations

7.1 Evasion Techniques

Sophisticated malware may avoid RWX (using RX only), encrypt payloads until execution, fragment payloads across multiple VADs, or blend into legitimate JIT regions (.NET, V8).

7.2 Kernel Tampering

With kernel access, VAD structures can be altered and regions hidden from traversal. Mitigation relies on cross-view analysis and memory integrity checks.

7.3 VAD Tagging and Pool Analysis

Each VAD is backed by kernel pool allocations, tagged "Vad " — useful for detecting hidden or orphaned nodes, and relevant in DKOM and rootkit scenarios.

7.4 Where VAD Analysis Falls Short

VAD analysis is powerful, but it is not a silver bullet, and a good analyst treats it as one layer among several:

• It’s still heuristic. RWX and unbacked-private-executable patterns are strong signals, not proof. Legitimate JIT engines, packers, and some AV/EDR agents produce similar shapes.
• Disciplined attackers avoid RWX entirely. Allocating RW, writing the payload, then flipping to RX — or going straight to RX after writing through a separate writable mapping of the same pages — sidesteps the most obvious heuristic.
• Fragmentation defeats single-VAD signatures. Splitting a payload across several smaller VADs, each individually unremarkable, forces analysts toward aggregate and cross-VAD analysis rather than single-node rules.
• Kernel-level compromise moves the goalposts. As noted in 3.3 and 7.2, an attacker with kernel code execution can, in principle, tamper with VAD structures directly — at that point VAD analysis alone is no longer trustworthy ground truth, and integrity checks or cross-view analysis become necessary.
• A snapshot has no history. A single memory image shows the VAD tree at one point in time; short-lived staging regions that were allocated and freed before acquisition leave no VAD trace at all — this is where ETW/Sysmon telemetry fills the gap VAD analysis structurally cannot.

None of this makes VAD analysis less valuable — it means it belongs in a layered detection strategy, not as a standalone verdict.

---

8. Synthesis: Building Reliable Detection

No single heuristic is sufficient. Strong detection combines:

• VAD anomaly detection
• PE validation in memory
• Thread start analysis
• Telemetry (ETW / Sysmon)
• Behavioral correlation

Core Principle: malicious memory stands out not by presence — but by inconsistency.

Closing Thoughts

Fileless malware doesn’t eliminate artifacts — it relocates them. VAD analysis provides a rare advantage: a kernel-enforced, difficult-to-forge map of memory.

For analysts operating at this level, memory is not opaque — it is structured, enumerable, and rich with signals. Even the most evasive implants leave traces, if you know where to look.

---

References

1. Microsoft Sysinternals / Windows Internals (Russinovich et al.) — learn.microsoft.com/sysinternals