Unpacking Packers

Unpacking Packers

GL4U Miqang
GL4U Miqang
malware

How Packers in Malware Are Detected

Malware packers are one of the most common defense‑evasion techniques in modern threats. Instead of exposing their true code, many malware families wrap themselves in layers of obfuscation — making static detection ineffective and human analysis cumbersome.

Detecting that a binary is packed, and ideally which packer is used, is a critical first step in any malware analysis pipeline. This post explains how detection works, what analysts can practically do, and what current research says about the state of the art.

A Short Taxonomy You Can Use

Think of packers along a simple hierarchy relevant for detection:

  1. Compression‑based packers
    These simply compress a binary and unpack at runtime.

    • Examples: UPX, MPRESS, NSPack1
    • Easy to detect via patterns and entropy; common in the wild.

  2. Encryption/Stub‑based packers
    These encrypt sections of the executable and use a small stub at startup to decrypt in memory.

    • Often custom or modified by malware families to evade signature rules2.

  3. Virtualization / Protector‑style packers
    These don’t just compress — they translate the code into a custom virtual machine format or insert anti‑analysis features.

    • Examples: VMProtect, Themida, ASProtect3
    • Hard to unpack without dynamic analysis and often trigger AV heuristics.

This taxonomy matters because detection methods behave differently depending on how the packer alters the binary.

The MITRE ATT&CK catalog lists many real malware families that use packing in the wild, highlighting why detection is crucial:

  • Emotet uses custom packers to shield its payload4
  • Tomiris and ZeroT have been observed packed with UPX5
  • APT38 campaigns have used commercial protectors like Themida and VMProtect6
  • Squirrelwaffle uses custom packing to delay analysis7

These are not theoretical; they come from real threat intelligence reports and show packers being actively used for evasion.

How Detection Works in Practice

1) Entropy and Section Heuristics

One of the earliest and still powerful signals that a file might be packed is entropy — a statistical measure of randomness.

  • Packed sections (e.g., encrypted payloads) show high entropy because they resemble random data.
  • Tools like Bintropy and custom scripts can flag anomalous sections quickly.

Most packers create unusual section names or leave few imports (often just LoadLibrary and GetProcAddress) because the unpacking stub does the real loading.

Example heuristics:

  • Entropy > 7.5 in .text or unnamed sections → likely packed
  • Import Table only contains loader APIs → indicates manual function resolution

These heuristics are fast and useful, especially during triage.

2) Signature Databases & Packer Detection Tools

Tools like PEiD, Detect It Easy (DIE), and ExeinfoPE use signature databases of known packers to identify patterns in binaries. In curated datasets, researchers benchmark detection tools against ~19 packers including ASPack, BeRoEXEPacker, UPX, and Yoda’s Protector8.

These tools work well for known packers but struggle with:

  • Custom variants
  • Packers that strip or alter identifiers

Professional analysts often combine multiple detectors for better coverage.

Detect It Easy identifying a packed PE file
Detect It Easy (DIE) identifying a packed executable using signature matching. Source: Detect It Easy project

3) Static Feature Machine Learning

Research shows static analysis with feature engineering can go beyond entropy and signatures:

  • Alkhateeb et al. propose a multilayer feature engine analyzing PE headers, section metadata, and patterns, achieving ~99.6% accuracy on known packers and ~91% on unknown ones9.

Typical features include:

  • PE header anomalies
  • Section size vs. entropy distributions
  • N‑gram opcode patterns

These feed into classifiers (random forests, SVMs, etc.) as reliable static indicators of packing.

4) Dynamic/Sandbox‑based Detection

Some packers unpack only at runtime or under specific conditions:

  • Running binaries in a sandbox (Cuckoo, Any.Run, or custom VM) and monitoring memory after the unpacking stub executes reveals the original code10
  • Memory dumps and heuristics (e.g., identifying the Original Entry Point) are powerful but computationally expensive

Dynamic analysis is especially useful for virtualization packers like VMProtect and Themida that hide control flow.

5) Deep Learning & Emerging Techniques

Recent research frames packed detection as a broader ML problem:

  • CNNs on grayscale images of binaries can differentiate packed vs non‑packed files11
  • Assembly language pretraining (Pack‑ALM) treats packed code as “pseudo instructions,” robust against unseen packers12

These methods suggest future automation where classical heuristics and deep learning combine to build generalized detectors.

Practical Detection Workflow (for analysts)

A recommended workflow:

  1. Initial triage: Compute entropy and check imports
  2. Signature detection: Run PEiD, DIE, or ExeinfoPE
  3. Static extraction: Feature‑based classifier to decide “packed vs not packed”
  4. Sandbox: Run the binary to observe memory unpacking
  5. Deep analysis: Use ML/DL models for challenging cases

This multi-stage process increases confidence and reduces false positives.

Closing Thoughts

Detecting packers isn’t just about spotting “UPX!” or high entropy; it’s about understanding how malware hides, why certain packers are chosen, and how tools can reliably detect them. Machine learning and research are closing gaps left by older signature lists, but fundamentals — entropy, structural anomalies, and runtime behaviors — remain key to effective detection.

Below is a YouTube video that clearly explains packers in practice, including polymorphism and common misconceptions:

Video courtesy of MalwareAnalysisForHedgehogs

References

Footnotes

  1. UPX, MPRESS, NSPack — common compression packers. UPX official ↩

  2. Encryption/Stub‑based packers overview — MITRE ATT&CK T1027. MITRE ATT&CK ↩

  3. VMProtect and Themida documentation. Themida ↩

  4. Emotet custom packers. Virus Bulletin ↩

  5. Tomiris malware packed with UPX. MITRE ATT&CK S0671 ↩

  6. APT38 packer use. PICUSS SECURITY ↩

  7. Squirrelwaffle packer usage. MITRE ATT&CK ↩

  8. PEiD/DIE signature databases and packing datasets. GitHub: FFRI PackerDetectionToolEvaluation ↩

  9. Alkhateeb et al., Identifying Malware Packers through Multilayer Feature Engineering. MDPI ↩

  10. Mal‑Flux memory unpacking techniques. ScienceDirect ↩

  11. CNN grayscale image representations for binaries. arXiv ↩

  12. Pack-ALM deep assembly pretraining for packed detection. arXiv ↩

#malware #packers #upx #rem