Technical Research · Windows Internals

Architectural Limits of Kernel-Level EDR:
Detecting In-Memory Execution in Modern Windows Systems

Kernel-mode EDR gives defenders the deepest vantage point available on Windows - a callback-driven view from inside the executive itself. We build the blueprint for a modern kernel sensor, then show exactly where that vantage point runs out against reflective loading, process hollowing, and raw shellcode staging.

Kernel-Mode EDR Windows Internals In-Memory Execution
Research By Amber Chalia, Security Researcher, ISECURION
Contents

Why Kernel-Mode EDR

Modern adversaries have largely stopped writing malware to disk. Reflective DLL injection, process hollowing, manual mapping, module stomping, and raw shellcode staged in private memory all share one defining trait: they never produce a file for a scanner to catch.

That shift has forced defenders to move their sensors deeper into the operating system - into a place where user-mode tricks can't reach. Kernel-mode EDR drivers run inside the Windows executive itself, observing process creation, image loads, and thread starts from what we call the source context: a vantage point above the reach of an attacker holding nothing more than administrator credentials.

The real question: Placement inside the kernel isn't the same as resolution. Can a callback-driven sensor actually observe - and act on - code that only ever exists as bytes inside an already-running process?

The answer, as this post documents, is partially - and the gap is architectural, not a bug waiting on better engineering.

SubjectWindows Kernel-Mode EDR
Threat ModelIn-Memory / Fileless Execution
Primary CallbacksProcess, Image Load, Thread Creation
Operating ConstraintsPatchGuard & HVCI
Kernel Callback CoverageStructurally Incomplete
Architectural Ceilings3 (Section 07)
5
In-memory evasion techniques covered by the threat model
3
Kernel callbacks forming the spine of the sensor
3
Structural ceilings no purely kernel-resident sensor can close
0
Undocumented hooking techniques used - all patterns PatchGuard/HVCI safe

Driver Foundations & Lifecycle

A kernel-mode driver is a living extension of the operating system, and its lifecycle determines whether it improves or destabilizes the host. Inside DriverEntry, the architect performs four duties: assign the DriverUnload routine, register kernel callbacks, initialize synchronization primitives, and create device objects for user-mode communication.

The RegistryPath parameter is more than a formality - it's a stable, file-system-independent place to persist configuration, policy, and audit logs, which matters when the EDR itself may be the thing inspecting the file system.

The most dangerous phase is unloading. Every callback registered in DriverEntry must be unregistered in strict reverse order, every pool freed, every I/O drained. Leave one callback registered and the next matching kernel event dispatches into a function address that no longer maps to live code - an immediate Bug Check, typically PAGE_FAULT_IN_NONPAGED_AREA or SYSTEM_SERVICE_EXCEPTION.

Three WDK headers are foundational: ntddk.h for basic kernel-mode declarations, ntifs.h for file/section object structures, and ntimage.h for the PE structure definitions used throughout header parsing. For development-time visibility, route telemetry through DbgPrintEx on the DPFLTR_IHVDRIVER_ID channel - conventionally reserved for hardware vendor drivers, so it stays quiet under normal OS noise and supports per-component filtering in DebugView.

Process Creation & Static PE Inspection

PsSetCreateProcessNotifyRoutineEx is where meaningful detection starts. Use the Ex variant only - the legacy routine doesn't deliver enough context (parent PID, command line, underlying file object) for modern detection logic.

ImageFileName
CommandLine
ParentProcessId
CreationStatus (veto)
Creation Timestamp

CommandLine is arguably the single most valuable field for catching Living-off-the-Land attacks, where adversaries abuse signed binaries like cmd.exe or powershell.exe - the binary is legitimate, the arguments are malicious. CreationStatus can veto process creation outright, but production deployments typically disable veto by default: a bad policy that blocks legitimate children of svchost.exe renders the host unusable in a way that looks a lot like malware to an end user.

Process names lie, though - which is why static inspection matters. Opening a file from kernel mode means always setting OBJ_KERNEL_HANDLE and using ZwCreateFile rather than NtCreateFile, which correctly sets the previous-mode field to KernelMode.

MZ-header triage isn't just "look for the bytes M and Z" - it's three checks, in order: confirm the read returned at least sizeof(IMAGE_DOS_HEADER) bytes, confirm the magic bytes, and confirm e_lfanew falls within the bytes actually read. Skip any one and the next parsing stage inherits a buffer overrun - which in kernel mode is a Bug Check, not a graceful segfault.

Section Tables & the Import RVA Trap

Two of the sharpest static signals available to a kernel EDR - and one of the most common mistakes made while building it.

Signal 01 The W⊕X Violation Static Signal

A section's Characteristics bitmask can mark it both writable and executable at once - a violation of the principle that memory should be one or the other, never both. Legitimate compiled code almost never does this; packers, self-decrypting loaders, and shellcode stagers do it constantly, because they need to write their real payload into memory before executing it.

Section names add a second tell: .text, .data, .rdata are standard. .UPX0, .aspack, .vmp0 signal packing - not inherently malicious, but a strong prior that static content no longer represents runtime behavior.

Section Characteristics Bitmask Packer / Loader Signature
Signal 02 Entry Point Locality Static Signal

In a normally compiled binary, AddressOfEntryPoint falls within .text, near its start, in a section marked executable but not writable. Three deviations are suspicious: an entry point inside .data or .rsrc; an entry point in a section that's both writable and executable; or an entry point that falls within no section at all.

AddressOfEntryPoint Validation Loader Stub Signature
Trap Parsing Imports From a Disk Buffer Rite of Passage

PE import tables reference everything by relative virtual address (RVA) - offsets relative to the image base after the loader maps it. On disk, sections sit at file offsets that don't correspond to those RVAs at all. Treat a disk-buffer offset as an in-memory offset and you'll dereference into kernel pool guard pages and crash the system.

The architectural fix: don't parse imports from disk. Register PsSetLoadImageNotifyRoutine instead and inspect imports from the mapped image, where RVAs are already valid. This also unlocks comparing the static import list against the dynamic load list - a DLL present at runtime but never statically declared means something bypassed the loader's normal binding machinery entirely, which is one of the most reliable signals a kernel EDR has.

RVA-to-File-Offset Translation Bug Check Risk if Mishandled Static vs Dynamic Import Diff

Synchronization & Target-Mode Filtering

Kernel callbacks fire on whatever thread happens to be inside the executive at that moment - on a busy multi-core host, that means callbacks can run simultaneously across every core. Any shared state is subject to concurrent mutation from an effectively unbounded number of writers.

For the typical EDR critical section, FAST_MUTEX is the right primitive. But a naive implementation - acquire the mutex, do all the work including PE parsing, release - breaks two ways: PE parsing calls ZwCreateFile/ZwReadFile, which can take milliseconds and serialize every process creation on the host; and parsing may trigger another callback in the same driver, producing a recursive deadlock that Bug Checks the system.

Locked phase

Acquire FAST_MUTEX, touch only the shared process table, set a DoParse flag if deeper inspection is warranted, release immediately.

Unlocked phase

If DoParse is true, run the heavyweight parsing routine with no lock held - any writes are reserved during the locked phase and committed in a brief second acquisition.

Result

Bounded lock hold time, no recursive deadlock, and clean composition with deferring heavy work to a system worker thread.

Global-mode telemetry doesn't scale well either: a typical desktop creates 2,000-4,000 processes a day, each with 40-60 image loads, producing hundreds of thousands of events per host - a volume where the marginal value of more data approaches zero. Target mode keeps every callback registered but filters which ones generate telemetry via a predicate consulting image name, parent image, command line, or user SID - concentrating analyst attention on a coherent intent graph while accepting that anything outside the target set is invisible.

Event-Driven Memory Scanning

Everything above builds a high-fidelity picture of what processes exist and what they loaded - none of it observes the runtime mutation of process memory, which is exactly where in-memory execution lives.

Periodic scanning - walking every process on a fixed timer - is a poor fit: average detection latency is half the scan interval, far too slow for a payload that executes and exits in milliseconds. Event-driven scanning triggers off thread creation instead, since a new thread is created every time injected code is about to run, whether via CreateRemoteThread, NtCreateThreadEx, or a queued APC.

The high-confidence signal is a Private memory region transitioning from writable to executable - PAGE_READWRITE to PAGE_EXECUTE_READ. Legitimate code lives in Image regions, backed by a mapped file section. Private regions are normally data. After excluding allowlisted JIT compilers, the residual population of RW→RX transitions in private memory is dominated by reflective loaders, manual mappers, and shellcode stagers.

Section X - Event-Based Memory Scanning

Implementation matters: the scan must run without holding any FAST_MUTEX, since walking a VAD tree may touch operations other callbacks need. The natural design defers the scan from the thread-creation callback to a system worker thread via IoQueueWorkItem, so the callback itself just queues the work and returns immediately - keeping thread creation responsive.

Three Architectural Ceilings

This is the part vendors are least eager to say out loud: no purely kernel-resident sensor can fully detect in-memory execution. Not because of a bug - because of the architecture itself.

Ceiling 01 Static Analysis Can't See What Never Touches Disk Structural

The PE parser is the sharpest tool available against malicious files - and completely blind to shellcode delivered over the network and decoded straight into a private memory region of an already-running process. It's not a flaw in the parser; it's a property of the threat model itself.

MITRE ATT&CK T1620 - Reflective Code Loading File-System-Blind by Design
Ceiling 02 Callback Telemetry Has Temporal & Semantic Gaps Structural

Process creation fires once. Image load fires once per image. Thread creation fires once per thread. Between those events, an attacker can freely allocate, write, decrypt, relocate, and re-protect memory without generating a single callback. A protection transition that happens and reverts between scans - protection laundering - is invisible by construction, and adversaries have built payloads that deliberately race the thread-creation scanner.

MITRE ATT&CK T1055 - Process Injection Protection Laundering Race Condition Exploitable
Ceiling 03 Performance Is a Hard Budget Structural

Every scan has a cost, and on busy server workloads that cost shows up as latency operators notice. Scan everything in full detail and the sensor eventually gets configured down or uninstalled. Any scan that's skipped is a scan an adversary can exploit - the architect is always choosing where to spend a finite budget, and every choice leaves some path uncovered.

Finite Scan Budget Coverage vs. Latency Trade-off
The critical insight: These aren't implementation defects waiting on better engineering of the same primitives. They're structural boundaries of kernel-resident sensing - closing them requires telemetry sources outside the conventional kernel callback set entirely.

Reading about these ceilings is one thing. Knowing whether your specific EDR deployment actually has them - and whether your SOC would catch an attacker exploiting them - is another question entirely, and it's not one a vendor datasheet can answer.

Want to find out? Talk to ISECURION's Red Team practice →

Closing the Gap: Beyond the Kernel Callback

What made this architecture instructive is that none of its three ceilings can be closed by writing a smarter kernel scanner. The gap is layered across sources the kernel callback set was never designed to cover.

🧩
Kernel Callbacks
Spine of the sensor. Structurally incomplete alone.
📡
ETW Threat-Intelligence
Streams VirtualAlloc, VirtualProtect, WriteProcessMemory events.
📜
AMSI
Catches script-based execution before interpretation.
🎯
User-Mode Stubs
Operation-level granularity on high-value APIs.
🛡️
PatchGuard / HVCI
Constrains what any of the above may do at runtime.

The ETW Threat-Intelligence provider, introduced in Windows 10, exposes high-fidelity events for memory operations that no callback surfaces - subscribing to it turns the EDR from a snapshot scanner into a streaming observer of memory mutation, eliminating the protection-laundering window. AMSI covers a different blind spot: script-based execution staged as text rather than bytes, seen just before interpretation. Minimal user-mode telemetry stubs add operation-level granularity that kernel callbacks don't carry - and while the stub itself is bypassable by an attacker with code execution in the same process, that bypass is itself detectable when corroborated against the kernel's view that the stub's own memory was modified.

All of this has to operate under two hard constraints. PatchGuard kills any EDR that achieves visibility by inline-hooking syscall handlers or patching the SSDT, within minutes. HVCI goes further - no page can be both writable and executable, no dynamic code generation, no unsigned drivers. Every pattern in this post is compatible with those constraints by design; anything relying on shadow page tables or runtime code synthesis is not.

🎯

Key Takeaway

A well-built kernel EDR covers a genuinely wide range of malicious behavior - and it still isn't the whole picture.

Code that never touches disk is invisible to the parser. Protection transitions that revert between scans are invisible to the scanner. The gap between thread creation and first execution is a window some adversaries already know how to race.

These are architectural limits of kernel-resident sensing, not bugs to be patched. The practical path forward isn't a smarter kernel scanner - it's a layered sensor, with kernel callbacks as the spine and ETW-TI, AMSI, and minimal user-mode stubs closing the memory and script blind spots no single layer can cover alone.

Building or Auditing a Kernel-Mode EDR?

ISECURION works with security teams and vendors on Windows kernel driver security, EDR architecture review, and detection-engineering assessments across India and internationally.

Talk to Our Research Team More Insights

Frequently Asked Questions: Kernel-Mode EDR & In-Memory Detection

Questions security architects and detection engineers ask when designing or evaluating kernel-resident EDR sensors.

Kernel callbacks are episodic - they fire once per process, once per image load, once per thread. Between those events an adversary can freely allocate, write, decrypt, and re-protect memory without triggering any callback. Static PE parsing is also inherently blind to payloads that are never written to disk.

This is a structural property of the callback model, not an implementation defect. Closing the gap requires telemetry sources - ETW Threat-Intelligence, AMSI, user-mode stubs - that sit outside the conventional kernel callback set.

Protection laundering refers to a memory region transitioning from writable to executable and back again in the window between two scans. Because event-driven scanning is triggered by thread creation rather than continuous monitoring, a transition that occurs and reverts between scan events leaves no trace for the scanner to observe.

Closing this window generally requires streaming telemetry - such as the ETW Threat-Intelligence provider - rather than point-in-time scans.

PE import tables use relative virtual addresses (RVAs), which are only valid once the loader has mapped the image into memory. On disk, section data sits at file offsets that don't correspond to those RVAs. Dereferencing a disk-buffer offset as though it were an in-memory offset reads from the wrong location entirely - which, in kernel mode, risks touching pool guard pages and triggering a Bug Check rather than a recoverable fault.

The standard fix is to inspect imports from the mapped image via PsSetLoadImageNotifyRoutine instead of parsing them from a disk read.

PatchGuard (Kernel Patch Protection) periodically verifies that core kernel structures and read-only pages haven't been modified, and Bug Checks the system if it detects unauthorized modification - killing any EDR that relies on inline-hooking syscall handlers or SSDT patching within minutes.

HVCI (Hypervisor-Protected Code Integrity) goes further: pages can't be both writable and executable, dynamic code generation is prohibited, and unsigned drivers are refused. Modern kernel EDR drivers must rely exclusively on documented callback registration, be compiled with /INTEGRITYCHECK, and be signed with an EV certificate cross-signed by Microsoft.

Three sources complement the kernel callback spine:

  • ETW Threat-Intelligence provider: streams memory-operation events (VirtualAlloc, VirtualProtect, WriteProcessMemory) that no kernel callback surfaces.
  • AMSI: inspects script content (PowerShell, JScript, macros) just before interpretation - covering in-memory execution staged as text rather than bytes.
  • Minimal user-mode telemetry stubs: hook a small set of high-value APIs and forward parameters to the kernel via a shared section, adding operation-level granularity.

Together with the kernel callback spine, these form a layered sensor where no single component needs to catch everything on its own.

Vendor claims and product datasheets aren't evidence - the only reliable way to know is to test the deployed configuration against the same techniques described in this post: reflective loading, process hollowing, manual mapping, and RW→RX protection transitions in private memory, executed the way a real adversary would chain them.

This is precisely what a Red Team Assessment is designed to answer. Rather than a checklist review, ISECURION's red team operators emulate real adversary TTPs mapped to MITRE ATT&CK, run them against your live EDR and SOC, and report exactly which techniques were detected, which were missed, and why - so you know your actual detection coverage, not your assumed coverage.

Penetration testing typically answers "can this system be compromised?" within a defined, often noisy, scope - it's optimized for finding as many vulnerabilities as possible in a fixed window. It doesn't tell you whether your detection stack would have noticed.

Red Teaming answers a different question: "would my people, process, and technology detect and respond to a real, stealthy adversary?" It's goal-oriented, avoids unnecessary noise, and specifically exercises the detection and response chain - EDR alerting, SOC triage, escalation - rather than just the initial exploit. For organisations that already have a kernel EDR deployed, Red Teaming is the more relevant exercise. Learn about ISECURION's Red Team Assessment methodology →

Because fileless techniques avoid disk artifacts, the signals are behavioral rather than file-based. Common indicators include: unexplained child processes spawned from powershell.exe, rundll32.exe, or mshta.exe; processes with network connections but no corresponding image on disk; EDR alerts referencing RWX or RW→RX memory regions that were dismissed as noise; unexpected LSASS access; and endpoints that are unusually slow or show high, sustained CPU from a process with no clear business purpose.

If you're seeing any of these and aren't fully confident in what happened, don't wait for confirmation - early containment matters more than certainty. ISECURION's Incident Response (DFIR) team can rapidly triage, determine scope, and contain active threats, including memory-resident and fileless intrusions. Reach out immediately if you suspect a live incident →

Yes, though it requires different techniques than traditional disk forensics. Memory forensics captures and analyzes RAM to recover injected code, decrypted payloads, network artifacts, and process/thread relationships that never existed as files. Combined with ETW traces, Windows Event Logs, and EDR telemetry captured at the time of the incident, investigators can frequently reconstruct the full attack chain - initial access, injection technique, and lateral movement - even when nothing was ever written to disk.

The catch is timing: volatile memory evidence degrades quickly, and once a compromised host is rebooted or the process exits, much of it is gone. This is why engaging DFIR specialists as early as possible in a suspected incident materially improves the odds of a conclusive investigation and a regulatory-ready report.

At minimum, annually - and additionally after any material change: a new EDR vendor or version, a SOC/MDR provider change, significant infrastructure changes, or following any real security incident. Detection coverage isn't static; EDR vendors patch gaps, adversaries develop new evasion techniques, and your own environment changes constantly, so a red team result from eighteen months ago tells you very little about today's actual exposure.

Organisations with a mature security programme often move to a continuous or twice-yearly cadence. ISECURION scopes Red Team engagements from a single detection-validation exercise up to a comprehensive 3-6 week full-scope adversary simulation - get a customised assessment plan and timeline →

Don't reboot or reimage the affected host if you can avoid it - that's frequently the single biggest mistake made in the first hour, and it destroys the volatile memory evidence needed to determine what actually happened. Instead: isolate the host at the network level (not by powering it off), preserve logs and EDR telemetry, and avoid taking remediation actions until scope is understood, since premature cleanup can tip off an attacker who still has access elsewhere in the environment.

ISECURION's Incident Response team provides rapid breach assessment, forensic investigation, and regulatory-ready reporting for ransomware, business email compromise, and intrusions involving memory-resident malware. Contact our IR team now → - early engagement materially reduces both dwell time and total incident cost.

Find Out What Your EDR Actually Catches

Don't take detection coverage on faith. ISECURION's Red Team Assessments emulate the exact evasion techniques covered in this post - reflective loading, process hollowing, manual mapping - against your live environment.

Request a Red Team Assessment
Suspect an Active Intrusion?

Fileless and in-memory attacks are hard to spot and easy to lose evidence for. ISECURION's DFIR team provides rapid triage, forensic investigation, and containment - available for urgent engagement.

Engage Incident Response
Have a question about kernel EDR architecture, Red Teaming, or Incident Response not answered here? Contact ISECURION at info@isecurion.com or submit an enquiry - we respond to all pre-engagement queries within one business day.

Evaluating Your EDR's Coverage Against In-Memory Threats?

ISECURION - Kernel Security Research, Detection Engineering & VAPT Across India and Internationally

Our research and technical services practice covers Windows internals, kernel driver security review, detection-engineering assessments, and adversarial evaluation of EDR and endpoint security products. CERT-In empanelled.

This post is produced for informational and educational purposes and reflects publicly documented Windows kernel APIs and callback behavior. It does not describe any specific commercial product or client deployment.

WhatsApp