Reading the MFT Directly: How DiskSleuth Scans a Drive in Seconds

The standard approach to building a disk space analyser is to walk the directory tree. Call ReadDirectoryChangesW, recurse through subdirectories, accumulate sizes, done. It works. It’s also slow, memory-hungry on large volumes, and requires the OS to construct the path hierarchy from the underlying data structures on every call.

DiskSleuth doesn’t do that. When it’s running elevated, it reads the NTFS Master File Table directly using FSCTL_ENUM_USN_DATA and reconstructs the directory tree in memory from raw MFT records. On a 500 GB volume with a million files, that takes a few seconds rather than minutes.

What the MFT actually is

NTFS stores every file and directory on a volume as a record in the Master File Table. Each record is 1 KB by default and contains the file’s attributes — name, size, timestamps, parent directory reference, data run locations. The directory tree that Windows Explorer shows you is not separately stored; it’s reconstructed from the parent references in each MFT record whenever you navigate somewhere.

The implications for scanning are significant. If you want to know the size of every file on a volume, you don’t need to walk the directory tree at all. You can enumerate the MFT directly, read the size from each record, and then reconstruct the parent-child relationships from the parent FRN (file reference number) in each record. You’re reading raw metadata in sequential disk order rather than following directory pointers.

FSCTL_ENUM_USN_DATA

The Windows API that exposes direct MFT enumeration is the USN journal interface. FSCTL_ENUM_USN_DATA lets you iterate over MFT records in bulk. You send it an MFT_ENUM_DATA_V0 structure specifying a starting USN and the file reference number range to enumerate, and it returns USN_RECORD_V2 structures describing each file.

let input = MFT_ENUM_DATA_V0 {
    StartFileReferenceNumber: 0,
    LowUsn: 0,
    HighUsn: i64::MAX,
};

DeviceIoControl(
    volume_handle,
    FSCTL_ENUM_USN_DATA,
    Some(&input as *const _ as *const _),
    size_of::<MFT_ENUM_DATA_V0>() as u32,
    Some(buffer.as_mut_ptr() as *mut _),
    buffer.len() as u32,
    &mut bytes_returned,
    None,
)?;

DiskSleuth uses a 256 KB buffer per call. The standard examples use 64 KB. Larger buffers mean fewer round-trips to the kernel — the gain levels off around 256 KB, so that’s where I settled.

This requires administrator elevation. CreateFile on the volume root (\\.\C:) needs GENERIC_READ with FILE_SHARE_READ | FILE_SHARE_WRITE | FILE_SHARE_DELETE, and the IOCTL itself requires the calling process to be elevated. If you’re not elevated, DiskSleuth falls back to parallel directory traversal.

The fallback: jwalk and rayon

When running without elevation — or on a drive where the USN journal isn’t enabled — DiskSleuth uses jwalk with rayon for parallel directory traversal. jwalk is a parallel directory iterator that sends work to a rayon thread pool, distributing subdirectory enumeration across CPU cores.

The throughput difference is real. On a developer workstation with a million files, MFT enumeration completes in 2–4 seconds. Parallel directory traversal with jwalk takes 15–30 seconds depending on disk and CPU. Both are faster than single-threaded directory walking, which can take several minutes on large volumes.

For most elevated use cases on typical admin machines, the MFT path is what runs.

Arena-allocated tree

Once you have the MFT records, you need to reconstruct the parent-child relationships. The naive approach is to allocate a HashMap<FRN, DirNode> with boxed children. Each allocation is an independent heap object. Iterating over children means following pointers scattered across the heap.

DiskSleuth uses an arena instead:

pub struct FileTree {
    nodes: Vec<FileNode>,
}

pub struct FileNode {
    pub name:    compact_str::CompactString,
    pub size:    u64,
    pub parent:  Option<NodeIndex>,
    pub children: Vec<NodeIndex>,
}

#[derive(Clone, Copy)]
pub struct NodeIndex(u32);

All nodes live in a single Vec<FileNode>. References between nodes are NodeIndex(u32) offsets into that vec. This is cache-friendly — iterating over a directory’s children is a sequential walk through the index array followed by direct offset lookups. There’s no pointer chasing, no heap fragmentation, no allocator overhead per node.

Aggregating sizes bottom-up is O(n) over the flat array. You sort nodes in reverse order (leaves first) and accumulate sizes upward through the parent references. No recursion, no stack overflow risk on deep directory hierarchies.

Keeping the UI responsive during a scan

The scan runs in a background thread. The UI thread needs to display progress and, when the scan completes, navigate the resulting tree. The shared state is Arc<RwLock<FileTree>>.

The naive implementation takes a write lock per node insertion. On a million-node tree, that’s a million lock acquisitions. The contention between the UI thread (which needs read access to show progress) and the scan thread (writing nodes) becomes the bottleneck.

DiskSleuth batches write operations: the scan thread accumulates 2,000 nodes before taking a write lock and inserting them all at once. The UI thread’s read locks happen between batches. Contention drops by a factor of 2,000.

The UI reads are also minimal during the scan — progress is a single counter, not the full tree. The full tree render only happens when the scan completes and the user navigates.

Treemap visualisation

The main view is a nested squarified treemap — each directory is a rectangle sized proportionally to its disk usage, subdivided by its contents. The algorithm is squarified layout, which produces rectangles closer to square than the older strip-based approach, making size comparisons more intuitive.

Navigation is click-to-drill-down. Clicking a directory rectangle zooms into that directory, showing its contents in detail. Back/forward/up buttons work like a browser. A breadcrumb trail shows your position in the hierarchy.

The treemap doesn’t clone the FileTree for navigation. It stores a NodeIndex for the current root and derives the layout on-demand. With the arena structure, computing the layout for a directory means reading a contiguous slice of child indices and looking up their sizes — fast enough to do on every frame in egui’s immediate-mode render loop without perceptible lag.

Virtual scrolling is applied to the tree view (the right-side panel showing files as a list). Rendering 100,000 rows would be slow; virtual scrolling renders only the visible rows. egui’s ScrollArea with a fixed row height makes this straightforward.

What it doesn’t do that you might expect

DiskSleuth only shows what the MFT can show — real files and directories by their stored sizes. It doesn’t show:

NTFS alternate data streams. A file can have hidden streams (used by IE’s “Mark of the Web”, some ransomware, Zone.Identifier data). They consume disk space but don’t appear in the main data run. A future version might surface them, but they’re edge-case enough that they’re not in v1.

Sparse files and compressed files. The AllocationSize and FileSize fields in the MFT can diverge significantly for sparse or NTFS-compressed files. DiskSleuth uses allocated size, which represents actual disk consumption. du on Linux also uses allocated size. This is the right number for “what’s eating my disk” but can differ from what Explorer shows.

System-protected files. Some files under System Volume Information require SYSTEM privileges even for metadata reads. They show up in the count but with size 0 if the read fails. Elevating to admin covers the vast majority of cases, but there’s a small residual that requires SE_BACKUP_NAME privilege.

The tool is on GitHub, MIT licensed.