Introduction

For developers training massive foundation models, the bottleneck is rarely just the GPU compute—it is often the storage subsystem. When moving terabytes of tensor data between NVMe SSDs and GPU memory, traditional Python I/O methods introduce significant latency and CPU overhead. DeepNVMe, a high-performance I/O optimization suite integrated into the DeepSpeed ecosystem, solves this by leveraging Linux Asynchronous I/O (AIO) and NVIDIA GPUDirect Storage (GDS) to saturate available hardware bandwidth. By bypassing the CPU bounce buffer in specific configurations, DeepNVMe allows researchers to achieve near-peak NVMe throughput, drastically reducing the time spent on model checkpointing and weight loading.

What Is DeepNVMe?

DeepNVMe is a high-performance C++ NVMe read/write library designed to accelerate the movement of raw data bytes between persistent storage and CPU or GPU tensors in PyTorch. It is part of the DeepSpeed ecosystem maintained by Microsoft and is licensed under the Apache-2.0 license. The tool provides a streamlined abstraction for complex I/O operations, allowing developers to move away from slow, synchronous Python file operations toward bulk asynchronous data transfers.

The library operates through two primary abstractions: the aio_handle, which utilizes Linux AIO and works on both host (CPU) and device (GPU) tensors, and the gds_handle, which leverages NVIDIA GPUDirect Storage to enable a direct DMA path between the NVMe drive and GPU memory, completely bypassing the CPU.

Why DeepNVMe Matters

As deep learning models scale to hundreds of billions of parameters, the volume of data required for optimizer states, gradients, and model weights exceeds the capacity of available GPU VRAM. This necessitates “offloading”—the process of storing these tensors on CPU RAM or NVMe SSDs and swapping them back into the GPU as needed. Without an optimized I/O path, this swapping process becomes a massive bottleneck, leaving expensive GPUs idle while they wait for data to arrive from the disk.

DeepNVMe fills this gap by providing the underlying I/O engine for critical DeepSpeed technologies like ZeRO-Infinity and ZeRO-Inference. By achieving near-peak NVMe bandwidth, it enables the training of models that are far larger than the aggregate GPU memory of a cluster. It transforms NVMe storage from a slow fallback into a high-speed extension of the memory hierarchy, making the training of trillion-parameter models computationally feasible on more modest hardware.

Key Features

• Bulk Asynchronous Data Transfer: DeepNVMe supports non-blocking I/O operations, allowing the system to queue multiple read/write requests and overlap storage transfers with GPU computation.
• NVIDIA GPUDirect Storage (GDS) Integration: By using the gds_handle, the library enables direct DMA transfers between NVMe and GPU memory, eliminating the CPU bounce buffer and reducing latency.
• Linux AIO Support: The aio_handle provides a high-performance alternative to standard Python I/O, utilizing the Linux kernel’s asynchronous I/O capabilities for both CPU and GPU tensors.
• Pinned Memory Management: The library includes a Pinned Memory Manager to optimize the limited supply of pinned memory, ensuring that data transfers are efficient and do not trigger unnecessary page faults.
• Dynamic Prefetching: DeepNVMe can trace operator sequences ahead of time and prefetch required tensors from NVMe to CPU/GPU memory before they are needed for computation, mitigating bandwidth bottlenecks.
• Near-Peak Bandwidth Saturation: In optimized environments, DeepNVMe can saturate the available NVMe bandwidth, achieving reads up to 10GB/sec and writes up to 5GB/sec on supported hardware.

How DeepNVMe Compares

When compared to traditional Python I/O or standard PyTorch tensor saving methods, DeepNVMe represents a fundamental shift in how data is moved. While Python’s open().read() is synchronous and relies on the CPU to manage every byte, DeepNVMe offloads the heavy lifting to the hardware and kernel.

The primary tradeoff is hardware requirement. Standard Python I/O works everywhere. DeepNVMe AIO requires libaio, and the GDS path requires specific NVIDIA hardware and the nvidia-fs driver. However, for those with the hardware, the performance gain is transformative. In microbenchmarks, DeepNVMe can achieve up to 16X faster tensor loading and 19X faster tensor storing compared to traditional Python methods.

Getting Started: Installation

DeepNVMe is integrated into the DeepSpeed library. To use it, you must first install DeepSpeed and ensure the underlying system libraries are present.

Prerequisites

You need a Linux environment with an NVMe SSD and DeepSpeed version >= 0.15.0.

Installing DeepSpeed

pip install deepspeed

System Dependencies

The async_io operator is required for all DeepNVMe functionality. If it is missing, install libaio:

apt install libaio-dev

Enabling GPUDirect Storage (GDS)

GDS requires the NVIDIA GDS driver and compatible hardware. Follow the official NVIDIA GDS installation guide to install the nvidia-fs driver and configure your file system for DMA transfers.

Verification

Run the following command to verify that the DeepNVMe operators are correctly installed and compatible:

ds_report

Check the output for async_io and gds operators; their status should be [OKAY].

How to Use DeepNVMe

DeepNVMe is accessed through handles that abstract the underlying I/O implementation. You create a handle and then use it to load or store tensors.

The basic workflow involves creating an AsyncIOBuilder or GDSBuilder to generate a handle, and then calling the load or store methods. These methods take a PyTorch tensor and a file path as arguments.

For best performance, it is highly recommended to use pinned memory (pinned host tensors or pinned device tensors) to avoid the kernel having to copy data to a temporary buffer before the DMA transfer.

Code Examples

Creating Handles

Depending on your hardware and requirements, you can initialize either an AIO or GDS handle:

from deepspeed.ops.op_builder import AsyncIOBuilder, GDSBuilder

# Create an AIO handle (works for CPU and GPU tensors)
aio_handle = AsyncIOBuilder().load().aio_handle()

# Create a GDS handle (works only for CUDA tensors, but is more efficient)
gds_handle = GDSBuilder().load().gds_handle()

Loading a Tensor from File

Once the handle is created, loading a tensor is a simple one-line operation:

# Load data from NVMe into a pre-allocated tensor
aio_handle.load(tensor=my_tensor, fp= "/path/to/tensor_data.bin")

Storing a Tensor to File

Similarly, storing a tensor is handled by the handle’s store method:

# Store tensor data to NVMe storage
aio_handle.store(tensor=my_tensor, fp= "/path/to/tensor_data.bin")

Real-World Use Cases

DeepNVMe is most effective in scenarios where the model size exceeds the available GPU memory, necessitating high-speed offloading.

• Massive Model Checkpointing: For researchers training models with trillions of parameters, saving a checkpoint can take minutes, stalling training. DeepNVMe (specifically via FastPersist) reduces this overhead to near-negligible levels by saturating NVMe bandwidth.
• ZeRO-Inference: When running inference on a massive model that cannot fit on a single GPU, DeepNVMe allows the model weights to be offloaded to NVMe and streamed into the GPU just-in-time, enabling the execution of 100B+ parameter models on a single H100.
• Dataset Loading for I/O Bound Workloads: In tasks where the dataset consists of massive raw tensors (e.g., high-resolution 3D medical imaging or climate data), DeepNVMe can be used to load data into the GPU faster than standard PyTorch DataLoader implementations.

Contributing to DeepNVMe

DeepNVMe is part of the DeepSpeedExamples repository. Contributions are welcome through the standard GitHub flow. Developers can report bugs or suggest new features by opening an issue in the DeepSpeedExamples repository.

The project follows the Microsoft Open Source Code of Conduct. To contribute, you can submit a pull request with performance benchmarks showing the improvement in I/O throughput or new example scripts for common tensor I/O patterns.

Community and Support

The primary hub for support is the DeepSpeed GitHub repository and its associated Discussions tab. Because DeepNVMe is a low-level system optimization, most troubleshooting involves verifying hardware compatibility and driver versions via ds_report.

Official documentation is available through the DeepSpeed tutorials and the Microsoft AI research blogs. For those seeking advanced tuning, the DeepSpeed team hosts regular office hours on the last Tuesday of each month to discuss development plans and features.

Conclusion

DeepNVMe transforms the way deep learning practitioners handle tensor I/O. By moving away from synchronous, CPU-bound Python operations and embracing asynchronous, hardware-accelerated paths, it removes one of the most significant bottlenecks in large-scale AI training. Whether you are implementing ZeRO-offload strategies or simply need to faster checkpointing, DeepNVMe provides the necessary primitives to saturate your NVMe hardware.

If you have the required NVIDIA hardware and NVMe SSDs, the performance gains are immediate and substantial. We recommend starting with the aio_handle for general compatibility and moving to the gds_handle for maximum throughput. Star the repo, try the quickstart examples, and join the DeepSpeed community to accelerate your I/O.

Resources

• DeepSpeedExamples GitHub Repository
• DeepSpeed Main Repository
• NVIDIA GPUDirect Storage Documentation