GPU utilization is the metric everyone monitors. When a training run underperforms, the instinct is to look at model architecture, batch size, gradient accumulation settings, or distributed training efficiency. Storage almost never appears in the first three rounds of debugging. In our experience working with AI infrastructure teams, storage is the undiagnosed bottleneck in a meaningful fraction of the cases where GPU utilization is persistently stuck below 70%.
This is not because storage is an exotic failure mode. It is because the symptoms look like compute problems. A data-starved GPU and a compute-bound GPU look the same in simple utilization metrics. Distinguishing them requires watching the right signals, and most training monitoring setups are not instrumented for it.
What GPU Starvation Looks Like
In a storage-bottlenecked training run, GPUs are idle waiting for data. The GPU itself reports high utilization (busy waiting or in a tight poll loop for the next batch) but is not doing useful compute. The forward pass completes quickly but the next batch is not ready; the GPU spins waiting for the data loader to deliver.
The most reliable signal: watch GPU memory utilization and GPU SM (streaming multiprocessor) active percentage simultaneously. A storage bottleneck typically shows high memory utilization (the model is loaded and the last batch is still in memory) with low SM active percentage (no compute is happening). A compute bottleneck shows both memory and SM utilization high. The gap between these two metrics is the diagnostic.
Additional signals to instrument:
- Data loader worker CPU utilization. If loader workers are pinned at 100% CPU while GPUs wait, the bottleneck is preprocessing, not storage. If loader workers are idle while GPUs wait, the bottleneck is I/O.
- Storage I/O saturation (iostat, or the equivalent in your monitoring stack). If the storage volume shows queue depth consistently above 32 and service times above 50ms, you are saturating the storage layer.
- Network bandwidth utilization on the storage network. If you are running a distributed filesystem and the storage network interface is at 90%+ of capacity, you have found the bottleneck.
Measuring these simultaneously for a 5-minute window during steady-state training (not during checkpoint writes, which temporarily saturate storage regardless of bottleneck status) gives you a clear picture of where time is being lost.
Why Storage Bottlenecks Occur on Modern Training Clusters
The proximate cause is usually one of three things: insufficient storage bandwidth relative to data consumption rate, excessive seek behavior from random access patterns in the data loader, or filesystem metadata overhead at high file count.
Insufficient bandwidth. A 64-GPU cluster with bf16 training at 2,048 token sequences and a batch size of 256 per GPU consumes data at roughly 500 to 800 MB/s of raw tokenized throughput, depending on model depth and step time. If your storage layer delivers 300 MB/s sustained, you will be training at 60 to 70% of achievable throughput regardless of how fast your GPUs are. The fix is straightforward: add storage nodes or upgrade to a higher-bandwidth storage architecture. The diagnosis is what teams often skip.
Random access in the data loader. Training data pipelines that shuffle data by accessing random file offsets rather than pre-shuffled sequential files impose random read patterns on the storage layer. Spinning disk storage, including SMR drives, handles sequential reads well and random reads poorly. A well-designed sequential storage layer that is forced into random access by the data loader behaves like a poorly-configured storage system. The fix is upstream: shuffle at the dataset construction stage, not at the per-batch sampling stage. Pre-shuffle your tokenized data into fixed-size sequential shards, then iterate shards in a random order at training time. This gives your data loader sequential reads per shard and per-epoch randomization at the shard selection level.
Metadata overhead at high file count. A training dataset stored as millions of small files imposes significant filesystem metadata overhead even if the total data volume is modest. Every open/close/stat call for a small file carries metadata overhead. On Lustre clusters, this is the most common cause of mysterious storage performance degradation at scale. The fix is to coalesce small files into larger container formats (TFDS shards, WebDataset tar files, Parquet files) before training. Leil's metadata layer handles high file counts more efficiently than Lustre's MDT architecture for typical AI training access patterns, but the general principle applies: fewer, larger files beat many small files for sequential training reads.
A Diagnostic Scenario
Consider a plausible scenario: a computer vision model trained on a dataset of 50 million images stored as individual JPEG files, accessed via a PyTorch DataLoader with 8 worker processes and random sampling. Training GPU utilization is stuck at 65%.
The storage layer is a single NFS volume backed by a RAID6 array of 10TB CMR drives. The array has 12 drives, giving roughly 800 MB/s theoretical read throughput. Measured sustained throughput on sequential reads: 750 MB/s. The training pipeline's data consumption rate: 400 to 500 MB/s of JPEG image data, plus decompression and preprocessing.
So bandwidth is not the bottleneck. The actual bottleneck: 50 million small files create a metadata access rate of thousands of open/stat/read/close operations per second on the NFS server. The NFS server CPU is saturated processing metadata requests, adding 5 to 15ms of latency per file access. With 8 data loader workers each accessing files asynchronously, the latency multiplies across workers but does not hide behind parallelism because the NFS server CPU is the serial constraint.
Diagnosis confirms: NFS server CPU at 95%, data loader workers showing I/O wait despite the RAID array being underutilized. Fix: convert the dataset to WebDataset tar shards of 1,000 images each, reducing file count from 50 million to 50,000. NFS metadata load drops by 3 orders of magnitude. GPU utilization rises to 88%.
We are not saying NFS is the cause of every storage bottleneck or that converting to large shards always solves it. The point is that storage bottlenecks have specific causes that require specific fixes, and diagnosing the cause takes instrumentation, not intuition.
The Data Loader Configuration Interaction
Data loader worker count, prefetch factor, and pin_memory settings interact with storage throughput in ways that are not obvious. More workers reduce per-batch latency variance but increase total I/O request count, which can saturate storage I/O queue depth at high worker counts. A prefetch factor of 4 with 8 workers means up to 32 batches are being read simultaneously, which can be either beneficial (hides individual I/O latency) or harmful (saturates the storage queue and causes backpressure).
Empirically, the worker count that maximizes GPU utilization is not always the maximum supported by your CPU. We typically find that the throughput-maximizing configuration is 4 to 6 workers per GPU (not per node) for large sequential training corpora, with a prefetch factor of 2. Higher worker counts improve latency hiding but degrade storage queue behavior at high I/O concurrency.
The pin_memory=True setting copies batches to pinned CPU memory before transfer to GPU, which accelerates host-to-GPU transfer but consumes additional CPU memory. On systems where CPU memory is not the constraint, pin_memory=True is almost always beneficial. The interaction with storage: pin_memory=True does not affect storage I/O rates but does reduce the latency between batch availability and GPU ingestion, making storage bottlenecks more visible rather than less.
When the Bottleneck Moves
Fixing a storage bottleneck can expose a secondary bottleneck that was hidden. If you increase storage throughput by 30% and GPU utilization rises from 65% to 75% rather than to 90%, you have revealed a secondary constraint: likely data preprocessing (JPEG decompression, augmentation, tokenization) that was previously masked by I/O wait. This is expected and correct. The process is iterative: instrument, diagnose the current bottleneck, fix it, and re-measure.
Storage is usually not the only bottleneck in an underperforming training pipeline. But it is frequently the first bottleneck and the least instrumented one. Starting the diagnostic process with explicit storage I/O measurements, rather than assuming the bottleneck is compute or preprocessing, saves significant debugging time on long training runs where each percentage point of GPU utilization translates directly to training cost.