LLM pre-training infrastructure conversations focus almost entirely on compute. GPU hours dominate the budget, and they dominate the engineering discussion. Storage gets treated as a commodity: provision enough NVMe to hold the dataset and checkpoints, pay the premium, move on. This works until the dataset grows to hundreds of terabytes, the training run extends to weeks, and the storage line item starts representing 15 to 20 percent of total infrastructure cost. At that point the question becomes: what are we actually paying for?
The Access Pattern That Drives Everything
LLM pre-training has a very specific storage access pattern. The training data pipeline reads tokenized training examples sequentially, in large buffers, across all dataset files during each epoch. There is minimal random access: the DataLoader workers advance a file pointer and read the next batch. Checksums and index lookups involve some small random reads, but these are a negligible fraction of total I/O by bytes transferred.
NVMe drives earn their cost premium primarily through random I/O performance. A high-end NVMe drive delivers sub-100 microsecond random read latency at hundreds of thousands of IOPS. That capability is useful for databases, inference-serving KV caches, and operating system workloads. For sequential streaming of tokenized training data, that same drive delivers roughly 3.5 GB/s of sequential read bandwidth. A high-density SMR HDD at one-third the cost per terabyte delivers approximately 250 to 280 MB/s sequential read throughput. With 12 drives striped across a node, that is 3 to 3.4 GB/s aggregate, before network. The IOPS advantage of NVMe is irrelevant; the sequential bandwidth is comparable.
This is the core economics argument for using high-density HDDs for training data storage. Not that SSDs are bad, but that you are paying for a performance characteristic that your workload does not use.
Where the Cost Gap Actually Comes From
The raw hardware cost differential between NVMe and high-density SMR drives is significant and has been widening, not narrowing, over the past several years. In early 2026, 20TB SMR drives are available in volume at approximately $90 to $120 each, representing roughly $5 to $6 per terabyte of raw capacity. Enterprise NVMe drives at equivalent capacity cost $250 to $350 or more per terabyte of raw capacity, depending on endurance tier.
The gap is compounded by density: a 2U storage node holding 24 SMR drives at 20TB each stores 480TB raw. A 2U NVMe all-flash array of comparable footprint typically stores 50 to 100TB raw. Rack space costs money in data centers; density matters. At $500 per terabyte per month all-in for a small colocation deployment, the density difference alone shifts the economics substantially.
Power consumption per terabyte also favors high-density HDDs for cold-to-warm storage tiers. A 20TB SMR drive draws approximately 7 to 9 watts during sustained reads. A comparable NVMe drive draws 5 to 10 watts but stores 2TB instead of 20TB. Per-terabyte power consumption is 5 to 8 times lower for SMR at current drive densities. This matters at scale: a 500TB raw SMR cluster draws roughly 25 to 40 drives-worth of power; the equivalent NVMe cluster requires 5 to 10 times more drives for the same capacity, multiplying the power draw accordingly.
The Throughput Argument Is More Nuanced Than It Appears
We need to be careful about oversimplifying the throughput comparison. Sequential read bandwidth from a high-density HDD cluster can absolutely keep up with typical GPU training data consumption rates. But there are workload configurations where this is less straightforward.
Large training clusters with very high GPU-to-storage ratios need proportionally more storage nodes to maintain throughput. If a cluster runs 64 GPUs consuming data at 12 GB/s aggregate, a 6-node SMR cluster delivering 2.5 GB/s per node aggregate will not keep pace. You need either more nodes or a network-level prefetch buffer (local NVMe cache tier) to absorb the burst. Dimensioning the storage cluster to the GPU count is work that an all-flash purchase sidesteps, at the cost of 3 to 5 times the price per terabyte.
Dataset shuffling is a second nuance. Random-shuffle dataloaders that read arbitrary file offsets across a large corpus turn sequential access into random access, and random access on HDDs is painful: 8 to 12 millisecond seek latency per operation. If your DataLoader is reading individual small examples at random positions from disk, HDD performance will be poor regardless of the filesystem. The standard mitigation is to pre-shuffle and tokenize the dataset into large sequential chunks so that each read fetches 1 to 4 MB of contiguous data. Most production training pipelines already do this; if yours does not, that is something to address before evaluating storage economics.
Two-Tier Architecture: What Actually Works
The storage architecture that makes most sense for training data at scale is not a pure choice between NVMe and HDD. It is a two-tier system: large-capacity SMR storage for the canonical dataset and checkpoint archive, with a smaller NVMe cache tier on each compute node for hot-window data that will be read repeatedly within an epoch.
This architecture has been used in ML training infrastructure for several years under various names: dataset caching, local shuffle buffer, prefetch ring. The key insight is that the hot tier does not need to hold the full dataset, only the portion being actively read by the current batch of data loader workers. A well-sized local NVMe cache on each compute node absorbs the latency of the underlying network storage and allows the GPU to always have data available, while the canonical dataset lives on high-density SMR storage at 60 percent lower cost per terabyte.
In our early-access configurations, we recommend sizing the local cache tier at two to four times the batch buffer that the DataLoader workers maintain in memory. For a cluster training a 13B parameter model with 128 workers and 4MB per-worker read buffers, the useful local cache per node is roughly 2 to 8 GB, which fits comfortably on even a small NVMe drive. The SMR cluster serves the full dataset at sequential throughput; the local cache eliminates the tail latency from individual slow reads.
Checkpoint Storage Is Simpler Than Training Data
Training checkpoints have an even more favorable access pattern for HDD storage than the training data itself. Checkpoints are written once, infrequently, and then read back only when a training run needs to resume from a failure or when a specific model state is needed for evaluation. The write is sequential, large, and predictable. The reads are infrequent and sequential.
The only real question for checkpoint storage is whether the write bandwidth of the SMR cluster is sufficient to complete the checkpoint before the training run needs to resume. For a 40GB checkpoint (typical for a 7B parameter model in bf16), the Leil cluster can complete a band-aligned write in under 30 seconds at 12 nodes. The training loop pause during checkpoint is in this range regardless of storage medium; the difference between 30 seconds on SMR and 10 seconds on NVMe is not a meaningful bottleneck in a run that takes 10 to 14 days.
Putting the Numbers Together
For a team running 200TB of training data and 20TB of checkpoint capacity at roughly $500 per TB per year all-in on NVMe (hardware depreciation, software, power, rack), the annual storage cost is approximately $110,000. The same configuration on high-density SMR storage at roughly $200 per TB per year all-in runs approximately $44,000. That $66,000 annual difference is compute-adjacent money: it buys roughly 2,000 to 4,000 additional GPU-hours depending on your cluster rate.
We are not saying the transition is free. Migration from NVMe to SMR storage requires evaluating your DataLoader's access pattern, potentially adjusting dataset pre-processing, sizing the local cache tier, and validating that checkpoint write times are acceptable for your checkpoint frequency. Those are engineering hours with a real cost. For teams that are past their first production training run and starting to see storage as a meaningful budget line, the engineering investment typically pays back within a quarter.