Technical writing on SMR drives, distributed filesystems, checkpoint strategies, and the real economics of petabyte-scale AI data pipelines.
We benchmarked a 200TB usable SMR-native Leil cluster against an NVMe all-flash reference build. Here is what actually happened to throughput, checkpoint latency, and total cost per terabyte.
Object storage APIs require code changes. NFS adds latency layers. POSIX-native filesystems let AI training code run unchanged. We explain why the filesystem layer matters more than most teams realize.
LLM pre-training pipelines read hundreds of terabytes sequentially. All-flash storage is fast but expensive, and it is fast in ways that do not matter for sequential reads. Here is how to think about this tradeoff.
Checkpoints are written once and read occasionally. The latency profile of NVMe adds nothing here. We make the case for why high-density HDDs, with the right filesystem, beat flash on this workload.
SMR drives are not just slower hard drives. They have a specific write geometry that generic filesystems handle poorly. This post explains how SMR tracks work and what it takes to use them efficiently.
NFS is easy to deploy and nearly universal. It is also built on a 1984 protocol that adds latency layers your training data pipeline does not need. We measured both against the same GPU workload.
Compute dominates the AI training budget discussion. Storage gets optimized last. This is a line-item breakdown of storage costs across a mid-size pre-training run: hardware, software, and operational overhead.
At petabyte scale, every architectural decision multiplies. We walk through the key design choices in Leil's distributed architecture: erasure coding ratios, metadata replication strategy, and failure domain design.
Checkpoint frequency is a tradeoff between recovery time and training overhead. Checkpoint storage is a tradeoff between speed and cost. This post covers both dimensions for clusters running week-long training runs.
GPU utilization drops below 70% and everyone looks at the model, the data loader, the batch size. The actual bottleneck is often the storage layer. Here is how to diagnose and fix it.
Estonia is an unusual place to build storage infrastructure. Cold climate, strong engineering culture, EU regulatory clarity, and a government that takes digital infrastructure seriously. A few thoughts on building deep tech from a small country.
The $/TB gap between NVMe and high-density HDDs was $40 in 2022. It is $55-70 now and widening. This post looks at the capacity density trajectory and what it means for AI storage architecture decisions made today.