NFS shows up in almost every AI training cluster we have evaluated. It is nearly universal as a shared storage solution for ML infrastructure because it works with everything, requires no client software beyond the kernel module, and is supported by every Linux distribution without additional packages. These are real operational advantages. The argument for a native distributed filesystem is not that NFS is broken, it is that NFS adds several layers of unnecessary overhead that accumulate into measurable throughput loss and operational risk at training scale. This post covers what we measured and where the differences actually matter.
What NFS Is and Is Not
NFS (Network File System) is a remote procedure call protocol that maps POSIX filesystem operations to network calls. When your training worker calls open(path), the NFS client sends a LOOKUP RPC to the NFS server to resolve the path, then a CREATE or ACCESS RPC, then begins data transfer. When it calls read(fd, buf, count), the client sends one or more READ RPCs to the server, each completing a round trip before the next can be issued (unless the client uses asynchronous readahead, which NFSv4 supports with some constraints).
The critical architectural characteristic of NFS is that it has a single logical server (or a small pNFS cluster) that handles all metadata operations. All LOOKUP, GETATTR, READDIR, and related operations serialize through the metadata server. This is not a bug in NFS; it is the design. NFS was built for networked workstation access to shared files, where a small number of clients access a shared NAS device. It was not designed for hundreds of worker processes issuing thousands of READDIR operations per second across a large dataset directory tree.
The Measurement Setup
We compared NFS on NVMe against Leil (native distributed filesystem on SMR) using the same 8-node training cluster on a 25GbE network. The NFS configuration used a dedicated NFS server with 8 NVMe drives, exporting via NFSv4.1 with pNFS enabled. Leil used 8 storage nodes with SMR drives, accessed via its native protocol with POSIX mount semantics. Both shared the same switch fabric. The workload was a DataLoader-style scan: 128 concurrent worker processes scanning a 50TB dataset composed of 2MB tokenized data files.
All measurements are from our internal test cluster, October 2025. These figures reflect this specific configuration and network topology; other configurations will produce different absolute numbers.
Throughput: Where NFS Performs Better Than Expected
On pure sequential read throughput, the NFS and Leil configurations were closer than we expected. NFS on NVMe delivered 4.1 GB/s aggregate throughput across 128 workers. Leil on SMR delivered 3.6 GB/s aggregate throughput. The NFS configuration was 14 percent faster on raw throughput, reflecting both the NVMe speed advantage on reads and the relatively efficient pNFS data path for large sequential reads.
This is worth noting directly: NFS on a fast backend can match or exceed a native distributed filesystem on slower drives for sequential throughput. The NVMe vs SMR hardware difference accounts for most of this gap. If the comparison were NFS on NVMe vs Leil on NVMe, the gap would close significantly. The storage medium matters more than the protocol for sequential read-dominated workloads.
Metadata Operations: Where NFS Falls Apart
The throughput number only tells part of the story. The 50TB dataset in our test was organized as a large number of files, typical of how tokenized training datasets are structured (many shards, each a few hundred MB to a few GB). Opening each file, stat-ing it, and listing directories requires metadata operations. At 128 concurrent workers each advancing through the dataset independently, the metadata operation rate was approximately 8,000 to 12,000 operations per second during active loading.
On NFS, metadata operations are serialized through the metadata server. Under this load, GETATTR latency rose from under 1ms at idle to 12 to 18ms under load. LOOKUP latency rose from under 2ms to 20 to 35ms. The workers began spending a measurable fraction of their time waiting for metadata responses rather than transferring data. Effective throughput during metadata-heavy phases dropped to 2.4 GB/s, roughly 41 percent below the theoretical peak. GPU utilization during these phases dropped from 76 percent to 61 percent.
On Leil, metadata operations are distributed across all 8 storage nodes via consistent hashing. A LOOKUP operation for a path is resolved by hashing the path to a node and making a direct request to that node. Under the same 128-worker load, LOOKUP latency remained at 1.4 to 2.2ms throughout. Effective throughput stayed within 5 percent of the sequential read peak. GPU utilization held at 74 to 77 percent.
Failure Modes Are Fundamentally Different
The operational risk profile of NFS and a native distributed filesystem diverges significantly when things go wrong. NFS has a single point of failure at the metadata server. If the NFS server becomes unavailable, all clients stall until the server recovers. NFSv4 has session recovery semantics that allow clients to reconnect and resume after a server restart, but during the outage window every training worker blocks at its next filesystem call. A training run that has been running for 60 hours and hits an NFS server failure will pause until the server recovers, with no way for the individual workers to fail over to an alternative path.
pNFS improves data availability by distributing data across multiple data servers, but metadata operations remain centralized. A pNFS deployment with metadata server failure still stalls all clients.
A native distributed filesystem with distributed metadata handles individual node failures differently. When a metadata node fails, the path lookup for affected paths routes to a replica node. The client may experience a brief reconnection delay (milliseconds to low seconds depending on failure detection timing) but does not stall the entire cluster. Data availability depends on the erasure coding or replication configuration. In our Leil configuration with 4+2 erasure coding, any two nodes can fail simultaneously without data loss; any single node failure is transparent to active read operations within the read retry window.
For week-long training runs, the mean time to a storage component event is not negligible. A cluster with 48 SMR drives running a 14-day training run is operating for 16,128 drive-hours. Drive AFRs in this class run at 1 to 3 percent annually; statistically, you expect 0.5 to 1.5 drive failures across a 14-day run at this scale. The question is not whether component failures will occur; it is whether the storage layer handles them without pausing training.
When NFS Is the Right Choice
We are not arguing that NFS is the wrong choice in every situation. There are several cases where NFS makes sense even for training infrastructure.
Small clusters where operational simplicity outweighs peak performance benefit from NFS. If you have 4 to 8 GPU nodes and a dedicated NFS server, the administrative overhead of deploying and managing a distributed filesystem is not justified by the throughput improvement. At this scale, a well-tuned NFS server with read-ahead enabled will serve most training pipelines adequately.
Development and experiment workloads where training runs are short and reproducibility matters more than throughput also fit NFS well. NFS is debuggable, well-documented, and compatible with every tool in the ML ecosystem. If your training infrastructure is shared with model evaluation, data processing, and notebook environments, a single NFS share is simpler to manage than a distributed filesystem with a POSIX mount on each host.
The switch to a native distributed filesystem makes operational sense when the cluster grows to 16 or more GPU nodes, when training runs extend to multiple days, when metadata-heavy workloads cause measurable GPU underutilization, and when single-node storage failure tolerance is a real operational requirement rather than a theoretical concern. Those criteria describe most serious LLM pre-training deployments.