Advanced Storage Features

Bottom line: Storage bottlenecks rarely come from insufficient aggregate bandwidth — they come from three structural faults: tenants stealing IO from each other, metadata hotspots overwhelmed by small-file storms, and checkpoints mixed with cold data on the same tier. This article walks through "isolate first, govern second, tier last" as the production upgrade path for large AI/HPC clusters.

1. Multi-Tenant IO Isolation and QoS

In a shared cluster, one tenant's full checkpoint flush can saturate the entire storage link and stall every other tenant's training. The goal of QoS is not to cap throughput but to give each tenant a committable bandwidth/IOPS floor while allowing peak bursts.

1.1 Three Layers of QoS Controls

Layer	Mechanism	Granularity	Typical Tools
Client	Rate limiting at compute nodes	Process / container	tc, cgroup blkio, ionice
File system	Server-side quota & throttling	Directory / user / tenant	BeeGFS beegfs-ctl quota, Lustre QoS (ladvise), GPFS storage pools
Network	Storage-network bandwidth isolation	Traffic class	RoCE PFC/ETS, InfiniBand SL, switch QoS queues

The recommended production combo is server-side file-system throttling + client-side cgroup as a backstop:

• Server-side throttling protects global fairness (one tenant cannot take down the shared metadata service);
• Client-side cgroup prevents tasks on the same node from starving each other internally;
• Never rely on client-side limiting alone — clients can migrate across nodes and containers.

1.2 Key Parameters and Metrics

• Bandwidth metric: QoS must be based on observable real-time throughput, not static config. Training writes a checkpoint every 30–60 minutes; instantaneous demand is 5–10× the average. Quotas must allow bursting, or they are equivalent to forced throttling.
• IOPS metric: Small-file workloads (data preprocessing, logs) are IOPS-bound, not bandwidth-bound. Metadata-service IOPS is the hard ceiling.
• Isolation granularity: Model tenants as "a set of directories + a set of users/groups", not as single client IPs — training tasks drift across nodes.

# BeeGFS per-tenant bandwidth quota example (by user group)
beegfs-ctl --quota --addquota --storage --ugid 20001 --limitby 8G --timelimit 3600
# Inspect real-time tenant usage
beegfs-ctl --quota --list --storage --ugid 20001

2. Metadata Hotspot Governance

Metadata performance decides how many files a task can open concurrently. The "massive small-file" shape of AI training datasets (hundreds of thousands to tens of millions of sample files) is the most common source of metadata hotspots.

2.1 Where Hotspots Come From

1. Poor directory fan-out: datasets organized in a single flat layer by label/date — a single directory with millions of files degrades hashing to linear scans;
2. File creation storms: parallel prefetch in DataLoaders has every worker open/stat simultaneously, saturating the metadata service;
3. Periodic scans: monitoring and archival jobs stat the whole tree daily, flushing every cache.

2.2 A Four-Step Governance Checklist

• Directory fan-out: layer by label/subset/first-two-chars, keep files per directory under ~10K (1K–10K is the common guidance for BeeGFS/Lustre);
• Prefetch alignment: switch DataLoaders to torchdata/WebDataset tar/shard sequential reads — turning "millions of random small-file reads" into "large sequential reads" typically removes 80% of metadata pressure;
• Metadata acceleration: metadata services on NVMe + NVRAM write cache; enable mdstats/ladvise metadata prefetch;
• Stagger scans: run full-stat monitoring/archival during training lulls (e.g. between checkpoints), and use batch interfaces to reduce per-file RPCs.

3. Checkpoint and Cold-Data Tiering

3.1 Checkpoint Bandwidth Budget

A 384-GPU cluster full checkpoint is ~0.5–2 TB. At a 20 GB/s storage allocation, a full save costs 30–100 seconds — training must pause. Production-standard practice:

• Incremental saves: write only deltas (sharded optimizer states) every N steps; full checkpoints go to lulls;
• Async staging: checkpoints land on local node NVMe first, then flush to the parallel file system asynchronously;
• Budget formula: checkpoint size ÷ tolerable pause time = minimum bandwidth required. Negotiate SLAs from this number, not from "bigger is better."

3.2 Cold-Data Tiering

Data lifecycles naturally split into three tiers; mixing them is the biggest hidden waste:

Tier	Medium	Use	Lifecycle
L1 hot	Cluster NVMe / local compute disks	Training set, active checkpoints	Days
L2 warm	Parallel FS (BeeGFS/Lustre/GPFS)	All training data, historical checkpoints	Months
L3 cold	Object storage (S3-compatible) / tape	Archival, non-reproducible experiment outputs	Years

Implementation: policy-based migration between L1 and L2 (e.g. demote to read-only L2 after 3 days untouched); L2→L3 via object-storage Lifecycle rules tagged by directory. Archived objects must carry metadata manifests (experiment name, code version, data version) — otherwise they become unusable data in three months.

4. Production Rollout Order

1. Build storage observability first: per-tenant bandwidth/IOPS/latency dashboards (beegfs-ctl, Grafana);
2. Then tenant QoS: server-side quotas + client cgroup backstop; run 72 hours to verify no mutual starvation;
3. Then metadata governance: small-file packaging + directory fan-out + NVMe metadata;
4. Finally tiering: incremental checkpoints + L1/L2/L3 auto-migration.

Success criterion: committability. If every tenant can answer "how much bandwidth and latency can I get right now" at any moment, that matters more than any peak benchmark — committability is the core output of large-cluster storage engineering.