AI Storage Systems

Parallel File Systems

• GPFS (Spectrum Scale): High-performance parallel file system by IBM.
• Lustre: Open-source high-performance file system used in Supercomputers.

Storage Tiering

• Hot Data (NVMe SSD)
• Warm Data (SATA SSD)
• Cold Data (HDD/Tape)

AI Challenges

• Small file random reads (ImageNet/Audio)
• Fast Checkpoint writing (LLM Fault Tolerance)