03. Cluster Architecture
Chapter focus: from capacity planning, network architecture (IB/RoCE) to MPI parallel programming and the L40S cluster delivery series — the core chapter of Part 1, covering the full path from design to production.
Outcomes
- Cluster capacity planning: from workload to racks and power
- IB/RoCE network architecture, deployment and operations (incl. 10K-GPU parameter plane)
- MPI environment setup and parallel programming
- 384-GPU L40S IB cluster end-to-end delivery
🧭 Reading Map
- Prerequisites: 02. Hardware & Chips — hardware topology and selection come first
- Role: core chapter of Part 1, L1 — planning (capacity/network) → network (IB/RoCE architecture & ops) → parallel programming (MPI) → project delivery (L40S series)
- Order: capacity planning → network architecture (10K fat-tree / RoCE) → network deployment & ops → MPI practice → L40S series
- Next: 04. Performance Benchmarking — set the baseline with reproducible benchmarks
Articles
Cluster Planning & Design
Network: IB & RoCE
A 10K-GPU IB Parameter Plane: 3-Tier Fat-Tree with 1,250 HGX H100
10,000 GPUs, 320 Leaf + 320 Spine + 256 Core non-blocking fat-tree: topology, POD internals and packet paths.
Read →
ROCE AI Fabric Architecture Guide
A deep dive into the 400G Ethernet solution for massive GPU clusters, featuring Adaptive Routing (AR), Rail-Optimized topology, and SONiC configuration.
Read →
Mellanox IB Operations & Tuning Guide
A comprehensive guide covering OFED installation, switch firmware upgrades, MPI/OSU benchmarking, and BER troubleshooting.
Read →
HPC-X ClusterKit IB Network Testing Guide
A practical guide to using NVIDIA HPC-X ClusterKit for InfiniBand bandwidth and latency validation.
Read →
InfiniBand Performance Tuning for AI Workloads
Production-grade guide to optimizing InfiniBand for distributed AI training, covering NCCL tuning, GPUDirect RDMA, NUMA affinity, and real benchmarking methodology.
Read →
UFM Enterprise 6.16 Bare Metal Installation
Installation guide for NVIDIA UFM Enterprise 6.16 on RHEL 7.9 bare metal, covering prerequisites, dependencies, and service startup.
Read →
GPU Training Network Troubleshooting: Packet Loss, Congestion & NCCL Timeouts
Three high-frequency faults: symptom/diagnosis/root cause/fix + decision tree.
Read →
Collective Communication: From MPI to NCCL
Communicators & primitives, per-primitive traffic, MPI vs NCCL positioning, NCCL topology-aware architecture.
Read →
MPI Parallel Programming in Practice
MPI Getting Started: Installing oneAPI and Your First Parallel Environment
offline install of Intel oneAPI Base + HPC Toolkit, silent commands, environment loading, and common pitfalls.
Read article →
MPI Hello World in Practice
compile, single-node multi-rank, and multi-node runs from hello.c, with rank output and fabric flags explained.
Read article →
MPI Programming in Practice: from Hello to Parallel Reduce
compilable code from point-to-point Send/Recv to MPI_Reduce, implementing parallel summation with verification.
Read article →
MPI Point-to-Point in Practice
ring relay with MPI_Sendrecv and a ping-pong latency/bandwidth probe, with cross-node fabric notes.
Read article →
MPI Collectives in Practice: from Bcast to Allreduce
broadcast, parallel sum, and Allreduce global statistics with compilable code, mapped to the HPL P×Q grid.
Read article →
MPI + Slurm in Practice: Scheduling and Performance Tuning
srun/sbatch submission, process layout and pinning, and an HPL tuning retrospective from 1 to 80 nodes.
Read article →
Project: 384-GPU L40S IB Cluster
384-GPU L40S IB Cluster: Engineering Thinking
48 nodes × 8×L40S + 4×200G HDR IB: three-plane separation, rail topology, lossless IB and the acceptance philosophy.
Read →
IB Three-Plane Separation: PKey, SL/VL & IPoIB
Physical isolation of compute/storage/management, PKey 0x100/0x200 isolation, OpenSM failover and verification commands.
Read →
Rail Topology + 1.5:1 Oversubscription
Design a 48-node cluster like a 10K-GPU one: rail-optimized Fat-Tree, 2×200G breakout, SM auto-discovery and growth.
Read →
MLNX_OFED & VPI LINK_TYPE in Practice
Switch dual-mode adapters to IB: mstconfig burning, driver stack, ibstat/ibv_devinfo verification and traps.
Read →
Kickstart + Mirrors + Ansible Delivery
Version consistency across 48 nodes: bulk install, internal YUM/pip mirrors, Ansible rollout and acceptance.
Read →
openEuler Kernel & System Tuning
The BIOS checklist, kernel parameters, NUMA and IRQ affinity — each step verified.
Read →
CUDA Stack Pinning & Bulk Install
Driver/CUDA/cuDNN/NCCL version matrix, internal-mirror installs, NCCL knobs and verification.
Read →
L40S Without NVLink: NCCL & Node Communication
Three implicit constraints, NUMA affinity, busbw baselines and an actionable tuning checklist.
Read →
Acceptance & Performance Baselines
Layered verification, archived baselines and layer-by-layer debugging — falsify, don't just run.
Read →
Power & Cooling: The 4.6kW Ledger
Per-node power math, rack density, power sequencing and full-load verification.
Read →