AI Accelerator Overview: Architecture, Generations & Selection

Abstract: Choosing an accelerator is not about "newest/most expensive" — it is about matching your workload type to the three pillars. This article builds the analysis framework first, then walks through the evolution logic of mainstream GPUs, and finally gives a selection matrix for training, inference, and scientific computing with engineering judgment.

1. The Framework: Three Pillars of an Accelerator

Any accelerator's paper spec can be compressed into three numbers:

① Compute TFLOPS   —— floating-point ops per second (compute density)
② Memory bandwidth TB/s —— data movement per second (data supply)
③ Interconnect bandwidth GB/s —— exchange with other GPUs per second (scalability)

Engineering iron law: training throughput = min(compute supply, data supply, communication supply). All three must match; any short board becomes the real bottleneck — buy a compute monster with weak memory bandwidth and the GPU just starves.

2. The Three Pillars, Broken Down

2.1 Compute: Tensor Cores and Precision

• Modern GPUs do matrix multiply on Tensor Cores; key metrics split by precision: FP64 (scientific) / FP32 / TF32 / BF16 / FP8 / FP4.
• BF16 is the training workhorse: FP32-equivalent dynamic range, stable training; FP8 is the emerging training/inference compromise.
• Evaluate compute by TFLOPS at your target precision, not the marketing FP16 peak (conversions: FP16 ≈ 2×FP8, FP8 ≈ 2×FP4; vendor conventions differ — always align precision before comparing).

2.2 Memory Bandwidth: The Real Lever for Training Throughput

Matrix multiply has extremely high arithmetic intensity; GPUs spend most of their time waiting on data. Memory bandwidth sets the per-token cost:

• Throughput ≈ memory bandwidth ÷ weight bytes read per token. As models grow, weight reads grow linearly; insufficient bandwidth directly caps throughput.
• This is why H100's most valuable upgrade over A100 is HBM3: 3.35 TB/s vs 2.0 TB/s (+67%) — a bigger direct win for training throughput than the ~3× compute bump.

2.3 Interconnect: From NVLink to Scale-Up

• NVLink (in-node): 900 GB/s on H100, 600 GB/s on A100. TP depends on it; 8-GPU full mesh is standard.
• NVSwitch (in-chassis): 8×H100 fully connected via NVSwitch; peer bandwidth between any two GPUs.
• Scale-Up networks (cross-chassis): NVLink domain extension (NVLink72 / rack-scale interconnect) turns "nodes" into "racks" — the next stop for 100-GPU training; IB/RoCE remains the backbone of 10K-scale Scale-Out.

3. Generational Evolution: A100 → H100 → H200 → B200

Spec	A100 80G	H100 SXM 80G	H200 141G	B200 192G (reference)
FP16/BF16 dense TFLOPS	312	989	989	~4500*
Memory	80 GB	80 GB	141 GB	192 GB
Memory bandwidth	2.0 TB/s	3.35 TB/s	4.8 TB/s	8.0 TB/s*
NVLink	600 GB/s	900 GB/s	900 GB/s	1.8 TB/s*
Process	7nm	4N	4N	custom 4NP
TDP	400 W	700 W	700 W	~1000 W*

* Engineering references; verify against the latest official spec sheet before procurement.

Evolution logic:

1. A100→H100: ~3× compute, 1.67× bandwidth, FP8 introduced — direct training win.
2. H100→H200: more memory, more bandwidth (141 GB / 4.8 TB/s), same compute — pure bandwidth upgrade; long-context and large-batch inference benefit most.
3. H200→B200: compute and bandwidth roughly double, TDP approaches 1kW — cooling and power become new constraints; facility readiness (liquid cooling) becomes a prerequisite for selection for the first time.

4. Selection Matrix: Decide by Workload

Workload	Most sensitive metric	Recommendation	Anti-pattern
LLM pretraining (1000-GPU)	Interconnect + memory bandwidth	H100/H200 high-bandwidth models; Scale-Up first	Chasing FP8 peaks while ignoring cross-node network
Long-context fine-tuning	Memory capacity	H200 (141 GB)-class	Forcing 80 GB GPUs → ZeRO-3 communication blowup
Inference (high concurrency)	Memory bandwidth + latency	Dedicated inference cards or high-bandwidth flagships	Using training flagships for low-concurrency small batches (wasted bandwidth)
Scientific computing (HPC)	FP64 compute + efficiency	General-purpose compute cards (native FP64)	Training cards (FP64 stripped; simulation code crawls)
Multimodal (video/3D)	Memory capacity + bandwidth	Large-memory high-bandwidth models	Only looking at TFLOPS

5. Ecosystem Compatibility: The Hidden Selection Variable

Beyond paper specs, software ecosystem determines real utilization:

• CUDA ecosystem maturity: do frameworks, NCCL, vLLM/TensorRT, and profilers work out of the box?
• Driver/firmware lifecycle: new card = new driver = new bugs; production clusters commonly adopt new architectures 1–2 quarters late as risk control.
• Heterogeneous mixing complexity: mixed architectures mean two driver stacks, two schedulers, two operator implementations — delivery cost doubles. Avoid mixing unless there's a clear business need.
• Training/inference framework support: FlashAttention, quantization kernels, inference-engine adaptation speed determines how much of the "paper compute" is real.

6. Common Misconceptions

• "Pick a card by TFLOPS": training is bandwidth-sensitive; without memory bandwidth, TFLOPS idle.
• "New architecture to production immediately": launch drivers + framework lag; validate for ≥ 1 quarter.
• "Heterogeneous mixing saves money": two ecosystems' ops and scheduling costs usually swallow the hardware savings.
• "Use training cards for HPC": FP64 is deliberately downrated; molecular dynamics / CFD workloads get unacceptable results.

Next: once architecture and selection are settled, enter delivery — see 10K-GPU Cluster Hardware Selection & Acceptance Checklist.