13. Inference Engines
Chapter focus:
🧭 Reading Map
- Prerequisites: 12. Distributed Training — inference plans tie directly to the trained model's characteristics
- Role: closing chapter of Part 4, L4 — vLLM/TensorRT-LLM principles and production, long context and speculative decoding
- Order: engine principles (vLLM → comparisons) → production deployment → frontier (long context/speculative)
- Next: 14. Industry Apps — turn inference capability into business
Articles
Long-Context Inference: From RoPE Extrapolation to Efficient KV Cache Serving
Reproducible configs and scripts dissecting long-context inference: RoPE extrapolation (PI/NTK/YaRN), the KV cache memory math, chunked prefill and efficient serving strategies, with engineering takeaways.
Read article →
TensorRT-LLM in Practice: From Model Export to FP8 Inference Acceleration
Reproducible commands walking the TensorRT-LLM production chain: HF-to-TRT-LLM conversion, FP8 quantization, engine build flags, inference serving, and benchmark comparison, with engineering takeaways.
Read article →
Transformers in Distributed Deployment: Performance Comparison with vLLM
In 2017, Google introduced the Transformer architecture in "Attention Is All You Need", replacing recurrent networks with multi-head self-attention me
Read article →
Deep Dive into vLLM: PagedAttention & High-Performance Inference
A deep analysis of vLLM's core technology PagedAttention, revealing memory management optimization mechanisms, with a complete hands-on guide.
Read article →
vLLM Production Deployment: From Launch Flags to High-Concurrency Service Governance
Reproducible commands dissecting vLLM production deployment: launch flag semantics, throughput/latency tuning, KV cache and quantization, OpenAI-compatible serving, Prometheus monitoring and multi-replica load balancing, with engineering takeaways.
Read article →
LLM Inference Benchmarking: Throughput, TTFT & Cost in Practice
metric semantics, concurrency modeling, external load, cost.
Read article →
Speculative Decoding: Principles, Engineering & Benefit Boundaries
small-model draft + large-model verification, consistency checks, acceptance-rate judgment.
Read article →
Output Sampling Ledger: Greedy, Beam, top-k/p
Four strategy tiers, temperature & top-p calibration, where sampling sits in cost, speculative-decoding consistency.
Read article →
KV Cache Compression: GQA, Quantization & Paging
GQA/MQA sharing, KV INT8/INT4, PagedAttention & prefix sharing — 95% memory saving combined.
Read article →
MoE Inference Ledger: Routing, Load Balancing & Expert Parallelism
Sparse-activation / dense-residency memory ledger, routing-imbalance cost, All-to-All communication ledger, and prefetch/offload optimization.
Read →
Multimodal Inference Ledger: Visual Tokens & Cost
Visual tokenization (one image = 1K tokens), encoder+projection, KV/compute amplification, engineering checklist.
Read article →