GPU Resource Partitioning in Practice: MIG, vGPU and Multi-Tenant Sharing

Scenario: The most common waste in multi-tenant clusters — an A100 below 50% utilization 90% of the time: training jobs intermittently saturate, inference jobs only use 10% of memory. GPU resource partitioning answers "how to divide one card among multiple people/jobs" — the next layer below Slurm Heterogeneous Scheduling: that article covers "how whole cards are allocated to jobs"; this one covers "how to slice a single card."

1. Three Partitioning Routes

Route	Principle	Isolation strength	Best for
MIG (hardware)	A100/H100 physically split into instances, compute+memory hard-isolated	Strongest (near-independent GPUs)	mixed training/inference, security-sensitive multi-tenancy
vGPU (virtualization)	NVIDIA vGPU Manager virtualizes a physical GPU into multiple vGPUs	Medium (memory isolated, compute shared or throttled)	virtual desktops / light multi-tenant workloads
Soft partitioning (memory/time slice)	containerized + CUDA limits (memory cap, MPS time slicing)	Weak (shared compute, mutual interference)	test environments, low-cost sharing

In one sentence: choose MIG for "real isolation"; soft partitioning for "flexible and light"; vGPU sits between and depends on a virtualization platform (vSphere/OpenStack).

2. MIG Deep-Dive (Focus)

2.1 What MIG Is

MIG (Multi-Instance GPU) on A100/H100 (Ampere+ architecture) slices a GPU into multiple compute+memory physically isolated instances, each behaving like a near-independent GPU:

A100-80G common slicing example
┌──────────────────────────────┐
│  MIG 2g.20gb  MIG 2g.20gb   │  ← 2 small instances
├──────────────────────────────┤
│  MIG 3g.40gb                 │  ← 1 medium instance
├──────────────────────────────┤
│  MIG 7g.80gb                 │  ← whole-card single instance
└──────────────────────────────┘

2.2 Key Features and Constraints

• Compute+memory dual isolation: one instance OOM/crashes without affecting others — something soft partitioning cannot do;
• Instance combinations are constrained: the set of simultaneously active instance profiles is fixed by GPU spec (1g/2g/3g/4g/7g tiers); profiles cannot be freely combined;
• NCCL communication limits: MIG instances communicate over PCIe/network, not full NVLink bandwidth — cross-instance data parallelism performs poorly; MIG instances are unsuitable for tensor parallelism between them;
• Driver/container requirements: requires NVIDIA container runtime + a driver supporting MIG; different instances can run different CUDA versions (containerized).

2.3 Typical Usage

Scenario	Slicing suggestion
Multiple inference service replicas	1g/2g small instances × N, each running a vLLM replica
Small training jobs	3g/4g medium instances, tenants train independently
Critical large jobs	7g whole-card instance, guaranteed exclusive

3. Slurm Integration: MIG and gres

Slurm 23.02+ natively supports MIG via gres declarations. Config points:

# declare MIG capability in slurm.conf
NodeName=gpu01 Gres=gpu:8,mig:2g.20gb:4,mig:3g.40gb:2
# request a MIG instance from a job
#SBATCH --gres=mig:2g.20gb:1

Engineering notes:

• Regular jobs requesting gpus consume whole-card gres and conflict with MIG jobs — use separate partitions/separate gres keywords to distinguish whole-card jobs from sliced jobs;
• MIG instance "health checks" are more complex than whole cards (instance-level fault detection); schedulers and monitoring must work at instance granularity;
• For mixed deployment follow the "partition nodes by type" principle from Slurm Heterogeneous Scheduling: plan whole-card nodes and MIG nodes separately, so the two job classes don't trample each other on a node.

4. Pitfalls and Tuning (Field Experience)

1. MIG is not a panacea: large-model training (70B+ multi-GPU parallel) almost always needs whole cards + NVLink; hard slicing actually cuts communication bandwidth. Partitioning serves "small jobs + inference + multi-tenancy", not "large training".

2. Memory isolation ≠ performance isolation (soft route): under time-slice/soft-memory limits, one busy job can slow other tenants on the same card several-fold — soft partitioning suits only low-sensitivity scenarios; production multi-tenancy prefers MIG.

3. Failure domain is a hidden cost: MIG depends on the whole card's hardware; whole-card failure = all instances on it fail together; "instance isolation" is runtime isolation, not hardware redundancy. Critical services still need replicas across physical cards.

4. The MIG-NCCL trap: cross-instance NCCL takes the slow path; hybrid parallelism (in-instance parallel + cross-instance communication) hits a performance cliff. When designing a parallel scheme confirm the communication path first — don't expect NVLink full speed between MIG instances.

5. Driver upgrades affect instance topology: MIG profiles, combinations and drivers are tightly coupled; upgrading drivers requires re-validating instance configs and rolling job verification — treat upgrade windows as maintenance downtime.

5. Quantified Results and Retrospective

Metric	Reference definition
Utilization gain	single card from ~40% to ~70–85% (partitioning + backpressure scheduling)
Tenant count	one A100-80G serves 2–7 isolated tenants simultaneously (by instance tier)
Isolation guarantee	compute/memory hard isolation; instance crash doesn't affect co-resident instances
Cost	no cross-instance NVLink, higher config & ops complexity

Note: figures are engineering-order-of-magnitude estimates; actuals follow your own job profile (job-size distribution, peak characteristics) — stat the job-size distribution first, then decide whether to slice.

Takeaway: GPU partitioning trades operational complexity for utilization. Job profile dominated by small jobs/inference → MIG pays off; dominated by large training → whole cards + scheduling backpressure is better. There is no one partitioning scheme, only the job profile it fits.

Engineering Takeaways

• GPU partitioning is the depth dimension of Slurm Heterogeneous Scheduling: that article solves "how to allocate cards," this one solves "how to slice cards fine" — together they make complete heterogeneous resource management;
• Strongly linked to 13 · Private LLM Serving: inference multi-replicas + MIG small instances is the standard combo for cutting private-serving costs;
• Beyond partitioning, the "physical exam" of a heterogeneous cluster (utilization, queueing, faults) is an ops basic — next planned: "Heterogeneous Cluster Monitoring & Job-Efficiency Analysis."