k8wiz
Playground

Reserve a GPU node for GPU work with taints and tolerations

Advanced · Kubernetes · about 15 min

The cluster got a GPU node, gpu-1: 4 CPUs, 16 GiB and one nvidia.com/gpu. Nobody tainted it, so the scheduler treats it like any other node, and some web replicas landed there.

Loading the playground...

About this scenario

The cluster got a GPU node, gpu-1: 4 CPUs, 16 GiB and one nvidia.com/gpu. Nobody tainted it, so the scheduler treats it like any other node, and some web replicas landed there.

The training Job needs the GPU and 3.5 CPUs. Make gpu-1 a node only GPU work can use, without taking web below its 6 ready replicas.

Objectives

  1. Read why the training Pod is Pending
  2. Taint gpu-1 so only Pods that tolerate it land there
  3. No web Pod runs on gpu-1
  4. The training Job completes on gpu-1
  5. web keeps 6 ready replicas

What you'll practice

  • An untainted special node is just another node to the scheduler. Its spare CPU attracts ordinary Pods, and then the work it was bought for no longer fits.
  • A taint keeps Pods off a node; a toleration only lets a Pod in, it doesn't send it there. Pair the toleration with a nodeSelector or node affinity to pin the work to those nodes.
  • NoSchedule only affects new Pods. NoExecute also evicts running Pods that don't tolerate the taint.
  • Extended resources such as nvidia.com/gpu are set in limits and can't be overcommitted. A Pod asking for one can only go to a node that advertises it.

How the playground works

The cluster runs entirely in your browser: a simulated API server, scheduler, kubelets and controllers, with virtual nodes in three zones. kubectl output follows the real formats, errors use the real messages, and the add-ons (CSI storage, Traefik, Envoy Gateway, cert-manager, Flux, Argo CD) behave like their real counterparts for the common paths. It's a simulation, so some behavior is simplified. Your progress is saved in this browser.