K8S-Schedule: Taints and Tolerations
Intro
In Kubernetes, Taints and Tolerations provide a mechanism to control Pod placement by restricting which Pods can be scheduled on specific Nodes. Unlike Node Affinity, which expresses preferences, Taints enforce restrictions by preventing unwanted Pods from running on designated Nodes unless they have the appropriate Tolerations.
What Are Taints?
A Taint is applied to a Node and acts as a repelling force that prevents certain Pods from being scheduled onto it. Each Taint consists of:
A key (e.g.,
dedicated)A value (e.g.,
gpu)An effect, which determines how the Taint behaves:
NoSchedule– Prevents scheduling unless the Pod has a matching Toleration.PreferNoSchedule– Tries to avoid scheduling Pods on the Node, but does not enforce it strictly.NoExecute– Evicts running Pods that do not have a matching Toleration.
What Are Tolerations?
A Toleration allows a Pod to "tolerate" a Node’s Taint, permitting it to be scheduled there despite the restriction. However, a Toleration alone does not force scheduling onto a tainted Node—it simply allows the Pod to bypass the restriction if necessary.
Why Use Taints and Tolerations?
Taints and Tolerations are commonly used in scenarios such as:
Dedicated Nodes – Preventing general workloads from running on Nodes reserved for specific tasks (e.g., GPU workloads).
Node Maintenance – Temporarily evicting Pods from a Node for updates or repairs.
High-Priority Workloads – Ensuring that only critical applications run on specific Nodes while blocking lower-priority workloads.
By effectively using Taints and Tolerations, Kubernetes administrators can fine-tune scheduling, improve resource management, and ensure workloads are deployed in the right locations within a cluster. In the next section, we will explore how to apply Taints and Tolerations with practical examples.
Demo
In this demo, we will explore Taints and Tolerations by:
Applying a Taint to a Node to restrict scheduling.
Creating a Pod without a Toleration (which should not get scheduled).
Creating a Pod with a Toleration (which should get scheduled on the tainted Node).
Apply a Taint to a Node
Now, let's taint all nodes so that only Pods with a matching Toleration can be scheduled on it.
$ kubectl taint nodes kind-control-plane dedicated=high-priority:NoSchedule
node/kind-control-plane tainted
$ kubectl taint nodes kind-worker dedicated=high-priority:NoSchedule
node/kind-worker tainted
$ kubectl taint nodes kind-worker2 dedicated=high-priority:NoSchedule
node/kind-worker2 tainted
Explanation of the Taint:
dedicated=high-priority: The key-value pair for the Taint.NoSchedule: Prevents scheduling unless the Pod has a Toleration.
Verify that the Taint was applied:
$ kubectl describe node kind-control-plane | grep Taint
Taints: dedicated=high-priority:NoSchedule
$ kubectl describe node kind-worker | grep Taint
Taints: dedicated=high-priority:NoSchedule
$ kubectl describe node kind-worker2 | grep Taint
Taints: dedicated=high-priority:NoSchedule
Deploy a Pod Without a Toleration (Expected to Fail)
Let's create a Pod without a Toleration and see if it gets scheduled.
apiVersion: v1
kind: Pod
metadata:
name: no-toleration-pod
spec:
containers:
- name: nginx-container
image: nginx-k8s:latest
imagePullPolicy: Never
Apply the Pod and check the status of the pod:
# Check pod status
$ kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
no-toleration-pod 0/1 Pending 0 5s <none> <none> <none> <none>
# Describe pod
$ kubectl describe pod no-toleration-pod
Name: no-toleration-pod
Namespace: default
Priority: 0
Service Account: default
Node: <none>
Labels: <none>
Annotations: <none>
Status: Pending
IP:
IPs: <none>
Containers:
nginx-container:
Image: nginx-k8s:latest
Port: <none>
Host Port: <none>
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-65sds (ro)
Conditions:
Type Status
PodScheduled False
Volumes:
kube-api-access-65sds:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 19s default-scheduler 0/3 nodes are available: 3 node(s) had untolerated taint {dedicated: high-priority}. preemption: 0/3 nodes are available: 3 Preemption is not helpful for scheduling.
From the describe message we can see that: the Pod remains in Pending state because it is not allowed to run on any pods.
Deploy a Pod With a Matching Toleration
Now, let's create a Pod with a Toleration that allows it to be scheduled on any nodes in my cluster, because I tainted them all with the same taint.
apiVersion: v1
kind: Pod
metadata:
name: toleration-pod
spec:
tolerations:
- key: "dedicated"
operator: "Equal"
value: "high-priority"
effect: "NoSchedule"
containers:
- name: nginx-container
image: nginx-k8s:latest
imagePullPolicy: Never
Apply the file and check if the pod is scheduled:
$ kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
toleration-pod 1/1 Running 0 3m8s 10.244.1.10 kind-worker2 <none> <none>
Remove the Taint
If you want to remove the Taint from the Node:
$ kubectl taint nodes kind-control-plane dedicated=high-priority:NoSchedule-
node/kind-control-plane untainted
$ kubectl taint nodes kind-worker dedicated=high-priority:NoSchedule-
node/kind-worker untainted
$ kubectl taint nodes kind-worker2 dedicated=high-priority:NoSchedule-
node/kind-worker2 untainted
This removes the Taint, allowing all Pods to be scheduled normally.
Conclusion
In this article, we learned how to:
Apply a Taint to restrict Pod scheduling.
Create a Pod without a Toleration, which remains pending.
Create a Pod with a Toleration, which gets scheduled on the tainted Node.
Remove the Taint to restore normal scheduling.