Taints and tolerations are a Kubernetes mechanism for controlling how Pods schedule to the Nodes in your cluster. Taints are applied to Nodes and act as a repelling barrier against new Pods. Tainted Nodes will only accept Pods that have been marked with a corresponding toleration.

Taints are one of the more advanced Kubernetes scheduling mechanisms. They facilitate many different use cases where you want to prevent Pods ending up on undesirable Nodes. In this article, you’ll learn what taints and tolerations are and how you can utilize them in your own cluster.

How Scheduling Works

Kubernetes is a distributed system where you can deploy containerized applications (Pods) across multiple physical hosts (Nodes). When you create a new Pod, Kubernetes needs to determine the set of Nodes it can be placed on. This is what scheduling refers to.

Thescheduler considers many different factorsto establish a suitable placement for each Pod. It’ll default to selecting a Node that can provide sufficient resources to satisfy the Pod’sCPU and memory requests.

The selected Node won’t necessarily be appropriate for your deployment though. It could lack required hardware or be reserved for development use. Node taints are a mechanism for enforcing these constraints by preventing arbitrary assignation of Pods to Nodes.

Taint Use Cases

Tainting a Nodemeans it will start to repelPods, forcing the scheduler to consider the next candidate Node instead. You can overcome the taint by setting a matching toleration on the Pod. This provides a mechanism for allowing specific Pods onto the Node.

Taints are often used to keep Pods away from Nodes that are reserved for specific purposes. Some Kubernetes clusters might host several environments, such as staging and production. In this situation you’ll want to prevent staging deployments from ending up on the dedicated production hardware.

you may achieve the desired behavior by tainting the production Node and setting a matching toleration on production Pods. Staging Pods will be confined to the other Nodes in your cluster, preventing them from consuming production resources.

Taints can also help distinguish between Nodes with particular hardware. Operators might deploy a subset of Nodes with dedicated GPUs for use with AI workloads. Tainting these Nodes ensures Pods that don’t need the GPU can’t schedule onto them.

Taint Effects

Each Node taint can have one of three different effects on Kubernetes scheduling decisions:

TheNoExecuteeffect is useful when you’re changing the role of a Node that’s already running some workloads.NoScheduleis more appropriate if you want to guard the Node against receiving new Pods, without disrupting existing deployments.

Tainting a Node

Taints are applied to Nodes using the

command. It takes the name of the target Node, a key and value for the taint, and an effect.

Here’s an example of tainting a Node to allocate it to a specific environment:

node/demo-node tainted

You can apply multiple taints to a Node by repeating the command. The key value is optional - you can create binary taints by omitting it:

To remove a previously applied taint, repeat the command but append a hyphen (-) to the effect name:

node/demo-node untainted

This will delete the matching taint if it exists.

You can retrieve a list of all the taints applied to a Node using thedescribecommand. The taints will be shown near the top of the output, after the Node’s labels and annotations:

Name: demo-node

Taints: env=production:NoSchedule

…

Adding Tolerations to Pods

The example above tainteddemo-nodewith the intention of reserving it for production workloads. The next step is to add an equivalent toleration to your production Pods so that they’re permitted to schedule onto the Node.

kind: Pod

name: api

  • name: api

image: example.com/api:latest

  • key: env

operator: Equals

value: production

effect: NoSchedule

This toleration allows theapiPod to schedule to Nodes that have anenvtaint with a value ofproductionandNoScheduleas the effect. The example Pod can now be scheduled todemo-node.

  • key: has-gpu

operator: Exists

The Pod now tolerates thehas-gputaint, whether or not a value has been set.

Tolerations do not require that the Pod is scheduled to a tainted Node. This is a common misconception around taints and tolerations. The mechanism only says that a Node can’t host a Pod; it does not express the alternative view that a Pod must be placed on a particular Node. Taints are commonly combinedwith affinitiesto achieve this bi-directional behavior.

Taint and Toleration Matching Rules

Tainted Nodes only receive Pods that tolerate all of their taints. Kubernetes first discovers the taints on the Node, then filters out taints that are tolerated by the Pod. The effects requested by the remaining set of taints will be applied to the Pod.

There’s a special case for theNoExecuteeffect. Pods that tolerate this kind of taint will usually get to stay on the Node after the taint is applied. You can modify this behavior so that Pods are voluntarily evicted after a given time, despite tolerating the trait:

effect: NoExecute

tolerationSeconds: 900

A Node that’s hosting this Pod but is subsequently tainted withenv=production:NoExecutewill allow the Pod to remain present for up to 15 minutes after the taint’s applied. The Pod will then be evicted despite having theNoExecutetoleration.

Automatic Taints

Nodes areautomatically tainted bythe Kubernetes control plane to evict Pods and prevent scheduling when resource contention occurs. Taints such asnode.kubernetes.io/memory-pressureandnode.kubernetes.io/disk-pressuremean Kubernetes is blocking the Node from taking new Pods because it lacks sufficient resources.

Other commonly applied taints includenode.kubernetes.io/not-ready, when a new Node isn’t accepting Pods, andnode.kubernetes.io/unschedulable. The latter is applied tocordoned Nodesto halt all Pod scheduling activity.

These taints implement the Kubernetes eviction and Node management systems. You don’t normally need to think about them and you shouldn’t manage these taints manually. If you see them on a Node, it’s because Kubernetes has applied them in response to changing conditions or another command you’ve issued. It is possible to create Pod tolerations for these taints but doing so could lead to resource exhaustion and unexpected behavior.

Summary

Taints and tolerations are a mechanism for repelling Pods away from individual Kubernetes Nodes. They help you avoid undesirable scheduling outcomes by preventing Pods from being automatically assigned to arbitrary Nodes.

Tainting isn’t the only mechanism that provides control over scheduling behavior. Podaffinities and anti-affinitiesare a related technique for constraining the Nodes that can receive a Pod. Affinity can also be defined at an inter-Pod level, allowing you to make scheduling decisions based on the Pods already running on a Node. You can combine affinity with taints and tolerations to set up advanced scheduling rules.