Karpenter – Concepts

V1.5: NodeClasses

Mon, 01 Jan 0001 00:00:00 +0000

Node Classes enable configuration of AWS specific settings. Each NodePool must reference an EC2NodeClass using spec.template.spec.nodeClassRef. Multiple NodePools may point to the same EC2NodeClass.

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
 name: default
spec:
 template:
 spec:
 nodeClassRef:
 group: karpenter.k8s.aws
 kind: EC2NodeClass
 name: default
---
apiVersion: karpenter.k8s.aws/v1
kind: EC2NodeClass
metadata:
 name: default
spec:
 kubelet:
 podsPerCore: 2
 maxPods: 20
 systemReserved:
 cpu: 100m
 memory: 100Mi
 ephemeral-storage: 1Gi
 kubeReserved:
 cpu: 200m
 memory: 100Mi
 ephemeral-storage: 3Gi
 evictionHard:
 memory.available: 5%
 nodefs.available: 10%
 nodefs.inodesFree: 10%
 evictionSoft:
 memory.available: 500Mi
 nodefs.available: 15%
 nodefs.inodesFree: 15%
 evictionSoftGracePeriod:
 memory.available: 1m
 nodefs.available: 1m30s
 nodefs.inodesFree: 2m
 evictionMaxPodGracePeriod: 60
 imageGCHighThresholdPercent: 85
 imageGCLowThresholdPercent: 80
 cpuCFSQuota: true
 clusterDNS: ["10.0.1.100"]
 # Optional, dictates UserData generation and default block device mappings.
 # May be ommited when using an `alias` amiSelectorTerm, otherwise required.
 amiFamily: AL2

 # Required, discovers subnets to attach to instances
 # Each term in the array of subnetSelectorTerms is ORed together
 # Within a single term, all conditions are ANDed
 subnetSelectorTerms:
 # Select on any subnet that has the "karpenter.sh/discovery: ${CLUSTER_NAME}"
 # AND the "environment: test" tag OR any subnet with ID "subnet-09fa4a0a8f233a921"
 - tags:
 karpenter.sh/discovery: "${CLUSTER_NAME}"
 environment: test
 - id: subnet-09fa4a0a8f233a921

 # Required, discovers security groups to attach to instances
 # Each term in the array of securityGroupSelectorTerms is ORed together
 # Within a single term, all conditions are ANDed
 securityGroupSelectorTerms:
 # Select on any security group that has both the "karpenter.sh/discovery: ${CLUSTER_NAME}" tag
 # AND the "environment: test" tag OR any security group with the "my-security-group" name
 # OR any security group with ID "sg-063d7acfb4b06c82c"
 - tags:
 karpenter.sh/discovery: "${CLUSTER_NAME}"
 environment: test
 - name: my-security-group
 - id: sg-063d7acfb4b06c82c

 # Optional, IAM role to use for the node identity.
 # The "role" field is immutable after EC2NodeClass creation. This may change in the
 # future, but this restriction is currently in place today to ensure that Karpenter
 # avoids leaking managed instance profiles in your account.
 # Must specify one of "role" or "instanceProfile" for Karpenter to launch nodes
 role: "KarpenterNodeRole-${CLUSTER_NAME}"

 # Optional, IAM instance profile to use for the node identity.
 # Must specify one of "role" or "instanceProfile" for Karpenter to launch nodes
 instanceProfile: "KarpenterNodeInstanceProfile-${CLUSTER_NAME}"

 # Each term in the array of amiSelectorTerms is ORed together
 # Within a single term, all conditions are ANDed
 amiSelectorTerms:
 # Select on any AMI that has both the `karpenter.sh/discovery: ${CLUSTER_NAME}`
 # AND `environment: test` tags OR any AMI with the name `my-ami` OR an AMI with
 # ID `ami-123` OR an AMI with ID matching the value of my-custom-parameter
 - tags:
 karpenter.sh/discovery: "${CLUSTER_NAME}"
 environment: test
 - name: my-ami
 - id: ami-123
 - ssmParameter: my-custom-parameter # ssm parameter name or ARN
 # Select EKS optimized AL2023 AMIs with version `v20240703`. This term is mutually
 # exclusive and can't be specified with other terms.
 # - alias: al2023@v20240703

 # Optional, each term in the array of capacityReservationSelectorTerms is ORed together.
 capacityReservationSelectorTerms:
 - tags:
 karpenter.sh/discovery: ${CLUSTER_NAME}
 - id: cr-123

 # Optional, propagates tags to underlying EC2 resources
 tags:
 team: team-a
 app: team-a-app

 # Optional, configures IMDS for the instance
 metadataOptions:
 httpEndpoint: enabled
 httpProtocolIPv6: disabled
 httpPutResponseHopLimit: 1 # This is changed to disable IMDS access from containers not on the host network
 httpTokens: required

 # Optional, configures storage devices for the instance
 blockDeviceMappings:
 - deviceName: /dev/xvda
 ebs:
 volumeSize: 100Gi
 volumeType: gp3
 iops: 10000
 encrypted: true
 kmsKeyID: "1234abcd-12ab-34cd-56ef-1234567890ab"
 deleteOnTermination: true
 throughput: 125
 snapshotID: snap-0123456789
 volumeInitializationRate: 100

 # Optional, use instance-store volumes for node ephemeral-storage
 instanceStorePolicy: RAID0

 # Optional, overrides autogenerated userdata with a merge semantic
 userData: |
 echo "Hello world"

 # Optional, configures detailed monitoring for the instance
 detailedMonitoring: true

 # Optional, configures if the instance should be launched with an associated public IP address.
 # If not specified, the default value depends on the subnet's public IP auto-assign setting.
 associatePublicIPAddress: true
status:
 # Resolved subnets
 subnets:
 - id: subnet-0a462d98193ff9fac
 zone: us-east-2b
 - id: subnet-0322dfafd76a609b6
 zone: us-east-2c
 - id: subnet-0727ef01daf4ac9fe
 zone: us-east-2b
 - id: subnet-00c99aeafe2a70304
 zone: us-east-2a
 - id: subnet-023b232fd5eb0028e
 zone: us-east-2c
 - id: subnet-03941e7ad6afeaa72
 zone: us-east-2a

 # Resolved security groups
 securityGroups:
 - id: sg-041513b454818610b
 name: ClusterSharedNodeSecurityGroup
 - id: sg-0286715698b894bca
 name: ControlPlaneSecurityGroup-1AQ073TSAAPW

 # Resolved AMIs
 amis:
 - id: ami-01234567890123456
 name: custom-ami-amd64
 requirements:
 - key: kubernetes.io/arch
 operator: In
 values:
 - amd64
 - id: ami-01234567890123456
 name: custom-ami-arm64
 requirements:
 - key: kubernetes.io/arch
 operator: In
 values:
 - arm64

 # Capacity Reservations
 capacityReservations:
 - availabilityZone: us-west-2a
 id: cr-01234567890123456
 instanceMatchCriteria: targeted
 instanceType: g6.48xlarge
 ownerID: "012345678901"
 - availabilityZone: us-west-2c
 id: cr-12345678901234567
 instanceMatchCriteria: open
 instanceType: g6.48xlarge
 ownerID: "98765432109"

 # Generated instance profile name from "role"
 instanceProfile: "${CLUSTER_NAME}-0123456778901234567789"
 conditions:
 - lastTransitionTime: "2024-02-02T19:54:34Z"
 status: "True"
 type: InstanceProfileReady
 - lastTransitionTime: "2024-02-02T19:54:34Z"
 status: "True"
 type: SubnetsReady
 - lastTransitionTime: "2024-02-02T19:54:34Z"
 status: "True"
 type: SecurityGroupsReady
 - lastTransitionTime: "2024-02-02T19:54:34Z"
 status: "True"
 type: AMIsReady
 - lastTransitionTime: "2024-02-02T19:54:34Z"
 status: "True"
 type: Ready

Refer to the NodePool docs for settings applicable to all providers. To explore various EC2NodeClass configurations, refer to the examples provided in the Karpenter Github repository.

spec.kubelet

Karpenter provides the ability to specify a few additional Kubelet arguments. These are all optional and provide support for additional customization and use cases. Adjust these only if you know you need to do so. For more details on kubelet settings, see the KubeletConfiguration reference. The implemented fields are a subset of the full list of upstream kubelet configuration arguments.

kubelet:
 podsPerCore: 2
 maxPods: 20
 systemReserved:
 cpu: 100m
 memory: 100Mi
 ephemeral-storage: 1Gi
 kubeReserved:
 cpu: 200m
 memory: 100Mi
 ephemeral-storage: 3Gi
 evictionHard:
 memory.available: 5%
 nodefs.available: 10%
 nodefs.inodesFree: 10%
 evictionSoft:
 memory.available: 500Mi
 nodefs.available: 15%
 nodefs.inodesFree: 15%
 evictionSoftGracePeriod:
 memory.available: 1m
 nodefs.available: 1m30s
 nodefs.inodesFree: 2m
 evictionMaxPodGracePeriod: 60
 imageGCHighThresholdPercent: 85
 imageGCLowThresholdPercent: 80
 cpuCFSQuota: true
 clusterDNS: ["10.0.1.100"]

Note

If you need to specify a field that isn’t present in spec.kubelet, you can set it via custom UserData. For example, if you wanted to configure maxPods and registryPullQPS you would set the former through spec.kubelet and the latter through UserData. The following example achieves this with AL2023:

apiVersion: karpenter.k8s.aws/v1
kind: EC2NodeClass
spec:
 amiSelectorTerms:
 - alias: al2023@v20240807
 kubelet:
 maxPods: 42
 userData: |
 apiVersion: node.eks.aws/v1alpha1
 kind: NodeConfig
 spec:
 kubelet:
 config:
 # Configured through UserData since unavailable in `spec.kubelet`
 registryPullQPS: 10

Note that when using the Custom AMIFamily you will need to specify fields both in spec.kubelet and spec.userData.

Pods Per Core

An alternative way to dynamically set the maximum density of pods on a node is to use the .spec.kubelet.podsPerCore value. Karpenter will calculate the pod density during scheduling by multiplying this value by the number of logical cores (vCPUs) on an instance type. This value will also be passed through to the --pods-per-core value on kubelet startup to configure the number of allocatable pods the kubelet can assign to the node instance.

The value generated from podsPerCore cannot exceed maxPods, meaning, if both are set, the minimum of the podsPerCore dynamic pod density and the static maxPods value will be used for scheduling.

Note

maxPods may not be set in the kubelet of an EC2NodeClass, but may still be restricted by the ENI_LIMITED_POD_DENSITY value. You may want to ensure that the podsPerCore value that will be used for instance families associated with the EC2NodeClass will not cause unexpected behavior by exceeding the maxPods value.

Max Pods

For small instances that require an increased pod density or large instances that require a reduced pod density, you can override this default value with .spec.kubelet.maxPods. This value will be used during Karpenter pod scheduling and passed through to --max-pods on kubelet startup.

Note

When using small instance types, it may be necessary to enable prefix assignment mode in the AWS VPC CNI plugin to support a higher pod density per node. Prefix assignment mode was introduced in AWS VPC CNI v1.9 and allows ENIs to manage a broader set of IP addresses. Much higher pod densities are supported as a result.

Windows Support Notice

Presently, Windows worker nodes do not support using more than one ENI. As a consequence, the number of IP addresses, and subsequently, the number of pods that a Windows worker node can support is limited by the number of IPv4 addresses available on the primary ENI. Currently, Karpenter will only consider individual secondary IP addresses when calculating the pod density limit.

Reserved Resources

Karpenter will automatically configure the system and kube reserved resource requests on the fly on your behalf. These requests are used to configure your node and to make scheduling decisions for your pods. If you have specific requirements or know that you will have additional capacity requirements, you can optionally override the --system-reserved configuration defaults with the .spec.kubelet.systemReserved values and the --kube-reserved configuration defaults with the .spec.kubelet.kubeReserved values.

Note

Karpenter considers these reserved resources when computing the allocatable ephemeral storage on a given instance type. If kubeReserved is not specified, Karpenter will compute the default reserved CPU and memory resources for the purpose of ephemeral storage computation. These defaults are based on the defaults on Karpenter’s supported AMI families, which are not the same as the kubelet defaults. You should be aware of the CPU and memory default calculation when using Custom AMI Families. If they don’t align, there may be a difference in Karpenter’s computed allocatable ephemeral storage and the actually ephemeral storage available on the node.

Eviction Thresholds

The kubelet supports eviction thresholds by default. When enough memory or file system pressure is exerted on the node, the kubelet will begin to evict pods to ensure that system daemons and other system processes can continue to run in a healthy manner.

Kubelet has the notion of hard evictions and soft evictions. In hard evictions, pods are evicted as soon as a threshold is met, with no grace period to terminate. Soft evictions, on the other hand, provide an opportunity for pods to be terminated gracefully. They do so by sending a termination signal to pods that are planning to be evicted and allowing those pods to terminate up to their grace period.

Karpenter supports hard evictions through the .spec.kubelet.evictionHard field and soft evictions through the .spec.kubelet.evictionSoft field. evictionHard and evictionSoft are configured by listing signal names with either percentage values or resource values.

kubelet:
 evictionHard:
 memory.available: 500Mi
 nodefs.available: 10%
 nodefs.inodesFree: 10%
 imagefs.available: 5%
 imagefs.inodesFree: 5%
 pid.available: 7%
 evictionSoft:
 memory.available: 1Gi
 nodefs.available: 15%
 nodefs.inodesFree: 15%
 imagefs.available: 10%
 imagefs.inodesFree: 10%
 pid.available: 10%

Supported Eviction Signals

Eviction Signal	Description
memory.available	memory.available := node.status.capacity[memory] - node.stats.memory.workingSet
nodefs.available	nodefs.available := node.stats.fs.available
nodefs.inodesFree	nodefs.inodesFree := node.stats.fs.inodesFree
imagefs.available	imagefs.available := node.stats.runtime.imagefs.available
imagefs.inodesFree	imagefs.inodesFree := node.stats.runtime.imagefs.inodesFree
pid.available	pid.available := node.stats.rlimit.maxpid - node.stats.rlimit.curproc

For more information on eviction thresholds, view the Node-pressure Eviction section of the official Kubernetes docs.

Soft Eviction Grace Periods

Soft eviction pairs an eviction threshold with a specified grace period. With soft eviction thresholds, the kubelet will only begin evicting pods when the node exceeds its soft eviction threshold over the entire duration of its grace period. For example, if you specify evictionSoft[memory.available] of 500Mi and a evictionSoftGracePeriod[memory.available] of 1m30, the node must have less than 500Mi of available memory over a minute and a half in order for the kubelet to begin evicting pods.

Optionally, you can specify an evictionMaxPodGracePeriod which defines the administrator-specified maximum pod termination grace period to use during soft eviction. If a namespace-owner had specified a pod terminationGracePeriodInSeconds on pods in their namespace, the minimum of evictionPodGracePeriod and terminationGracePeriodInSeconds would be used.

kubelet:
 evictionSoftGracePeriod:
 memory.available: 1m
 nodefs.available: 1m30s
 nodefs.inodesFree: 2m
 imagefs.available: 1m30s
 imagefs.inodesFree: 2m
 pid.available: 2m
 evictionMaxPodGracePeriod: 60

Pod Density

By default, the number of pods on a node is limited by both the number of networking interfaces (ENIs) that may be attached to an instance type and the number of IP addresses that can be assigned to each ENI. See IP addresses per network interface per instance type for a more detailed information on these instance types’ limits.

Note

By default, the VPC CNI allocates IPs for a node and pods from the same subnet. With VPC CNI Custom Networking, the pods will receive IP addresses from another subnet dedicated to pod IPs. This approach makes it easier to manage IP addresses and allows for separate Network Access Control Lists (NACLs) applied to your pods. VPC CNI Custom Networking reduces the pod density of a node since one of the ENI attachments will be used for the node and cannot share the allocated IPs on the interface to pods. Karpenter supports VPC CNI Custom Networking and similar CNI setups where the primary node interface is separated from the pods interfaces through a global environment variable RESERVED_ENIS, see Settings. In the common case, RESERVED_ENIS should be set to “1” if using Custom Networking.

Windows Support Notice

It’s currently not possible to specify custom networking with Windows nodes.

spec.amiFamily

AMIFamily dictates the default bootstrapping logic for nodes provisioned through this EC2NodeClass. An amiFamily is only required if you don’t specify a spec.amiSelectorTerms.alias object. For example, if you specify alias: al2023@v20240807, the amiFamily is implicitly AL2023.

AMIFamily does not impact which AMI is discovered, only the UserData generation and default BlockDeviceMappings. To automatically discover EKS optimized AMIs, use the new alias field in amiSelectorTerms.

Ubuntu Support Dropped at v1

Support for the Ubuntu AMIFamily has been dropped at Karpenter v1.0.0. This means Karpenter no longer supports automatic AMI discovery and UserData generation for Ubuntu. To continue using Ubuntu AMIs, you will need to select Ubuntu AMIs using amiSelectorTerms.

Additionally, you will need to either maintain UserData yourself using the Custom AMIFamily, or you can use the AL2 AMIFamily and custom blockDeviceMappings (as shown below). The AL2 family has an identical UserData format, but this compatibility isn’t guaranteed long term. Changes to AL2’s or Ubuntu’s UserData format could result in incompatibility, at which point the Custom AMIFamily must be used.

Ubuntu NodeClass Example:

apiVersion: karpenter.k8s.aws/v1
kind: EC2NodeClass
spec:
 amiFamily: AL2
 amiSelectorTerms:
 - id: ami-placeholder
 blockDeviceMappings:
 - deviceName: '/dev/sda1'
 rootVolume: true
 ebs:
 encrypted: true
 volumeType: gp3
 volumeSize: 20Gi

AL2

AL2 support dropped at Kubernetes 1.33

Kubernetes version 1.32 is the last version for which Amazon EKS will release Amazon Linux 2 (AL2) AMIs.

Note

Note that Karpenter will automatically generate a call to the /etc/eks/bootstrap.sh script as part of its generated UserData. When using amiFamily: AL2 you should not call this script yourself in .spec.userData. If you need to, use the Custom AMI family instead.

MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="//"

--//
Content-Type: text/x-shellscript; charset="us-ascii"

#!/bin/bash -xe
exec > >(tee /var/log/user-data.log|logger -t user-data -s 2>/dev/console) 2>&1
/etc/eks/bootstrap.sh 'test-cluster' --apiserver-endpoint 'https://test-cluster' --b64-cluster-ca 'ca-bundle' \
--dns-cluster-ip '10.100.0.10' \
--use-max-pods false \
--kubelet-extra-args '--node-labels=karpenter.sh/capacity-type=on-demand,karpenter.sh/nodepool=test --max-pods=110'
--//--

AL2023

MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="//"

--//
Content-Type: application/node.eks.aws

# Karpenter Generated NodeConfig
apiVersion: node.eks.aws/v1alpha1
kind: NodeConfig
spec:
 cluster:
 name: test-cluster
 apiServerEndpoint: https://example.com
 certificateAuthority: ca-bundle
 cidr: 10.100.0.0/16
 kubelet:
 config:
 maxPods: 110
 flags:
 - --node-labels=karpenter.sh/capacity-type=on-demand,karpenter.sh/nodepool=test

--//--

Bottlerocket

[settings]
[settings.kubernetes]
api-server = 'https://test-cluster'
cluster-certificate = 'ca-bundle'
cluster-name = 'test-cluster'
cluster-dns-ip = '10.100.0.10'
max-pods = 110

[settings.kubernetes.node-labels]
'karpenter.sh/capacity-type' = 'on-demand'
'karpenter.sh/nodepool' = 'test'

Windows2019

<powershell>
[string]$EKSBootstrapScriptFile = "$env:ProgramFiles\Amazon\EKS\Start-EKSBootstrap.ps1"
& $EKSBootstrapScriptFile -EKSClusterName 'test-cluster' -APIServerEndpoint 'https://test-cluster' -Base64ClusterCA 'ca-bundle' -KubeletExtraArgs '--node-labels="karpenter.sh/capacity-type=on-demand,karpenter.sh/nodepool=test" --max-pods=110' -DNSClusterIP '10.100.0.10'
</powershell>

Windows2022

<powershell>
[string]$EKSBootstrapScriptFile = "$env:ProgramFiles\Amazon\EKS\Start-EKSBootstrap.ps1"
& $EKSBootstrapScriptFile -EKSClusterName 'test-cluster' -APIServerEndpoint 'https://test-cluster' -Base64ClusterCA 'ca-bundle' -KubeletExtraArgs '--node-labels="karpenter.sh/capacity-type=on-demand,karpenter.sh/nodepool=test" --max-pods=110' -DNSClusterIP '10.100.0.10'
</powershell>

Custom

The Custom AMIFamily ships without any default userData to allow you to configure custom bootstrapping for control planes or images that don’t support the default methods from the other families. For this AMIFamily, kubelet must add the taint karpenter.sh/unregistered:NoExecute via the --register-with-taints flag (flags) or the KubeletConfiguration spec (options and docs). Karpenter will fail to register nodes that do not have this taint.

spec.subnetSelectorTerms

Subnet Selector Terms allow you to specify selection logic for a set of subnet options that Karpenter can choose from when launching an instance from the EC2NodeClass. Karpenter discovers subnets through the EC2NodeClass using ids or tags. When launching nodes, a subnet is automatically chosen that matches the desired zone. If multiple subnets exist for a zone, the one with the most available IP addresses will be used.

This selection logic is modeled as terms, where each term contains multiple conditions that must all be satisfied for the selector to match. Effectively, all requirements within a single term are ANDed together. It’s possible that you may want to select on two different subnets that have unrelated requirements. In this case, you can specify multiple terms which will be ORed together to form your selection logic. The example below shows how this selection logic is fulfilled.

subnetSelectorTerms:
 # Select on any subnet that has the "karpenter.sh/discovery: ${CLUSTER_NAME}"
 # AND the "environment: test" tag OR any subnet with ID "subnet-09fa4a0a8f233a921"
 - tags:
 karpenter.sh/discovery: "${CLUSTER_NAME}"
 environment: test
 - id: subnet-09fa4a0a8f233a921

Tip

Subnets may be specified by any tag, including Name. Selecting tag values using wildcards (*) is supported.

Examples

Select all with a specified tag key:

spec:
 subnetSelectorTerms:
 - tags:
 karpenter.sh/discovery/MyClusterName: '*'

Select by name and tag (all criteria must match):

spec:
 subnetSelectorTerms:
 - tags:
 Name: my-subnet
 MyTag: '' # matches all resources with the tag

Select using multiple tag terms:

spec:
 subnetSelectorTerms:
 - tags:
 Name: "my-subnet-1"
 - tags:
 Name: "my-subnet-2"

Select using wildcards:

spec:
 subnetSelectorTerms:
 - tags:
 Name: "*Public*"

Select using ids:

spec:
 subnetSelectorTerms:
 - id: "subnet-09fa4a0a8f233a921"
 - id: "subnet-0471ca205b8a129ae"

spec.securityGroupSelectorTerms

Security Group Selector Terms allow you to specify selection logic for all security groups that will be attached to an instance launched from the EC2NodeClass. The security group of an instance is comparable to a set of firewall rules. EKS creates at least two security groups by default.

This selection logic is modeled as terms, where each term contains multiple conditions that must all be satisfied for the selector to match. Effectively, all requirements within a single term are ANDed together. It’s possible that you may want to select on two different security groups that have unrelated requirements. In this case, you can specify multiple terms which will be ORed together to form your selection logic. The example below shows how this selection logic is fulfilled.

securityGroupSelectorTerms:
 # Select on any security group that has both the "karpenter.sh/discovery: ${CLUSTER_NAME}" tag
 # AND the "environment: test" tag OR any security group with the "my-security-group" name
 # OR any security group with ID "sg-063d7acfb4b06c82c"
 - tags:
 karpenter.sh/discovery: "${CLUSTER_NAME}"
 environment: test
 - name: my-security-group
 - id: sg-063d7acfb4b06c82c

Tip

Security groups may be specified by any tag, including “Name”. Selecting tags using wildcards (*) is supported.

Note

When launching nodes, Karpenter uses all the security groups that match the selector. If you choose to use the kubernetes.io/cluster/$CLUSTER_NAME tag for discovery, note that this may result in failures using the AWS Load Balancer controller. The Load Balancer controller only supports a single security group having that tag key. See this issue for more details.

To verify if this restriction affects you, run the following commands.

CLUSTER_VPC_ID="$(aws eks describe-cluster --name $CLUSTER_NAME --query cluster.resourcesVpcConfig.vpcId --output text)"

aws ec2 describe-security-groups --filters Name=vpc-id,Values=$CLUSTER_VPC_ID Name=tag-key,Values=kubernetes.io/cluster/$CLUSTER_NAME --query 'SecurityGroups[].[GroupName]' --output text

If multiple securityGroups are printed, you will need more specific securityGroupSelectorTerms. We generally recommend that you use the karpenter.sh/discovery: $CLUSTER_NAME tag selector instead.

Examples

Select all assigned to a cluster:

spec:
 securityGroupSelectorTerms:
 - tags:
 kubernetes.io/cluster/$CLUSTER_NAME: "owned"

Select all with a specified tag key:

spec:
 securityGroupSelectorTerms:
 - tags:
 MyTag: '*'

Select by name and tag (all criteria must match):

spec:
 securityGroupSelectorTerms:
 - name: my-security-group
 tags:
 MyTag: '*' # matches all resources with the tag

Select using multiple tag terms:

spec:
 securityGroupSelectorTerms:
 - tags:
 Name: "my-security-group-1"
 - tags:
 Name: "my-security-group-2"

Select by name using a wildcard:

spec:
 securityGroupSelectorTerms:
 - name: "*Public*"

Select using ids:

spec:
 securityGroupSelectorTerms:
 - id: "sg-063d7acfb4b06c82c"
 - id: "sg-06e0cf9c198874591"

spec.role

Role is an optional field and tells Karpenter which IAM identity nodes should assume. You must specify one of role or instanceProfile when creating a Karpenter EC2NodeClass. If using the Karpenter Getting Started Guide to deploy Karpenter, you can use the KarpenterNodeRole-$CLUSTER_NAME role provisioned by that process.

spec:
 role: "KarpenterNodeRole-$CLUSTER_NAME"

spec.instanceProfile

InstanceProfile is an optional field and tells Karpenter which IAM identity nodes should assume. You must specify one of role or instanceProfile when creating a Karpenter EC2NodeClass. If you use the instanceProfile field instead of role, Karpenter will not manage the InstanceProfile on your behalf; instead, it expects that you have pre-provisioned an IAM instance profile and assigned it a role.

You can provision and assign a role to an IAM instance profile using CloudFormation or by using the aws iam create-instance-profile and aws iam add-role-to-instance-profile commands in the CLI.

Note

For private clusters that do not have access to the public internet, using spec.instanceProfile is required. spec.role cannot be used since Karpenter needs to access IAM endpoints to manage a generated instance profile. IAM doesn’t support private endpoints to enable accessing the service without going to the public internet.

spec.amiSelectorTerms

AMI Selector Terms are required and are used to configure AMIs for Karpenter to use. AMIs are discovered through alias, id, owner, name, and tags.

This selection logic is modeled as terms, where each term contains multiple conditions that must all be satisfied for the selector to match. Effectively, all requirements within a single term are ANDed together. It’s possible that you may want to select on two different AMIs that have unrelated requirements. In this case, you can specify multiple terms which will be ORed together to form your selection logic. The example below shows how this selection logic is fulfilled.

amiSelectorTerms:
 # Select on any AMI that has both the `karpenter.sh/discovery: ${CLUSTER_NAME}`
 # AND `environment: test` tags OR any AMI with the name `my-ami` OR an AMI with
 # ID `ami-123` OR an AMI with ID matching the value of my-custom-parameter
 - tags:
 karpenter.sh/discovery: "${CLUSTER_NAME}"
 environment: test
 - name: my-ami
 - id: ami-123
 - ssmParameter: my-custom-parameter # ssm parameter name or ARN
 # Select EKS optimized AL2023 AMIs with version `v20240807`. This term is mutually
 # exclusive and can't be specified with other terms.
 # - alias: al2023@v20240807

An alias term can be used to select EKS-optimized AMIs. An alias is formatted as family@version. Family can be one of the following values:

al2
al2023
bottlerocket
windows2019
windows2022

The version string can be set to latest, or pinned to a specific AMI using the format of that AMI’s GitHub release tags. For example, AL2 and AL2023 use dates for their release, so they can be pinned as follows:

alias: al2023@v20240703

Bottlerocket uses a semantic version for their releases. You can pin bottlerocket as follows:

alias: bottlerocket@v1.20.4

The Windows family does not support pinning, so only latest is supported.

The following commands can be used to determine the versions availble for an alias in your region:

export K8S_VERSION="1.32"
aws ssm get-parameters-by-path --path "/aws/service/eks/optimized-ami/$K8S_VERSION/amazon-linux-2023/" --recursive | jq -cr '.Parameters[].Name' | grep -v "recommended" | awk -F '/' '{print $10}' | sed -r 's/.*(v[[:digit:]]+)$/\1/' | sort | uniq

export K8S_VERSION="1.32"
aws ssm get-parameters-by-path --path "/aws/service/eks/optimized-ami/$K8S_VERSION/amazon-linux-2/" --recursive | jq -cr '.Parameters[].Name' | grep -v "recommended" | awk -F '/' '{print $8}' | sed -r 's/.*(v[[:digit:]]+)$/\1/' | sort | uniq

export K8S_VERSION="1.32"
aws ssm get-parameters-by-path --path "/aws/service/bottlerocket/aws-k8s-$K8S_VERSION" --recursive | jq -cr '.Parameters[].Name' | grep -v "latest" | awk -F '/' '{print $7}' | sort | uniq

Warning

Karpenter supports automatic AMI selection and upgrades using the latest version pin, but this is not recommended for production environments. When using latest, a new AMI release will cause Karpenter to drift all out-of-date nodes in the cluster, replacing them with nodes running the new AMI. We strongly recommend evaluating new AMIs in a lower environment before rolling them out into a production environment. More details on Karpenter’s recommendations for managing AMIs can be found here.

To select an AMI by name, use the name field in the selector term. To select an AMI by id, use the id field in the selector term. To select AMIs that are not owned by amazon or the account that Karpenter is running in, use the owner field - you can use a combination of account aliases (e.g. self amazon, your-aws-account-name) and account IDs.

If owner is not set for name, it defaults to self,amazon, preventing Karpenter from inadvertently selecting an AMI that is owned by a different account. Tags don’t require an owner as tags can only be discovered by the user who created them.

Tip

AMIs may be specified by any AWS tag, including Name. Selecting by tag or by name using wildcards (*) is supported.

Note

If amiSelectorTerms match more than one AMI, Karpenter will automatically determine which AMI best fits the workloads on the launched worker node under the following constraints:

When launching nodes, Karpenter automatically determines which architecture a custom AMI is compatible with and will use images that match an instanceType’s requirements.
- Unless using an alias, Karpenter cannot detect requirements other than architecture. If you need to specify different AMIs for different kind of nodes (e.g. accelerated GPU AMIs), you should use a separate EC2NodeClass.
If multiple AMIs are found that can be used, Karpenter will choose the latest one.
If no AMIs are found that can be used, then no nodes will be provisioned.

Examples

Select by AMI family and version:

 amiSelectorTerms:
 - alias: al2023@v20240807

Select all with a specified tag:

 amiSelectorTerms:
 - tags:
 karpenter.sh/discovery/MyClusterName: '*'

Select by name:

 amiSelectorTerms:
 - name: my-ami

Select by Name tag:

 amiSelectorTerms:
 - tags:
 Name: my-ami

Select by name and owner:

 amiSelectorTerms:
 - name: my-ami
 owner: self
 - name: my-ami
 owner: "0123456789"

Select by name using a wildcard:

spec:
 amiSelectorTerms:
 - name: "*EKS*"

Select by all under an owner:

spec:
 amiSelectorTerms:
 - name: "*"
 owner: self

Specify using ids:

 amiSelectorTerms:
 - id: "ami-123"
 - id: "ami-456"

Specify using custom ssm parameter name or ARN:

 amiSelectorTerms:
 - ssmParameter: "my-custom-parameter"

Note

When using a custom SSM parameter, you’ll need to expand the ssm:GetParameter permissions on the Karpenter IAM role to include your custom parameter, as the default policy only allows access to the AWS public parameters.

spec.capacityReservationSelectorTerms

Feature State: Alpha

Capacity Reservation Selector Terms allow you to select on-demand capacity reservations, which will be made available to NodePools which select the given EC2NodeClass. Karpenter will prioritize utilizing the capacity in these reservations before falling back to on-demand and spot. Capacity reservations can be discovered using ids or tags.

This selection logic is modeled as terms. A term can specify an ID or a set of tags to select against. When specifying tags, it will select all capacity reservations accessible from the account with matching tags. This can be further restricted by specifying an owner ID.

Note

Note that the IAM role Karpenter assumes should have a permissions policy associated with it that grants it permissions to use the ec2:DescribeCapacityReservations action to discover capacity reservations and the ec2:RunInstances action to run instances in those capacity reservations.

Examples

Select the reservations with the given IDs:

spec:
 capacityReservationSelectorTerms:
 - id: cr-123
 - id: cr-456

Select the reservations by tags:

spec:
 capacityReservationSelectorTerms:
 # Select all capacity reservations which have both matching tags
 - tags:
 key1: foo
 key2: bar
 # Additionally, select all capacity reservations with the following matching tag
 - tags:
 key3: foobar

Select by tags and owner ID:

spec:
 # Select all capacity reservations with the matching tags which are also owned by
 # the specified account.
 capacityReservationSelectorTerms:
 - tags:
 key: foo
 ownerID: 012345678901

spec.tags

Karpenter adds tags to all resources it creates, including EC2 Instances, EBS volumes, and Launch Templates. The default set of tags are listed below.

Name: <node-name>
karpenter.sh/nodeclaim: <nodeclaim-name>
karpenter.sh/nodepool: <nodepool-name>
karpenter.k8s.aws/ec2nodeclass: <ec2nodeclass-name>
kubernetes.io/cluster/<cluster-name>: owned
eks:eks-cluster-name: <cluster-name>

Additional tags can be added in the tags section, which will be merged with the default tags specified above.

spec:
 tags:
 InternalAccountingTag: 1234
 dev.corp.net/app: Calculator
 dev.corp.net/team: MyTeam

Note

Karpenter allows overrides of the default “Name” tag but does not allow overrides to restricted domains (such as “karpenter.sh”, “karpenter.k8s.aws”, and “kubernetes.io/cluster”). This ensures that Karpenter is able to correctly auto-discover nodes that it owns.

spec.metadataOptions

Control the exposure of Instance Metadata Service on EC2 Instances launched by this EC2NodeClass using a generated launch template.

Refer to recommended, security best practices for limiting exposure of Instance Metadata and User Data to pods.

If metadataOptions are omitted from this EC2NodeClass, the following default settings are applied:

spec:
 metadataOptions:
 httpEndpoint: enabled
 httpProtocolIPv6: disabled
 httpPutResponseHopLimit: 1
 httpTokens: required

spec.blockDeviceMappings

The blockDeviceMappings field in an EC2NodeClass can be used to control the Elastic Block Storage (EBS) volumes that Karpenter attaches to provisioned nodes. Karpenter uses default block device mappings for the AMIFamily specified. For example, the Bottlerocket AMI Family defaults with two block device mappings, one for Bottlerocket’s control volume and the other for container resources such as images and logs.

spec:
 blockDeviceMappings:
 - deviceName: /dev/xvda
 ebs:
 volumeSize: 100Gi
 volumeType: gp3
 iops: 10000
 encrypted: true
 kmsKeyID: "1234abcd-12ab-34cd-56ef-1234567890ab"
 deleteOnTermination: true
 throughput: 125
 snapshotID: snap-0123456789
 volumeInitializationRate: 100

The following blockDeviceMapping defaults are used for each AMIFamily if no blockDeviceMapping overrides are specified in the EC2NodeClass

AL2

spec:
 blockDeviceMappings:
 - deviceName: /dev/xvda
 ebs:
 volumeSize: 20Gi
 volumeType: gp3
 encrypted: true

AL2023

spec:
 blockDeviceMappings:
 - deviceName: /dev/xvda
 ebs:
 volumeSize: 20Gi
 volumeType: gp3
 encrypted: true

Bottlerocket

spec:
 blockDeviceMappings:
 # Root device
 - deviceName: /dev/xvda
 ebs:
 volumeSize: 4Gi
 volumeType: gp3
 encrypted: true
 # Data device: Container resources such as images and logs
 - deviceName: /dev/xvdb
 ebs:
 volumeSize: 20Gi
 volumeType: gp3
 encrypted: true

Windows2019/Windows2022

spec:
 blockDeviceMappings:
 - deviceName: /dev/sda1
 ebs:
 volumeSize: 50Gi
 volumeType: gp3
 encrypted: true

Custom

The Custom AMIFamily ships without any default blockDeviceMappings.

spec.instanceStorePolicy

The instanceStorePolicy field controls how instance-store volumes are handled. By default, Karpenter and Kubernetes will simply ignore them.

RAID0

If you intend to use these volumes for faster node ephemeral-storage, set instanceStorePolicy to RAID0:

spec:
 instanceStorePolicy: RAID0

This will set the allocatable ephemeral-storage of each node to the total size of the instance-store volume(s). This configuration is likely to be useful for workloads that leverage dense storage instance types or require the low latency from instance-stores that are nvme ssd based.

The disks must be formatted & mounted in a RAID0 and be the underlying filesystem for the Kubelet & Containerd. Even if you already configure your volumes with RAID0, Karpenter won’t recognize this by default unless you set the instanceStorePolicy to RAID0. Without this, scheduling workloads that depend on ephemeral-storage from the instance-stores may result in a deadlock due to insufficient storage.

Instructions for each AMI family are listed below:

AL2

On AL2, Karpenter automatically configures the disks through an additional boostrap argument (--local-disks raid0). The device name is /dev/md/0 and its mount point is /mnt/k8s-disks/0. You should ensure any additional disk setup does not interfere with these.

AL2023

On AL2023, Karpenter automatically configures the disks via the generated NodeConfig object. Like AL2, the device name is /dev/md/0 and its mount point is /mnt/k8s-disks/0. You should ensure any additional disk setup does not interfere with these.

Others

For all other AMI families, you must configure the disks yourself. Check out the setup-local-disks script in amazon-eks-ami to see how this is done for AL2.

Tip

Since the Kubelet & Containerd will be using the instance-store filesystem, you may consider using a more minimal root volume size.

spec.userData

You can control the UserData that is applied to your worker nodes via this field. This allows you to run custom scripts or pass-through custom configuration to Karpenter instances on start-up.

apiVersion: karpenter.k8s.aws/v1
kind: EC2NodeClass
metadata:
 name: bottlerocket-example
spec:
 ...
 amiFamily: Bottlerocket
 userData: |
 [settings.kubernetes]
 "kube-api-qps" = 30
 "shutdown-grace-period" = "30s"
 "shutdown-grace-period-for-critical-pods" = "30s"
 [settings.kubernetes.eviction-hard]
 "memory.available" = "20%"

This example adds SSH keys to allow remote login to the node (replace my-authorized_keys with your public key file):

Note

Instead of using SSH as set up in this example, you can use Session Manager (SSM) or EC2 Instance Connect to gain shell access to Karpenter nodes. See Node NotReady troubleshooting for an example of starting an SSM session from the command line or EC2 Instance Connect documentation to connect to nodes using SSH.

Also, my-authorized_key key is the public key. See Retrieve the public key material.

apiVersion: karpenter.k8s.aws/v1
kind: EC2NodeClass
metadata:
 name: al2-example
spec:
 ...
 amiFamily: AL2
 userData: |
 #!/bin/bash
 mkdir -p ~ec2-user/.ssh/
 touch ~ec2-user/.ssh/authorized_keys
 cat >> ~ec2-user/.ssh/authorized_keys <<EOF
 {{ insertFile "../my-authorized_keys" | indent 4 }}
 EOF
 chmod -R go-w ~ec2-user/.ssh/authorized_keys
 chown -R ec2-user ~ec2-user/.ssh

Alternatively, you can save the key in your SSM Parameter Store and use the get-parameter command mentioned below to retrieve the key for authorized_keys.

aws ssm get-parameter --name "<parameter-name>" --region <region> --with-decryption --query "Parameter.Value" --output text > /home/ec2-user/.ssh/authorized_keys

For more examples on configuring fields for different AMI families, see the examples here.

Karpenter will merge the userData you specify with the default userData for that AMIFamily. See the AMIFamily section for more details on these defaults. View the sections below to understand the different merge strategies for each AMIFamily.

AL2

Your UserData can be in the MIME multi part archive format.
Karpenter will transform your custom user-data as a MIME part, if necessary, and then merge a final MIME part to the end of your UserData parts which will bootstrap the worker node. Karpenter will have full control over all the parameters being passed to the bootstrap script.
- Karpenter will continue to set MaxPods, ClusterDNS and all other parameters defined in spec.kubeletConfiguration as before.

Consider the following example to understand how your custom UserData will be merged -

Passed-in UserData (bash)

#!/bin/bash
echo "Running custom user data script (bash)"

Merged UserData (bash)

MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="//"

--//
Content-Type: text/x-shellscript; charset="us-ascii"

#!/bin/bash
echo "Running custom user data script (bash)"

--//
Content-Type: text/x-shellscript; charset="us-ascii"

#!/bin/bash -xe
exec > >(tee /var/log/user-data.log|logger -t user-data -s 2>/dev/console) 2>&1
/etc/eks/bootstrap.sh 'test-cluster' --apiserver-endpoint 'https://test-cluster' --b64-cluster-ca 'ca-bundle' \
--use-max-pods false \
--kubelet-extra-args '--node-labels=karpenter.sh/capacity-type=on-demand,karpenter.sh/nodepool=test --max-pods=110'
--//--

Passed-in UserData (MIME)

MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="BOUNDARY"

--BOUNDARY
Content-Type: text/x-shellscript; charset="us-ascii"

#!/bin/bash
echo "Running custom user data script (mime)"

--BOUNDARY--

Merged UserData (MIME)

MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="//"

--//
Content-Type: text/x-shellscript; charset="us-ascii"

#!/bin/bash
echo "Running custom user data script (mime)"

--//
Content-Type: text/x-shellscript; charset="us-ascii"

#!/bin/bash -xe
exec > >(tee /var/log/user-data.log|logger -t user-data -s 2>/dev/console) 2>&1
/etc/eks/bootstrap.sh 'test-cluster' --apiserver-endpoint 'https://test-cluster' --b64-cluster-ca 'ca-bundle' \
--use-max-pods false \
--kubelet-extra-args '--node-labels=karpenter.sh/capacity-type=on-demand,karpenter.sh/nodepool=test --max-pods=110'
--//--

Tip

You can set additional kubelet configuration properties, unavailable through spec.kubelet, by updating the kubelet-config.json file:

apiVersion: karpenter.k8s.aws/v1
kind: EC2NodeClass
metadata:
 name: kubelet-config-example
spec:
 amiFamily: AL2
 userData: |
 #!/bin/bash
 echo "$(jq '.kubeAPIQPS=50' /etc/kubernetes/kubelet/kubelet-config.json)" > /etc/kubernetes/kubelet/kubelet-config.json

AL2023

Your UserData may be in one of three formats: a MIME multi part archive, a NodeConfig YAML / JSON string, or a shell script.
Karpenter will transform your custom UserData into a MIME part, if necessary, and then create a MIME multi-part archive. This archive will consist of a generated NodeConfig, containing Karpenter’s default values, followed by the transformed custom UserData. For more information on the NodeConfig spec, refer to the AL2023 EKS Optimized AMI docs.

Warning

Any values configured by the Karpenter generated NodeConfig object will take precedent over values specifed in spec.userData. This includes cluster name, cluster CIDR, cluster endpoint, certificate authority, taints, labels, and any value in spec.kubelet. These fields must be configured natively through Karpenter rather than through UserData.

Passed-in UserData (NodeConfig)

apiVersion: node.eks.aws/v1alpha1
kind: NodeConfig
spec:
 kubelet:
 config:
 maxPods: 42

Merged UserData (NodeConfig)

MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="//"

--//
Content-Type: application/node.eks.aws

apiVersion: node.eks.aws/v1alpha1
kind: NodeConfig
spec:
 kubelet:
 config:
 maxPods: 42

--//
Content-Type: application/node.eks.aws

# Karpenter Generated NodeConfig
apiVersion: node.eks.aws/v1alpha1
kind: NodeConfig
spec:
 cluster:
 apiServerEndpoint: https://test-cluster
 certificateAuthority: cluster-ca
 cidr: 10.100.0.0/16
 name: test-cluster
 kubelet:
 config:
 clusterDNS:
 - 10.100.0.10
 maxPods: 118
 flags:
 - --node-labels="karpenter.sh/capacity-type=on-demand,karpenter.sh/nodepool=default"

--//--

Passed-in UserData (bash)

#!/bin/bash
echo "Hello, AL2023!"

Merged UserData (bash)

MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="//"

--//
Content-Type: text/x-shellscript; charset="us-ascii"

#!/bin/bash
echo "Hello, AL2023!"

--//
Content-Type: application/node.eks.aws

# Karpenter Generated NodeConfig
apiVersion: node.eks.aws/v1alpha1
kind: NodeConfig
spec:
 cluster:
 apiServerEndpoint: https://test-cluster
 certificateAuthority: cluster-ca
 cidr: 10.100.0.0/16
 name: test-cluster
 kubelet:
 config:
 clusterDNS:
 - 10.100.0.10
 maxPods: 118
 flags:
 - --node-labels="karpenter.sh/capacity-type=on-demand,karpenter.sh/nodepool=default"

--//--

Passed-in UserData (MIME)

MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="//"

--//
Content-Type: text/x-shellscript; charset="us-ascii"

#!/bin/bash
echo "Hello, AL2023!"

--//
Content-Type: application/node.eks.aws

apiVersion: node.eks.aws/v1alpha1
kind: NodeConfig
spec:
 kubelet:
 config:
 maxPods: 42
--//

Merged UserData (MIME)

MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="//"

--//
Content-Type: application/node.eks.aws

apiVersion: node.eks.aws/v1alpha1
kind: NodeConfig
spec:
 kubelet:
 config:
 maxPods: 42
--//
Content-Type: text/x-shellscript; charset="us-ascii"

#!/bin/bash
echo "Hello, AL2023!"

--//
Content-Type: application/node.eks.aws

# Karpenter Generated NodeConfig
apiVersion: node.eks.aws/v1alpha1
kind: NodeConfig
spec:
 cluster:
 apiServerEndpoint: https://test-cluster
 certificateAuthority: cluster-ca
 cidr: 10.100.0.0/16
 name: test-cluster
 kubelet:
 config:
 clusterDNS:
 - 10.100.0.10
 maxPods: 118
 flags:
 - --node-labels="karpenter.sh/capacity-type=on-demand,karpenter.sh/nodepool=default"

--//--

Bottlerocket

Your UserData must be valid TOML.
Unknown TOML fields will be ignored when the final merged UserData is generated by Karpenter.

Warning

Any values configured by Karpenter will take precedent over values specifed in spec.userData. This includes cluster name, cluster endpoint, cluster certificate, taints, labels, and any value in spec.kubelet. These fields must be configured natively through Karpenter rather than through UserData.

Consider the following example to understand how your custom UserData settings will be merged in.

Passed-in UserData

[settings.kubernetes.eviction-hard]
"memory.available" = "12%"
[settings.kubernetes]
"unknown-setting" = "unknown"
[settings.kubernetes.node-labels]
'field.controlled.by/karpenter' = 'will-be-overridden'

Merged UserData

[settings]
[settings.kubernetes]
api-server = 'https://cluster'
cluster-certificate = 'ca-bundle'
cluster-name = 'cluster'

[settings.kubernetes.node-labels]
'karpenter.sh/capacity-type' = 'on-demand'
'karpenter.sh/nodepool' = 'default'

[settings.kubernetes.node-taints]

[settings.kubernetes.eviction-hard]
'memory.available' = '12%%'

Device ownership in Bottlerocket

Bottlerocket v1.30.0+ supports device ownership using the security context provided in the Kubernetes specfile. To enable this, you will need the following user-data configurations:

[settings]
[settings.kubernetes]
device-ownership-from-security-context = true

This allows the container to take ownership of devices allocated to the pod via device-plugins based on the runAsUser and runAsGroup values provided in the spec. For more details on this, see the Kubernetes documentation

This setting helps you enable Neuron workloads on Bottlerocket instances. See Accelerators/GPU Resources for more details.

Windows2019/Windows2022

Your UserData must be specified as PowerShell commands.
The UserData specified will be prepended to a Karpenter managed section that will bootstrap the kubelet.
Karpenter will continue to set ClusterDNS and all other parameters defined in spec.kubeletConfiguration as before.

Consider the following example to understand how your custom UserData settings will be merged in.

Passed-in UserData

Write-Host "Running custom user data script"

Merged UserData

<powershell>
Write-Host "Running custom user data script"
[string]$EKSBootstrapScriptFile = "$env:ProgramFiles\Amazon\EKS\Start-EKSBootstrap.ps1"
& $EKSBootstrapScriptFile -EKSClusterName 'test-cluster' -APIServerEndpoint 'https://test-cluster' -Base64ClusterCA 'ca-bundle' -KubeletExtraArgs '--node-labels="karpenter.sh/capacity-type=spot,karpenter.sh/nodepool=windows2022" --max-pods=110' -DNSClusterIP '10.0.100.10'
</powershell>

Windows Support Notice

Currently, Karpenter does not specify -ServiceCIDR to EKS Windows AMI Bootstrap script. Windows worker nodes will use 172.20.0.0/16 or 10.100.0.0/16 for Kubernetes service IP address ranges based on the IP address of the primary interface. The effective ServiceCIDR can be verified at $env:ProgramData\Amazon\EKS\cni\config\vpc-bridge.conf on the worker node.

Support for the Windows ServiceCIDR argument can be tracked in a Karpenter Github Issue. Currently, if the effective ServiceCIDR is incorrect for your windows worker nodes, you can add the following userData as a workaround.

spec:
 userData: |
 $global:EKSCluster = Get-EKSCluster -Name my-cluster

Custom

No merging is performed, your UserData must perform all setup required of the node to allow it to join the cluster.
Custom UserData must meet the following requirements to work correctly with Karpenter:
- It must ensure the node is registered with the karpenter.sh/unregistered:NoExecute taint (via kubelet configuration field registerWithTaints)
- It must set kubelet config options to match those configured in spec.kubelet

spec.detailedMonitoring

Enabling detailed monitoring controls the EC2 detailed monitoring feature. If you enable this option, the Amazon EC2 console displays monitoring graphs with a 1-minute period for the instances that Karpenter launches.

spec:
 detailedMonitoring: true

spec.associatePublicIPAddress

You can explicitly set AssociatePublicIPAddress: false when you are only launching into private subnets. Previously, Karpenter auto-set associatePublicIPAddress on the primary ENI to false if a user’s subnet options were all private subnets. This value is a boolean field that controls whether instances created by Karpenter for this EC2NodeClass will have an associated public IP address. This overrides the MapPublicIpOnLaunch setting applied to the subnet the node is launched in. If this field is not set, the MapPublicIpOnLaunch field will be respected.

Note

If a NodeClaim requests vpc.amazonaws.com/efa resources, spec.associatePublicIPAddress is respected. However, if this NodeClaim requests multiple EFA resources and the value for spec.associatePublicIPAddress is true, the instance will fail to launch. This is due to an EC2 restriction which requires that the field is only set to true when configuring an instance with a single ENI at launch. When using this field, it is advised that users segregate their EFA workload to use a separate NodePool / EC2NodeClass pair.

status.subnets

status.subnets contains the resolved id and zone of the subnets that were selected by the spec.subnetSelectorTerms for the node class. The subnets will be sorted by the available IP address count in decreasing order.

Examples

spec:
 subnetSelectorTerms:
 - tags:
 karpenter.sh/discovery: "${CLUSTER_NAME}"
status:
 subnets:
 - id: subnet-0a462d98193ff9fac
 zone: us-east-2b
 - id: subnet-0322dfafd76a609b6
 zone: us-east-2c
 - id: subnet-0727ef01daf4ac9fe
 zone: us-east-2b
 - id: subnet-00c99aeafe2a70304
 zone: us-east-2a
 - id: subnet-023b232fd5eb0028e
 zone: us-east-2c
 - id: subnet-03941e7ad6afeaa72
 zone: us-east-2a

status.securityGroups

status.securityGroups contains the resolved id and name of the security groups that were selected by the spec.securityGroupSelectorTerms for the node class. The subnets will be sorted by the available IP address count in decreasing order.

Examples

spec:
 securityGroupSelectorTerms:
 - tags:
 karpenter.sh/discovery: "${CLUSTER_NAME}"
status:
 securityGroups:
 - id: sg-041513b454818610b
 name: ClusterSharedNodeSecurityGroup
 - id: sg-0286715698b894bca
 name: ControlPlaneSecurityGroup-1AQ073TSAAPW

status.amis

status.amis contains the resolved id, name, requirements, and the deprecated status of either the default AMIs for the spec.amiFamily or the AMIs selected by the spec.amiSelectorTerms if this field is specified. The deprecated status will be shown for resolved AMIs that are deprecated.

Examples

AMIs resolved with an AL2 alias:

spec:
 amiSelectorTerms:
 - alias: al2@v20240807
status:
 amis:
 - id: ami-03c3a3dcda64f5b75
 name: amazon-linux-2-gpu
 requirements:
 - key: kubernetes.io/arch
 operator: In
 values:
 - amd64
 - key: karpenter.k8s.aws/instance-gpu-count
 operator: Exists
 - id: ami-03c3a3dcda64f5b75
 name: amazon-linux-2-gpu
 requirements:
 - key: kubernetes.io/arch
 operator: In
 values:
 - amd64
 - key: karpenter.k8s.aws/instance-accelerator-count
 operator: Exists
 - id: ami-06afb2d101cc4b8bd
 name: amazon-linux-2-arm64
 requirements:
 - key: kubernetes.io/arch
 operator: In
 values:
 - arm64
 - key: karpenter.k8s.aws/instance-gpu-count
 operator: DoesNotExist
 - key: karpenter.k8s.aws/instance-accelerator-count
 operator: DoesNotExist
 - id: ami-0e28b76d768af234e
 name: amazon-linux-2
 requirements:
 - key: kubernetes.io/arch
 operator: In
 values:
 - amd64
 - key: karpenter.k8s.aws/instance-gpu-count
 operator: DoesNotExist
 - key: karpenter.k8s.aws/instance-accelerator-count
 operator: DoesNotExist

AMIs resolved from tags:

spec:
 amiSelectorTerms:
 - tags:
 karpenter.sh/discovery: "${CLUSTER_NAME}"
status:
 amis:
 - id: ami-01234567890123456
 name: custom-ami-amd64
 deprecated: true
 requirements:
 - key: kubernetes.io/arch
 operator: In
 values:
 - amd64
 - id: ami-01234567890123456
 name: custom-ami-arm64
 requirements:
 - key: kubernetes.io/arch
 operator: In
 values:
 - arm64

status.capacityReservations

status.capacityReservations contains the following information for each resolved capacity reservation:

Field	Example	Description
`availabilityZone`	`us-east-1a`	The availability zone the capacity reservation is available in
`id`	`cr-56fac701cc1951b03`	The ID of the capacity reservation
`instanceMatchCriteria`	`open`	The instanceMatchCriteria for the capacity reservation. Can be `open` or `targeted`.
`instanceType`	`m5.large`	The EC2 instance type of the capacity reservation
`ownerID`	`459763720645`	The account ID that owns the capacity reservation

Examples

status:
 capacityReservations:
 - availabilityZone: us-west-2a
 id: cr-01234567890123456
 instanceMatchCriteria: targeted
 instanceType: g6.48xlarge
 ownerID: "012345678901"
 - availabilityZone: us-west-2c
 id: cr-12345678901234567
 instanceMatchCriteria: open
 instanceType: g6.48xlarge
 ownerID: "98765432109"

status.instanceProfile

status.instanceProfile contains the resolved instance profile generated by Karpenter from the spec.role

spec:
 role: "KarpenterNodeRole-${CLUSTER_NAME}"
status:
 instanceProfile: "${CLUSTER_NAME}-0123456778901234567789"

status.conditions

status.conditions indicates EC2NodeClass readiness. This will be Ready when Karpenter successfully discovers AMIs, Instance Profile, Subnets, Cluster CIDR (AL2023 only) and SecurityGroups for the EC2NodeClass.

NodeClasses have the following status conditions:

Condition Type	Description
SubnetsReady	Subnets are discovered.
SecurityGroupsReady	Security Groups are discovered.
InstanceProfileReady	Instance Profile is discovered.
AMIsReady	AMIs are discovered.
Ready	Top level condition that indicates if the nodeClass is ready. If any of the underlying conditions is `False` then this condition is set to `False` and `Message` on the condition indicates the dependency that was not resolved.

If a NodeClass is not ready, NodePools that reference it through their nodeClassRef will not be considered for scheduling.

V1.5: NodePools

Mon, 01 Jan 0001 00:00:00 +0000

When you first installed Karpenter, you set up a default NodePool. The NodePool sets constraints on the nodes that can be created by Karpenter and the pods that can run on those nodes. The NodePool can be set to do things like:

Define taints to limit the pods that can run on nodes Karpenter creates
Define any startup taints to inform Karpenter that it should taint the node initially, but that the taint is temporary.
Limit node creation to certain zones, instance types, and computer architectures
Set defaults for node expiration

You can change your NodePool or add other NodePools to Karpenter. Here are things you should know about NodePools:

Karpenter won’t do anything if there is not at least one NodePool configured.
Each NodePool that is configured is looped through by Karpenter.
If Karpenter encounters a taint in the NodePool that is not tolerated by a Pod, Karpenter won’t use that NodePool to provision the pod.
If Karpenter encounters a startup taint in the NodePool it will be applied to nodes that are provisioned, but pods do not need to tolerate the taint. Karpenter assumes that the taint is temporary and some other system will remove the taint.
It is recommended to create NodePools that are mutually exclusive. So no Pod should match multiple NodePools. If multiple NodePools are matched, Karpenter will use the NodePool with the highest weight.

Note

Objects for setting Kubelet features have been moved from the NodePool spec to the EC2NodeClasses spec, to not require other Karpenter providers to support those features.

For some example NodePool configurations, see the examples in the Karpenter GitHub repository.

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
 name: default
spec:
 # Template section that describes how to template out NodeClaim resources that Karpenter will provision
 # Karpenter will consider this template to be the minimum requirements needed to provision a Node using this NodePool
 # It will overlay this NodePool with Pods that need to schedule to further constrain the NodeClaims
 # Karpenter will provision to launch new Nodes for the cluster
 template:
 metadata:
 # Labels are arbitrary key-values that are applied to all nodes
 labels:
 billing-team: my-team

 # Annotations are arbitrary key-values that are applied to all nodes
 annotations:
 example.com/owner: "my-team"
 spec:
 # References the Cloud Provider's NodeClass resource, see your cloud provider specific documentation
 nodeClassRef:
 group: karpenter.k8s.aws  # Updated since only a single version will be served
 kind: EC2NodeClass
 name: default

 # Provisioned nodes will have these taints
 # Taints may prevent pods from scheduling if they are not tolerated by the pod.
 taints:
 - key: example.com/special-taint
 effect: NoSchedule

 # Provisioned nodes will have these taints, but pods do not need to tolerate these taints to be provisioned by this
 # NodePool. These taints are expected to be temporary and some other entity (e.g. a DaemonSet) is responsible for
 # removing the taint after it has finished initializing the node.
 startupTaints:
 - key: example.com/another-taint
 effect: NoSchedule

 # The amount of time a Node can live on the cluster before being removed
 # Avoiding long-running Nodes helps to reduce security vulnerabilities as well as to reduce the chance of issues that can plague Nodes with long uptimes such as file fragmentation or memory leaks from system processes
 # You can choose to disable expiration entirely by setting the string value 'Never' here

 # Note: changing this value in the nodepool will drift the nodeclaims.
 expireAfter: 720h | Never

 # The amount of time that a node can be draining before it's forcibly deleted. A node begins draining when a delete call is made against it, starting
 # its finalization flow. Pods with TerminationGracePeriodSeconds will be deleted preemptively before this terminationGracePeriod ends to give as much time to cleanup as possible.
 # If your pod's terminationGracePeriodSeconds is larger than this terminationGracePeriod, Karpenter may forcibly delete the pod
 # before it has its full terminationGracePeriod to cleanup.

 # Note: changing this value in the nodepool will drift the nodeclaims.
 terminationGracePeriod: 48h

 # Requirements that constrain the parameters of provisioned nodes.
 # These requirements are combined with pod.spec.topologySpreadConstraints, pod.spec.affinity.nodeAffinity, pod.spec.affinity.podAffinity, and pod.spec.nodeSelector rules.
 # Operators { In, NotIn, Exists, DoesNotExist, Gt, and Lt } are supported.
 # https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#operators
 requirements:
 - key: "karpenter.k8s.aws/instance-category"
 operator: In
 values: ["c", "m", "r"]
 # minValues here enforces the scheduler to consider at least that number of unique instance-category to schedule the pods.
 # This field is ALPHA and can be dropped or replaced at any time
 minValues: 2
 - key: "karpenter.k8s.aws/instance-family"
 operator: In
 values: ["m5","m5d","c5","c5d","c4","r4"]
 minValues: 5
 - key: "karpenter.k8s.aws/instance-cpu"
 operator: In
 values: ["4", "8", "16", "32"]
 - key: "karpenter.k8s.aws/instance-hypervisor"
 operator: In
 values: ["nitro"]
 - key: "karpenter.k8s.aws/instance-generation"
 operator: Gt
 values: ["2"]
 - key: "topology.kubernetes.io/zone"
 operator: In
 values: ["us-west-2a", "us-west-2b"]
 - key: "kubernetes.io/arch"
 operator: In
 values: ["arm64", "amd64"]
 - key: "karpenter.sh/capacity-type"
 operator: In
 values: ["spot", "on-demand", "reserved"]

 # Disruption section which describes the ways in which Karpenter can disrupt and replace Nodes
 # Configuration in this section constrains how aggressive Karpenter can be with performing operations
 # like rolling Nodes due to them hitting their maximum lifetime (expiry) or scaling down nodes to reduce cluster cost
 disruption:
 # Describes which types of Nodes Karpenter should consider for consolidation
 # If using 'WhenEmptyOrUnderutilized', Karpenter will consider all nodes for consolidation and attempt to remove or replace Nodes when it discovers that the Node is empty or underutilized and could be changed to reduce cost
 # If using `WhenEmpty`, Karpenter will only consider nodes for consolidation that contain no workload pods
 consolidationPolicy: WhenEmptyOrUnderutilized | WhenEmpty

 # The amount of time Karpenter should wait to consolidate a node after a pod has been added or removed from the node.
 # You can choose to disable consolidation entirely by setting the string value 'Never' here
 consolidateAfter: 1m | Never # Added to allow additional control over consolidation aggressiveness

 # Budgets control the speed Karpenter can scale down nodes.
 # Karpenter will respect the minimum of the currently active budgets, and will round up
 # when considering percentages. Duration and Schedule must be set together.
 budgets:
 - nodes: 10%
 # On Weekdays during business hours, don't do any deprovisioning.
 - schedule: "0 9 * * mon-fri"
 duration: 8h
 nodes: "0"

 # Resource limits constrain the total size of the pool.
 # Limits prevent Karpenter from creating new instances once the limit is exceeded.
 limits:
 cpu: "1000"
 memory: 1000Gi

 # Priority given to the NodePool when the scheduler considers which NodePool
 # to select. Higher weights indicate higher priority when comparing NodePools.
 # Specifying no weight is equivalent to specifying a weight of 0.
 weight: 10
status:
 conditions:
 - type: Initialized
 status: "False"
 observedGeneration: 1
 lastTransitionTime: "2024-02-02T19:54:34Z"
 reason: NodeClaimNotLaunched
 message: "NodeClaim hasn't succeeded launch"
 resources:
 cpu: "20"
 memory: "8192Mi"
 ephemeral-storage: "100Gi"

metadata.name

The name of the NodePool.

spec.template.metadata.labels

Arbitrary key/value pairs to apply to all nodes.

spec.template.metadata.annotations

Arbitrary key/value pairs to apply to all nodes.

spec.template.spec.nodeClassRef

This field points to the Cloud Provider NodeClass resource. See EC2NodeClasses for details.

spec.template.spec.taints

Taints to add to provisioned nodes. Pods that don’t tolerate those taints could be prevented from scheduling. See Taints and Tolerations for details.

spec.template.spec.startupTaints

Taints that are added to nodes to indicate that a certain condition must be met, such as starting an agent or setting up networking, before the node is can be initialized. These taints must be cleared before pods can be deployed to a node.

spec.template.spec.expireAfter

The amount of time a Node can live on the cluster before being deleted by Karpenter. Nodes will begin draining once it’s expiration has been hit.

spec.template.spec.terminationGracePeriod

The amount of time a Node can be draining before Karpenter forcibly cleans up the node. Pods blocking eviction like PDBs and do-not-disrupt will be respected during draining until the terminationGracePeriod is reached, where those pods will be forcibly deleted.

spec.template.spec.requirements

Kubernetes defines the following Well-Known Labels, and cloud providers (e.g., AWS) implement them. They are defined at the “spec.requirements” section of the NodePool API.

In addition to the well-known labels from Kubernetes, Karpenter supports AWS-specific labels for more advanced scheduling. See the full list here.

These well-known labels may be specified at the NodePool level, or in a workload definition (e.g., nodeSelector on a pod.spec). Nodes are chosen using both the NodePool’s and pod’s requirements. If there is no overlap, nodes will not be launched. In other words, a pod’s requirements must be within the NodePool’s requirements. If a requirement is not defined for a well known label, any value available to the cloud provider may be chosen.

For example, an instance type may be specified using a nodeSelector in a pod spec. If the instance type requested is not included in the NodePool list and the NodePool has instance type requirements, Karpenter will not create a node or schedule the pod.

Well-Known Labels

Instance Types

key: node.kubernetes.io/instance-type
key: karpenter.k8s.aws/instance-family
key: karpenter.k8s.aws/instance-category
key: karpenter.k8s.aws/instance-generation

Generally, instance types should be a list and not a single value. Leaving these requirements undefined is recommended, as it maximizes choices for efficiently placing pods.

Review AWS instance types. Most instance types are supported with the exclusion of non-HVM.

Availability Zones

key: topology.kubernetes.io/zone
value example: us-east-1c
value list: aws ec2 describe-availability-zones --region <region-name>

Karpenter can be configured to create nodes in a particular zone. Note that the Availability Zone us-east-1a for your AWS account might not have the same location as us-east-1a for another AWS account.

Learn more about Availability Zone IDs.

Architecture

key: kubernetes.io/arch
values
- amd64
- arm64

Karpenter supports amd64 nodes, and arm64 nodes.

Operating System

key: kubernetes.io/os
values
- linux
- windows

Karpenter supports linux and windows operating systems.

Capacity Type

key: karpenter.sh/capacity-type
values
- spot
- on-demand
- reserved

Karpenter supports specifying capacity type, which is analogous to EC2 purchase options.

If a NodePool is compatible with multiple capacity types, Karpenter will prioritize reserved capacity, followed by spot, then finally on-demand. If the provider API (e.g. EC2 Fleet’s API) indicates capacity is unavailable, Karpenter caches that result across all attempts to provision EC2 capacity for that instance type and zone for the next 3 minutes. If there are no other possible offerings available for a higher priority capacity type, Karpenter will attempt to fallback to a lower priority capacity type, generally within milliseconds.

Karpenter also allows karpenter.sh/capacity-type to be used as a topology key for enforcing topology-spread.

Note

There is currently a limit of 100 on the total number of requirements on both the NodePool and the NodeClaim. It’s important to note that spec.template.metadata.labels are also propagated as requirements on the NodeClaim when it’s created, meaning that you can’t have more than 100 requirements and labels combined set on your NodePool.

Min Values

Along with the combination of [key,operator,values] in the requirements, Karpenter also supports minValues in the NodePool requirements block, allowing the scheduler to be aware of user-specified flexibility minimums while scheduling pods to a cluster. If Karpenter cannot meet this minimum flexibility for each key when scheduling a pod, it will fail the scheduling loop for that NodePool, either falling back to another NodePool which meets the pod requirements or failing scheduling the pod altogether.

For example, the below spec will use spot instance type for all provisioned instances and enforces minValues to various keys where it is defined i.e at least 2 unique instance families from [c,m,r], 5 unique instance families [eg: “m5”,“m5d”,“r4”,“c5”,“c5d”,“c4” etc], 10 unique instance types [eg: “c5.2xlarge”,“c4.xlarge” etc] is required for scheduling the pods.

spec:
 template:
 spec:
 requirements:
 - key: kubernetes.io/arch
 operator: In
 values: ["amd64"]
 - key: kubernetes.io/os
 operator: In
 values: ["linux"]
 - key: karpenter.k8s.aws/instance-category
 operator: In
 values: ["c", "m", "r"]
 minValues: 2
 - key: karpenter.k8s.aws/instance-family
 operator: Exists
 minValues: 5
 - key: node.kubernetes.io/instance-type
 operator: Exists
 minValues: 10
 - key: karpenter.k8s.aws/instance-generation
 operator: Gt
 values: ["2"]

Note that minValues can be used with multiple operators and multiple requirements. And if the minValues are defined with multiple operators for the same requirement key, scheduler considers the max of all the minValues for that requirement. For example, the below spec requires scheduler to consider at least 5 instance-family to schedule the pods.

spec:
 template:
 spec:
 requirements:
 - key: kubernetes.io/arch
 operator: In
 values: ["amd64"]
 - key: kubernetes.io/os
 operator: In
 values: ["linux"]
 - key: karpenter.k8s.aws/instance-category
 operator: In
 values: ["c", "m", "r"]
 minValues: 2
 - key: karpenter.k8s.aws/instance-family
 operator: Exists
 minValues: 5
 - key: karpenter.k8s.aws/instance-family
 operator: In
 values: ["m5","m5d","c5","c5d","c4","r4"]
 minValues: 3
 - key: node.kubernetes.io/instance-type
 operator: Exists
 minValues: 10
 - key: karpenter.k8s.aws/instance-generation
 operator: Gt
 values: ["2"]

Karpenter allows you to be extremely flexible with your NodePools by only constraining your instance types in ways that are absolutely necessary for your cluster. By default, Karpenter will enforce that you specify the spec.template.spec.requirements field, but will not enforce that you specify any requirements within the field. If you choose to specify requirements: [], this means that you will completely flexible to all instance types that your cloud provider supports.

Though Karpenter doesn’t enforce these defaults, for most use-cases, we recommend that you specify some requirements to avoid odd behavior or exotic instance types. Below, is a high-level recommendation for requirements that should fit the majority of use-cases for generic workloads

spec:
 template:
 spec:
 requirements:
 - key: kubernetes.io/arch
 operator: In
 values: ["amd64"]
 - key: kubernetes.io/os
 operator: In
 values: ["linux"]
 - key: karpenter.sh/capacity-type
 operator: In
 values: ["on-demand"]
 - key: karpenter.k8s.aws/instance-category
 operator: In
 values: ["c", "m", "r"]
 - key: karpenter.k8s.aws/instance-generation
 operator: Gt
 values: ["2"]

spec.disruption

You can configure Karpenter to disrupt Nodes through your NodePool in multiple ways. You can use spec.disruption.consolidationPolicy, spec.disruption.consolidateAfter, or spec.template.spec.expireAfter. You can also rate limit Karpenter’s disruption through the NodePool’s spec.disruption.budgets. Read Disruption for more.

spec.limits

The NodePool spec includes a limits section (spec.limits), which constrains the maximum amount of resources that the NodePool can consume.

If the NodePool.spec.limits section is unspecified, it means that there is no default limitation on resource allocation. In this case, the maximum resource consumption is governed by the quotas set by your cloud provider. If a limit has been exceeded, nodes provisioning is prevented until some nodes have been terminated.

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
 name: default
spec:
 template:
 spec:
 requirements:
 - key: karpenter.sh/capacity-type
 operator: In
 values: ["spot"]
 limits:
 cpu: 1000
 memory: 1000Gi
 nvidia.com/gpu: 2

Note

Karpenter provisioning is highly parallel. Because of this, limit checking is eventually consistent, which can result in overrun during rapid scale outs.

CPU limits are described with a DecimalSI value. Note that the Kubernetes API will coerce this into a string, so we recommend against using integers to avoid GitOps skew.

Memory limits are described with a BinarySI value, such as 1000Gi.

You can view the current consumption of cpu and memory on your cluster by running:

kubectl get nodepool -o=jsonpath='{.items[0].status}'

Review the Kubernetes core API (k8s.io/api/core/v1) for more information on resources.

spec.weight

Karpenter allows you to describe NodePool preferences through a weight mechanism similar to how weight is described with pod and node affinities.

For more information on weighting NodePools, see the Weighted NodePools section in the scheduling docs.

status.conditions

Conditions objects add observability features to Karpenter.

The status.conditions.type object reflects node status, such as Initialized or Available.
The status of the condition, status.conditions.status, indicates if the condition is True or False.
The status.conditions.observedGeneration indicates if the instance is out of date with the current state of .metadata.generation.
The status.conditions.lastTransitionTime object contains a programatic identifier that indicates the time of the condition’s previous transition.
The status.conditions.reason object indicates the reason for the condition’s previous transition.
The status.conditions.message object provides human-readable details about the condition’s previous transition.

NodePools have the following status conditions:

Condition Type	Description
NodeClassReady	Underlying nodeClass is ready
ValidationSucceeded	NodePool CRD validation succeeded
Ready	Top level condition that indicates if the nodePool is ready. This condition will not be true until all the other conditions on nodePool are true.

If a NodePool is not ready, it will not be considered for scheduling.

status.resources

Objects under status.resources provide information about the status of resources such as cpu, memory, and ephemeral-storage.

Examples

Isolating Expensive Hardware

A NodePool can be set up to only provision nodes on particular processor types. The following example sets a taint that only allows pods with tolerations for Nvidia GPUs to be scheduled:

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
 name: gpu
spec:
 disruption:
 consolidationPolicy: WhenEmptyOrUnderutilized
 template:
 spec:
 requirements:
 - key: node.kubernetes.io/instance-type
 operator: In
 values: ["p3.8xlarge", "p3.16xlarge"]
 taints:
 - key: nvidia.com/gpu
 value: "true"
 effect: NoSchedule

In order for a pod to run on a node defined in this NodePool, it must tolerate nvidia.com/gpu in its pod spec.

Cilium Startup Taint

Per the Cilium docs, it’s recommended to place a taint of node.cilium.io/agent-not-ready=true:NoExecute on nodes to allow Cilium to configure networking prior to other pods starting. This can be accomplished via the use of Karpenter startupTaints. These taints are placed on the node, but pods aren’t required to tolerate these taints to be considered for provisioning.

Failure to provide accurate startupTaints can result in Karpenter continually provisioning new nodes. When the new node joins and the startup taint that Karpenter is unaware of is added, Karpenter now considers the pending pod to be unschedulable to this node. Karpenter will attempt to provision yet another new node to schedule the pending pod.

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
 name: cilium-startup
spec:
 disruption:
 consolidationPolicy: WhenEmptyOrUnderutilized
 template:
 spec:
 startupTaints:
 - key: node.cilium.io/agent-not-ready
 value: "true"
 effect: NoExecute

V1.5: NodeClaims

Mon, 01 Jan 0001 00:00:00 +0000

Karpenter uses NodeClaims to manage the lifecycle of Kubernetes Nodes with the underlying cloud provider. Karpenter will create and delete NodeClaims in response to the demands of Pods in the cluster. It does this by evaluating the requirements of pending pods, finding a compatible NodePool and NodeClass pair, and creating a NodeClaim which meets both sets of requirements. Although NodeClaims are immutable resources managed by Karpenter, you can monitor NodeClaims to keep track of the status of your Nodes.

In addition to tracking the lifecycle of Nodes, NodeClaims serve as requests for capacity. Karpenter creates NodeClaims in response to provisioning and disruption needs (pre-spin). Whenever Karpenter creates a NodeClaim, it asks the cloud provider to create the instance (launch), register and link the created node with the NodeClaim (registration), and wait for the node and its resources to be ready (initialization).

This page describes how NodeClaims integrate throughout Karpenter and the cloud provider implementation.

If you want to learn more about the nodes being managed by Karpenter, you can either look directly at the NodeClaim or at the nodes they are associated with:

Checking NodeClaims: If something goes wrong in the process of creating a node, you can look at the NodeClaim to see where the node creation process might have failed. kubectl get nodeclaims will show you the NodeClaims for the cluster, and its linked node. Using kubectl describe nodeclaim <nodeclaim> will show the status of a particular NodeClaim. For example, if the node is NotReady, you might see statuses indicating that the NodeClaim failed to launch, register, or initialize. There will be logs emitted by the Karpenter controller to indicate this too.
Checking nodes: Use commands such as kubectl get node and kubectl describe node <nodename> to see the actual resources, labels, and other attributes associated with a particular node.

NodeClaim roles in node creation

NodeClaims provide a critical role in the Karpenter workflow for provisioning capacity, and in node disruptions.

The following diagram illustrates how NodeClaims interact with other components during Karpenter-driven node creation.

Note

Configure the KARPENTER_NAMESPACE environment variable to the namespace where you’ve installed Karpenter (kube-system is the default). Follow along with the Karpenter logs in your cluster and do the following:

export KARPENTER_NAMESPACE="kube-system"
kubectl logs -f -n "${KARPENTER_NAMESPACE}" \
 -l app.kubernetes.io/name=karpenter

In a separate terminal, start some pods that would require Karpenter to create nodes to handle those pods. For example, start up some inflate pods as described in Scale up deployment.

As illustrated in the previous diagram, Karpenter interacts with NodeClaims and related components when creating a node:

Watches for pods and monitors NodePools and NodeClasses:

Checks the pod scheduling constraints and resource requests.
Cross-references the requirements with the existing NodePools and NodeClasses, (e.g. zones, arch, os)

Example log:

{
 "level": "INFO",
 "time": "2024-06-22T02:24:16.114Z",
 "message": "found provisionable pod(s)",
 "commit": "490ef94",
 "Pods": "default/inflate-66fb68585c-xvs86, default/inflate-66fb68585c-hpcdz, default/inflate-66fb68585c-8xztf,01234567adb205c7e default/inflate-66fb68585c-t29d8, default/inflate-66fb68585c-nxflz",
 "duration": "100.761702ms"
}

Computes the shape and size of a NodeClaim (or NodeClaims) to create in the cluster to fit the set of pods from step 1.

Example log:

{
 "level": "INFO",
 "time": "2024-06-22T02:24:16.114Z",
 "message": "computed new nodeclaim(s) to fit pod(s)",
 "controller": "provisioner",
 "nodeclaims": 1,
 "pods": 5
}

Creates the NodeClaim object in the cluster.

Example log:

{
 "level": "INFO",
 "time": "2024-06-22T02:24:16.128Z",
 "message": "created nodeclaim",
 "controller": "provisioner",
 "NodePool": {
 "name":"default"
 },
 "NodeClaim": {
 "name":"default-sfpsl"
 },
 "requests": {
 "cpu":"5150m",
 "pods":"8"
 },
 "instance-types": "c3.2xlarge, c4.2xlarge, c4.4xlarge, c5.2xlarge, c5.4xlarge and 55 other(s)"
}

Finds the new NodeClaim and translates it into an API call to create a cloud provider instance, logging the response of the API call.

If the API response is an unrecoverable error, such as an Insufficient Capacity Error, Karpenter will delete the NodeClaim, mark that instance type as temporarily unavailable, and create another NodeClaim if necessary.

Example log:

{
 "level": "INFO",
 "time": "2024-06-22T02:24:19.028Z",
 "message": "launched nodeclaim",
 "controller": "nodeclaim.lifecycle",
 "NodeClaim": {
 "name": "default-sfpsl"
 },
 "provider-id": "aws:///us-west-2b/i-01234567adb205c7e",
 "instance-type": "c3.2xlarge",
 "zone": "us-west-2b",
 "capacity-type": "spot",
 "allocatable": {
 "cpu": "7910m",
 "ephemeral-storage": "17Gi",
 "memory": "13215Mi",
 "pods": "58"
 }
}

Karpenter watches for the instance to register itself with the cluster as a node, and updates the node’s labels, annotations, taints, owner refs, and finalizer to match what was defined in the NodePool and NodeClaim. Once this step is completed, Karpenter will remove the karpenter.sh/unregistered taint from the Node.

If this fails to succeed within 15 minutes, Karpenter will remove the NodeClaim from the cluster and delete the underlying instance, creating another NodeClaim if necessary.

Example log:
```
{
 "level": "INFO",
 "time": "2024-06-22T02:26:19.028Z",
 "message": "registered nodeclaim",
 "controller": "nodeclaim.lifecycle",
 "NodeClaim": {
 "name": "default-sfpsl"
 },
 "provider-id": "aws:///us-west-2b/i-01234567adb205c7e",
 "Node": {
 "name": "ip-xxx-xxx-xx-xxx.us-west-2.compute.internal"
 }
}
```

Karpenter continues to watch the node, waiting until the node becomes ready, has all its startup taints removed, and has all requested resources registered on the node.

Example log:

{
 "level": "INFO",
 "time": "2024-06-22T02:24:52.642Z",
 "message": "initialized nodeclaim",
 "controller": "nodeclaim.lifecycle",
 "NodeClaim": {
 "name": "default-sfpsl"
 },
 "provider-id": "aws:///us-west-2b/i-01234567adb205c7e",
 "Node": {
 "name": "ip-xxx-xxx-xx-xxx.us-west-2.compute.internal"
 },
 "allocatable": {
 "cpu": "7910m",
 "ephemeral-storage": "18242267924",
 "hugepages-2Mi": "0",
 "memory": "14320468Ki",
 "pods": "58"
 }
}

NodeClaim example

The following is an example of a NodeClaim. Keep in mind that you cannot modify a NodeClaim. To see the contents of a NodeClaim, get the name of your NodeClaim, then run kubectl describe to see its contents:

kubectl get nodeclaim
NAME TYPE ZONE NODE READY AGE
default-m6pzn c7i-flex.2xlarge us-west-1a ip-xxx-xxx-xx-xxx.us-west-1.compute.internal True 7m50s
kubectl describe nodeclaim default-m6pzn

Starting at the bottom of this example, here are some highlights of what the NodeClaim contains:

The Node Name (ip-xxx-xxx-xx-xxx.us-west-1.compute.internal) and Provider ID (aws:///us-west-1a/i-xxxxxxxxxxxxxxxxx) identify the instance that is fulfilling this NodeClaim.
Image ID (ami-0ccbbed159cce4e37) represents the operating system image running on the node.
Status shows the resources that are available on the node (CPU, memory, and so on) as well as the conditions associated with the node. The conditions show the status of the node, including whether the node is launched, registered, and initialized. This is particularly useful if Pods are not deploying to the node and you want to determine the cause.
Spec contains the metadata required for Karpenter to launch and manage an instance. This includes any scheduling requirements, resource requirements, the NodeClass reference, taints, and immutable disruption fields (expireAfter and terminationGracePeriod).
Additional information includes annotations and labels which should be synced to the Node, creation metadata, the termination finalizer, and the owner reference.

Name: default-x9wxq
Namespace:
Labels: karpenter.k8s.aws/instance-category=c
karpenter.k8s.aws/instance-cpu=8
karpenter.k8s.aws/instance-cpu-manufacturer=amd
karpenter.k8s.aws/instance-ebs-bandwidth=3170
karpenter.k8s.aws/instance-encryption-in-transit-supported=true
karpenter.k8s.aws/instance-family=c5a
karpenter.k8s.aws/instance-generation=5
karpenter.k8s.aws/instance-hypervisor=nitro
karpenter.k8s.aws/instance-memory=16384
karpenter.k8s.aws/instance-network-bandwidth=2500
karpenter.k8s.aws/instance-size=2xlarge
karpenter.sh/capacity-type=spot
karpenter.sh/nodepool=default
kubernetes.io/arch=amd64
kubernetes.io/os=linux
node.kubernetes.io/instance-type=c5a.2xlarge
topology.k8s.aws/zone-id=usw2-az3
topology.kubernetes.io/region=us-west-2
topology.kubernetes.io/zone=us-west-2c
Annotations: compatibility.karpenter.k8s.aws/cluster-name-tagged: true
compatibility.karpenter.k8s.aws/kubelet-drift-hash: 15379597991425564585
karpenter.k8s.aws/ec2nodeclass-hash: 5763643673275251833
karpenter.k8s.aws/ec2nodeclass-hash-version: v3
karpenter.k8s.aws/tagged: true
karpenter.sh/nodepool-hash: 377058807571762610
karpenter.sh/nodepool-hash-version: v3
API Version: karpenter.sh/v1
Kind: NodeClaim
Metadata:
Creation Timestamp: 2024-08-07T05:37:30Z
Finalizers:
karpenter.sh/termination
Generate Name: default-
Generation: 1
Owner References:
API Version: karpenter.sh/v1
Block Owner Deletion: true
Kind: NodePool
Name: default
UID: 6b9c6781-ac05-4a4c-ad6a-7551a07b2ce7
Resource Version: 19600526
UID: 98a2ba32-232d-45c4-b7c0-b183cfb13d93
Spec:
Expire After: 720h0m0s
Node Class Ref:
Group:
Kind: EC2NodeClass
Name: default
Requirements:
Key: kubernetes.io/arch
Operator: In
Values:
amd64
Key: kubernetes.io/os
Operator: In
Values:
linux
Key: karpenter.sh/capacity-type
Operator: In
Values:
spot
Key: karpenter.k8s.aws/instance-category
Operator: In
Values:
c
m
r
Key: karpenter.k8s.aws/instance-generation
Operator: Gt
Values:
2
Key: karpenter.sh/nodepool
Operator: In
Values:
default
Key: node.kubernetes.io/instance-type
Operator: In
Values:
c3.xlarge
c4.xlarge
c5.2xlarge
c5.xlarge
c5a.xlarge
c5ad.2xlarge
c5ad.xlarge
c5d.2xlarge
Resources:
Requests:
Cpu: 3150m
Pods: 6
Startup Taints:
Effect: NoSchedule
Key: app.dev/example-startup
Taints:
Effect: NoSchedule
Key: app.dev/example
Termination Grace Period: 1h0m0s
Status:
Allocatable:
Cpu: 7910m
Ephemeral - Storage: 17Gi
Memory: 14162Mi
Pods: 58
vpc.amazonaws.com/pod-eni: 38
Capacity:
Cpu: 8
Ephemeral - Storage: 20Gi
Memory: 15155Mi
Pods: 58
vpc.amazonaws.com/pod-eni: 38
Conditions:
Last Transition Time: 2024-08-07T05:38:08Z
Message:
Reason: Consolidatable
Status: True
Type: Consolidatable
Last Transition Time: 2024-08-07T05:38:07Z
Message:
Reason: Initialized
Status: True
Type: Initialized
Last Transition Time: 2024-08-07T05:37:33Z
Message:
Reason: Launched
Status: True
Type: Launched
Last Transition Time: 2024-08-07T05:38:07Z
Message:
Reason: Ready
Status: True
Type: Ready
Last Transition Time: 2024-08-07T05:37:55Z
Message:
Reason: Registered
Status: True
Type: Registered
Image ID: ami-08946d4d49fc3f27b
Node Name: ip-xxx-xxx-xxx-xxx.us-west-2.compute.internal
Provider ID: aws:///us-west-2c/i-01234567890123
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Launched 70s karpenter Status condition transitioned, Type: Launched, Status: Unknown -> True, Reason: Launched
Normal DisruptionBlocked 70s karpenter Cannot disrupt NodeClaim: state node doesn't contain both a node and a nodeclaim
Normal Registered 48s karpenter Status condition transitioned, Type: Registered, Status: Unknown -> True, Reason: Registered
Normal Initialized 36s karpenter Status condition transitioned, Type: Initialized, Status: Unknown -> True, Reason: Initialized
Normal Ready 36s karpenter Status condition transitioned, Type: Ready, Status: Unknown -> True, Reason: Ready

V1.5: Scheduling

Mon, 01 Jan 0001 00:00:00 +0000

If your pods have no requirements for how or where to run, you can let Karpenter choose nodes from the full range of available cloud provider resources. However, by taking advantage of Karpenter’s model of layered constraints, you can be sure that the precise type and amount of resources needed are available to your pods. Reasons for constraining where your pods run could include:

Needing to run in zones where dependent applications or storage are available
Requiring certain kinds of processors or other hardware
Wanting to use techniques like topology spread to help ensure high availability

Your Cloud Provider defines the first layer of constraints, including all instance types, architectures, zones, and purchase types available to its cloud. The cluster administrator adds the next layer of constraints by creating one or more NodePools. The final layer comes from you adding specifications to your Kubernetes pod deployments. Pod scheduling constraints must fall within a NodePool’s constraints or the pods will not deploy. For example, if the NodePool sets limits that allow only a particular zone to be used, and a pod asks for a different zone, it will not be scheduled.

Constraints you can request include:

Resource requests: Request that certain amount of memory or CPU be available.
Node selection: Choose to run on a node that is has a particular label (nodeSelector).
Node affinity: Draws a pod to run on nodes with particular attributes (affinity).
Topology spread: Use topology spread to help ensure availability of the application.
Pod affinity/anti-affinity: Draws pods towards or away from topology domains based on the scheduling of other pods.

Karpenter supports standard Kubernetes scheduling constraints. This allows you to define a single set of rules that apply to both existing and provisioned capacity.

Note

Karpenter supports specific Well-Known Labels, Annotations and Taints that are useful for scheduling.

Resource requests

Within a Pod spec, you can both make requests and set limits on resources a pod needs, such as CPU and memory. For example:

apiVersion: v1
kind: Pod
metadata:
 name: myapp
spec:
 containers:
 - name: app
 image: myimage
 resources:
 requests:
 memory: "128Mi"
 cpu: "500m"
 limits:
 memory: "256Mi"
 cpu: "1000m"

In this example, the container is requesting 128MiB of memory and .5 CPU. Its limits are set to 256MiB of memory and 1 CPU. Instance type selection math only uses requests, but limits may be configured to enable resource oversubscription.

See Managing Resources for Containers for details on resource types supported by Kubernetes, Specify a memory request and a memory limit for examples of memory requests, and NodePools for a list of supported resources.

Accelerators/GPU Resources

Accelerator (e.g., GPU) values include

nvidia.com/gpu
amd.com/gpu
aws.amazon.com/neuron
aws.amazon.com/neuroncore
habana.ai/gaudi

Karpenter supports accelerators, such as GPUs.

Additionally, include a resource requirement in the workload manifest. This will cause the GPU dependent pod to be scheduled onto the appropriate node.

Here is an example of an accelerator resource in a workload manifest (e.g., pod):

spec:
 template:
 spec:
 containers:
 - resources:
 limits:
 nvidia.com/gpu: "1"

Note

If you are provisioning nodes that will utilize accelerators/GPUs, you need to deploy the appropriate device plugin daemonset. Without the respective device plugin daemonset, Karpenter will not see those nodes as initialized. Refer to general Kubernetes GPU docs and the following specific GPU docs:

nvidia.com/gpu: NVIDIA device plugin for Kubernetes
amd.com/gpu: AMD GPU device plugin for Kubernetes
aws.amazon.com/neuron/aws.amazon.com/neuroncore: AWS Neuron device plugin for Kubernetes
habana.ai/gaudi: Habana device plugin for Kubernetes

AWS Neuron Resources

The Neuron scheduler extension is required for pods that require more than one Neuron core (aws.amazon.com/neuroncore) or device (aws.amazon.com/neuron) resource, but less than all available Neuron cores or devices on a node. From the AWS Neuron documentation:

The Neuron scheduler extension finds sets of directly connected devices with minimal communication latency when scheduling containers. On Inf1 and Inf2 instance types where Neuron devices are connected through a ring topology, the scheduler finds sets of contiguous devices. For example, for a container requesting 3 Neuron devices the scheduler might assign Neuron devices 0,1,2 to the container if they are available but never devices 0,2,4 because those devices are not directly connected. On Trn1.32xlarge and Trn1n.32xlarge instance types where devices are connected through a 2D torus topology, the Neuron scheduler enforces additional constraints that containers request 1, 4, 8, or all 16 devices. If your container requires a different number of devices, such as 2 or 5, we recommend that you use an Inf2 instance instead of Trn1 to benefit from more advanced topology.

However, Karpenter is not aware of the decisions made by the Neuron scheduler extension which precludes it from making any optimizations to consolidate and bin pack pods requiring Neuron resources. To ensure Karpenter’s bin-packing is consistent with the decisions made by the scheduler extension, containers must have like-sized, power of 2 requests (e.g. 1, 2, 4, etc). Failing to do so may result in permanently pending pods.

Pod ENI Resources (Security Groups for Pods)

Pod ENI is a feature of the AWS VPC CNI Plugin which allows an Elastic Network Interface (ENI) to be allocated directly to a Pod. When enabled, the vpc.amazonaws.com/pod-eni extended resource is added to supported nodes. The Pod ENI feature can be used independently, but is most often used in conjunction with Security Groups for Pods. Follow the below instructions to enable support for Pod ENI and/or Security Groups for Pods in Karpenter.

Note

You must enable Pod ENI support in the AWS VPC CNI Plugin before enabling Pod ENI support in Karpenter. Please refer to the Security Groups for Pods documentation for instructions.

Note

If you’ve enabled Security Groups per Pod, one of the instance’s ENIs is reserved. To avoid discrepancies between the maxPods value and the node’s supported pod density, you need to set RESERVED_ENIS=1.

Here is an example of a pod-eni resource defined in a deployment manifest:

spec:
template:
spec:
containers:
- resources:
limits:
vpc.amazonaws.com/pod-eni: "1"

Windows Support Notice

Security groups for pods are currently unsupported for Windows nodes

Selecting nodes

With nodeSelector you can ask for a node that matches selected key-value pairs. This can include well-known labels or custom labels you create yourself.

You can use affinity to define more complicated constraints, see Node Affinity for the complete specification.

Labels

Well-known labels may be specified as NodePool requirements or pod scheduling constraints. You can also define your own custom labels by specifying requirements or labels on your NodePool and select them using nodeAffinity or nodeSelectors on your Pods.

Warning

Take care to ensure the label domains are correct. A well known label like karpenter.k8s.aws/instance-family will enforce node properties, but may be confused with node.kubernetes.io/instance-family, which is unknown to Karpenter, and treated as a custom label which will not enforce node properties.

Well-Known Labels

Label	Example	Description
topology.kubernetes.io/zone	us-east-2a	Zones are defined by your cloud provider (aws)
node.kubernetes.io/instance-type	g4dn.8xlarge	Instance types are defined by your cloud provider (aws)
node.kubernetes.io/windows-build	10.0.17763	Windows OS build in the format “MajorVersion.MinorVersion.BuildNumber”. Can be `10.0.17763` for WS2019, or `10.0.20348` for WS2022. (k8s)
kubernetes.io/os	linux	Operating systems are defined by GOOS values (`KnownOS`) on the instance
kubernetes.io/arch	amd64	Architectures are defined by GOARCH values (`KnownArch`) on the instance
karpenter.sh/capacity-type	spot	Capacity types include `reserved`, `spot`, and `on-demand`
karpenter.k8s.aws/capacity-reservation-id	cr-56fac701cc1951b03	[AWS Specific] The capacity reservation ID. Only present on reserved nodes.
karpenter.k8s.aws/instance-hypervisor	nitro	[AWS Specific] Instance types that use a specific hypervisor
karpenter.k8s.aws/instance-encryption-in-transit-supported	true	[AWS Specific] Instance types that support (or not) in-transit encryption
karpenter.k8s.aws/instance-category	g	[AWS Specific] Instance types of the same category, usually the string before the generation number
karpenter.k8s.aws/instance-generation	4	[AWS Specific] Instance type generation number within an instance category
karpenter.k8s.aws/instance-family	g4dn	[AWS Specific] Instance types of similar properties but different resource quantities
karpenter.k8s.aws/instance-size	8xlarge	[AWS Specific] Instance types of similar resource quantities but different properties
karpenter.k8s.aws/instance-cpu	32	[AWS Specific] Number of CPUs on the instance
karpenter.k8s.aws/instance-cpu-manufacturer	aws	[AWS Specific] Name of the CPU manufacturer
karpenter.k8s.aws/instance-cpu-sustained-clock-speed-mhz	3600	[AWS Specific] The CPU clock speed, in MHz
karpenter.k8s.aws/instance-memory	131072	[AWS Specific] Number of mebibytes of memory on the instance
karpenter.k8s.aws/instance-ebs-bandwidth	9500	[AWS Specific] Number of maximum megabits of EBS available on the instance
karpenter.k8s.aws/instance-network-bandwidth	131072	[AWS Specific] Number of baseline megabits available on the instance
karpenter.k8s.aws/instance-pods	110	[AWS Specific] Number of pods the instance supports
karpenter.k8s.aws/instance-gpu-name	t4	[AWS Specific] Name of the GPU on the instance, if available
karpenter.k8s.aws/instance-gpu-manufacturer	nvidia	[AWS Specific] Name of the GPU manufacturer
karpenter.k8s.aws/instance-gpu-count	1	[AWS Specific] Number of GPUs on the instance
karpenter.k8s.aws/instance-gpu-memory	16384	[AWS Specific] Number of mebibytes of memory on the GPU
karpenter.k8s.aws/instance-local-nvme	900	[AWS Specific] Number of gibibytes of local nvme storage on the instance

Note

Karpenter translates the following deprecated labels to their stable equivalents: failure-domain.beta.kubernetes.io/zone, failure-domain.beta.kubernetes.io/region, beta.kubernetes.io/arch, beta.kubernetes.io/os, and beta.kubernetes.io/instance-type.

User-Defined Labels

Karpenter is aware of several well-known labels, deriving them from instance type details. If you specify a nodeSelector or a required nodeAffinity using a label that is not well-known to Karpenter, it will not launch nodes with these labels and pods will remain pending. For Karpenter to become aware that it can schedule for these labels, you must specify the label in the NodePool requirements with the Exists operator:

requirements:
 - key: user.defined.label/type
 operator: Exists

Note

Node selectors

Here is an example of a nodeSelector for selecting nodes:

nodeSelector:
 topology.kubernetes.io/zone: us-west-2a
 karpenter.sh/capacity-type: spot

This example features a well-known label (topology.kubernetes.io/zone) and a label that is well known to Karpenter (karpenter.sh/capacity-type).

If you want to create a custom label, you should do that at the NodePool level. Then the pod can declare that custom label.

See nodeSelector in the Kubernetes documentation for details.

Preferences

Karpenter is aware of preferences (node affinity, pod affinity, pod anti-affinity, and pod topology) and treats them as requirements in most circumstances. Karpenter uses these preferences when determining if a pod can schedule on a node (absent topology requirements), or when determining if a pod can be shifted to a new node.

Karpenter starts by treating preferred affinities as required affinities when constructing requirements for a pod. When these requirements cannot be met, the pod’s preferences are relaxed one-at-a-time by ascending weight (lowest weight is relaxed first), and the remaining requirements are tried again.

Warning

Karpenter does not interpret preferred affinities as required when constructing topology requirements for scheduling to a node. If these preferences are necessary, required affinities should be used as documented in Node Affinity.

Node affinity

Examples below illustrate how to use Node affinity to include (In) and exclude (NotIn) objects. See Node affinity for details. When setting rules, the following Node affinity types define how hard or soft each rule is:

requiredDuringSchedulingIgnoredDuringExecution: This is a hard rule that must be met.
preferredDuringSchedulingIgnoredDuringExecution: This is a preference, but the pod can run on a node where it is not guaranteed.

Note

Preferred affinities on pods can result in more nodes being created than expected because Karpenter will prefer to create new nodes to satisfy preferences, see the preferences documentation for details.

The IgnoredDuringExecution part of each tells the pod to keep running, even if conditions change on the node so the rules no longer matched. You can think of these concepts as required and preferred, since Kubernetes never implemented other variants of these rules.

All examples below assume that the NodePool doesn’t have constraints to prevent those zones from being used. The first constraint says you could use us-west-2a or us-west-2b, the second constraint makes it so only us-west-2b can be used.

 affinity:
 nodeAffinity:
 requiredDuringSchedulingIgnoredDuringExecution:
 nodeSelectorTerms:
 - matchExpressions:
 - key: "topology.kubernetes.io/zone"
 operator: "In"
 values: ["us-west-2a", "us-west-2b"]
 - key: "topology.kubernetes.io/zone"
 operator: "In"
 values: ["us-west-2b"]

Changing the second operator to NotIn would allow the pod to run in us-west-2a only:

 - key: "topology.kubernetes.io/zone"
 operator: "In"
 values: ["us-west-2a", "us-west-2b"]
 - key: "topology.kubernetes.io/zone"
 operator: "NotIn"
 values: ["us-west-2b"]

Continuing to add to the example, nodeAffinity lets you define terms so if one term doesn’t work it goes to the next one. Here, if us-west-2a is not available, the second term will cause the pod to run on a spot instance in us-west-2d.

 affinity:
 nodeAffinity:
 requiredDuringSchedulingIgnoredDuringExecution:
 nodeSelectorTerms:
 - matchExpressions: # OR
 - key: "topology.kubernetes.io/zone" # AND
 operator: "In"
 values: ["us-west-2a", "us-west-2b"]
 - key: "topology.kubernetes.io/zone" # AND
 operator: "NotIn"
 values: ["us-west-2b"]
 - matchExpressions: # OR
 - key: "karpenter.sh/capacity-type" # AND
 operator: "In"
 values: ["spot"]
 - key: "topology.kubernetes.io/zone" # AND
 operator: "In"
 values: ["us-west-2d"]

In general, Karpenter will go through each of the nodeSelectorTerms in order and take the first one that works. However, if Karpenter fails to provision on the first nodeSelectorTerms, it will try again using the second one. If they all fail, Karpenter will fail to provision the pod. Karpenter will backoff and retry over time. So if capacity becomes available, it will schedule the pod without user intervention.

Taints and tolerations

Taints are the opposite of affinity. Setting a taint on a node tells the scheduler to not run a pod on it unless the pod has explicitly said it can tolerate that taint. This example shows a NodePool that was set up with a taint for only running pods that require a GPU, such as the following:

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
 name: gpu
spec:
 template:
 spec:
 requirements:
 - key: karpenter.k8s.aws/instance-family
 operator: In
 values:
 - p3
 taints:
 - key: nvidia.com/gpu
 value: "true"
 effect: "NoSchedule"

For a pod to request to run on a node that has this NodePool, it could set a toleration as follows:

apiVersion: v1
kind: Pod
metadata:
 name: mygpupod
spec:
 containers:
 - name: gpuapp
 resources:
 requests:
 nvidia.com/gpu: 1
 limits:
 nvidia.com/gpu: 1
 image: mygpucontainer
 tolerations:
 - key: "nvidia.com/gpu"
 operator: "Exists"
 effect: "NoSchedule"

See Taints and Tolerations in the Kubernetes documentation for details.

Topology Spread

By using the Kubernetes topologySpreadConstraints you can ask the NodePool to have pods push away from each other to limit the blast radius of an outage. Think of it as the Kubernetes evolution for pod affinity: it lets you relate pods with respect to nodes while still allowing spread.

Note

Preferred topology spread (ScheduleAnyway) can result in more nodes being created than expected because Karpenter will prefer to create new nodes to satisfy spread constraints, see the preferences documentation for details.

For example:

spec:
 topologySpreadConstraints:
 - maxSkew: 1
 topologyKey: "topology.kubernetes.io/zone"
 whenUnsatisfiable: ScheduleAnyway
 labelSelector:
 matchLabels:
 dev: jjones
 - maxSkew: 1
 topologyKey: "kubernetes.io/hostname"
 whenUnsatisfiable: ScheduleAnyway
 labelSelector:
 matchLabels:
 dev: jjones
 - maxSkew: 1
 topologyKey: "karpenter.sh/capacity-type"
 whenUnsatisfiable: ScheduleAnyway
 labelSelector:
 matchLabels:
 dev: jjones

Adding this to your podspec would result in:

Pods being spread across zones, hosts, and capacity-type (topologyKey).
The dev labelSelector will include all pods with the label of dev=jjones in topology calculations. It is recommended to use a selector to match all pods in a deployment.
No more than one pod difference in the number of pods on each host (maxSkew). For example, if there were three nodes and five pods the pods could be spread 1, 2, 2 or 2, 1, 2 and so on. If instead the maxSkew were 5, pods could be spread 5, 0, 0 or 3, 2, 0, or 2, 1, 2 and so on.

The three supported topologyKey values that Karpenter supports are:

topology.kubernetes.io/zone
kubernetes.io/hostname
karpenter.sh/capacity-type

See Pod Topology Spread Constraints for details.

Note

NodePools do not attempt to balance or rebalance the availability zones for their nodes. Availability zone balancing may be achieved by defining zonal Topology Spread Constraints for Pods that require multi-zone durability, and NodePools will respect these constraints while optimizing for compute costs.

Pod affinity/anti-affinity

By using the podAffinity and podAntiAffinity configuration on a pod spec, you can inform the Karpenter scheduler of your desire for pods to schedule together or apart with respect to different topology domains.

Note

For example:

spec:
 affinity:
 podAffinity:
 requiredDuringSchedulingIgnoredDuringExecution:
 - labelSelector:
 matchExpressions:
 - key: system
 operator: In
 values:
 - backend
 topologyKey: topology.kubernetes.io/zone
 podAntiAffinity:
 requiredDuringSchedulingIgnoredDuringExecution:
 - labelSelector:
 matchLabels:
 app: inflate
 topologyKey: kubernetes.io/hostname

The above pod affinity rule would cause the pod to only schedule in zones where a pod with the label system=backend is already running.

The anti-affinity rule would cause it to avoid running on any node with a pod labeled app=inflate. If this anti-affinity term was on a deployment pod spec along with a matching app=inflate label, it would prevent more than one pod from the deployment from running on any single node.

See Inter-pod affinity and anti-affinity in the Kubernetes documentation for details.

Persistent Volume Topology

Karpenter automatically detects storage scheduling requirements and includes them in node launch decisions.

In the following example, the StorageClass defines zonal topologies for us-west-2a and us-west-2b and binding mode WaitForFirstConsumer. When the pod is created, Karpenter follows references from the Pod to PersistentVolumeClaim to StorageClass and identifies that this pod requires storage in us-west-2a and us-west-2b. It randomly selects us-west-2a, provisions a node in that zone, and waits for kube-scheduler to bind the pod to the node. The CSI driver creates a PersistentVolume according to the PersistentVolumeClaim and gives it a node affinity rule for us-west-2a.

Later on, the pod is deleted and a new pod is created that requests the same claim. This time, Karpenter identifies that a PersistentVolume already exists for the PersistentVolumeClaim, and includes its zone us-west-2a in the pod’s scheduling requirements.

apiVersion: v1
kind: Pod
metadata:
 name: app
spec:
 containers: ...
 volumes:
 - name: storage
 persistentVolumeClaim:
 claimName: ebs-claim
---
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
 name: ebs
provisioner: ebs.csi.aws.com
volumeBindingMode: WaitForFirstConsumer
allowedTopologies:
- matchLabelExpressions:
 - key: topology.ebs.csi.aws.com/zone
 values: ["us-west-2a", "us-west-2b"]
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
 name: ebs-claim
spec:
 accessModes:
 - ReadWriteOnce
 storageClassName: ebs
 resources:
 requests:
 storage: 4Gi

Note

☁️ AWS Specific

The EBS CSI driver uses topology.ebs.csi.aws.com/zone instead of the standard topology.kubernetes.io/zone label. Karpenter is aware of label aliasing and translates this label into topology.kubernetes.io/zone in memory. When configuring a StorageClass for the EBS CSI Driver, you must use topology.ebs.csi.aws.com/zone.

Note

The topology key topology.kubernetes.io/region is not supported. Legacy in-tree CSI providers specify this label. Instead, install an out-of-tree CSI provider. Learn more about moving to CSI providers.

Weighted NodePools

Karpenter allows you to order your NodePools using the .spec.weight field so that the Karpenter scheduler will attempt to schedule one NodePool before another.

Savings Plans and Reserved Instances

If you have purchased a Savings Plan or Reserved Instances, you may want to tell Karpenter to prioritize this reserved capacity ahead of other instance types.

To enable this, you will need to tell the Karpenter controllers which instance types to prioritize and what is the maximum amount of capacity that should be provisioned using those instance types. We can set the .spec.limits field on the NodePool to limit the capacity that can be launched by this NodePool. Combined with the .spec.weight value, we can tell Karpenter to pull from instance types in the reserved NodePool before defaulting to generic instance types.

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
 name: reserved-instance
spec:
 weight: 50
 limits:
 cpu: 100
 template:
 spec:
 requirements:
 - key: "node.kubernetes.io/instance-type"
 operator: In
 values: ["c4.large"]
---
apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
 name: default
spec:
 template:
 spec:
 requirements:
 - key: karpenter.sh/capacity-type
 operator: In
 values: ["spot", "on-demand"]
 - key: kubernetes.io/arch
 operator: In
 values: ["amd64"]

Fallback

Pods that do not specify node selectors or affinities can potentially be assigned to any node with any configuration. There may be cases where you require these pods to schedule to a specific capacity type or architecture but assigning the relevant node selectors or affinities to all these workload pods may be too tedious or infeasible. Instead, we want to define a cluster-wide default configuration for nodes launched using Karpenter.

By assigning a higher .spec.weight value and restricting a NodePool to a specific capacity type or architecture, we can set default configuration for the nodes launched by pods that don’t have node configuration restrictions.

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
 name: default
spec:
 weight: 50
 template:
 spec:
 requirements:
 - key: karpenter.sh/capacity-type
 operator: In
 values: ["spot", "on-demand"]
 - key: kubernetes.io/arch
 operator: In
 values: ["amd64"]
---
apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
 name: arm64-specific
spec:
 template:
 spec:
 requirements:
 - key: karpenter.sh/capacity-type
 operator: In
 values: ["spot", "on-demand"]
 - key: kubernetes.io/arch
 operator: In
 values: ["arm64"]
 - key: node.kubernetes.io/instance-type
 operator: In
 values: ["a1.large", "a1.xlarge"]

Note

Based on the way that Karpenter performs pod batching and bin packing, it is not guaranteed that Karpenter will always choose the highest priority NodePool given specific requirements. For example, if a pod can’t be scheduled with the highest priority NodePool, it will force creation of a node using a lower priority NodePool, allowing other pods from that batch to also schedule on that node. The behavior may also occur if existing capacity is available, as the kube-scheduler will schedule the pods instead of allowing Karpenter to provision a new node.

Advanced Scheduling Techniques

Scheduling based on Node Resources

You may want pods to be able to request resources of nodes that Kubernetes natively does not provide as a schedulable resource or that are aspects of certain nodes like High Performance Networking or NVME Local Storage. You can use Karpenter’s Well-Known Labels to accomplish this.

These can further be applied at the NodePool or Workload level using Requirements, NodeSelectors or Affinities

Pod example of requiring any NVME disk:

...
 affinity:
 nodeAffinity:
 requiredDuringSchedulingIgnoredDuringExecution:
 nodeSelectorTerms:
 - matchExpressions:
 - key: "karpenter.k8s.aws/instance-local-nvme"
 operator: "Exists"
...

NodePool Example:

...
requirement:
 - key: "karpenter.k8s.aws/instance-local-nvme"
 operator: "Exists"
...

Pod example of requiring at least 100GB of NVME disk:

...
 affinity:
 nodeAffinity:
 requiredDuringSchedulingIgnoredDuringExecution:
 nodeSelectorTerms:
 - matchExpressions:
 - key: "karpenter.k8s.aws/instance-local-nvme"
 operator: Gt
 values: ["99"]
...

NodePool Example:

...
requirement:
 - key: "karpenter.k8s.aws/instance-local-nvme"
 operator: Gt
 values: ["99"]
...

Note

Karpenter cannot yet take into account ephemeral-storage requests while scheduling pods, we’re purely requesting attributes of nodes and getting X amount of resources as a side effect. You may need to tweak schedulable resources like CPU or Memory to achieve desired fit, especially if Consolidation is enabled.

Your NodeClass will also need to support automatically formatting and mounting NVME Instance Storage if available.

Pod example of requiring at least 50 Gbps of network bandwidth:

...
 affinity:
 nodeAffinity:
 requiredDuringSchedulingIgnoredDuringExecution:
 nodeSelectorTerms:
 - matchExpressions:
 - key: "karpenter.k8s.aws/instance-network-bandwidth"
 operator: Gt
 values: ["49999"]
...

NodePool Example:

...
requirement:
 - key: "karpenter.k8s.aws/instance-network-bandwidth"
 operator: Gt
 values: ["49999"]
...

Note

If using Gt/Lt operators, make sure to use values under the actual label values of the desired resource.

`Exists` Operator

The Exists operator can be used on a NodePool to provide workload segregation across nodes.

apiVersion: karpenter.sh/v1
kind: NodePool
spec:
 template:
 spec:
 requirements:
 - key: company.com/team
 operator: Exists
...

With this requirement on the NodePool, workloads can specify the same key (e.g. company.com/team) with custom values (e.g. team-a, team-b, etc.) as a required nodeAffinity or nodeSelector. Karpenter will then apply the key/value pair to nodes it launches dynamically based on the pod’s node requirements.

If each set of pods that can schedule with this NodePool specifies this key in its nodeAffinity or nodeSelector, you can isolate pods onto different nodes based on their values. This provides a way to more dynamically isolate workloads without requiring a unique NodePool for each workload subset.

For example, providing the following nodeSelectors would isolate the pods for each of these deployments on different nodes.

Team A Deployment

apiVersion: v1
kind: Deployment
metadata:
 name: team-a-deployment
spec:
 replicas: 5
 template:
 spec:
 nodeSelector:
 company.com/team: team-a

Team A Node

apiVersion: v1
kind: Node
metadata:
 labels:
 company.com/team: team-a

Team B Deployment

apiVersion: apps/v1
kind: Deployment
metadata:
 name: team-b-deployment
spec:
 replicas: 5
 template:
 spec:
 nodeSelector:
 company.com/team: team-b

Team B Node

apiVersion: v1
kind: Node
metadata:
 labels:
 company.com/team: team-b

Note

If a workload matches the NodePool but doesn’t specify a label, Karpenter will generate a random label for the node.

On-Demand/Spot Ratio Split

Taking advantage of Karpenter’s ability to assign labels to node and using a topology spread across those labels enables a crude method for splitting a workload across on-demand and spot instances in a desired ratio.

To do this, we create one NodePool each for spot and on-demand with disjoint values for a unique new label called capacity-spread. In the example below, we provide four unique values for the spot NodePool and one value for the on-demand NodePool. When we spread across our new label evenly, we’ll end up with a ratio of 4:1 spot to on-demand nodes.

Warning

This is not identical to a topology spread with a specified ratio. We are constructing ‘virtual domains’ to spread evenly across and the ratio of those ‘virtual domains’ to spot and on-demand happen to coincide with the desired spot to on-demand ratio. As an example, if you launch pods using the provided example, Karpenter will launch nodes with capacity-spread labels of 1, 2, 3, 4, and 5. kube-scheduler will then schedule evenly across those nodes to give the desired ratio.

NodePools

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
 name: spot
spec:
 template:
 spec:
 requirements:
 - key: "karpenter.sh/capacity-type"
 operator: In
 values: ["spot"]
 - key: capacity-spread
 operator: In
 values:
 - "2"
 - "3"
 - "4"
 - "5"
---
apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
 name: on-demand
spec:
 template:
 spec:
 requirements:
 - key: "karpenter.sh/capacity-type"
 operator: In
 values: ["on-demand"]
 - key: capacity-spread
 operator: In
 values:
 - "1"

Workload Topology Spread Constraint

topologySpreadConstraints:
- maxSkew: 1
 topologyKey: capacity-spread
 whenUnsatisfiable: DoNotSchedule
 labelSelector:
 ...

V1.5: Disruption

Mon, 01 Jan 0001 00:00:00 +0000

Control Flow

Karpenter sets a Kubernetes finalizer on each node and node claim it provisions. The finalizer blocks deletion of the node object while the Termination Controller taints and drains the node, before removing the underlying NodeClaim. Disruption is triggered by the Disruption Controller, by the user through manual disruption, or through an external system that sends a delete request to the node object.

Disruption Controller

Karpenter automatically discovers disruptable nodes and spins up replacements when needed. Karpenter disrupts nodes by executing one automated method at a time, first doing Drift then Consolidation. Each method varies slightly, but they all follow the standard disruption process. Karpenter uses disruption budgets to control the speed at which these disruptions begin.

Identify a list of prioritized candidates for the disruption method.
- If there are pods that cannot be evicted on the node, Karpenter will ignore the node and try disrupting it later.
- If there are no disruptable nodes, continue to the next disruption method.
For each disruptable node:
1. Check if disrupting it would violate its NodePool’s disruption budget.
2. Execute a scheduling simulation with the pods on the node to find if any replacement nodes are needed.
Add the karpenter.sh/disrupted:NoSchedule taint to the node(s) to prevent pods from scheduling to it.
Pre-spin any replacement nodes needed as calculated in Step (2), and wait for them to become ready.
- If a replacement node fails to initialize, un-taint the node(s), and restart from Step (1), starting at the first disruption method again.
Delete the node(s) and wait for the Termination Controller to gracefully shutdown the node(s).
Once the Termination Controller terminates the node, go back to Step (1), starting at the first disruption method again.

Termination Controller

When a Karpenter node is deleted, the Karpenter finalizer will block deletion and the APIServer will set the DeletionTimestamp on the node, allowing Karpenter to gracefully shutdown the node, modeled after Kubernetes Graceful Node Shutdown. Karpenter’s graceful shutdown process will:

Add the karpenter.sh/disrupted:NoSchedule taint to the node to prevent pods from scheduling to it.
Begin evicting the pods on the node with the Kubernetes Eviction API to respect PDBs, while ignoring all static pods, pods tolerating the karpenter.sh/disrupted:NoSchedule taint, and succeeded/failed pods. Wait for the node to be fully drained before proceeding to Step (3).
- While waiting, if the underlying NodeClaim for the node no longer exists, remove the finalizer to allow the APIServer to delete the node, completing termination.
Terminate the NodeClaim in the Cloud Provider.
Remove the finalizer from the node to allow the APIServer to delete the node, completing termination.

Manual Methods

Node Deletion: You can use kubectl to manually remove a single Karpenter node or nodeclaim. Since each Karpenter node is owned by a NodeClaim, deleting either the node or the nodeclaim will cause cascade deletion of the other:

# Delete a specific nodeclaim
kubectl delete nodeclaim $NODECLAIM_NAME

# Delete a specific node
kubectl delete node $NODE_NAME

# Delete all nodeclaims
kubectl delete nodeclaims --all

# Delete all nodes owned by any nodepool
kubectl delete nodes -l karpenter.sh/nodepool

# Delete all nodeclaims owned by a specific nodepoolXS
kubectl delete nodeclaims -l karpenter.sh/nodepool=$NODEPOOL_NAME

NodePool Deletion: NodeClaims are owned by the NodePool through an owner reference that launched them. Karpenter will gracefully terminate nodes through cascading deletion when the owning NodePool is deleted.

Note

By adding the finalizer, Karpenter improves the default Kubernetes process of node deletion. When you run kubectl delete node on a node without a finalizer, the node is deleted without triggering the finalization logic. The instance will continue running in EC2, even though there is no longer a node object for it. The kubelet isn’t watching for its own existence, so if a node is deleted, the kubelet doesn’t terminate itself. All the pod objects get deleted by a garbage collection process later, because the pods’ node is gone.

Automated Graceful Methods

Automated graceful methods, can be rate limited through NodePool Disruption Budgets

Consolidation: Karpenter works to actively reduce cluster cost by identifying when:
- Nodes can be removed because the node is empty
- Nodes can be removed as their workloads will run on other nodes in the cluster.
- Nodes can be replaced with lower priced variants due to a change in the workloads.
Drift: Karpenter will mark nodes as drifted and disrupt nodes that have drifted from their desired specification. See Drift to see which fields are considered.

Defaults

Disruption is configured through the NodePool’s disruption block by the consolidationPolicy, and consolidateAfter fields. Karpenter will configure these fields with the following values by default if they are not set:

spec:
 disruption:
 consolidationPolicy: WhenEmptyOrUnderutilized
 consolidateAfter: 0s

Consolidation

Consolidation is configured by consolidationPolicy and consolidateAfter. consolidationPolicy determines the pre-conditions for nodes to be considered consolidatable, and are WhenEmpty or WhenEmptyOrUnderutilized. If a node has no running non-daemon pods, it is considered empty. consolidateAfter can be set to indicate how long Karpenter should wait after a pod schedules or is removed from the node before considering the node consolidatable. With WhenEmptyOrUnderutilized, Karpenter will consider a node consolidatable when its consolidateAfter has been reached, empty or not.

Karpenter has two mechanisms for cluster consolidation:

Deletion - A node is eligible for deletion if all of its pods can run on free capacity of other nodes in the cluster.
Replace - A node can be replaced if all of its pods can run on a combination of free capacity of other nodes in the cluster and a single lower price replacement node.

Consolidation has three mechanisms that are performed in order to attempt to identify a consolidation action:

Empty Node Consolidation - Delete any entirely empty nodes in parallel
Multi Node Consolidation - Try to delete two or more nodes in parallel, possibly launching a single replacement whose price is lower than that of all nodes being removed
Single Node Consolidation - Try to delete any single node, possibly launching a single replacement whose price is lower than that of the node being removed

It’s impractical to examine all possible consolidation options for multi-node consolidation, so Karpenter uses a heuristic to identify a likely set of nodes that can be consolidated. For single-node consolidation we consider each node in the cluster individually.

When there are multiple nodes that could be potentially deleted or replaced, Karpenter chooses to consolidate the node that overall disrupts your workloads the least by preferring to terminate:

Nodes running fewer pods
Nodes that will expire soon
Nodes with lower priority pods

If consolidation is enabled, Karpenter periodically reports events against nodes that indicate why the node can’t be consolidated. These events can be used to investigate nodes that you expect to have been consolidated, but still remain in your cluster.

Events:
 Type Reason Age From Message
 ---- ------ ---- ---- -------
 Normal Unconsolidatable 66s karpenter pdb default/inflate-pdb prevents pod evictions
 Normal Unconsolidatable 33s (x3 over 30m) karpenter can't replace with a lower-priced node

Warning

Using preferred anti-affinity and topology spreads can reduce the effectiveness of consolidation. At node launch, Karpenter attempts to satisfy affinity and topology spread preferences. In order to reduce node churn, consolidation must also attempt to satisfy these constraints to avoid immediately consolidating nodes after they launch. This means that consolidation may not disrupt nodes in order to avoid violating preferences, even if kube-scheduler can fit the host pods elsewhere. Karpenter reports these pods via logging to bring awareness to the possible issues they can cause (e.g. pod default/inflate-anti-self-55894c5d8b-522jd has a preferred Anti-Affinity which can prevent consolidation).

Spot consolidation

For spot nodes, Karpenter has deletion consolidation enabled by default. If you would like to enable replacement with spot consolidation, you need to enable the feature through the SpotToSpotConsolidation feature flag.

Lower priced spot instance types are selected with the price-capacity-optimized strategy. Sometimes, the lowest priced spot instance type is not launched due to the likelihood of interruption. As a result, Karpenter uses the number of available instance type options with a price lower than the currently launched spot instance as a heuristic for evaluating whether it should launch a replacement for the current spot node.

We refer to the number of instances that Karpenter has within its launch decision as a launch’s “instance type flexibility.” When Karpenter is considering performing a spot-to-spot consolidation replacement, it will check whether replacing the instance type will lead to enough instance type flexibility in the subsequent launch request. As a result, we get the following properties when evaluating for consolidation:

We shouldn’t continually consolidate down to the lowest priced spot instance which might have very high rates of interruption.
We launch with enough instance types that there’s high likelihood that our replacement instance has comparable availability to our current one.

Karpenter requires a minimum instance type flexibility of 15 instance types when performing single node spot-to-spot consolidations (1 node to 1 node). It does not have the same instance type flexibility requirement for multi-node spot-to-spot consolidations (many nodes to 1 node) since doing so without requiring flexibility won’t lead to “race to the bottom” scenarios.

Drift

Drift handles changes to the NodePool/EC2NodeClass. For Drift, values in the NodePool/EC2NodeClass are reflected in the NodeClaimTemplateSpec/EC2NodeClassSpec in the same way that they’re set. A NodeClaim will be detected as drifted if the values in its owning NodePool/EC2NodeClass do not match the values in the NodeClaim. Similar to the upstream deployment.spec.template relationship to pods, Karpenter will annotate the owning NodePool and EC2NodeClass with a hash of the NodeClaimTemplateSpec to check for drift. Some special cases will be discovered either from Karpenter or through the CloudProvider interface, triggered by NodeClaim/Instance/NodePool/EC2NodeClass changes.

Special Cases on Drift

In special cases, drift can correspond to multiple values and must be handled differently. Drift on resolved fields can create cases where drift occurs without changes to CRDs, or where CRD changes do not result in drift. For example, if a NodeClaim has node.kubernetes.io/instance-type: m5.large, and requirements change from node.kubernetes.io/instance-type In [m5.large] to node.kubernetes.io/instance-type In [m5.large, m5.2xlarge], the NodeClaim will not be drifted because its value is still compatible with the new requirements. Conversely, if a NodeClaim is using a NodeClaim image ami: ami-abc, but a new image is published, Karpenter’s EC2NodeClass.spec.amiSelectorTerms will discover that the new correct value is ami: ami-xyz, and detect the NodeClaim as drifted.

NodePool

Fields
spec.template.spec.requirements

EC2NodeClass

Fields
spec.subnetSelectorTerms
spec.securityGroupSelectorTerms
spec.amiSelectorTerms

Behavioral Fields

Behavioral Fields are treated as over-arching settings on the NodePool to dictate how Karpenter behaves. These fields don’t correspond to settings on the NodeClaim or instance. They’re set by the user to control Karpenter’s Provisioning and disruption logic. Since these don’t map to a desired state of NodeClaims, behavioral fields are not considered for Drift.

NodePool

Fields
spec.weight
spec.limits
spec.disruption.*

Read the Drift Design for more.

Karpenter will add the Drifted status condition on NodeClaims if the NodeClaim is drifted from its owning NodePool. Karpenter will also remove the Drifted status condition if either:

The Drift feature gate is not enabled but the NodeClaim is drifted, Karpenter will remove the status condition.
The NodeClaim isn’t drifted, but has the status condition, Karpenter will remove it.

Automated Forceful Methods

Automated forceful methods will begin draining nodes as soon as the condition is met. Unlike the graceful methods mentioned above, these methods can not be rate-limited using NodePool Disruption Budgets, and do not wait for a pre-spin replacement node to be healthy for the pods to reschedule. Pod disruption budgets may be used to rate-limit application disruption.

Expiration

A node is expired once it’s lifetime exceeds the duration set on the owning NodeClaim’s spec.expireAfter field. Changes to spec.template.spec.expireAfter on the owning NodePool will not update the field for existing NodeClaims - it will induce NodeClaim drift and the replacements will have the updated value. Expiration can be used, in conjunction with terminationGracePeriod, to enforce a maximum Node lifetime. By default, expireAfter is set to 720h (30 days).

Warning

Misconfigured PDBs and pods with the karpenter.sh/do-not-disrupt annotation may block draining indefinitely. For this reason, it is not recommended to set expireAfter without also setting terminationGracePeriod if your cluster has pods with the karpenter.sh/do-not-disrupt annotation. Doing so can result in partially drained nodes stuck in the cluster, driving up cluster cost and potentially requiring manual intervention to resolve.

Interruption

If interruption-handling is enabled, Karpenter will watch for upcoming involuntary interruption events that would cause disruption to your workloads. These interruption events include:

Spot Interruption Warnings
Scheduled Change Health Events (Maintenance Events)
Instance Terminating Events
Instance Stopping Events

When Karpenter detects one of these events will occur to your nodes, it automatically taints, drains, and terminates the node(s) ahead of the interruption event to give the maximum amount of time for workload cleanup prior to compute disruption. This enables scenarios where the terminationGracePeriod for your workloads may be long or cleanup for your workloads is critical, and you want enough time to be able to gracefully clean-up your pods.

For Spot interruptions, the NodePool will start a new node as soon as it sees the Spot interruption warning. Spot interruptions have a 2 minute notice before Amazon EC2 reclaims the instance. Once Karpenter has received this warning it will begin draining the node while in parallel provisioning a new node. Karpenter’s average node startup time means that, generally, there is sufficient time for the new node to become ready before EC2 initiates termination for the spot instance.

Note

Karpenter publishes Kubernetes events to the node for all events listed above in addition to Spot Rebalance Recommendations. Karpenter does not currently support taint, drain, and terminate logic for Spot Rebalance Recommendations.

If you require handling for Spot Rebalance Recommendations, you can use the AWS Node Termination Handler (NTH) alongside Karpenter; however, note that the AWS Node Termination Handler cordons and drains nodes on rebalance recommendations, potentially causing more node churn in the cluster than with interruptions alone. Further information can be found in the Troubleshooting Guide.

Karpenter enables this feature by watching an SQS queue which receives critical events from AWS services which may affect your nodes. Karpenter requires that an SQS queue be provisioned and EventBridge rules and targets be added that forward interruption events from AWS services to the SQS queue. Karpenter provides details for provisioning this infrastructure in the CloudFormation template in the Getting Started Guide.

To enable interruption handling, configure the --interruption-queue CLI argument with the name of the interruption queue provisioned to handle interruption events.

Node Auto Repair

Feature State: Karpenter v1.1.0 alpha

Node Auto Repair is a feature that automatically identifies and replaces unhealthy nodes in your cluster, helping to maintain overall cluster health. Nodes can experience various types of failures affecting their hardware, file systems, or container environments. These failures may be surfaced through node conditions such as network unavailability, disk pressure, memory pressure, or other conditions reported by node diagnostic agents. When Karpenter detects these unhealthy conditions, it automatically replaces the affected nodes based on cloud provider-defined repair policies. Once a node has been in an unhealthy state beyond its configured toleration duration, Karpenter will forcefully terminate the node and its corresponding NodeClaim, bypassing the standard drain and grace period procedures to ensure swift replacement of problematic nodes. To prevent cascading failures, Karpenter includes safety mechanisms: it will not perform repairs if more than 20% of nodes in a NodePool are unhealthy, and for standalone NodeClaims, it evaluates this threshold against all nodes in the cluster. This ensures your cluster remains in a healthy state with minimal manual intervention, even in scenarios where normal node termination procedures might be impacted by the node’s unhealthy state.

To enable Node Auto Repair:

Ensure you have a Node Monitoring Agent deployed or any agent that will add status conditions to nodes that are supported (e.g., Node Problem Detector)
Enable the feature flag: NodeRepair=true
Node AutoRepair will automatically terminate nodes when they have unhealthy status conditions based on your cloud provider’s repair policies

Karpenter monitors nodes for the following node status conditions when initiating repair actions:

Kubelet Node Conditions

Type	Status	Toleration Duration
Ready	False	30 minutes
Ready	Unknown	30 minutes

Node Monitoring Agent Conditions

Type	Status	Toleration Duration
AcceleratedHardwareReady	False	10 minutes
StorageReady	False	30 minutes
NetworkingReady	False	30 minutes
KernelReady	False	30 minutes
ContainerRuntimeReady	False	30 minutes

To enable the NodeRepair feature flag, refer to the Feature Gates.

Controls

TerminationGracePeriod

To configure a maximum termination duration, terminationGracePeriod should be used. It is configured through a NodePool’s spec.template.spec.terminationGracePeriod field, and is persisted to created NodeClaims (spec.terminationGracePeriod). Changes to the spec.template.spec.terminationGracePeriod field on the NodePool will not result in a change for existing NodeClaims - it will induce NodeClaim drift and the replacements will have the updated terminationGracePeriod.

Once a node is disrupted, via either a graceful or forceful disruption method, Karpenter will begin draining the node. At this point, the countdown for terminationGracePeriod begins. Once the terminationGracePeriod elapses, remaining pods will be forcibly deleted and the underlying instance will be terminated. A node may be terminated before the terminationGracePeriod has elapsed if all disruptable pods have been drained.

In conjunction with expireAfter, terminationGracePeriod can be used to enforce an absolute maximum node lifetime. The node will begin to drain once its expireAfter has elapsed, and it will be forcibly terminated once its terminationGracePeriod has elapsed, making the maximum node lifetime the sum of the two fields.

Additionally, configuring terminationGracePeriod changes the eligibility criteria for disruption via Drift. When configured, a node may be disrupted via drift even if there are pods with blocking PDBs or the karpenter.sh/do-not-disrupt annotation scheduled to it. This enables cluster administrators to ensure crucial updates (e.g. AMI updates addressing CVEs) can’t be blocked by misconfigured applications.

Warning

To ensure that the terminationGracePeriodSeconds value for draining pods is respected, pods will be preemptively deleted before the Node’s terminationGracePeriod has elapsed. This includes pods with blocking pod disruption budgets or the karpenter.sh/do-not-disrupt annotation.

Consider the following example: a Node with a 1 hour terminationGracePeriod has been disrupted and begins to drain. A pod with the karpenter.sh/do-not-disrupt annotation and a 300 second (5 minute) terminationGracePeriodsSeconds is scheduled to it. If the pod is still running 55 minutes after the Node begins to drain, the pod will be deleted to ensure its terminationGracePeriodSeconds value is respected.

If a pod’s terminationGracePeriodSeconds value exceeds that of the Node it is scheduled to, Karpenter will prioritize the Node’s terminationGracePeriod. The pod will be deleted as soon as the Node begins to drain, and it will not receive it’s full terminationGracePeriodSeconds.

NodePool Disruption Budgets

You can rate limit Karpenter’s disruption through the NodePool’s spec.disruption.budgets. If undefined, Karpenter will default to one budget with nodes: 10%. Budgets will consider nodes that are actively being deleted for any reason, and will only block Karpenter from disrupting nodes voluntarily through drift, emptiness, and consolidation. Note that NodePool Disruption Budgets do not prevent Karpenter from terminating expired nodes.

Reasons

Karpenter allows specifying if a budget applies to any of Drifted, Underutilized, or Empty. When a budget has no reasons, it’s assumed that it applies to all reasons. When calculating allowed disruptions for a given reason, Karpenter will take the minimum of the budgets that have listed the reason or have left reasons undefined.

Nodes

When calculating if a budget will block nodes from disruption, Karpenter lists the total number of nodes owned by a NodePool, subtracting out the nodes owned by that NodePool that are currently being deleted and nodes that are NotReady. If the number of nodes being deleted by Karpenter or any other processes is greater than the number of allowed disruptions, disruption for this node will not proceed.

If the budget is configured with a percentage value, such as 20%, Karpenter will calculate the number of allowed disruptions as allowed_disruptions = roundup(total * percentage) - total_deleting - total_notready. If otherwise defined as a non-percentage value, Karpenter will simply use that number as a static ceiling non_percentage_value - total_deleting - total_notready. For multiple budgets in a NodePool, Karpenter will take the minimum value (most restrictive) of each of the budgets.

For example, the following NodePool with three budgets defines the following requirements:

The first budget will only allow 20% of nodes owned by that NodePool to be disrupted if it’s empty or drifted. For instance, if there were 19 nodes owned by the NodePool, 4 empty or drifted nodes could be disrupted, rounding up from 19 * .2 = 3.8.
The second budget acts as a ceiling to the previous budget, only allowing 5 disruptions when there are more than 25 nodes.
The last budget only blocks disruptions during the first 10 minutes of the day, where 0 disruptions are allowed, only applying to underutilized nodes.

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
 name: default
spec:
 disruption:
 consolidationPolicy: WhenEmptyOrUnderutilized
 budgets:
 - nodes: "20%"
 reasons:
 - "Empty"
 - "Drifted"
 - nodes: "5"
 - nodes: "0"
 schedule: "@daily"
 duration: 10m
 reasons:
 - "Underutilized"

Schedule

Schedule is a cronjob schedule. Generally, the cron syntax is five space-delimited values with options below, with additional special macros like @yearly, @monthly, @weekly, @daily, @hourly. Follow the Kubernetes documentation for more information on how to follow the cron syntax. Timezones are not currently supported. Schedules are always in UTC.

# ┌───────────── minute (0 - 59)
# │ ┌───────────── hour (0 - 23)
# │ │ ┌───────────── day of the month (1 - 31)
# │ │ │ ┌───────────── month (1 - 12)
# │ │ │ │ ┌───────────── day of the week (0 - 6) (Sunday to Saturday;
# │ │ │ │ │ 7 is also Sunday on some systems)
# │ │ │ │ │ OR sun, mon, tue, wed, thu, fri, sat
# │ │ │ │ │
# * * * * *

Duration

Duration allows compound durations with minutes and hours values such as 10h5m or 30m or 160h. Since cron syntax does not accept denominations smaller than minutes, users can only define minutes or hours.

Note

Duration and Schedule must be defined together. When omitted, the budget is always active. When defined, the schedule determines a starting point where the budget will begin being enforced, and the duration determines how long from that starting point the budget will be enforced.

Pod-Level Controls

Pods with blocking PDBs will not be evicted by the Termination Controller or be considered for voluntary disruption actions. When multiple pods on a node have different PDBs, none of the PDBs may be blocking for Karpenter to voluntary disrupt a node. This can create complex eviction scenarios:

If a pod matches multiple PDBs (via label selectors), ALL of these PDBs must allow for disruption
When different pods on the same node belong to different PDBs, ALL PDBs must simultaneously permit eviction
A single blocking PDB can prevent the entire node from being voluntary disrupted

For example, consider a node with these pods and PDBs:

Pod A: Matches PDB-1 (maxUnavailable: 0) and PDB-2 (maxUnavailable: 1)
Pod B: Matches PDB-3 (minAvailable: 100%)
Pod C: No PDB

In this scenario, Karpenter cannot voluntary disrupt the node because:

Pod A is blocked by PDB-1 even though PDB-2 would allow disruption
Pod B is blocked by PDB-3’s requirement for 100% availability

As seen in this example, the more PDBs there are affecting a Node, the more difficult it will be for Karpenter to find an opportunity to perform voluntary disruption actions.

Secondly, you can block Karpenter from voluntarily disrupting and draining pods by adding the karpenter.sh/do-not-disrupt: "true" annotation to the pod. You can treat this annotation as a single-pod, permanently blocking PDB. This has the following consequences:

Nodes with karpenter.sh/do-not-disrupt pods will be excluded from Consolidation, and conditionally excluded from Drift.
- If the Node’s owning NodeClaim has a terminationGracePeriod configured, it will still be eligible for disruption via drift.
Like pods with a blocking PDB, pods with the karpenter.sh/do-not-disrupt annotation will not be gracefully evicted by the Termination Controller. Karpenter will not be able to complete termination of the node until one of the following conditions is met:
- All pods with the karpenter.sh/do-not-disrupt annotation are removed.
- All pods with the karpenter.sh/do-not-disrupt annotation have entered a terminal phase (Succeeded or Failed).
- The owning NodeClaim’s terminationGracePeriod has elapsed.

This is useful for pods that you want to run from start to finish without disruption. Examples of pods that you might want to opt-out of disruption include an interactive game that you don’t want to interrupt or a long batch job (such as you might have with machine learning) that would need to start over if it were interrupted.

apiVersion: apps/v1
kind: Deployment
spec:
 template:
 metadata:
 annotations:
 karpenter.sh/do-not-disrupt: "true"

Note

The karpenter.sh/do-not-disrupt annotation does not exclude nodes from the forceful disruption methods: Expiration, Interruption, Node Repair, and manual deletion (e.g. kubectl delete node ...). While both interruption and node repair have implicit upper-bounds on termination time, expiration and manual termination do not. Manual intervention may be required to unblock node termination, by removing pods with the karpenter.sh/do-not-disrupt annotation. For this reason, it is not recommended to use the karpenter.sh/do-not-disrupt annotation with expireAfter if you have not also configured terminationGracePeriod.

Node-Level Controls

You can block Karpenter from voluntarily choosing to disrupt certain nodes by setting the karpenter.sh/do-not-disrupt: "true" annotation on the node. This will prevent voluntary disruption actions against the node.

apiVersion: v1
kind: Node
metadata:
 annotations:
 karpenter.sh/do-not-disrupt: "true"

Example: Disable Disruption on a NodePool

To disable disruption for all nodes launched by a NodePool, you can configure its .spec.disruption.budgets. Setting a budget of zero nodes will prevent any of those nodes from being considered for voluntary disruption.

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
 name: default
spec:
 disruption:
 budgets:
 - nodes: "0"