Karpenter – Upgrading

V0.32: Upgrade Guide

Mon, 01 Jan 0001 00:00:00 +0000

Karpenter is a controller that runs in your cluster, but it is not tied to a specific Kubernetes version, as the Cluster Autoscaler is. Use your existing upgrade mechanisms to upgrade your core add-ons in Kubernetes and keep Karpenter up to date on bug fixes and new features. This guide contains information needed to upgrade to the latest release of Karpenter, along with compatibility issues you need to be aware of when upgrading from earlier Karpenter versions.

When upgrading Karpenter in production environments, implementing a robust CI/CD pipeline approach is crucial. Improper upgrades can lead to significant disruptions including failed node provisioning, orphaned nodes, interrupted workloads, and potential cost implications from unmanaged scaling. Given Karpenter’s critical role in cluster scaling and workload management, untested upgrades could result in production outages or resource allocation issues that directly impact application availability and performance. Therefore, we recommend following these structured steps:

Pre-upgrade Validation

Validate all required IAM permissions (node role, controller role)
Check webhook configurations
Back up existing NodePool and NodeClass configurations
Document current version and settings

Staging Environment Setup

Create or verify staging environment
Update version tags in Helm values or manifests
Configure automated validation tests

Staging Deployment

Deploy to staging environment
Run comprehensive tests including node provisioning
Verify controller health
Test NodePool and NodeClass functionality
Monitor system behavior

Production Approval and Deployment

Require manual approval/review
Schedule maintenance window if needed
Execute production deployment
Monitor deployment progress
Verify all components are functioning

Post-Deployment

Monitor system health
Verify node provisioning
Keep rollback configurations accessible
Update documentation

Here are few recommended CI/CD Pipeline Options:

GitHub Actions - Excellent for GitHub-hosted repositories with built-in Kubernetes support
GitLab CI - Strong container-native pipeline with integrated Kubernetes functionality
ArgoCD - Specialized for GitOps workflows with Kubernetes
AWS CodePipeline - Native integration with EKS and AWS services
Flux - Open-source GitOps tool for Kubernetes with automatic deployment capabilities

Each pipeline tool can be configured to handle the Karpenter upgrade workflow, but choose based on your existing infrastructure, team expertise, and specific requirements for automation and integration.

CRD Upgrades

Karpenter ships with a few Custom Resource Definitions (CRDs). These CRDs are published:

As an independent Helm chart karpenter-crd (source) that can be used by Helm to manage the lifecycle of these CRDs. To upgrade or install karpenter-crd run:

KARPENTER_NAMESPACE=kube-system
helm upgrade --install karpenter-crd oci://public.ecr.aws/karpenter/karpenter-crd --version vx.y.z --namespace "${KARPENTER_NAMESPACE}" --create-namespace

As part of the helm chart karpenter (source). Helm does not manage the lifecycle of CRDs using this method - the tool will only install the CRD during the first installation of the Helm chart. Subsequent chart upgrades will not add or remove CRDs, even if the CRDs have changed.

CRDs are coupled to the version of Karpenter, and should be updated along with Karpenter. For this reason, we recommend using the independent karpenter-crd chart to manage CRDs.

Note

If you get the error invalid ownership metadata; label validation error: while installing the karpenter-crd chart from an older version of Karpenter, follow the Troubleshooting Guide for details on how to resolve these errors.

Upgrading to v0.32.0+

Warning

Karpenter v0.32.0 introduces v1beta1 APIs, including significant changes to the API and installation procedures for the Karpenter controllers. Do not upgrade to v0.32.0+ without referencing the v1beta1 Migration Upgrade Procedure.

This version includes dual support for both alpha and beta APIs to ensure that you can slowly migrate your existing Provisioner, AWSNodeTemplate, and Machine alpha APIs to the newer NodePool, EC2NodeClass, and NodeClaim beta APIs.

Note that if you are rolling back after upgrading to v0.32.0, note that only versions v0.31.4+ support handling rollback after you have deployed the v1beta1 APIs to your cluster.

Karpenter now uses settings.InterruptionQueue instead of settings.aws.InterruptionQueueName in its helm chart. The CLI argument also changed to --interruption-queue.
Karpenter now serves the webhook prometheus metrics server on port 8001. If this port is already in-use on the pod or you are running in hostNetworking mode, you may need to change this port value. You can configure this port value through the WEBHOOK_METRICS_PORT environment variable or the webhook.metrics.port value if installing via Helm.
Karpenter now exposes the ability to disable webhooks through the webhook.enabled=false value. This value will disable the webhook server and will prevent any permissions, mutating or validating webhook configurations from being deployed to the cluster.
Karpenter now moves all logging configuration for the Zap logger into the logConfig values block. Configuring Karpenter logging with this mechanism is deprecated and will be dropped at v1. Karpenter now only surfaces logLevel through the logLevel helm value. If you need more advanced configuration due to log parsing constraints, we recommend configuring your log parser to handle Karpenter’s Zap JSON logging.
The default log encoding changed from console to json. If you were previously not setting the type of log encoding, this default will change with the helm chart. If you were setting the value through logEncoding, this value will continue to work until v0.33.x but it is deprecated in favor of logConfig.logEncoding
Karpenter now uses the karpenter.sh/disruption:NoSchedule=disrupting taint instead of the upstream node.kubernetes.io/unschedulable taint for nodes spawned with a NodePool to prevent pods from scheduling to nodes being disrupted. Pods that previously tolerated the node.kubernetes.io/unschedulable taint that previously weren’t evicted during termination will now be evicted. This most notably affects DaemonSets, which have the node.kubernetes.io/unschedulable toleration by default, where Karpenter will now remove these pods during termination. If you want your specific pods to not be evicted when nodes are scaled down, you should add a toleration to the pods with the following: Key=karpenter.sh/disruption, Effect=NoSchedule, Operator=Equals, Values=disrupting.
- Note: Karpenter will continue to use the old node.kubernetes.io/unschedulable taint for nodes spawned with a Provisioner.

Upgrading to v0.31.0+

Karpenter moved its securityContext constraints from pod-wide to only applying to the Karpenter container exclusively. If you were previously relying on the pod-wide securityContext for your sidecar containers, you will now need to set these values explicitly in your sidecar container configuration.

Upgrading to v0.30.0+

Karpenter will now statically drift on both Provisioner and AWSNodeTemplate Fields. For Provisioner Static Drift, the karpenter.sh/provisioner-hash annotation must be present on both the Provisioner and Machine. For AWSNodeTemplate drift, the karpenter.k8s.aws/nodetemplate-hash annotation must be present on the AWSNodeTemplate and Machine. Karpenter will not add these annotations to pre-existing nodes, so each of these nodes will need to be recycled one time for the annotations to be added.
Karpenter will now fail validation on AWSNodeTemplates and Provisioner spec.provider that have amiSelectors, subnetSelectors, or securityGroupSelectors set with a combination of id selectors (aws-ids, aws::ids) and other selectors.
Karpenter now statically sets the securityContext at both the pod and container-levels and doesn’t allow override values to be passed through the helm chart. This change was made to adhere to Restricted Pod Security Standard, which follows pod hardening best practices.

Note

If you have sidecar containers configured to run alongside Karpenter that cannot tolerate the pod-wide securityContext constraints, you will need to specify overrides to the sidecar securityContext in your deployment.

Upgrading to v0.29.0+

Warning

Karpenter v0.29.1 contains a file descriptor and memory leak bug that leads to Karpenter getting OOMKilled and restarting at the point that it hits its memory or file descriptor limit. Karpenter >v0.29.2 fixes this leak.

Karpenter has changed the default metrics service port from 8080 to 8000 and the default webhook service port from 443 to 8443. In v0.28.0, the Karpenter pod port was changed to 8000, but referenced the service by name, allowing users to scrape the service at port 8080 for metrics. v0.29.0 aligns the two ports so that service and pod metrics ports are the same. These ports are set by the controller.metrics.port and webhook.port helm chart values, so if you have previously set these to non-default values, you may need to update your Prometheus scraper to match these new values.
Karpenter will now reconcile nodes that are drifted due to their Security Groups or their Subnets. If your AWSNodeTemplate’s Security Groups differ from the Security Groups used for an instance, Karpenter will consider it drifted. If the Subnet used by an instance is not contained in the allowed list of Subnets for an AWSNodeTemplate, Karpenter will also consider it drifted.
- Since Karpenter uses tags for discovery of Subnets and SecurityGroups, check the Threat Model to see how to manage this IAM Permission.

Upgrading to v0.28.0+

Warning

Karpenter v0.28.0 is incompatible with Kubernetes version 1.26+, which can result in additional node scale outs when using --cloudprovider=external, which is the default for the EKS Optimized AMI. See: https://github.com/aws/karpenter-core/pull/375. Karpenter >v0.28.1 fixes this issue and is compatible with Kubernetes version 1.26+.

The extraObjects value is now removed from the Helm chart. Having this value in the chart proved to not work in the majority of Karpenter installs and often led to anti-patterns, where the Karpenter resources installed to manage Karpenter’s capacity were directly tied to the install of the Karpenter controller deployments. The Karpenter team recommends that, if you want to install Karpenter manifests alongside the Karpenter helm chart, to do so by creating a separate chart for the manifests, creating a dependency on the controller chart.
The aws.nodeNameConvention setting is now removed from the karpenter-global-settings ConfigMap. Because Karpenter is now driving its orchestration of capacity through Machines, it no longer needs to know the node name, making this setting obsolete. Karpenter ignores configuration that it doesn’t recognize in the karpenter-global-settings ConfigMap, so leaving the aws.nodeNameConvention in the ConfigMap will simply cause this setting to be ignored.
Karpenter now defines a set of “restricted tags” which can’t be overridden with custom tagging in the AWSNodeTemplate or in the karpenter-global-settings ConfigMap. If you are currently using any of these tag overrides when tagging your instances, webhook validation will now fail. These tags include:
- karpenter.sh/managed-by
- karpenter.sh/provisioner-name
- kubernetes.io/cluster/${CLUSTER_NAME}
The following metrics changed their meaning, based on the introduction of the Machine resource:
- karpenter_nodes_terminated: Use karpenter_machines_terminated if you are interested in the reason why a Karpenter machine was deleted. karpenter_nodes_terminated now only tracks the count of terminated nodes without any additional labels.
- karpenter_nodes_created: Use karpenter_machines_created if you are interested in the reason why a Karpenter machine was created. karpenter_nodes_created now only tracks the count of created nodes without any additional labels.
- karpenter_deprovisioning_replacement_node_initialized_seconds: This metric has been replaced in favor of karpenter_deprovisioning_replacement_machine_initialized_seconds.
v0.28.0 introduces the Machine CustomResource into the karpenter.sh API Group and requires this CustomResourceDefinition to run properly. Karpenter now orchestrates its CloudProvider capacity through these in-cluster Machine CustomResources. When performing a scheduling decision, Karpenter will create a Machine, resulting in launching CloudProvider capacity. The kubelet running on the new capacity will then register the node to the cluster shortly after launch.
- If you are using Helm to upgrade between versions of Karpenter, note that Helm does not automate the process of upgrading or install the new CRDs into your cluster. To install or upgrade the existing CRDs, follow the guidance under the Custom Resource Definition (CRD) Upgrades section of the upgrade guide.
- Karpenter will hydrate Machines on startup for existing capacity managed by Karpenter into the cluster. Existing capacity launched by an older version of Karpenter is discovered by finding CloudProvider capacity with the karpenter.sh/provisioner-name tag or the karpenter.sh/provisioner-name label on nodes.
The metrics port for the Karpenter deployment was changed from 8080 to 8000. Users who scrape the pod directly for metrics rather than the service will need to adjust the commands they use to reference port 8000. Any users who scrape metrics from the service should be unaffected.

Warning

Karpenter creates a mapping between CloudProvider machines and CustomResources in the cluster for capacity tracking. To ensure this mapping is consistent, Karpenter utilizes the following tag keys:

karpenter.sh/managed-by
karpenter.sh/provisioner-name
kubernetes.io/cluster/${CLUSTER_NAME}

Because Karpenter takes this dependency, any user that has the ability to Create/Delete these tags on CloudProvider machines will have the ability to orchestrate Karpenter to Create/Delete CloudProvider machines as a side effect. Check the Threat Model to see how this might affect you, and ways to mitigate this.

Rolling Back

If, after upgrading to v0.28.0+, a rollback to an older version of Karpenter needs to be performed, Karpenter will continue to function normally, though you will still have the Machine CustomResources on your cluster. You will need to manually delete the Machines and patch out the finalizers to fully complete the rollback.

Karpenter marks CloudProvider capacity as “managed by” a Machine using the karpenter-sh/managed-by tag on the CloudProvider machine. It uses this tag to ensure that the Machine CustomResources in the cluster match the CloudProvider capacity managed by Karpenter. If these states don’t match, Karpenter will garbage collect the capacity. Because of this, if performing an upgrade, followed by a rollback, followed by another upgrade to v0.28.0+, ensure you remove the karpenter.sh/managed-by tags from existing capacity; otherwise, Karpenter will deprovision the capacity without a Machine CR counterpart.

Upgrading to v0.27.3+

The defaulting.webhook.karpenter.sh mutating webhook was removed in v0.27.3. If you are coming from an older version of Karpenter where this webhook existed and the webhook was not managed by Helm, you may need to delete the stale webhook.

kubectl delete mutatingwebhookconfigurations defaulting.webhook.karpenter.sh

Upgrading to v0.27.0+

The Karpenter controller pods now deploy with kubernetes.io/hostname self anti-affinity by default. If you are running Karpenter in HA (high-availability) mode and you do not have enough nodes to match the number of pod replicas you are deploying with, you will need to scale-out your nodes for Karpenter.
The following controller metrics changed and moved under the controller_runtime metrics namespace:
- karpenter_metricscraper_...
- karpenter_deprovisioning_...
- karpenter_provisioner_...
- karpenter_interruption_...
The following controller metric names changed, affecting the controller label value under controller_runtime_... metrics. These metrics include:
- podmetrics -> pod_metrics
- provisionermetrics -> provisioner_metrics
- metricscraper -> metric_scraper
- provisioning -> provisioner_trigger
- node-state -> node_state
- pod-state -> pod_state
- provisioner-state -> provisioner_state
The karpenter_allocation_controller_scheduling_duration_seconds metric name changed to karpenter_provisioner_scheduling_duration_seconds

Upgrading to v0.26.0+

The karpenter.sh/do-not-evict annotation no longer blocks node termination when running kubectl delete node. This annotation on pods will only block automatic deprovisioning that is considered “voluntary,” that is, disruptions that can be avoided. Disruptions that Karpenter deems as “involuntary” and will ignore the karpenter.sh/do-not-evict annotation include spot interruption and manual deletion of the node. See Disabling Deprovisioning for more details.
Default resources requests and limits are removed from the Karpenter’s controller deployment through the Helm chart. If you have not set custom resource requests or limits in your helm values and are using Karpenter’s defaults, you will now need to set these values in your helm chart deployment.
The controller.image value in the helm chart has been broken out to a map consisting of controller.image.repository, controller.image.tag, and controller.image.digest. If manually overriding the controller.image, you will need to update your values to the new design.

Upgrading to v0.25.0+

Cluster Endpoint can now be automatically discovered. If you are using Amazon Elastic Kubernetes Service (EKS), you can now omit the clusterEndpoint field in your configuration. In order to allow the resolving, you have to add the permission eks:DescribeCluster to the Karpenter Controller IAM role.

Upgrading to v0.24.0+

Settings are no longer updated dynamically while Karpenter is running. If you manually make a change to the karpenter-global-settings ConfigMap, you will need to reload the containers by restarting the deployment with kubectl rollout restart -n karpenter deploy/karpenter
Karpenter no longer filters out instance types internally. Previously, g2 (not supported by the NVIDIA device plugin) and FPGA instance types were filtered. The only way to filter instance types now is to set requirements on your provisioner or pods using well-known node labels described here. If you are currently using overly broad requirements that allows all of the g instance-category, you will want to tighten the requirement, or add an instance-generation requirement.
aws.tags in karpenter-global-settings ConfigMap is now a top-level field and expects the value associated with this key to be a JSON object of string to string. This is change from previous versions where keys were given implicitly by providing the key-value pair aws.tags.<key>: value in the ConfigMap.

Upgrading to v0.22.0+

Do not upgrade to this version unless you are on Kubernetes >= v1.21. Karpenter no longer supports Kubernetes v1.20, but now supports Kubernetes v1.25. This change is due to the v1 PDB API, which was introduced in K8s v1.20 and subsequent removal of the v1beta1 API in K8s v1.25.

Upgrading to v0.20.0+

Prior to v0.20.0, Karpenter would prioritize certain instance type categories absent of any requirements in the Provisioner. v0.20.0+ removes prioritizing these instance type categories (“m”, “c”, “r”, “a”, “t”, “i”) in code. Bare Metal and GPU instance types are still deprioritized and only used if no other instance types are compatible with the node requirements. Since Karpenter does not prioritize any instance types, if you do not want exotic instance types and are not using the runtime Provisioner defaults, you will need to specify this in the Provisioner.

Upgrading to v0.19.0+

The karpenter webhook and controller containers are combined into a single binary, which requires changes to the helm chart. If your Karpenter installation (helm or otherwise) currently customizes the karpenter webhook, your deployment tooling may require minor changes.
Karpenter now supports native interruption handling. If you were previously using Node Termination Handler for spot interruption handling and health events, you will need to remove the component from your cluster before enabling aws.interruptionQueueName. For more details on Karpenter’s interruption handling, see the Interruption Handling Docs.
Instance category defaults are now explicitly persisted in the Provisioner, rather than handled implicitly in memory. By default, Provisioners will limit instance category to c,m,r. If any instance type constraints are applied, it will override this default. If you have created Provisioners in the past with unconstrained instance type, family, or category, Karpenter will now more flexibly use instance types than before. If you would like to apply these constraints, they must be included in the Provisioner CRD.
Karpenter CRD raw YAML URLs have migrated from https://raw.githubusercontent.com/aws/karpenter-provider-aws/v0.19.3/charts/karpenter/crds/... to https://raw.githubusercontent.com/aws/karpenter-provider-aws/v0.19.3/pkg/apis/crds/.... If you reference static Karpenter CRDs or rely on kubectl replace -f to apply these CRDs from their remote location, you will need to migrate to the new location.
Pods without an ownerRef (also called “controllerless” or “naked” pods) will now be evicted by default during node termination and consolidation. Users can prevent controllerless pods from being voluntarily disrupted by applying the karpenter.sh/do-not-evict: "true" annotation to the pods in question.
The following CLI options/environment variables are now removed and replaced in favor of pulling settings dynamically from the karpenter-global-settings ConfigMap. See the Settings docs for more details on configuring the new values in the ConfigMap.
- CLUSTER_NAME -> settings.aws.clusterName
- CLUSTER_ENDPOINT -> settings.aws.clusterEndpoint
- AWS_DEFAULT_INSTANCE_PROFILE -> settings.aws.defaultInstanceProfile
- AWS_ENABLE_POD_ENI -> settings.aws.enablePodENI
- AWS_ENI_LIMITED_POD_DENSITY -> settings.aws.enableENILimitedPodDensity
- AWS_ISOLATED_VPC -> settings.aws.isolatedVPC
- AWS_NODE_NAME_CONVENTION -> settings.aws.nodeNameConvention
- VM_MEMORY_OVERHEAD -> settings.aws.vmMemoryOverheadPercent

Upgrading to v0.18.0+

v0.18.0 removes the karpenter_consolidation_nodes_created and karpenter_consolidation_nodes_terminated prometheus metrics in favor of the more generic karpenter_nodes_created and karpenter_nodes_terminated metrics. You can still see nodes created and terminated by consolidation by checking the reason label on the metrics. Check out all the metrics published by Karpenter here.

Upgrading to v0.17.0+

Karpenter’s Helm chart package is now stored in Karpenter’s OCI (Open Container Initiative) registry. The Helm CLI supports the new format since v3.8.0+. With this change charts.karpenter.sh is no longer updated but preserved to allow using older Karpenter versions. For examples on working with the Karpenter helm charts look at Install Karpenter Helm Chart.

Users who have scripted the installation or upgrading of Karpenter need to adjust their scripts with the following changes:

There is no longer a need to add the Karpenter helm repo to helm
The full URL of the Helm chart needs to be present when using the helm commands
If you were not prepending a v to the version (i.e. 0.17.0), you will need to do so with the OCI chart, v0.17.0.

Upgrading to v0.16.2+

v0.16.2 adds new kubeletConfiguration fields to the provisioners.karpenter.sh v1alpha5 CRD. The CRD will need to be updated to use the new parameters:

kubectl replace -f https://raw.githubusercontent.com/aws/karpenter-provider-aws/v0.16.2/charts/karpenter/crds/karpenter.sh_provisioners.yaml

Upgrading to v0.16.0+

v0.16.0 adds a new weight field to the provisioners.karpenter.sh v1alpha5 CRD. The CRD will need to be updated to use the new parameters:

kubectl replace -f https://raw.githubusercontent.com/aws/karpenter-provider-aws/v0.16.0/charts/karpenter/crds/karpenter.sh_provisioners.yaml

Upgrading to v0.15.0+

v0.15.0 adds a new consolidation field to the provisioners.karpenter.sh v1alpha5 CRD. The CRD will need to be updated to use the new parameters:

kubectl replace -f https://raw.githubusercontent.com/aws/karpenter-provider-aws/v0.15.0/charts/karpenter/crds/karpenter.sh_provisioners.yaml

Upgrading to v0.14.0+

v0.14.0 adds new fields to the provisioners.karpenter.sh v1alpha5 and awsnodetemplates.karpenter.k8s.aws v1alpha1 CRDs. The CRDs will need to be updated to use the new parameters:

kubectl replace -f https://raw.githubusercontent.com/aws/karpenter-provider-aws/v0.14.0/charts/karpenter/crds/karpenter.sh_provisioners.yaml

kubectl replace -f https://raw.githubusercontent.com/aws/karpenter-provider-aws/v0.14.0/charts/karpenter/crds/karpenter.k8s.aws_awsnodetemplates.yaml

v0.14.0 changes the way Karpenter discovers its dynamically generated AWS launch templates to use a tag rather than a Name scheme. The previous name scheme was Karpenter-${CLUSTER_NAME}-* which could collide with user created launch templates that Karpenter should not manage. The new scheme uses a tag on the launch template karpenter.k8s.aws/cluster: ${CLUSTER_NAME}. As a result, Karpenter will not clean-up dynamically generated launch templates using the old name scheme. You can manually clean these up with the following commands:

## Find launch templates that match the naming pattern and you do not want to keep
aws ec2 describe-launch-templates --filters="Name=launch-template-name,Values=Karpenter-${CLUSTER_NAME}-*"

## Delete launch template(s) that match the name but do not have the "karpenter.k8s.aws/cluster" tag
aws ec2 delete-launch-template --launch-template-id <LAUNCH_TEMPLATE_ID>

v0.14.0 introduces additional instance type filtering if there are no node.kubernetes.io/instance-type or karpenter.k8s.aws/instance-family or karpenter.k8s.aws/instance-category requirements that restrict instance types specified on the provisioner. This prevents Karpenter from launching bare metal and some older non-current generation instance types unless the provisioner has been explicitly configured to allow them. If you specify an instance type or family requirement that supplies a list of instance-types or families, that list will be used regardless of filtering. The filtering can also be completely eliminated by adding an Exists requirement for instance type or family.

 - key: node.kubernetes.io/instance-type
 operator: Exists

v0.14.0 introduces support for custom AMIs without the need for an entire launch template. You must add the ec2:DescribeImages permission to the Karpenter Controller Role for this feature to work. This permission is needed for Karpenter to discover custom images specified. Read the Custom AMI documentation here to get started
v0.14.0 adds an an additional default toleration (CriticalAddonOnly=Exists) to the Karpenter helm chart. This may cause Karpenter to run on nodes with that use this Taint which previously would not have been schedulable. This can be overridden by using --set tolerations[0]=null.
v0.14.0 deprecates the AWS_ENI_LIMITED_POD_DENSITY environment variable in-favor of specifying spec.kubeletConfiguration.maxPods on the Provisioner. AWS_ENI_LIMITED_POD_DENSITY will continue to work when maxPods is not set on the Provisioner. If maxPods is set, it will override AWS_ENI_LIMITED_POD_DENSITY on that specific Provisioner.

Upgrading to v0.13.0+

v0.13.0 introduces a new CRD named AWSNodeTemplate which can be used to specify AWS Cloud Provider parameters. Everything that was previously specified under spec.provider in the Provisioner resource, can now be specified in the spec of the new resource. The use of spec.provider is deprecated but will continue to function to maintain backwards compatibility for the current API version (v1alpha5) of the Provisioner resource. v0.13.0 also introduces support for custom user data that doesn’t require the use of a custom launch template. The user data can be specified in-line in the AWSNodeTemplate resource.

If you are upgrading from v0.10.1 - v0.11.1, a new CRD awsnodetemplate was added. In v0.12.0, this crd was renamed to awsnodetemplates. Since helm does not manage the lifecycle of CRDs, you will need to perform a few manual steps for this CRD upgrade:
1. Make sure any awsnodetemplate manifests are saved somewhere so that they can be reapplied to the cluster.
2. kubectl delete crd awsnodetemplate
3. kubectl apply -f https://raw.githubusercontent.com/aws/karpenter-provider-aws/v0.13.2/charts/karpenter/crds/karpenter.k8s.aws_awsnodetemplates.yaml
4. Perform the Karpenter upgrade to v0.13.x, which will install the new awsnodetemplates CRD.
5. Reapply the awsnodetemplate manifests you saved from step 1, if applicable.
v0.13.0 also adds EC2/spot price fetching to Karpenter to allow making more accurate decisions regarding node deployments. Our getting started guide documents this, but if you are upgrading Karpenter you will need to modify your Karpenter controller policy to add the pricing:GetProducts and ec2:DescribeSpotPriceHistory permissions.

Upgrading to v0.12.0+

v0.12.0 adds an OwnerReference to each Node created by a provisioner. Previously, deleting a provisioner would orphan nodes. Now, deleting a provisioner will cause Kubernetes cascading delete logic to gracefully terminate the nodes using the Karpenter node finalizer. You may still orphan nodes by removing the owner reference.
If you are upgrading from v0.10.1 - v0.11.1, a new CRD awsnodetemplate was added. In v0.12.0, this crd was renamed to awsnodetemplates. Since helm does not manage the lifecycle of CRDs, you will need to perform a few manual steps for this CRD upgrade:
1. Make sure any awsnodetemplate manifests are saved somewhere so that they can be reapplied to the cluster.
2. kubectl delete crd awsnodetemplate
3. kubectl apply -f https://raw.githubusercontent.com/aws/karpenter-provider-aws/v0.12.1/charts/karpenter/crds/karpenter.k8s.aws_awsnodetemplates.yaml
4. Perform the Karpenter upgrade to v0.12.x, which will install the new awsnodetemplates CRD.
5. Reapply the awsnodetemplate manifests you saved from step 1, if applicable.

Upgrading to v0.11.0+

v0.11.0 changes the way that the vpc.amazonaws.com/pod-eni resource is reported. Instead of being reported for all nodes that could support the resources regardless of if the cluster is configured to support it, it is now controlled by a command line flag or environment variable. The parameter defaults to false and must be set if your cluster uses security groups for pods. This can be enabled by setting the environment variable AWS_ENABLE_POD_ENI to true via the helm value controller.env.

Other extended resources must be registered on nodes by their respective device plugins which are typically installed as DaemonSets (e.g. the nvidia.com/gpu resource will be registered by the NVIDIA device plugin. Previously, Karpenter would register these resources on nodes at creation and they would be zeroed out by kubelet at startup. By allowing the device plugins to register the resources, pods will not bind to the nodes before any device plugin initialization has occurred.

v0.11.0 adds a providerRef field in the Provisioner CRD. To use this new field you will need to replace the Provisioner CRD manually:

kubectl replace -f https://raw.githubusercontent.com/aws/karpenter-provider-aws/v0.11.0/charts/karpenter/crds/karpenter.sh_provisioners.yaml

Upgrading to v0.10.0+

v0.10.0 adds a new field, startupTaints to the provisioner spec. Standard Helm upgrades do not upgrade CRDs so the field will not be available unless the CRD is manually updated. This can be performed prior to the standard upgrade by applying the new CRD manually:

kubectl replace -f https://raw.githubusercontent.com/aws/karpenter-provider-aws/v0.10.0/charts/karpenter/crds/karpenter.sh_provisioners.yaml

📝 If you don’t perform this manual CRD update, Karpenter will work correctly except for rejecting the creation/update of provisioners that use startupTaints.

Upgrading to v0.6.2+

If using Helm, the variable names have changed for the cluster’s name and endpoint. You may need to update any configuration that sets the old variable names.

controller.clusterName is now clusterName
controller.clusterEndpoint is now clusterEndpoint

V0.32: Compatibility

Mon, 01 Jan 0001 00:00:00 +0000

Compatibility

To make upgrading easier we aim to minimize the introduction of breaking changes. Before you begin upgrading Karpenter, consider Karpenter compatibility issues related to Kubernetes and the NodePool API (previously Provisioner).

Compatibility Matrix

KUBERNETES	1.23	1.24	1.25	1.26	1.27	1.28	1.29
karpenter	>= 0.21	>= 0.21	>= 0.25	>= 0.28	>= 0.28	>= 0.31	>= 0.34.0

Note

Karpenter currently does not support the following new topologySpreadConstraints keys, promoted to beta in Kubernetes 1.27:

matchLabelKeys
nodeAffinityPolicy
nodeTaintsPolicy

For more information on Karpenter’s support for these keys, view this tracking issue.

Compatibility issues

When we introduce a breaking change, we do so only as described in this document.

Karpenter follows Semantic Versioning 2.0.0 in its stable release versions, while in major version zero (v0.y.z) anything may change at any time. However, to further protect users during this phase we will only introduce breaking changes in minor releases (releases that increment y in x.y.z). Note this does not mean every minor upgrade has a breaking change as we will also increment the minor version when we release a new feature.

Users should therefore check to see if there is a breaking change every time they are upgrading to a new minor version.

How Do We Break Incompatibility?

When there is a breaking change we will:

Increment the minor version when in major version 0
Add a permanent separate section named upgrading to vx.y.z+ under release upgrade notes clearly explaining the breaking change and what needs to be done on the user side to ensure a safe upgrade
Add the sentence “This is a breaking change, please refer to the above link for upgrade instructions” to the top of the release notes and in all our announcements

How Do We Find Incompatibilities?

Besides the peer review process for all changes to the code base we also do the followings in order to find incompatibilities:

(To be implemented) To check the compatibility of the application, we will automate tests for installing, uninstalling, upgrading from an older version, and downgrading to an older version
(To be implemented) To check the compatibility of the documentation with the application, we will turn the commands in our documentation into scripts that we can automatically run

Security Patches

While we are in major version 0 we will not release security patches to older versions. Rather we provide the patches in the latest versions. When at major version 1 we will have an EOL (end of life) policy where we provide security patches for a subset of older versions and deprecate the others.

Release Types

Karpenter offers three types of releases. This section explains the purpose of each release type and how the images for each release type are tagged in our public image repository.

Stable Releases

Stable releases are the only recommended versions for production environments. Stable releases are tagged with a semantic version (e.g. 0.35.0). Note that stable releases prior to 0.35.0 are prefixed with a v (e.g. v0.34.0).

Release Candidates

We consider having release candidates for major and important minor versions. Our release candidates are tagged like x.y.z-rc.0, x.y.z-rc.1. The release candidate will then graduate to x.y.z as a stable release. By adopting this practice we allow our users who are early adopters to test out new releases before they are available to the wider community, thereby providing us with early feedback resulting in more stable releases. Note that, like the stable releases, release candidates prior to 0.35.0 are prefixed with a v.

Snapshot Releases

We release a snapshot release for every commit that gets merged into aws/karpenter-provider-aws. This enables users to immediately try a new feature or fix right after it’s merged rather than waiting days or weeks for release.

Snapshot releases are not made available in the same public ECR repository as other release types, they are instead published to a separate private ECR repository. Helm charts are published to oci://021119463062.dkr.ecr.us-east-1.amazonaws.com/karpenter/snapshot/karpenter and are tagged with the git commit hash prefixed by the Karpenter major version (e.g. v0-fc17bfc89ebb30a3b102a86012b3e3992ec08adf). Anyone with an AWS account can pull from this repository, but must first authenticate:

aws ecr get-login-password --region us-east-1 | docker login --username AWS --password-stdin 021119463062.dkr.ecr.us-east-1.amazonaws.com

Note

Snapshot releases are suitable for testing, and troubleshooting but they should not be used in production environments. Snapshot releases are ephemeral and will be removed 90 days after they were published.

V0.32: v1beta1 Migration

Mon, 01 Jan 0001 00:00:00 +0000

Significant changes to the Karpenter APIs have been introduced in Karpenter v0.32.x. In this release, Karpenter APIs have advanced to v1beta1, in preparation for Karpenter v1 in the near future. The v1beta1 changes are meant to simplify and improve ease of use of those APIs, as well as solidify the APIs for the v1 release. Use this document as a reference to the changes that were introduced in the current release and as a guide to how you need to update the manifests and other Karpenter objects you created in previous Karpenter releases.

Here is some information you should know about upgrading the Karpenter controller to v0.32.x:

Towards a v1 release: The latest version of Karpenter sets the stage for Karpenter v1. Karpenter v0.32.x implements the Karpenter v1beta1 API spec. The intention is to have v1beta1 be used as the v1 spec, with only minimal changes needed.
Path to upgrading: This procedure assumes that you are upgrading from Karpenter v0.31.x to v0.32.x. If you are on an earlier version of Karpenter, review the Release Upgrade Notes for earlier versions’ breaking changes.
Enhancing and renaming components: For v1beta1, APIs have been enhanced to improve and solidify Karpenter APIs. Part of these enhancements includes renaming the Kinds for all Karpenter CustomResources. The following name changes have been made:
- Provisioner -> NodePool
- Machine -> NodeClaim
- AWSNodeTemplate -> EC2NodeClass
Running v1alpha1 alongside v1beta1: Having different Kind names for v1alpha5 and v1beta1 allows them to coexist for the same Karpenter controller for v0.32.x. This gives you time to transition to the new v1beta1 APIs while existing Provisioners and other objects stay in place. Keep in mind that there is no guarantee that the two versions will be able to coexist in future Karpenter versions.

Upgrade Procedure

This procedure assumes you are running the Karpenter controller on cluster and want to upgrade that cluster to v0.32.x.

NOTE: Please read through the entire procedure before beginning the upgrade. There are major changes in this upgrade, so you should carefully evaluate your cluster and workloads before proceeding.

Determine the current cluster version: Run the following to make sure that your Karpenter version is v0.31.x:
```
kubectl get pod -A | grep karpenter
kubectl describe pod -n karpenter karpenter-xxxxxxxxxx-xxxxx | grep Image: | grep v0.....
```
Sample output:
```
Image: public.ecr.aws/karpenter/controller:v0.31.0@sha256:d29767fa9c5c0511a3812397c932f5735234f03a7a875575422b712d15e54a77
```
Warning
v0.31.2 introduces minor changes to Karpenter so that rollback from v0.32.0 is supported. If you are coming from some other patch version of minor version v0.31.x, note that only versions v0.31.4+ support rollback for v1beta1.
Review for breaking changes: If you are already running Karpenter v0.31.x, you can skip this step. If you are running an earlier Karpenter version, you need to review the upgrade notes for each minor release.

Set environment variables for your cluster:

export KARPENTER_NAMESPACE=karpenter
export KARPENTER_VERSION=v0.32.10
export AWS_PARTITION="aws" # if you are not using standard partitions, you may need to configure to aws-cn / aws-us-gov
export CLUSTER_NAME="${USER}-karpenter-demo"
export AWS_REGION="us-west-2"
export AWS_ACCOUNT_ID="$(aws sts get-caller-identity --query Account --output text)"
export KARPENTER_IAM_ROLE_ARN="arn:${AWS_PARTITION}:iam::${AWS_ACCOUNT_ID}:role/${CLUSTER_NAME}-karpenter"
export CLUSTER_ENDPOINT="$(aws eks describe-cluster --name ${CLUSTER_NAME} --query "cluster.endpoint" --output text)"

Apply the new Karpenter policy and assign it to the existing Karpenter role:

TEMPOUT=$(mktemp)
curl -fsSL https://raw.githubusercontent.com/aws/karpenter-provider-aws/v0.32.10/website/content/en/preview/upgrading/v1beta1-controller-policy.json > ${TEMPOUT}

AWS_REGION=${AWS_REGION:=$AWS_DEFAULT_REGION} # use the default region if AWS_REGION isn't defined
POLICY_DOCUMENT=$(envsubst < ${TEMPOUT})
POLICY_NAME="KarpenterControllerPolicy-${CLUSTER_NAME}-v1beta1"
ROLE_NAME="${CLUSTER_NAME}-karpenter"

POLICY_ARN=$(aws iam create-policy --policy-name "${POLICY_NAME}" --policy-document "${POLICY_DOCUMENT}" | jq -r .Policy.Arn)
aws iam attach-role-policy --role-name "${ROLE_NAME}" --policy-arn "${POLICY_ARN}"

Apply the v0.32.10 Custom Resource Definitions (CRDs):

 kubectl apply -f https://raw.githubusercontent.com/aws/karpenter-provider-aws/v0.32.10/pkg/apis/crds/karpenter.sh_provisioners.yaml
 kubectl apply -f https://raw.githubusercontent.com/aws/karpenter-provider-aws/v0.32.10/pkg/apis/crds/karpenter.sh_machines.yaml
 kubectl apply -f https://raw.githubusercontent.com/aws/karpenter-provider-aws/v0.32.10/pkg/apis/crds/karpenter.k8s.aws_awsnodetemplates.yaml
 kubectl apply -f https://raw.githubusercontent.com/aws/karpenter-provider-aws/v0.32.10/pkg/apis/crds/karpenter.sh_nodepools.yaml
 kubectl apply -f https://raw.githubusercontent.com/aws/karpenter-provider-aws/v0.32.10/pkg/apis/crds/karpenter.sh_nodeclaims.yaml
 kubectl apply -f https://raw.githubusercontent.com/aws/karpenter-provider-aws/v0.32.10/pkg/apis/crds/karpenter.k8s.aws_ec2nodeclasses.yaml

Upgrade Karpenter to the new version:

helm registry logout public.ecr.aws

helm upgrade --install karpenter oci://public.ecr.aws/karpenter/karpenter --version ${KARPENTER_VERSION} --namespace karpenter --create-namespace \
 --set serviceAccount.annotations."eks\.amazonaws\.com/role-arn"=${KARPENTER_IAM_ROLE_ARN} \
 --set settings.aws.defaultInstanceProfile=KarpenterNodeInstanceProfile-${CLUSTER_NAME} \
 --set settings.clusterName=${CLUSTER_NAME} \
 --set settings.interruptionQueue=${CLUSTER_NAME} \
 --set controller.resources.requests.cpu=1 \
 --set controller.resources.requests.memory=1Gi \
 --set controller.resources.limits.cpu=1 \
 --set controller.resources.limits.memory=1Gi \
 --wait

Note

Karpenter has deprecated and moved a number of Helm values as part of the v1beta1 release. Ensure that you upgrade to the newer version of these helm values during your migration to v1beta1. You can find detail for all the settings that were moved in the v1beta1 Upgrade Reference.

Install the karpenter-convert tool to help convert the alpha Karpenter manifests to beta manifests:

go install github.com/aws/karpenter/tools/karpenter-convert/cmd/karpenter-convert@release-v0.32.x

Convert each AWSNodeTemplate to an EC2NodeClass. To convert your v1alpha Karpenter manifests to v1beta1, you can either manually apply changes to API components or use the karpenter-convert CLI tool. See the AWSNodeTemplate to EC2NodeClass section of the Karpenter Upgrade Reference for details on how to update to Karpenter AWSNodeTemplate objects.

For each EC2NodeClass, specify the $KARPENTER_NODE_ROLE you will use for nodes launched with this node class. Karpenter v1beta1 drops the need for managing your own instance profile and uses node roles directly. The example below shows how to migrate your AWSNodeTemplate to an EC2NodeClass if your node role is the same role that was used when creating your cluster with the Getting Started Guide.
```
export KARPENTER_NODE_ROLE="KarpenterNodeRole-${CLUSTER_NAME}"
karpenter-convert -f awsnodetemplate.yaml | envsubst > ec2nodeclass.yaml
```
When you are satisfied with your EC2NodeClass file, apply it as follows:
```
kubectl apply -f ec2nodeclass.yaml
```
Convert each Provisioner to a NodePool. Again, either manually update your Provisioner manifests or use the karpenter-convert CLI tool:
```
karpenter-convert -f provisioner.yaml > nodepool.yaml
```
When you are satisfied with your NodePool file, apply it as follows:
```
kubectl apply -f nodepool.yaml
```

Note

The karpenter-convert CLI tool will auto-inject the previous requirement defaulting logic that was orchestrated by webhooks in alpha. This defaulting logic set things like the karpenter.sh/capacity-type, karpenter.k8s.aws/instance-generation, karpenter.k8s.aws/instance-category, etc. These defaults are no longer set by the webhooks and need to be explicitly defined by the user in the NodePool.

Roll over nodes: With the new NodePool yaml in hand, there are several ways you can begin to roll over your nodes to use the new NodePool:
1. Periodic Rolling with Expiration:
  - Add the following taint to the old Provisioner: karpenter.sh/legacy=true:NoSchedule
  - Set the Expiration in your Provisioner to a small value like ttlSecondsUntilExpired: 300 to mark all nodes older than 5 minutes as expired. Please note that in beta APIs, this is the same as disruption.expireAfter
  - Watch as replacement nodes are launched from the new NodePool resource.
  Because Karpenter will only roll one node at a time, it may take some time for Karpenter to completely roll all nodes under a Provisioner.
2. Forced Deletion: For each Provisioner in your cluster:
  - Delete the old Provisioner with: kubectl delete provisioner <provisioner-name> --cascade=foreground
  - Wait as Karpenter deletes all the Provisioner’s nodes. All nodes will drain simultaneously. New nodes are launched after the old ones have been drained.
3. Manual Rolling: For each Provisioner in your cluster:
  - Add the following taint to the old Provisioner: karpenter.sh/legacy=true:NoSchedule
  - For all the nodes owned by the Provisioner, delete one at a time as follows: kubectl delete node <node-name>
Update workload labels: Old alpha labels (karpenter.sh/do-not-consolidate and karpenter.sh/do-not-evict) are deprecated, but will not be dropped until Karpenter v1. However, you can begin updating those labels at any time with karpenter.sh/do-not-disrupt.

Any pods that specified a karpenter.sh/provisioner-name:DoesNotExist requirement also need to add a karpenter.sh/nodepool:DoesNotExist requirement to ensure that the pods continue to not schedule to nodes unmanaged by Karpenter while migrating to v1beta1.
Ensure that there are no more Machine resources on your cluster. You should see No resources found when running the following command:
```
 kubectl get machines
```
Once there are no more Machines on your cluster, it is safe to delete the other Karpenter configuration resources. You can do so by running the following commands:
```
kubectl delete provisioners --all
kubectl delete awsnodetemplates --all
```

Remove the alpha instance profile(s). If you were just using the InstanceProfile deployed through the Getting Started Guide, delete the KarpenterNodeInstanceProfile section from the CloudFormation. Alternatively, if you want to remove the instance profile manually, you can run the following command

ROLE_NAME="KarpenterNodeRole-${CLUSTER_NAME}"
INSTANCE_PROFILE_NAME="KarpenterNodeInstanceProfile-${CLUSTER_NAME}"
aws iam remove-role-from-instance-profile --instance-profile-name "${INSTANCE_PROFILE_NAME}" --role-name "${ROLE_NAME}"
aws iam delete-instance-profile --instance-profile-name "${INSTANCE_PROFILE_NAME}"

Finally, remove the alpha policy from the controller role: This will remove any remaining permissions from the alpha APIs. You can orchestrate the removal of this policy with the following command:

ROLE_NAME="${CLUSTER_NAME}-karpenter"
POLICY_NAME="KarpenterControllerPolicy-${CLUSTER_NAME}"
POLICY_ARN=$(aws iam list-policies --query "Policies[?PolicyName=='${POLICY_NAME}'].Arn" --output text)
aws iam detach-role-policy --role-name "${ROLE_NAME}" --policy-arn "${POLICY_ARN}"

Note

If you are using some IaC for managing your policy documents attached to the controller role, you may want to attach this new beta policy to the same CloudFormation stack. You can do this by removing the old alpha policy, ensuring that the Karpenter controller continues to work with just the beta policy, and then updating the stack to contain the new beta policy rather than having that policy managed separately.

Changelog

Provisioner -> NodePool

Karpenter v1beta1 moves almost all top-level fields under the NodePool template field. Similar to Deployments (which template Pods that are orchestrated by the deployment controller), Karpenter NodePool templates NodeClaims (that are orchestrated by the Karpenter controller). Here is an example of a Provisioner (v1alpha5) migrated to a NodePool (v1beta1):

Note that:

The Limits and Weight fields sit outside of the template section. The Labels and Annotations fields from the Provisioner are now under the spec.template.metadata section. All other fields including requirements, taints, kubelet, and so on, are specified under the spec.template.spec section.
Support for spec.template.spec.kubelet.containerRuntime has been dropped. If you are using EKS 1.23 you should upgrade to containerd before using Karpenter v0.32.0, as this field in the kubelet block of the NodePool is not supported. EKS 1.24+ only supports containerd as a supported runtime.

Provisioner example (v1alpha)

apiVersion: karpenter.sh/v1alpha5
kind: Provisioner
 ...
spec:
 providerRef:
 name: default
 annotations:
 custom-annotation: custom-value
 labels:
 team: team-a
 custom-label: custom-value
 requirements:
 - key: karpenter.k8s.aws/instance-generation
 operator: Gt
 values: ["3"]
 - key: karpenter.k8s.aws/instance-category
 operator: In
 values: ["c", "m", "r"]
 - key: karpenter.sh/capacity-type
 operator: In
 values: ["spot"]
 taints:
 - key: example.com/special-taint
 value: "true"
 effect: NoSchedule
 startupTaints:
 - key: example.com/another-taint
 value: "true"
 effect: NoSchedule
 kubelet:
 systemReserved:
 cpu: 100m
 memory: 100Mi
 ephemeral-storage: 1Gi
 maxPods: 20
 limits:
 resources:
 cpu: 1000
 memory: 1000Gi
 weight: 50

NodePool example (v1beta1)

apiVersion: karpenter.sh/v1beta1
kind: NodePool
...
spec:
 template:
 metadata:
 annotations:
 custom-annotation: custom-value
 labels:
 team: team-a
 custom-label: custom-value
 spec:
 requirements:
 - key: karpenter.k8s.aws/instance-generation
 operator: Gt
 values: ["3"]
 - key: karpenter.k8s.aws/instance-category
 operator: In
 values: ["c", "m", "r"]
 - key: karpenter.sh/capacity-type
 operator: In
 values: ["spot"]
 taints:
 - key: example.com/special-taint
 value: "true"
 effect: NoSchedule
 startupTaints:
 - key: example.com/another-taint
 value: "true"
 effect: NoSchedule
 kubelet:
 systemReserved:
 cpu: 100m
 memory: 100Mi
 ephemeral-storage: 1Gi
 maxPods: 20
 limits:
 cpu: 1000
 memory: 1000Gi
 weight: 50

Provider

The Karpenter spec.provider field has been deprecated since version v0.7.0 and is now removed in the new NodePool resource. Any of the fields that you could specify within the spec.provider field are now available in the separate NodeClass resource.

Provider example (v1alpha)

apiVersion: karpenter.sh/v1alpha5
kind: Provisioner
...
spec:
 provider:
 amiFamily: Bottlerocket
 tags:
 test-tag: test-value

Nodepool example (v1beta1)

apiVersion: karpenter.sh/v1beta1
kind: NodePool
...
nodeClassRef:
 apiVersion: karpenter.k8s.aws/v1beta1
 kind: EC2NodeClass
 name: default

EC2NodeClass example (v1beta1)

apiVersion: karpenter.k8s.aws/v1beta1
kind: EC2NodeClass
metadata:
 name: default
spec:
 amiFamily: Bottlerocket
 tags:
 test-tag: test-value

TTLSecondsAfterEmpty

The Karpenter spec.ttlSecondsAfterEmpty field has been removed in favor of a consolidationPolicy and consolidateAfter field.

As part of the v1beta1 migration, Karpenter has chosen to collapse the concepts of emptiness and underutilization into a single concept: consolidation. You can now define the types of consolidation that you want to support in your consolidationPolicy field. The current values for this field are WhenEmpty or WhenUnderutilized (defaulting to WhenUnderutilized if not specified). If specifying WhenEmpty, you can define how long you wish to wait for consolidation to act on your empty nodes by tuning the consolidateAfter parameter. This field works the same as the ttlSecondsAfterEmpty field except this field now accepts either of the following values:

Never: This disables empty consolidation.
Duration String (e.g. “10m”, “1s”): This enables empty consolidation for the time specified.

ttlSecondsAfterEmpty example (v1alpha)

apiVersion: karpenter.sh/v1alpha5
kind: Provisioner
...
spec:
 ttlSecondsAfterEmpty: 120

consolidationPolicy and consolidateAfter examples (v1beta1)

apiVersion: karpenter.sh/v1beta1
kind: NodePool
...
spec:
 disruption:
 consolidationPolicy: WhenEmpty
 consolidateAfter: 2m

Consolidation

The Karpenter spec.consolidation block has also been shifted under consolidationPolicy. If you were previously enabling Karpenter’s consolidation feature for underutilized nodes using the consolidation.enabled flag, you now enable consolidation through the consolidationPolicy.

consolidation enabled example (v1alpha)

apiVersion: karpenter.sh/v1alpha5
kind: Provisioner
...
spec:
 consolidation:
 enabled: true

consolidationPolicy WhenUnderutilized example (v1beta1)

apiVersion: karpenter.sh/v1beta1
kind: NodePool
...
spec:
 disruption:
 consolidationPolicy: WhenUnderutilized

Note: You currently can’t set the consolidateAfter field when specifying consolidationPolicy: WhenUnderutilized. Karpenter will use a 15s consolidateAfter runtime default.

TTLSecondsUntilExpired

The Karpenter spec.ttlSecondsUntilExpired field has been removed in favor of the expireAfter field inside of the disruption block. This field works the same as it did before except this field now accepts either of the following values:

Never: This disables expiration.
Duration String (e.g. “10m”, “1s”): This enables expiration for the time specified.

consolidation ttlSecondsUntilExpired example (v1alpha)

apiVersion: karpenter.sh/v1alpha5
kind: Provisioner
...
spec:
 ttlSecondsUntilExpired: 2592000 # 30 Days = 60 * 60 * 24 * 30 Seconds

consolidationPolicy WhenUnderutilized example (v1beta1)

apiVersion: karpenter.sh/v1beta1
kind: NodePool
...
spec:
 disruption:
 expireAfter: 720h # 30 days = 30 * 24 Hours

Defaults

Karpenter now statically defaults some fields in the v1beta1 if they are not specified when applying the NodePool configuration. The following fields are defaulted if unspecified.

Field	Default
spec.disruption	{“consolidationPolicy: WhenUnderutilized”, expireAfter: “720h”}
spec.disruption.consolidationPolicy	WhenUnderutilized
spec.disruption.expireAfter	720h

spec.template.spec.requirements Defaults Dropped

Karpenter v1beta1 drops the defaulting logic for the node requirements that were shipped by default with Provisioners in v1alpha5. Previously, Karpenter would create dynamic defaulting in the following cases. If multiple of these cases were satisfied, those default requirements would be combined:

If you didn’t specify any instance type requirement:

spec:
 requirements:
 - key: karpenter.k8s.aws/instance-category
 operator: In
 values: ["c", "m", "r"]
 - key: karpenter.k8s.aws/instance-generation
 operator: In
 values: ["2"]

If you didn’t specify any capacity type requirement (karpenter.sh/capacity-type):

spec:
 requirements:
 - key: karpenter.sh/capacity-type
 operator: In
 values: ["on-demand"]

If you didn’t specify any OS requirement (kubernetes.io/os):

spec:
 requirements:
 - key: kubernetes.io/os
 operator: In
 values: ["linux"]

If you didn’t specify any architecture requirement (kubernetes.io/arch):

spec:
 requirements:
 - key: kubernetes.io/arch
 operator: In
 values: ["amd64"]

If you were previously relying on this defaulting logic, you will now need to explicitly specify these requirements in your NodePool.

AWSNodeTemplate -> EC2NodeClass

To configure AWS-specific settings, AWSNodeTemplate (v1alpha) is being changed to EC2NodeClass (v1beta1). Below are ways in which you can update your manifests for the new version.

Note

Tag-based AMI Selection is no longer supported for the new v1beta1 EC2NodeClass API. That feature allowed users to tag their AMIs using EC2 tags to express “In” requirements on selected. We recommend using different NodePools with different EC2NodeClasses with your various AMI requirement constraints to appropriately constrain your AMIs based on the instance types you’ve selected for a given NodePool.

Note

Karpenter will now tag EC2 instances with their node name instead of with karpenter.sh/provisioner-name: $PROVISIONER_NAME. This makes it easier to map nodes to instances if you are not currently using resource-based naming.

InstanceProfile

The Karpenter spec.instanceProfile field has been removed from the EC2NodeClass in favor of the spec.role field. Karpenter is also removing support for the defaultInstanceProfile specified globally in the karpenter-global-settings, making the spec.role field required for all EC2NodeClasses.

Karpenter will now auto-generate the instance profile in your EC2NodeClass, given the role that you specify. To find the role, type:

export INSTANCE_PROFILE_NAME=KarpenterNodeInstanceProfile-bob-karpenter-demo
aws iam get-instance-profile --instance-profile-name $INSTANCE_PROFILE_NAME --query "InstanceProfile.Roles[0].RoleName"
KarpenterNodeRole-bob-karpenter-demo

instanceProfile example (v1alpha)

apiVersion: karpenter.k8s.aws/v1alpha1
kind: AWSNodeTemplate
...
spec:
 instanceProfile: KarpenterNodeInstanceProfile-karpenter-demo

role example (v1beta1)

apiVersion: karpenter.k8s.aws/v1beta1
kind: EC2NodeClass
...
spec:
 role: KarpenterNodeRole-karpenter-demo

SubnetSelector, SecurityGroupSelector, and AMISelector

Karpenter’s spec.subnetSelector, spec.securityGroupSelector, and spec.amiSelector fields have been modified to support multiple terms and to first-class keys like id and name. If using comma-delimited strings in your tag, id, or name values, you may need to create separate terms for the new fields.

subnetSelector example (v1alpha)

apiVersion: karpenter.k8s.aws/v1alpha1
kind: AWSNodeTemplate
...
spec:
 subnetSelector:
 karpenter.sh/discovery: karpenter-demo

SubnetSelectorTerms.tags example (v1beta1)

apiVersion: karpenter.k8s.aws/v1beta1
kind: EC2NodeClass
...
spec:
 subnetSelectorTerms:
 - tags:
 karpenter.sh/discovery: karpenter-demo

subnetSelector example (v1alpha)

apiVersion: karpenter.k8s.aws/v1alpha1
kind: AWSNodeTemplate
...
spec:
 subnetSelector:
 aws::ids: subnet-123,subnet-456

subnetSelectorTerms.id example (v1beta1)

apiVersion: karpenter.k8s.aws/v1beta1
kind: EC2NodeClass
...
spec:
 subnetSelectorTerms:
 - id: subnet-123
 - id: subnet-456

securityGroupSelector example (v1alpha)

apiVersion: karpenter.k8s.aws/v1alpha1
kind: AWSNodeTemplate
...
spec:
 securityGroupSelector:
 karpenter.sh/discovery: karpenter-demo

securityGroupSelectorTerms.tags example (v1beta1)

apiVersion: compute.k8s.aws/v1beta1
kind: EC2NodeClass
...
spec:
 securityGroupSelectorTerms:
 - tags:
 karpenter.sh/discovery: karpenter-demo

securityGroupSelector example (v1alpha)

apiVersion: karpenter.k8s.aws/v1alpha1
kind: AWSNodeTemplate
...
spec:
 securityGroupSelector:
 aws::ids: sg-123, sg-456

securityGroupSelectorTerms.id example (v1beta1)

apiVersion: compute.k8s.aws/v1beta1
kind: EC2NodeClass
...
spec:
 securityGroupSelectorTerms:
 - id: sg-123
 - id: sg-456

amiSelector example (v1alpha)

apiVersion: karpenter.k8s.aws/v1alpha1
kind: AWSNodeTemplate
...
spec:
 amiSelector:
 karpenter.sh/discovery: karpenter-demo

amiSelectorTerms.tags example (v1beta1)

apiVersion: compute.k8s.aws/v1beta1
kind: EC2NodeClass
...
spec:
 amiSelectorTerms:
 - tags:
 karpenter.sh/discovery: karpenter-demo

amiSelector example (v1alpha)

apiVersion: karpenter.k8s.aws/v1alpha1
kind: AWSNodeTemplate
...
spec:
 amiSelector:
 aws::ids: ami-123,ami-456

amiSelectorTerms example (v1beta1)

apiVersion: compute.k8s.aws/v1beta1
kind: EC2NodeClass
...
spec:
 amiSelectorTerms:
 - id: ami-123
 - id: ami-456

amiSelector example (v1alpha)

apiVersion: karpenter.k8s.aws/v1alpha1
kind: AWSNodeTemplate
...
spec:
 amiSelector:
 aws::name: my-name1,my-name2
 aws::owners: 123456789,amazon

amiSelectorTerms.name example (v1beta1)

apiVersion: compute.k8s.aws/v1beta1
kind: EC2NodeClass
...
spec:
 amiSelectorTerms:
 - name: my-name1
 owner: "123456789"
 - name: my-name2
 owner: "123456789"
 - name: my-name1
 owner: amazon
 - name: my-name2
 owner: amazon

LaunchTemplateName

The spec.launchTemplateName field for referencing unmanaged launch templates within Karpenter has been removed. Find a discussion of the decision to remove spec.launchTemplateName, see RFC: Unmanaged LaunchTemplate Removal.

AMIFamily

The AMIFamily field is now required. If you were previously not specifying the AMIFamily field, having Karpenter default the AMIFamily to AL2, you will now have to specify AL2 explicitly.

Annotations, Labels, and Status Conditions

Karpenter v1beta1 introduces changes to some common labels, annotations, and status conditions that are present in the project. The tables below lists the v1alpha5 values and their v1beta1 equivalent.

Karpenter Labels
v1alpha5	v1beta1
karpenter.sh/provisioner-name	karpenter.sh/nodepool
karpenter.k8s.aws/instance-pods	Dropped

Note: Previously, you could use the karpenter.sh/provisioner-name:DoesNotExist requirement on pods to specify that pods should schedule to nodes unmanaged by Karpenter. With the addition of the karpenter.sh/nodepool label key, you now need to specify the karpenter.sh/nodepool:DoesNotExist requirement on these pods as well to ensure they don’t schedule to nodes provisioned by the new NodePool resources.

Karpenter Annotations
v1alpha5	v1beta1
karpenter.sh/provisioner-hash	karpenter.sh/nodepool-hash
karpenter.k8s.aws/nodetemplate-hash	karpenter.k8s.aws/ec2nodeclass-hash
karpenter.sh/do-not-consolidate	karpenter.sh/do-not-disrupt
karpenter.sh/do-not-evict	karpenter.sh/do-not-disrupt

Note: Karpenter dropped the karpenter.sh/do-not-consolidate annotation in favor of the karpenter.sh/do-not-disrupt annotation on nodes. This annotation specifies that no voluntary disruption should be performed by Karpenter against this node.

StatusCondition Types
v1alpha5	v1beta1
MachineLaunched	Launched
MachineRegistered	Registered
MachineInitialized	Initialized
MachineEmpty	Empty
MachineExpired	Expired
MachineDrifted	Drifted

IAM Controller Permissions

v1beta1 introduces changes to the IAM permissions assigned to the Karpenter controller policy used when deploying Karpenter to your cluster with IRSA or EKS Pod Identity.

You can see a full example of the v1beta1 required controller permissions by viewing the v1beta1 Controller Policy.

Additionally, read more detail about the full set of permissions assigned to the Karpenter controller policy in the CloudFormation Reference Guide.

Updating Tag-Based Permissions

Since Karpenter v1beta1 introduces changes to custom resource, label, and annotation naming, this also changes the tag keys that Karpenter uses to tag instances, volumes launch templates, and any other resources that Karpenter deploys and manages.

By default, when using the Karpenter Getting Started Guide to setup Karpenter on your cluster, you will deploy an IAM policy that scopes Karpenter’s permissions based on tag keys and values using ABAC. Any part of the Karpenter alpha controller policy which previously referenced aws:RequestTag:karpenter.sh/provisioner-name or aws:ResourceTag:karpenter.sh/provisioner-name is now updated in v1beta1 to be aws:RequestTag:karpenter.sh/nodepool and aws:ResourceTag:karpenter.sh/nodepool.

Note

While migrating between alpha and beta, you will need to maintain the old tag permissions as well as the new permissions. You can remove the old tag key permissions from the controller policy when you have fully migrated to beta resources. This process for maintaining both sets of permissions while migrating is also mentioned in greater detail in the Upgrade Procedure shown above.

Updating IAM Instance Profile Permissions

Additionally, starting in v1beta1, Karpenter removes the need for you to manage your own instance profiles used to launch EC2 instances, allowing you to only specify the role that you want assigned to your instances in the spec.role of the EC2NodeClass. When you do this, Karpenter will generate and manage an instance profile on your behalf.

To enable this functionality, you need to add iam: permissions that give Karpenter permission to generate and managed instance profiles. These permissions include iam:CreateInstanceProfile, iam:TagInstanceProfile, iam:AddRoleToInstanceProfile, iam:RemoveRoleFromInstanceProfile, iam:DeleteInstanceProfile, and iam:GetInstanceProfile. Each of these permissions is scoped down to only operate on instance profiles generated from a single Karpenter instance on a single Karpenter cluster using ABAC.

Note

These iam: permissions are not required if you do not intend to use the spec.role field to enable the managed instance profile feature. Instead, you can use the spec.instanceProfile field to tell Karpenter to use an unmanaged instance profile explicitly. Note that this means that you have to manually provision and manage the instance profile yourself as you did in alpha.

Metrics

The following table shows v1alpha5 metrics and the v1beta1 version of each metric. All metrics on this table will exist simultaneously, while both v1alpha5 and v1beta1 are supported within the same version.

v1alpha5 Metric Name	v1beta1 Metric Name
karpenter_machines_created	karpenter_nodeclaims_created
karpenter_machines_disrupted	karpenter_nodeclaims_disrupted
karpenter_machines_drifted	karpenter_nodeclaims_drifted
karpenter_machines_initialized	karpenter_nodeclaims_initialized
karpenter_machines_launched	karpenter_nodeclaims_launched
karpenter_machines_registered	karpenter_nodeclaims_registered
karpenter_machines_terminated	karpenter_nodeclaims_terminated
karpenter_provisioners_limit	karpenter_nodepools_limit
karpenter_provisioners_usage	karpenter_nodepools_usage
karpenter_deprovisioning_evaluation_duration_seconds	karpenter_disruption_evaluation_duration_seconds
karpenter_deprovisioning_eligible_machines	karpenter_disruption_eligible_nodeclaims
karpenter_deprovisioning_replacement_machine_initialized_seconds	karpenter_disruption_replacement_nodeclaims_initialized_seconds
karpenter_deprovisioning_replacement_machine_launch_failure_counter	karpenter_disruption_replacement_nodeclaims_launch_failed_total
karpenter_deprovisioning_actions_performed	karpenter_disruption_actions_performed_total
karpenter_deprovisioning_consolidation_timeouts	karpenter_disruption_consolidation_timeouts_total
karpenter_nodes_leases_deleted	karpenter_leases_deleted
karpenter_provisioners_usage_pct	Dropped

In addition to these metrics, the MachineNotFound error returned by the karpenter_cloudprovider_errors_total values in the error label has been changed to NodeClaimNotFound. This is agnostic to the version of the API (Machine or NodeClaim) that actually owns the instance.

Global Settings

The v1beta1 specification removes the karpenter-global-settings ConfigMap in favor of setting all Karpenter configuration using environment variables. Along, with this change, Karpenter has chosen to remove certain global variables that can be configured with more specificity in the EC2NodeClass . These values are marked as removed below.

`karpenter-global-settings` ConfigMap Key	Environment Variable	CLI Argument
batchMaxDuration	BATCH_MAX_DURATION	–batch-max-duration
batchIdleDuration	BATCH_IDLE_DURATION	–batch-idle-duration
aws.assumeRoleARN	ASSUME_ROLE_ARN	–assume-role-arn
aws.assumeRoleDuration	ASSUME_ROLE_DURATION	–assume-role-duration
aws.clusterCABundle	CLUSTER_CA_BUNDLE	–cluster-ca-bundle
aws.clusterName	CLUSTER_NAME	–cluster-name
aws.clusterEndpoint	CLUSTER_ENDPOINT	–cluster-endpoint
aws.isolatedVPC	ISOLATED_VPC	–isolated-vpc
aws.vmMemoryOverheadPercent	VM_MEMORY_OVERHEAD_PERCENT	–vm-memory-overhead-percent
aws.interruptionQueueName	INTERRUPTION_QUEUE	–interruption-queue
aws.reservedENIs	RESERVED_ENIS	–reserved-enis
featureGates.driftEnabled	FEATURE_GATE=“Drift=true”	–feature-gates Drift=true
aws.defaultInstanceProfile	Dropped	Dropped
aws.enablePodENI	Dropped	Dropped
aws.enableENILimitedPodDensity	Dropped	Dropped

NOTE: The aws.defaultInstanceProfile was dropped because Karpenter no longer utilizes instance profiles but creates a managed version of an instance profile based on an EC2NodeClass role. The aws.enablePodENI was dropped since Karpenter will now always assume that vpc.amazonaws.com/pod-eni resource exists. The aws.enableENILimitedPodDensity was dropped since you can now override the --max-pods value for kubelet in the spec.kubelet.maxPods for NodeClaims or NodeClaimTemplates.

Helm Values

The v1beta1 helm chart comes with a number of changes to the values that were previously used in v0.31.x. Your older helm values will continue to work throughout v0.32.x but any values no longer specified in the chart will no longer be supported starting in v0.33.0.

< v0.32.x Key	>= v0.32.x Key
controller.outputPaths	logConfig.outputPaths
controller.errorOutputPaths	logConfig.errorOutputPaths
controller.logLevel	logConfig.logLevel.controller
webhook.logLevel	logConfig.logLevel.webhook
logEncoding	logConfig.logEncoding
settings.aws.assumeRoleARN	settings.assumeRoleARN
settings.aws.assumeRoleDuration	settings.assumeRoleDuration
settings.aws.clusterCABundle	settings.clusterCABundle
settings.aws.clusterName	settings.clusterName
settings.aws.clusterEndpoint	settings.clusterEndpoint
settings.aws.isolatedVPC	settings.isolatedVPC
settings.aws.vmMemoryOverheadPercent	settings.vmMemoryOverheadPercent
settings.aws.interruptionQueueName	settings.interruptionQueue
settings.aws.reservedENIs	settings.reservedENIs
settings.featureGates.driftEnabled	settings.featureGates.drift
settings.aws.defaultInstanceProfile	Dropped
settings.aws.enablePodENI	Dropped
settings.aws.enableENILimitedPodDensity	Dropped
settings.aws.tags	Dropped

Drift Enabled by Default

The drift feature will now be enabled by default starting from v0.33.0. If you don’t specify the Drift featureGate, the feature will be assumed to be enabled. You can disable the drift feature by specifying –feature-gates Drift=false. This feature gate is expected to be dropped when core APIs (NodePool, NodeClaim) are bumped to v1.

Logging Configuration is No Longer Dynamic

As part of this deprecation, Karpenter will no longer call out to the APIServer to discover the ConfigMap. Instead, Karpenter will expect the ConfigMap to be mounted on the filesystem at /etc/karpenter/logging/zap-logger-config. You can also still choose to override the individual log level of components of the system (webhook and controller) at the paths /etc/karpenter/logging/loglevel.webhook and /etc/karpenter/logging/loglevel.controller.

What you do to upgrade this feature depends on how you install Karpenter:

If you are using the helm chart to install Karpenter, you won’t need to make any changes for Karpenter to begin using this new mechanism for loading the config.

If you are manually configuring the deployment for Karpenter, you will need to add the following sections to your deployment:

apiVersion: apps/v1
kind: Deployment
spec:
 template:
 spec:
 ...
 containers:
 - name: controller
 volumeMounts:
 - name: config-logging
 mountPath: /etc/karpenter/logging
 volumes:
 - name: config-logging
 configMap:
 name: config-logging

Karpenter will drop support for ConfigMap discovery through the APIServer starting in v0.33.0, meaning that you will need to ensure that you are mounting the config file on the expected filepath by that version.

Webhook Support Deprecated in Favor of CEL

Karpenter v1beta1 APIs now support Common Expression Language (CEL) for validaiton directly through the APIServer. This change means that Karpenter’s validating webhooks are no longer needed to ensure that Karpenter’s NodePools and EC2NodeClasses are configured correctly.

As a result, Karpenter will now disable webhooks by default by setting the DISABLE_WEBHOOK environment variable to true starting in v0.33.0. If you are currently on a version of Kubernetes < less than 1.25, CEL validation for Custom Resources is not enabled. We recommend that you enable the webhooks on these versions with DISABLE_WEBHOOK=false to get proper validation support for any Karpenter configuration.