Karpenter – Upgrading

V1.0: Upgrade Guide

Mon, 01 Jan 0001 00:00:00 +0000

Karpenter is a controller that runs in your cluster, but it is not tied to a specific Kubernetes version, as the Cluster Autoscaler is. Use your existing upgrade mechanisms to upgrade your core add-ons in Kubernetes and keep Karpenter up to date on bug fixes and new features. This guide contains information needed to upgrade to the latest release of Karpenter, along with compatibility issues you need to be aware of when upgrading from earlier Karpenter versions.

Warning

With the release of Karpenter v1.0.0, the Karpenter team will be dropping support for karpenter versions v0.32 and below. We recommend upgrading to the latest version of Karpenter and keeping Karpenter up-to-date for bug fixes and new features.

When upgrading Karpenter in production environments, implementing a robust CI/CD pipeline approach is crucial. Improper upgrades can lead to significant disruptions including failed node provisioning, orphaned nodes, interrupted workloads, and potential cost implications from unmanaged scaling. Given Karpenter’s critical role in cluster scaling and workload management, untested upgrades could result in production outages or resource allocation issues that directly impact application availability and performance. Therefore, we recommend following these structured steps:

Pre-upgrade Validation

Validate all required IAM permissions (node role, controller role)
Check webhook configurations
Back up existing NodePool and NodeClass configurations
Document current version and settings

Staging Environment Setup

Create or verify staging environment
Update version tags in Helm values or manifests
Configure automated validation tests

Staging Deployment

Deploy to staging environment
Run comprehensive tests including node provisioning
Verify controller health
Test NodePool and NodeClass functionality
Monitor system behavior

Production Approval and Deployment

Require manual approval/review
Schedule maintenance window if needed
Execute production deployment
Monitor deployment progress
Verify all components are functioning

Post-Deployment

Monitor system health
Verify node provisioning
Keep rollback configurations accessible
Update documentation

Here are few recommended CI/CD Pipeline Options:

GitHub Actions - Excellent for GitHub-hosted repositories with built-in Kubernetes support
GitLab CI - Strong container-native pipeline with integrated Kubernetes functionality
ArgoCD - Specialized for GitOps workflows with Kubernetes
AWS CodePipeline - Native integration with EKS and AWS services
Flux - Open-source GitOps tool for Kubernetes with automatic deployment capabilities

Each pipeline tool can be configured to handle the Karpenter upgrade workflow, but choose based on your existing infrastructure, team expertise, and specific requirements for automation and integration.

CRD Upgrades

Karpenter ships with a few Custom Resource Definitions (CRDs). These CRDs are published:

As an independent Helm chart karpenter-crd (source) that can be used by Helm to manage the lifecycle of these CRDs. To upgrade or install karpenter-crd run:

KARPENTER_NAMESPACE=kube-system
helm upgrade --install karpenter-crd oci://public.ecr.aws/karpenter/karpenter-crd --version x.y.z --namespace "${KARPENTER_NAMESPACE}" --create-namespace

As part of the helm chart karpenter (source). Helm does not manage the lifecycle of CRDs using this method - the tool will only install the CRD during the first installation of the Helm chart. Subsequent chart upgrades will not add or remove CRDs, even if the CRDs have changed.

CRDs are coupled to the version of Karpenter, and should be updated along with Karpenter. For this reason, we recommend using the independent karpenter-crd chart to manage CRDs.

Note

If you get the error invalid ownership metadata; label validation error: while installing the karpenter-crd chart from an older version of Karpenter, follow the Troubleshooting Guide for details on how to resolve these errors.

Upgrading to `1.0.0`+

Warning

Karpenter 1.0.0 introduces the v1 APIs and uses conversion webhooks to support existing v1beta1 APIs. Do not upgrade to 1.0.0+ without following the v1 Migration Guide.

Refer to the v1 Migration Guide for the full changelog.

Upgrading to `0.37.0`+

Warning

0.33.0+ only supports Karpenter v1beta1 APIs and will not work with existing Provisioner, AWSNodeTemplate or Machine alpha APIs. Do not upgrade to 0.37.0+ without first upgrading to 0.32.x. This version supports both the alpha and beta APIs, allowing you to migrate all of your existing APIs to beta APIs without experiencing downtime.

Note

Webhooks have been re-enabled by default starting in 0.37.3 to faciliate migration to v1.0. If your cluster has network policies that block Ingress then ports 8000, 8001, 8081, 8443 will need to be allowlisted. You may still choose to disable webhooks through the helm chart.

Karpenter now adds a readiness status condition to the EC2NodeClass. Make sure to upgrade your Custom Resource Definitions before proceeding with the upgrade. Failure to do so will result in Karpenter being unable to provision new nodes.
Karpenter no longer updates the logger name when creating controller loggers. We now adhere to the controller-runtime standard, where the logger name will be set as "logger": "controller" always and the controller name will be stored in the structured value "controller"
Karpenter updated the NodeClass controller naming in the following way: nodeclass -> nodeclass.status, nodeclass.hash, nodeclass.termination
Karpenter’s NodeClaim status conditions no longer include the severity field

Upgrading to `0.36.0`+

Warning

0.33.0+ only supports Karpenter v1beta1 APIs and will not work with existing Provisioner, AWSNodeTemplate or Machine alpha APIs. Do not upgrade to 0.36.0+ without first upgrading to 0.32.x. This version supports both the alpha and beta APIs, allowing you to migrate all of your existing APIs to beta APIs without experiencing downtime.

Warning

v0.36.x introduces update to drift that restricts rollback. When rolling back from >=v0.36.0, note that v0.32.9+, v0.33.4+, v0.34.5+, v0.35.4+ are the patch versions that support rollback. If Karpenter is rolled back to an older patch version, Karpenter can potentially drift all the nodes in the cluster.

Note

Webhooks have been re-enabled by default starting in 0.36.5 to faciliate migration to v1.0. If your cluster has network policies that block Ingress then ports 8000, 8001, 8081, 8443 will need to be allowlisted. You may still choose to disable webhooks through the helm chart.

Karpenter changed the name of the karpenter_cloudprovider_instance_type_price_estimate metric to karpenter_cloudprovider_instance_type_offering_price_estimate to align with the new karpenter_cloudprovider_instance_type_offering_available metric. The region label was also dropped from the metric, since this can be inferred from the environment that Karpenter is running in.

Upgrading to `0.35.0`+

Warning

0.33.0+ only supports Karpenter v1beta1 APIs and will not work with existing Provisioner, AWSNodeTemplate or Machine alpha APIs. Do not upgrade to 0.35.0+ without first upgrading to 0.32.x. This version supports both the alpha and beta APIs, allowing you to migrate all of your existing APIs to beta APIs without experiencing downtime.

Note

Webhooks have been re-enabled by default starting in 0.35.8 to faciliate migration to v1.0. If your cluster has network policies that block Ingress then ports 8000, 8001, 8081, 8443 will need to be allowlisted. You may still choose to disable webhooks through the helm chart.

Karpenter OCI tags and Helm chart version are now valid semantic versions, meaning that the v prefix from the git tag has been removed and they now follow the x.y.z pattern.

Upgrading to `0.34.0`+

Warning

0.33.0+ only supports Karpenter v1beta1 APIs and will not work with existing Provisioner, AWSNodeTemplate or Machine alpha APIs. Do not upgrade to 0.34.0+ without first upgrading to 0.32.x. This version supports both the alpha and beta APIs, allowing you to migrate all of your existing APIs to beta APIs without experiencing downtime.

Warning

The Ubuntu EKS optimized AMI has moved from 20.04 to 22.04 for Kubernetes 1.29+. This new AMI version is not currently supported for users relying on AMI auto-discovery with the Ubuntu AMI family. More details can be found in this GitHub issue. Please review this issue before upgrading to Kubernetes 1.29 if you are using the Ubuntu AMI family. Upgrading to 1.29 without making any changes to your EC2NodeClass will result in Karpenter being unable to create new nodes.

Note

Webhooks have been re-enabled by default starting in 0.34.9 to faciliate migration to v1.0. If your cluster has network policies that block Ingress then ports 8000, 8001, 8081, 8443 will need to be allowlisted. You may still choose to disable webhooks through the helm chart.

Karpenter now supports nodepool.spec.disruption.budgets, which allows users to control the speed of disruption in the cluster. Since this requires an update to the Custom Resource, before upgrading, you should re-apply the new updates to the CRDs. Check out Disruption Budgets for more.
With Disruption Budgets, Karpenter will disrupt multiple batches of nodes simultaneously, which can result in overall quicker scale-down of your cluster. Before 0.34.0, Karpenter had a hard-coded parallelism limit for each type of disruption. In 0.34.0+, Karpenter will now disrupt at most 10% of nodes for a given NodePool. There is no setting that will be perfectly equivalent with the behavior prior to 0.34.0. When considering how to configure your budgets, please refer to the following limits for versions prior to 0.34.0:
- Empty Expiration / Empty Drift / Empty Consolidation: infinite parallelism
- Non-Empty Expiration / Non-Empty Drift / Single-Node Consolidation: one node at a time
- Multi-Node Consolidation: max 100 nodes
To support Disruption Budgets, 0.34.0+ includes critical changes to Karpenter’s core controllers, which allows Karpenter to consider multiple batches of disrupting nodes simultaneously. This increases Karpenter’s performance with the potential downside of higher CPU and memory utilization from the Karpenter pod. While the magnitude of this difference varies on a case-by-case basis, when upgrading to Karpenter 0.34.0+, please note that you may need to increase the resources allocated to the Karpenter controller pods.
Karpenter now adds a default podSecurityContext that configures the fsgroup: 65536 of volumes in the pod. If you are using sidecar containers, you should review if this configuration is compatible for them. You can disable this default podSecurityContext through helm by performing --set podSecurityContext=null when installing/upgrading the chart.
The dnsPolicy for the Karpenter controller pod has been changed back to the Kubernetes cluster default of ClusterFirst. Setting our dnsPolicy to Default (confusingly, this is not the Kubernetes cluster default) caused more confusion for any users running IPv6 clusters with dual-stack nodes or anyone running Karpenter with dependencies on cluster services (like clusters running service meshes). This change may be breaking for any users on Fargate or MNG who were allowing Karpenter to manage their in-cluster DNS service (core-dns on most clusters). If you still want the old behavior here, you can change the dnsPolicy to point to use Default by setting the helm value on install/upgrade with --set dnsPolicy=Default. More details on this issue can be found in the following Github issues: #2186 and #4947.
Karpenter now disallows nodepool.spec.template.spec.resources to be set. The webhook validation never allowed nodepool.spec.template.spec.resources. We are now ensuring that CEL validation also disallows nodepool.spec.template.spec.resources to be set. If you were previously setting the resources field on your NodePool, ensure that you remove this field before upgrading to the newest version of Karpenter or else updates to the resource may fail on the new version.

Upgrading to `0.33.0`+

Warning

0.33.0+ only supports Karpenter v1beta1 APIs and will not work with existing Provisioner, AWSNodeTemplate or Machine alpha APIs. Do not upgrade to 0.33.0+ without first upgrading to 0.32.x. This version supports both the alpha and beta APIs, allowing you to migrate all of your existing APIs to beta APIs without experiencing downtime.

Note

Webhooks have been re-enabled by default starting in 0.33.8 to faciliate migration to v1.0. If your cluster has network policies that block Ingress then ports 8000, 8001, 8081, 8443 will need to be allowlisted. You may still choose to disable webhooks through the helm chart.

Karpenter no longer supports using the karpenter.sh/provisioner-name label in NodePool labels and requirements or in application node selectors, affinities, or topologySpreadConstraints. If you were previously using this label to target applications to specific Provisioners, you should update your applications to use the karpenter.sh/nodepool label instead before upgrading. If you upgrade without changing these labels, you may begin to see pod scheduling failures for these applications.
Karpenter now tags spot-instances-request with the same tags that it tags instances, volumes, and primary ENIs. This means that you will now need to add ec2:CreateTags permission for spot-instances-request. You can also further scope your controller policy for the ec2:RunInstances action to require that it launches the spot-instances-request with these specific tags. You can view an example of scoping these actions in the Getting Started Guide’s default CloudFormation controller policy.
We now recommend that you set the installation namespace for your Karpenter controllers to kube-system to denote Karpenter as a critical cluster component. This ensures that requests from the Karpenter controllers are treated with higher priority by assigning them to a different PriorityLevelConfiguration than generic requests from other namespaces. For more details on API Priority and Fairness, read the Kubernetes API Priority and Fairness Conceptual Docs. Note: Changing the namespace for your Karpenter release will cause the service account namespace to change. If you are using IRSA for authentication with AWS, you will need to change scoping set in the controller’s trust policy from karpenter:karpenter to kube-system:karpenter.
0.33.0 disables mutating and validating webhooks by default in favor of using Common Expression Language for CRD validation. The Common Expression Language Validation Feature is enabled by default on EKS 1.25. If you are using Kubernetes version >= 1.25, no further action is required. If you are using a Kubernetes version below 1.25, you now need to set DISABLE_WEBHOOK=false in your container environment variables or --set webhook.enabled=true if using Helm. View the Webhook Support Deprecated in Favor of CEL Section of the v1beta1 Migration Guide.
0.33.0 drops support for passing settings through the karpenter-global-settings ConfigMap. You should pass settings through the container environment variables in the Karpenter deployment manifest. View the Global Settings Section of the v1beta1 Migration Guide for more details.
0.33.0 enables Drift=true by default in the FEATURE_GATES. If you previously didn’t enable the feature gate, Karpenter will now check if there is a difference between the desired state of your nodes declared in your NodePool and the actual state of your nodes. View the Drift Section of Disruption Conceptual Docs for more details.
0.33.0 drops looking up the zap-logger-config through ConfigMap discovery. Instead, Karpenter now expects the logging config to be mounted on the filesystem if you are using this to configure Zap logging. This is not enabled by default, but can be enabled through --set logConfig.enabled=true in the Helm values. If you are setting any values in the logConfig from the 0.32.x upgrade, such as logConfig.logEncoding, note that you will have to explicitly set logConfig.enabled=true alongside it. Also, note that setting the Zap logging config is a deprecated feature in beta and is planned to be dropped at v1. View the Logging Configuration Section of the v1beta1 Migration Guide for more details.
0.33.0 change the default LOG_LEVEL from debug to info by default. If you are still enabling logging configuration through the zap-logger-config, no action is required.
0.33.0 drops support for comma delimited lists on tags for SubnetSelectorTerm, SecurityGroupsSelectorTerm, and AMISelectorTerm. Karpenter now supports multiple terms for each of the selectors which means that we can specify a more explicit OR-based constraint through separate terms rather than a comma-delimited list of values.

Upgrading to `0.32.0`+

Warning

Karpenter 0.32.0 introduces v1beta1 APIs, including significant changes to the API and installation procedures for the Karpenter controllers. Do not upgrade to 0.32.0+ without referencing the v1beta1 Migration Upgrade Procedure.

This version includes dual support for both alpha and beta APIs to ensure that you can slowly migrate your existing Provisioner, AWSNodeTemplate, and Machine alpha APIs to the newer NodePool, EC2NodeClass, and NodeClaim beta APIs.

Note that if you are rolling back after upgrading to 0.32.0, note that only versions 0.31.4 support handling rollback after you have deployed the v1beta1 APIs to your cluster.

Karpenter now uses settings.InterruptionQueue instead of settings.aws.InterruptionQueueName in its helm chart. The CLI argument also changed to --interruption-queue.
Karpenter now serves the webhook prometheus metrics server on port 8001. If this port is already in-use on the pod or you are running in hostNetworking mode, you may need to change this port value. You can configure this port value through the WEBHOOK_METRICS_PORT environment variable or the webhook.metrics.port value if installing via Helm.
Karpenter now exposes the ability to disable webhooks through the webhook.enabled=false value. This value will disable the webhook server and will prevent any permissions, mutating or validating webhook configurations from being deployed to the cluster.
Karpenter now moves all logging configuration for the Zap logger into the logConfig values block. Configuring Karpenter logging with this mechanism is deprecated and will be dropped at v1. Karpenter now only surfaces logLevel through the logLevel helm value. If you need more advanced configuration due to log parsing constraints, we recommend configuring your log parser to handle Karpenter’s Zap JSON logging.
The default log encoding changed from console to json. If you were previously not setting the type of log encoding, this default will change with the Helm chart. If you were setting the value through logEncoding, this value will continue to work until 0.33.x but it is deprecated in favor of logConfig.logEncoding
Karpenter now uses the karpenter.sh/disruption:NoSchedule=disrupting taint instead of the upstream node.kubernetes.io/unschedulable taint for nodes spawned with a NodePool to prevent pods from scheduling to nodes being disrupted. Pods that previously tolerated the node.kubernetes.io/unschedulable taint that previously weren’t evicted during termination will now be evicted. This most notably affects DaemonSets, which have the node.kubernetes.io/unschedulable toleration by default, where Karpenter will now remove these pods during termination. If you want your specific pods to not be evicted when nodes are scaled down, you should add a toleration to the pods with the following: Key=karpenter.sh/disruption, Effect=NoSchedule, Operator=Equals, Values=disrupting.
- Note: Karpenter will continue to use the old node.kubernetes.io/unschedulable taint for nodes spawned with a Provisioner.

Upgrading to `0.31.0`+

Karpenter moved its securityContext constraints from pod-wide to only applying to the Karpenter container exclusively. If you were previously relying on the pod-wide securityContext for your sidecar containers, you will now need to set these values explicitly in your sidecar container configuration.

Upgrading to `0.30.0`+

Karpenter will now statically drift on both Provisioner and AWSNodeTemplate Fields. For Provisioner Static Drift, the karpenter.sh/provisioner-hash annotation must be present on both the Provisioner and Machine. For AWSNodeTemplate drift, the karpenter.k8s.aws/nodetemplate-hash annotation must be present on the AWSNodeTemplate and Machine. Karpenter will not add these annotations to pre-existing nodes, so each of these nodes will need to be recycled one time for the annotations to be added.
Karpenter will now fail validation on AWSNodeTemplates and Provisioner spec.provider that have amiSelectors, subnetSelectors, or securityGroupSelectors set with a combination of id selectors (aws-ids, aws::ids) and other selectors.
Karpenter now statically sets the securityContext at both the pod and container-levels and doesn’t allow override values to be passed through the Helm chart. This change was made to adhere to Restricted Pod Security Standard, which follows pod hardening best practices.

Note

If you have sidecar containers configured to run alongside Karpenter that cannot tolerate the pod-wide securityContext constraints, you will need to specify overrides to the sidecar securityContext in your deployment.

Upgrading to `0.29.0`+

Warning

Karpenter 0.29.1 contains a file descriptor and memory leak bug that leads to Karpenter getting OOMKilled and restarting at the point that it hits its memory or file descriptor limit. Karpenter 0.29.2+ fixes this leak.

Karpenter has changed the default metrics service port from 8080 to 8000 and the default webhook service port from 443 to 8443. In 0.28.0, the Karpenter pod port was changed to 8000, but referenced the service by name, allowing users to scrape the service at port 8080 for metrics. 0.29.0 aligns the two ports so that service and pod metrics ports are the same. These ports are set by the controller.metrics.port and webhook.port Helm chart values, so if you have previously set these to non-default values, you may need to update your Prometheus scraper to match these new values.
Karpenter will now reconcile nodes that are drifted due to their Security Groups or their Subnets. If your AWSNodeTemplate’s Security Groups differ from the Security Groups used for an instance, Karpenter will consider it drifted. If the Subnet used by an instance is not contained in the allowed list of Subnets for an AWSNodeTemplate, Karpenter will also consider it drifted.
- Since Karpenter uses tags for discovery of Subnets and SecurityGroups, check the Threat Model to see how to manage this IAM Permission.

Upgrading to `0.28.0`+

Warning

Karpenter 0.28.0 is incompatible with Kubernetes version 1.26+, which can result in additional node scale outs when using --cloudprovider=external, which is the default for the EKS Optimized AMI. See: https://github.com/aws/karpenter-core/pull/375. Karpenter 0.28.1+ fixes this issue and is compatible with Kubernetes version 1.26+.

The extraObjects value is now removed from the Helm chart. Having this value in the chart proved to not work in the majority of Karpenter installs and often led to anti-patterns, where the Karpenter resources installed to manage Karpenter’s capacity were directly tied to the install of the Karpenter controller deployments. The Karpenter team recommends that, if you want to install Karpenter manifests alongside the Karpenter Helm chart, to do so by creating a separate chart for the manifests, creating a dependency on the controller chart.
The aws.nodeNameConvention setting is now removed from the karpenter-global-settings ConfigMap. Because Karpenter is now driving its orchestration of capacity through Machines, it no longer needs to know the node name, making this setting obsolete. Karpenter ignores configuration that it doesn’t recognize in the karpenter-global-settings ConfigMap, so leaving the aws.nodeNameConvention in the ConfigMap will simply cause this setting to be ignored.
Karpenter now defines a set of “restricted tags” which can’t be overridden with custom tagging in the AWSNodeTemplate or in the karpenter-global-settings ConfigMap. If you are currently using any of these tag overrides when tagging your instances, webhook validation will now fail. These tags include:
- karpenter.sh/managed-by
- karpenter.sh/provisioner-name
- kubernetes.io/cluster/${CLUSTER_NAME}
The following metrics changed their meaning, based on the introduction of the Machine resource:
- karpenter_nodes_terminated: Use karpenter_machines_terminated if you are interested in the reason why a Karpenter machine was deleted. karpenter_nodes_terminated now only tracks the count of terminated nodes without any additional labels.
- karpenter_nodes_created: Use karpenter_machines_created if you are interested in the reason why a Karpenter machine was created. karpenter_nodes_created now only tracks the count of created nodes without any additional labels.
- karpenter_deprovisioning_replacement_node_initialized_seconds: This metric has been replaced in favor of karpenter_deprovisioning_replacement_machine_initialized_seconds.
0.28.0 introduces the Machine CustomResource into the karpenter.sh API Group and requires this CustomResourceDefinition to run properly. Karpenter now orchestrates its CloudProvider capacity through these in-cluster Machine CustomResources. When performing a scheduling decision, Karpenter will create a Machine, resulting in launching CloudProvider capacity. The kubelet running on the new capacity will then register the node to the cluster shortly after launch.
- If you are using Helm to upgrade between versions of Karpenter, note that Helm does not automate the process of upgrading or install the new CRDs into your cluster. To install or upgrade the existing CRDs, follow the guidance under the Custom Resource Definition (CRD) Upgrades section of the upgrade guide.
- Karpenter will hydrate Machines on startup for existing capacity managed by Karpenter into the cluster. Existing capacity launched by an older version of Karpenter is discovered by finding CloudProvider capacity with the karpenter.sh/provisioner-name tag or the karpenter.sh/provisioner-name label on nodes.
The metrics port for the Karpenter deployment was changed from 8080 to 8000. Users who scrape the pod directly for metrics rather than the service will need to adjust the commands they use to reference port 8000. Any users who scrape metrics from the service should be unaffected.

Warning

Karpenter creates a mapping between CloudProvider machines and CustomResources in the cluster for capacity tracking. To ensure this mapping is consistent, Karpenter utilizes the following tag keys:

karpenter.sh/managed-by
karpenter.sh/provisioner-name
kubernetes.io/cluster/${CLUSTER_NAME}

Because Karpenter takes this dependency, any user that has the ability to Create/Delete these tags on CloudProvider machines will have the ability to orchestrate Karpenter to Create/Delete CloudProvider machines as a side effect. Check the Threat Model to see how this might affect you, and ways to mitigate this.

Rolling Back

If, after upgrading to 0.28.0+, a rollback to an older version of Karpenter needs to be performed, Karpenter will continue to function normally, though you will still have the Machine CustomResources on your cluster. You will need to manually delete the Machines and patch out the finalizers to fully complete the rollback.

Karpenter marks CloudProvider capacity as “managed by” a Machine using the karpenter-sh/managed-by tag on the CloudProvider machine. It uses this tag to ensure that the Machine CustomResources in the cluster match the CloudProvider capacity managed by Karpenter. If these states don’t match, Karpenter will garbage collect the capacity. Because of this, if performing an upgrade, followed by a rollback, followed by another upgrade to 0.28.0+, ensure you remove the karpenter.sh/managed-by tags from existing capacity; otherwise, Karpenter will deprovision the capacity without a Machine CR counterpart.

Upgrading to `0.27.3`+

The defaulting.webhook.karpenter.sh mutating webhook was removed in 0.27.3. If you are coming from an older version of Karpenter where this webhook existed and the webhook was not managed by Helm, you may need to delete the stale webhook.

kubectl delete mutatingwebhookconfigurations defaulting.webhook.karpenter.sh

Upgrading to `0.27.0`+

The Karpenter controller pods now deploy with kubernetes.io/hostname self anti-affinity by default. If you are running Karpenter in HA (high-availability) mode and you do not have enough nodes to match the number of pod replicas you are deploying with, you will need to scale-out your nodes for Karpenter.
The following controller metrics changed and moved under the controller_runtime metrics namespace:
- karpenter_metricscraper_...
- karpenter_deprovisioning_...
- karpenter_provisioner_...
- karpenter_interruption_...
The following controller metric names changed, affecting the controller label value under controller_runtime_... metrics. These metrics include:
- podmetrics -> pod_metrics
- provisionermetrics -> provisioner_metrics
- metricscraper -> metric_scraper
- provisioning -> provisioner_trigger
- node-state -> node_state
- pod-state -> pod_state
- provisioner-state -> provisioner_state
The karpenter_allocation_controller_scheduling_duration_seconds metric name changed to karpenter_provisioner_scheduling_duration_seconds

Upgrading to `0.26.0`+

The karpenter.sh/do-not-evict annotation no longer blocks node termination when running kubectl delete node. This annotation on pods will only block automatic deprovisioning that is considered “voluntary,” that is, disruptions that can be avoided. Disruptions that Karpenter deems as “involuntary” and will ignore the karpenter.sh/do-not-evict annotation include spot interruption and manual deletion of the node. See Disabling Deprovisioning for more details.
Default resources requests and limits are removed from the Karpenter’s controller deployment through the Helm chart. If you have not set custom resource requests or limits in your Helm values and are using Karpenter’s defaults, you will now need to set these values in your Helm chart deployment.
The controller.image value in the Helm chart has been broken out to a map consisting of controller.image.repository, controller.image.tag, and controller.image.digest. If manually overriding the controller.image, you will need to update your values to the new design.

Upgrading to `0.25.0`+

Cluster Endpoint can now be automatically discovered. If you are using Amazon Elastic Kubernetes Service (EKS), you can now omit the clusterEndpoint field in your configuration. In order to allow the resolving, you have to add the permission eks:DescribeCluster to the Karpenter Controller IAM role.

Upgrading to `0.24.0`+

Settings are no longer updated dynamically while Karpenter is running. If you manually make a change to the karpenter-global-settings ConfigMap, you will need to reload the containers by restarting the deployment with kubectl rollout restart -n karpenter deploy/karpenter
Karpenter no longer filters out instance types internally. Previously, g2 (not supported by the NVIDIA device plugin) and FPGA instance types were filtered. The only way to filter instance types now is to set requirements on your provisioner or pods using well-known node labels described here. If you are currently using overly broad requirements that allows all of the g instance-category, you will want to tighten the requirement, or add an instance-generation requirement.
aws.tags in karpenter-global-settings ConfigMap is now a top-level field and expects the value associated with this key to be a JSON object of string to string. This is change from previous versions where keys were given implicitly by providing the key-value pair aws.tags.<key>: value in the ConfigMap.

Upgrading to `0.22.0`+

Do not upgrade to this version unless you are on Kubernetes >= v1.21. Karpenter no longer supports Kubernetes v1.20, but now supports Kubernetes v1.25. This change is due to the v1 PDB API, which was introduced in K8s v1.20 and subsequent removal of the v1beta1 API in K8s v1.25.

Upgrading to `0.20.0`+

Prior to 0.20.0, Karpenter would prioritize certain instance type categories absent of any requirements in the Provisioner. 0.20.0+ removes prioritizing these instance type categories (“m”, “c”, “r”, “a”, “t”, “i”) in code. Bare Metal and GPU instance types are still deprioritized and only used if no other instance types are compatible with the node requirements. Since Karpenter does not prioritize any instance types, if you do not want exotic instance types and are not using the runtime Provisioner defaults, you will need to specify this in the Provisioner.

Upgrading to `0.19.0`+

The karpenter webhook and controller containers are combined into a single binary, which requires changes to the Helm chart. If your Karpenter installation (Helm or otherwise) currently customizes the karpenter webhook, your deployment tooling may require minor changes.
Karpenter now supports native interruption handling. If you were previously using Node Termination Handler for spot interruption handling and health events, you will need to remove the component from your cluster before enabling aws.interruptionQueueName. For more details on Karpenter’s interruption handling, see the Interruption Handling Docs.
Instance category defaults are now explicitly persisted in the Provisioner, rather than handled implicitly in memory. By default, Provisioners will limit instance category to c,m,r. If any instance type constraints are applied, it will override this default. If you have created Provisioners in the past with unconstrained instance type, family, or category, Karpenter will now more flexibly use instance types than before. If you would like to apply these constraints, they must be included in the Provisioner CRD.
Karpenter CRD raw YAML URLs have migrated from https://raw.githubusercontent.com/aws/karpenter-provider-aws/v0.19.3/charts/karpenter/crds/... to https://raw.githubusercontent.com/aws/karpenter-provider-aws/v0.19.3/pkg/apis/crds/.... If you reference static Karpenter CRDs or rely on kubectl replace -f to apply these CRDs from their remote location, you will need to migrate to the new location.
Pods without an ownerRef (also called “controllerless” or “naked” pods) will now be evicted by default during node termination and consolidation. Users can prevent controllerless pods from being voluntarily disrupted by applying the karpenter.sh/do-not-evict: "true" annotation to the pods in question.
The following CLI options/environment variables are now removed and replaced in favor of pulling settings dynamically from the karpenter-global-settings ConfigMap. See the Settings docs for more details on configuring the new values in the ConfigMap.
- CLUSTER_NAME -> settings.aws.clusterName
- CLUSTER_ENDPOINT -> settings.aws.clusterEndpoint
- AWS_DEFAULT_INSTANCE_PROFILE -> settings.aws.defaultInstanceProfile
- AWS_ENABLE_POD_ENI -> settings.aws.enablePodENI
- AWS_ENI_LIMITED_POD_DENSITY -> settings.aws.enableENILimitedPodDensity
- AWS_ISOLATED_VPC -> settings.aws.isolatedVPC
- AWS_NODE_NAME_CONVENTION -> settings.aws.nodeNameConvention
- VM_MEMORY_OVERHEAD -> settings.aws.vmMemoryOverheadPercent

Upgrading to `0.18.0`+

0.18.0 removes the karpenter_consolidation_nodes_created and karpenter_consolidation_nodes_terminated prometheus metrics in favor of the more generic karpenter_nodes_created and karpenter_nodes_terminated metrics. You can still see nodes created and terminated by consolidation by checking the reason label on the metrics. Check out all the metrics published by Karpenter here.

Upgrading to `0.17.0`+

Karpenter’s Helm chart package is now stored in Karpenter’s OCI (Open Container Initiative) registry. The Helm CLI supports the new format since v3.8.0+. With this change charts.karpenter.sh is no longer updated but preserved to allow using older Karpenter versions. For examples on working with the Karpenter Helm charts look at Install Karpenter Helm Chart.

Users who have scripted the installation or upgrading of Karpenter need to adjust their scripts with the following changes:

There is no longer a need to add the Karpenter Helm repo with helm repo add
The full URL of the Helm chart needs to be present when using the helm CLI
If you were not prepending a v to the version (i.e. 0.17.0), you will need to do so with the OCI chart (i.e v0.17.0).

Upgrading to `0.16.2`+

0.16.2 adds new kubeletConfiguration fields to the provisioners.karpenter.sh v1alpha5 CRD. The CRD will need to be updated to use the new parameters:

kubectl replace -f https://raw.githubusercontent.com/aws/karpenter-provider-aws/v0.16.2/charts/karpenter/crds/karpenter.sh_provisioners.yaml

Upgrading to `0.16.0`+

0.16.0 adds a new weight field to the provisioners.karpenter.sh v1alpha5 CRD. The CRD will need to be updated to use the new parameters:

kubectl replace -f https://raw.githubusercontent.com/aws/karpenter-provider-aws/v0.16.0/charts/karpenter/crds/karpenter.sh_provisioners.yaml

Upgrading to `0.15.0`+

0.15.0 adds a new consolidation field to the provisioners.karpenter.sh v1alpha5 CRD. The CRD will need to be updated to use the new parameters:

kubectl replace -f https://raw.githubusercontent.com/aws/karpenter-provider-aws/v0.15.0/charts/karpenter/crds/karpenter.sh_provisioners.yaml

Upgrading to `0.14.0`+

0.14.0 adds new fields to the provisioners.karpenter.sh v1alpha5 and awsnodetemplates.karpenter.k8s.aws v1alpha1 CRDs. The CRDs will need to be updated to use the new parameters:

kubectl replace -f https://raw.githubusercontent.com/aws/karpenter-provider-aws/v0.14.0/charts/karpenter/crds/karpenter.sh_provisioners.yaml

kubectl replace -f https://raw.githubusercontent.com/aws/karpenter-provider-aws/v0.14.0/charts/karpenter/crds/karpenter.k8s.aws_awsnodetemplates.yaml

0.14.0 changes the way Karpenter discovers its dynamically generated AWS launch templates to use a tag rather than a Name scheme. The previous name scheme was Karpenter-${CLUSTER_NAME}-* which could collide with user created launch templates that Karpenter should not manage. The new scheme uses a tag on the launch template karpenter.k8s.aws/cluster: ${CLUSTER_NAME}. As a result, Karpenter will not clean-up dynamically generated launch templates using the old name scheme. You can manually clean these up with the following commands:

## Find launch templates that match the naming pattern and you do not want to keep
aws ec2 describe-launch-templates --filters="Name=launch-template-name,Values=Karpenter-${CLUSTER_NAME}-*"

## Delete launch template(s) that match the name but do not have the "karpenter.k8s.aws/cluster" tag
aws ec2 delete-launch-template --launch-template-id <LAUNCH_TEMPLATE_ID>

0.14.0 introduces additional instance type filtering if there are no node.kubernetes.io/instance-type or karpenter.k8s.aws/instance-family or karpenter.k8s.aws/instance-category requirements that restrict instance types specified on the provisioner. This prevents Karpenter from launching bare metal and some older non-current generation instance types unless the provisioner has been explicitly configured to allow them. If you specify an instance type or family requirement that supplies a list of instance-types or families, that list will be used regardless of filtering. The filtering can also be completely eliminated by adding an Exists requirement for instance type or family.

 - key: node.kubernetes.io/instance-type
 operator: Exists

0.14.0 introduces support for custom AMIs without the need for an entire launch template. You must add the ec2:DescribeImages permission to the Karpenter Controller Role for this feature to work. This permission is needed for Karpenter to discover custom images specified. Read the Custom AMI documentation here to get started
0.14.0 adds an an additional default toleration (CriticalAddonOnly=Exists) to the Karpenter Helm chart. This may cause Karpenter to run on nodes with that use this Taint which previously would not have been schedulable. This can be overridden by using --set tolerations[0]=null.
0.14.0 deprecates the AWS_ENI_LIMITED_POD_DENSITY environment variable in-favor of specifying spec.kubeletConfiguration.maxPods on the Provisioner. AWS_ENI_LIMITED_POD_DENSITY will continue to work when maxPods is not set on the Provisioner. If maxPods is set, it will override AWS_ENI_LIMITED_POD_DENSITY on that specific Provisioner.

Upgrading to `0.13.0`+

0.13.0 introduces a new CRD named AWSNodeTemplate which can be used to specify AWS Cloud Provider parameters. Everything that was previously specified under spec.provider in the Provisioner resource, can now be specified in the spec of the new resource. The use of spec.provider is deprecated but will continue to function to maintain backwards compatibility for the current API version (v1alpha5) of the Provisioner resource. 0.13.0 also introduces support for custom user data that doesn’t require the use of a custom launch template. The user data can be specified in-line in the AWSNodeTemplate resource.

If you are upgrading from 0.10.1 - 0.11.1, a new CRD awsnodetemplate was added. In 0.12.0, this crd was renamed to awsnodetemplates. Since Helm does not manage the lifecycle of CRDs, you will need to perform a few manual steps for this CRD upgrade:
1. Make sure any awsnodetemplate manifests are saved somewhere so that they can be reapplied to the cluster.
2. kubectl delete crd awsnodetemplate
3. kubectl apply -f https://raw.githubusercontent.com/aws/karpenter-provider-aws/v0.13.2/charts/karpenter/crds/karpenter.k8s.aws_awsnodetemplates.yaml
4. Perform the Karpenter upgrade to 0.13.0+, which will install the new awsnodetemplates CRD.
5. Reapply the awsnodetemplate manifests you saved from step 1, if applicable.
0.13.0 also adds EC2/spot price fetching to Karpenter to allow making more accurate decisions regarding node deployments. Our getting started guide documents this, but if you are upgrading Karpenter you will need to modify your Karpenter controller policy to add the pricing:GetProducts and ec2:DescribeSpotPriceHistory permissions.

Upgrading to `0.12.0`+

0.12.0 adds an OwnerReference to each Node created by a provisioner. Previously, deleting a provisioner would orphan nodes. Now, deleting a provisioner will cause Kubernetes cascading delete logic to gracefully terminate the nodes using the Karpenter node finalizer. You may still orphan nodes by removing the owner reference.
If you are upgrading from 0.10.1 - 0.11.1, a new CRD awsnodetemplate was added. In 0.12.0, this crd was renamed to awsnodetemplates. Since Helm does not manage the lifecycle of CRDs, you will need to perform a few manual steps for this CRD upgrade:
1. Make sure any awsnodetemplate manifests are saved somewhere so that they can be reapplied to the cluster.
2. kubectl delete crd awsnodetemplate
3. kubectl apply -f https://raw.githubusercontent.com/aws/karpenter-provider-aws/v0.12.1/charts/karpenter/crds/karpenter.k8s.aws_awsnodetemplates.yaml
4. Perform the Karpenter upgrade to 0.12.0+, which will install the new awsnodetemplates CRD.
5. Reapply the awsnodetemplate manifests you saved from step 1, if applicable.

Upgrading to `0.11.0`+

0.11.0 changes the way that the vpc.amazonaws.com/pod-eni resource is reported. Instead of being reported for all nodes that could support the resources regardless of if the cluster is configured to support it, it is now controlled by a command line flag or environment variable. The parameter defaults to false and must be set if your cluster uses security groups for pods. This can be enabled by setting the environment variable AWS_ENABLE_POD_ENI to true via the helm value controller.env.

Other extended resources must be registered on nodes by their respective device plugins which are typically installed as DaemonSets (e.g. the nvidia.com/gpu resource will be registered by the NVIDIA device plugin. Previously, Karpenter would register these resources on nodes at creation and they would be zeroed out by kubelet at startup. By allowing the device plugins to register the resources, pods will not bind to the nodes before any device plugin initialization has occurred.

0.11.0 adds a providerRef field in the Provisioner CRD. To use this new field you will need to replace the Provisioner CRD manually:

kubectl replace -f https://raw.githubusercontent.com/aws/karpenter-provider-aws/v0.11.0/charts/karpenter/crds/karpenter.sh_provisioners.yaml

Upgrading to `0.10.0`+

0.10.0 adds a new field, startupTaints to the provisioner spec. Standard Helm upgrades do not upgrade CRDs so the field will not be available unless the CRD is manually updated. This can be performed prior to the standard upgrade by applying the new CRD manually:

kubectl replace -f https://raw.githubusercontent.com/aws/karpenter-provider-aws/v0.10.0/charts/karpenter/crds/karpenter.sh_provisioners.yaml

📝 If you don’t perform this manual CRD update, Karpenter will work correctly except for rejecting the creation/update of provisioners that use startupTaints.

Upgrading to `0.6.2`+

If using Helm, the variable names have changed for the cluster’s name and endpoint. You may need to update any configuration that sets the old variable names.

controller.clusterName is now clusterName
controller.clusterEndpoint is now clusterEndpoint

V1.0: Compatibility

Mon, 01 Jan 0001 00:00:00 +0000

Compatibility

To make upgrading easier we aim to minimize the introduction of breaking changes. Before you begin upgrading Karpenter, consider Karpenter compatibility issues related to Kubernetes and the NodePool API (previously Provisioner).

Compatibility Matrix

KUBERNETES	1.25	1.26	1.27	1.28	1.29	1.30	1.31
karpenter	>= 0.25	>= 0.28	>= 0.28	>= 0.31	>= 0.34	>= 0.37	>= 1.0.5

Note

Karpenter currently does not support the following new topologySpreadConstraints keys, promoted to beta in Kubernetes 1.27:

matchLabelKeys
nodeAffinityPolicy
nodeTaintsPolicy

For more information on Karpenter’s support for these keys, view this tracking issue.

Note

Karpenter supports using Kubernetes Common Expression Language for validating its Custom Resource Definitions out-of-the-box; however, this feature is not supported on versions of Kubernetes < 1.25. If you are running an earlier version of Kubernetes, you will need to use the Karpenter admission webhooks for validation instead. You can enable these webhooks with --set webhook.enabled=true when applying the Karpenter Helm chart.

Compatibility issues

When we introduce a breaking change, we do so only as described in this document.

Karpenter follows Semantic Versioning 2.0.0 in its stable release versions, while in major version zero (0.y.z) anything may change at any time. However, to further protect users during this phase we will only introduce breaking changes in minor releases (releases that increment y in x.y.z). Note this does not mean every minor upgrade has a breaking change as we will also increment the minor version when we release a new feature.

Users should therefore check to see if there is a breaking change every time they are upgrading to a new minor version.

How Do We Break Incompatibility?

When there is a breaking change we will:

Increment the minor version when in major version 0
Add a permanent separate section named upgrading to x.y.z+ under release upgrade notes clearly explaining the breaking change and what needs to be done on the user side to ensure a safe upgrade
Add the sentence “This is a breaking change, please refer to the above link for upgrade instructions” to the top of the release notes and in all our announcements

How Do We Find Incompatibilities?

Besides the peer review process for all changes to the code base we also do the followings in order to find incompatibilities:

(To be implemented) To check the compatibility of the application, we will automate tests for installing, uninstalling, upgrading from an older version, and downgrading to an older version
(To be implemented) To check the compatibility of the documentation with the application, we will turn the commands in our documentation into scripts that we can automatically run

Security Patches

While we are in major version 0 we will not release security patches to older versions. Rather we provide the patches in the latest versions. When at major version 1 we will have an EOL (end of life) policy where we provide security patches for a subset of older versions and deprecate the others.

Release Types

Karpenter offers three types of releases. This section explains the purpose of each release type and how the images for each release type are tagged in our public image repository.

Stable Releases

Stable releases are the only recommended versions for production environments. Stable releases are tagged with a semantic version (e.g. 0.35.0). Note that stable releases prior to 0.35.0 are prefixed with a v (e.g. v0.34.0).

Release Candidates

We consider having release candidates for major and important minor versions. Our release candidates are tagged like x.y.z-rc.0, x.y.z-rc.1. The release candidate will then graduate to x.y.z as a stable release. By adopting this practice we allow our users who are early adopters to test out new releases before they are available to the wider community, thereby providing us with early feedback resulting in more stable releases. Note that, like the stable releases, release candidates prior to 0.35.0 are prefixed with a v.

Snapshot Releases

We release a snapshot release for every commit that gets merged into aws/karpenter-provider-aws. This enables users to immediately try a new feature or fix right after it’s merged rather than waiting days or weeks for release.

Snapshot releases are not made available in the same public ECR repository as other release types, they are instead published to a separate private ECR repository. Helm charts are published to oci://021119463062.dkr.ecr.us-east-1.amazonaws.com/karpenter/snapshot/karpenter and are tagged with the git commit hash prefixed by the Karpenter major version (e.g. 0-fc17bfc89ebb30a3b102a86012b3e3992ec08adf). Anyone with an AWS account can pull from this repository, but must first authenticate:

aws ecr get-login-password --region us-east-1 | docker login --username AWS --password-stdin 021119463062.dkr.ecr.us-east-1.amazonaws.com

Note

Snapshot releases are suitable for testing, and troubleshooting but they should not be used in production environments. Snapshot releases are ephemeral and will be removed 90 days after they were published.

V1.0: v1 Migration

Mon, 01 Jan 0001 00:00:00 +0000

This migration guide is designed to help you migrate to Karpenter v1.0.x from v0.33.x through v0.37.x. Use this document as a reference to the changes that were introduced in this release and as a guide to how you need to update the manifests and other Karpenter objects you created in previous Karpenter releases.

Before continuing with this guide, you should know that Karpenter v1.0.x only supports Karpenter v1 and v1beta1 APIs. Earlier Provisioner, AWSNodeTemplate, and Machine APIs are not supported. Do not upgrade to v1.0.x without first upgrading to v0.32.x and then upgrading to v0.33+.

Additionaly, validate that you are running at least Kubernetes 1.25. Use the compatibility matrix to confirm you are on a supported Kubernetes version.

Before You Start

Karpenter v1.0 is a major release and contains a number of breaking changes. The following section will highlight some of the major breaking changes, but you should review the full changelog before proceeding with the upgrade.

Forceful Expiration

Previously, Karpenter would only begin draining an expired node after ensuring graceful disruption was possible and replacement capacity was provisioned. Now, NodeClaim expiration is forceful. Once a NodeClaim expires, Karpenter will immediately begin draining the corresponding Node without first provisioning a replacement Node. This forceful approach means that all expired NodeClaims that were previously blocked from disruption (due to Pod Disruption Budgets (PDBs), do-not-disrupt pods, or other constraints) will be expired. The scheduler will then attempt to provision new Nodes for the disrupted workloads as they are draining. This behavior may lead to an increased number of pods in the “Pending” state while replacement capacity is being provisioned. Before upgrading, users should check for expired NodeClaims and resolve any disruption blockers to ensure a smooth upgrade process.

Deprecated Annotations, Labels, and Tags Removed

The following annotations, labels, and tags have been removed in v1.0.0:

Key	Type
`karpenter.sh/do-not-consolidate`	annotation
`karpenter.sh/do-not-evict`	annotation
`karpenter.sh/managed-by`	tag

Both the karpenter.sh/do-not-consolidate and the karpenter.sh/do-not-evict annotations were deprecated in v0.32.0. They have now been dropped in-favor of their replacement, karpenter.sh/do-not-disrupt.

The karpenter.sh/managed-by, which currently stores the cluster name in its value, is replaced by eks:eks-cluster-name, to allow for EKS Pod Identity ABAC policies.

Zap Logging Config Removed

Support for setting the Zap logging config was deprecated in v0.32.0 and has been been removed in v1.0.0. The following environment variables are now available to configure logging:

LOG_LEVEL
LOG_OUTPUT_PATHS
LOG_ERROR_OUTPUT_PATHS.

Refer to Settings for more details.

New MetadataOptions Defaults

The default value for httpPutResponseHopLimit has been reduced from 2 to 1. This prevents pods that are not using hostNetworking from accessing IMDS by default. If you have pods which rely on access to IMDS, and are not using hostNetworking, you will need to either update the pod’s networking config or configure httpPutResponseHopLimit on your EC2NodeClass. This change aligns Karpenter’s defaults with EKS’ Best Practices.

Ubuntu AMIFamily Removed

Support for automatic AMI selection and UserData generation for Ubuntu has been dropped in Karpenter v1.0.0. To continue using Ubuntu AMIs you will need to specify an AMI using amiSelectorTerms.

UserData generation can be achieved using amiFamily: AL2, which has an identical UserData format. However, compatibility is not guaranteed long-term and changes to either AL2 or Ubuntu’s UserData format may introduce incompatibilities. If this occurs, amiFamily: Custom should be used for Ubuntu AMIs and UserData will need to be entirely maintained by the user.

If you are upgrading to v1.0.x and already have v1beta1 Ubuntu EC2NodeClasses, all you need to do is specify amiSelectorTerms and Karpenter will translate your EC2NodeClasses to the v1 equivalent (as shown below). Failure to specify amiSelectorTerms will result in the EC2NodeClass and all referencing NodePools to become NotReady. These NodePools and EC2NodeClasses would then be ignored for provisioning and drift.

# Original v1beta1 EC2NodeClass
version: karpenter.k8s.aws/v1beta1
kind: EC2NodeClass
spec:
 amiFamily: Ubuntu
 amiSelectorTerms:
 - id: ami-foo
---
# Conversion Webhook Output
version: karpenter.k8s.aws/v1
kind: EC2NodeClass
metadata:
 annotations:
 compatibility.karpenter.k8s.aws/v1beta1-ubuntu: amiFamily,blockDeviceMappings
spec:
 amiFamily: AL2
 amiSelectorTerms:
 - id: ami-foo
 blockDeviceMappings:
 - deviceName: '/dev/sda1'
 rootVolume: true
 ebs:
 encrypted: true
 volumeType: gp3
 volumeSize: 20Gi

New Registration Taint

EC2NodeClasses using amiFamily: Custom must configure the kubelet to register with the karpenter.sh/unregistered:NoExecute taint. For example, to achieve this with an AL2023 AMI you would use the following UserData:

version: karpenter.k8s.aws/v1beta1
kind: EC2NodeClass
spec:
 amiFamily: Custom
 amiSelectorTerms:
 - id: ami-custom-al2023-ami
 userData: |
 apiVersion: node.eks.aws/v1alpha1
 kind: NodeConfig
 spec:
 # ...
 kubelet:
 config:
 # ...
 registerWithTaints:
 - key: karpenter.sh/unregistered
 effect: NoExecute

If you are using one of Karpenter’s managed AMI families, this will be handled for you by Karpenter’s generated UserData.

Upgrading

Before proceeding with the upgrade, be sure to review the changelog and review the upgrade procedure in its entirety. The procedure can be split into two sections:

Steps 1 through 6 will upgrade you to the latest patch release on your current minor version.
Steps 7 through 11 will then upgrade you to the latest v1.0 release.

While it is possible to upgrade directly from any patch version on versions v0.33 through v0.37, rollback from v1.0.x is only supported on the latest patch releases. Upgrading directly may leave you unable to rollback. For more information on the rollback procedure, refer to the downgrading section.

Note

The examples provided in the upgrade procedure assume you’ve installed Karpenter following the getting started guide. If you are using IaC / GitOps, you may need to adapt the procedure to fit your specific infrastructure solution. You should still review the upgrade procedure; the sequence of operations remains the same regardless of the solution used to roll out the changes.

Upgrade Procedure

Configure environment variables for the cluster you’re upgrading:

export AWS_PARTITION="aws" # if you are not using standard partitions, you may need to configure to aws-cn / aws-us-gov
export CLUSTER_NAME="${USER}-karpenter-demo"
export AWS_REGION="us-west-2"
export AWS_ACCOUNT_ID="$(aws sts get-caller-identity --query Account --output text)"
export KARPENTER_NAMESPACE=kube-system
export KARPENTER_IAM_ROLE_ARN="arn:${AWS_PARTITION}:iam::${AWS_ACCOUNT_ID}:role/${CLUSTER_NAME}-karpenter"

Determine your current Karpenter version:
```
kubectl get deployment -A -l app.kubernetes.io/name=karpenter -ojsonpath="{.items[0].metadata.labels['app\.kubernetes\.io/version']}{'\n'}"
```
To upgrade to v1, you must be running a Karpenter version between v0.33 and v0.37. If you are on an older version, you must upgrade before continuing with this guide.
Before upgrading to v1, we’re going to upgrade to a patch release that supports rollback. Set the KARPENTER_VERSION environment variable to the latest patch release for your current minor version. The following releases are the current latest:
- 0.37.7
- 0.36.9
- 0.35.12
- v0.34.13
- v0.33.12
```
# Note: v0.33.x and v0.34.x include the v prefix, omit it for versions v0.35+
export KARPENTER_VERSION="0.37.7" # Replace with your minor version
```

Upgrade Karpenter to the latest patch release for your current minor version. Note that webhooks must be enabled.

# Service account annotation can be dropped when using pod identity
helm upgrade --install karpenter oci://public.ecr.aws/karpenter/karpenter --version ${KARPENTER_VERSION} --namespace "${KARPENTER_NAMESPACE}" --create-namespace \
 --set serviceAccount.annotations."eks\.amazonaws\.com/role-arn"=${KARPENTER_IAM_ROLE_ARN} \
 --set settings.clusterName=${CLUSTER_NAME} \
 --set settings.interruptionQueue=${CLUSTER_NAME} \
 --set controller.resources.requests.cpu=1 \
 --set controller.resources.requests.memory=1Gi \
 --set controller.resources.limits.cpu=1 \
 --set controller.resources.limits.memory=1Gi \
 --set webhook.enabled=true \
 --set webhook.port=8443 \
 --wait

Apply the latest patch version of your current minor version’s Custom Resource Definitions (CRDs). Applying this version of the CRDs will enable the use of both the v1 and v1beta1 APIs on this version via the conversion webhooks. Note that this is only for rollback purposes, and new features available with the v1 APIs will not work on your minor version.
```
helm upgrade --install karpenter-crd oci://public.ecr.aws/karpenter/karpenter-crd --version "${KARPENTER_VERSION}" --namespace "${KARPENTER_NAMESPACE}" --create-namespace \
 --set webhook.enabled=true \
 --set webhook.serviceName="karpenter" \
 --set webhook.port=8443
```
Note

To properly template the conversion field in the CRD, the karpenter-crd chart must be used. If you’re using a GitOps solution to manage your Karpenter installation, you should use this chart to manage your CRDs. You should set skipCrds to true for the main karpenter chart (e.g. Argo CD).

Alternatively, you can install the CRDs with the main chart and apply the following patches. However, we strongly recommend using the dedicated CRD chart.
```
SERVICE_NAME="karpenter"
SERVICE_NAMESPACE="kube-system"
SERVICE_PORT="8443"
CRDS=("nodepools.karpenter.sh" "nodeclaims.karpenter.sh" "ec2nodeclasses.karpenter.k8s.aws")
for crd in ${CRDS[@]}; do
 kubectl patch customresourcedefinitions ${crd} --patch-file=/dev/stdin <<-EOF
spec:
 conversion:
 webhook:
 clientConfig:
 service:
 name: "${SERVICE_NAME}"
 namespace: "${SERVICE_NAMESPACE}"
 port: ${SERVICE_PORT}
EOF
done
```
Note
Helm uses annotations on resources it provisions to track ownership. Switching to the new chart may result in Helm failing to install the chart due to invalid ownership metadata. If you encounter errors at this step, consult this troubleshooting entry to resolve.
Validate that Karpenter is operating as expected on this patch release. If you need to rollback after upgrading to v1, this is the version you will need to rollback to.

Note
The conversion webhooks must be able to communicate with the API server to operate correctly. If you see errors related to the conversion webhooks, ensure that your security groups and network policies allow traffic between the webhooks and the API server.
We’re now ready to begin the upgrade to v1. Set the KARPENTER_VERSION environment variable to the latest v1.0.x release.
```
export KARPENTER_VERSION="1.0.10"
```

Attach the v1 policy to your existing NodeRole. Notable Changes to the IAM Policy include additional tag-scoping for the eks:eks-cluster-name tag for instances and instance profiles. We will remove this additional policy later once the controller has been migrated to v1 and we’ve updated the Karpenter cloudformation stack.

POLICY_DOCUMENT=$(mktemp)
curl -fsSL https://raw.githubusercontent.com/aws/karpenter-provider-aws/13d6fc014ea59019b1c3b1953184efc41809df11/website/content/en/v1.0/upgrading/get-controller-policy.sh | sh | envsubst > ${POLICY_DOCUMENT}
POLICY_NAME="KarpenterControllerPolicy-${CLUSTER_NAME}-v1"
ROLE_NAME="${CLUSTER_NAME}-karpenter"
POLICY_ARN="$(aws iam create-policy --policy-name "${POLICY_NAME}" --policy-document "file://${POLICY_DOCUMENT}" | jq -r .Policy.Arn)"
aws iam attach-role-policy --role-name "${ROLE_NAME}" --policy-arn "${POLICY_ARN}"

Apply the v1 Custom Resource Definitions (CRDs):

helm upgrade --install karpenter-crd oci://public.ecr.aws/karpenter/karpenter-crd --version "${KARPENTER_VERSION}" --namespace "${KARPENTER_NAMESPACE}" --create-namespace \
 --set webhook.enabled=true \
 --set webhook.serviceName="karpenter" \
 --set webhook.port=8443

Upgrade Karpenter to the latest v1.0.x release.
```
# Service account annotion can be dropped when using pod identity
helm upgrade --install karpenter oci://public.ecr.aws/karpenter/karpenter --version ${KARPENTER_VERSION} --namespace "${KARPENTER_NAMESPACE}" --create-namespace \
 --set serviceAccount.annotations."eks\.amazonaws\.com/role-arn"=${KARPENTER_IAM_ROLE_ARN} \
 --set settings.clusterName=${CLUSTER_NAME} \
 --set settings.interruptionQueue=${CLUSTER_NAME} \
 --set controller.resources.requests.cpu=1 \
 --set controller.resources.requests.memory=1Gi \
 --set controller.resources.limits.cpu=1 \
 --set controller.resources.limits.memory=1Gi \
 --wait
```
Note

Karpenter has deprecated and moved a number of Helm values as part of the v1 release. Ensure that you upgrade to the newer version of these helm values during your migration to v1. You can find detail for all the settings that were moved in the v1 Upgrade Reference.
Note

Karpenter versions 0.32.x through 0.37.x have a number of mutating webhooks and validating webhooks that are not present in 1.0.x; if you are only using helm to generate manifests and are not using it to deploy them, you must clean up these webhook configurations by hand: specifically, you will need to remove the following resources:
- defaulting.webhook.karpenter.k8s.aws
- validation.webhook.karpenter.sh
- validation.webhook.config.karpenter.sh
- validation.webhook.karpenter.k8s.aws

Upgrade your cloudformation stack and remove the temporary v1 controller policy.

TEMPOUT=$(mktemp)
curl -fsSL https://raw.githubusercontent.com/aws/karpenter-provider-aws/v"${KARPENTER_VERSION}"/website/content/en/preview/getting-started/getting-started-with-karpenter/cloudformation.yaml > "${TEMPOUT}"
aws cloudformation deploy \
 --stack-name "Karpenter-${CLUSTER_NAME}" \
 --template-file "${TEMPOUT}" \
 --capabilities CAPABILITY_NAMED_IAM \
 --parameter-overrides "ClusterName=${CLUSTER_NAME}"

ROLE_NAME="${CLUSTER_NAME}-karpenter"
POLICY_NAME="KarpenterControllerPolicy-${CLUSTER_NAME}-v1"
POLICY_ARN=$(aws iam list-policies --query "Policies[?PolicyName=='${POLICY_NAME}'].Arn" --output text)
aws iam detach-role-policy --role-name "${ROLE_NAME}" --policy-arn "${POLICY_ARN}"
aws iam delete-policy --policy-arn "${POLICY_ARN}"

Downgrading

Once you upgrade to Karpenter v1.0.x, both v1 and v1beta1 resources may be stored in ETCD. Due to this, you may only rollback to a version of Karpenter with the conversion webhooks. The following releases should be used as rollback targets:

v0.37.7
v0.36.9
v0.35.12
v0.34.13
v0.33.12

Warning

When rolling back from v1, Karpenter will not retain data that was only valid in the v1 APIs. For instance, if you upgraded from v0.33.5 to v1.0.x, updated the NodePool.Spec.Disruption.Budgets field, and then rolled back to v0.33.12, Karpenter would not retain the NodePool.Spec.Disruption.Budgets field, as that was introduced in v0.34.0.

If you have configured the kubelet field on your EC2NodeClass and have removed the compatibility.karpenter.sh/v1beta1-kubelet-conversion annotation from your NodePools, you must re-add the annotation before downgrading. For more information, refer to kubelet configuration migration.

Note

Since both v1beta1 and v1 will be served, kubectl will default to returning the v1 version of your CRs. To interact with the v1beta1 version of your CRs, you’ll need to add the full resource path (including api version) into kubectl calls. For example: kubectl get nodepool.v1beta1.karpenter.sh.

Downgrade Procedure

Configure environment variables for the cluster you’re downgrading:

export AWS_PARTITION="aws" # if you are not using standard partitions, you may need to configure to aws-cn / aws-us-gov
export CLUSTER_NAME="${USER}-karpenter-demo"
export AWS_REGION="us-west-2"
export AWS_ACCOUNT_ID="$(aws sts get-caller-identity --query Account --output text)"
export KARPENTER_NAMESPACE=kube-system
export KARPENTER_IAM_ROLE_ARN="arn:${AWS_PARTITION}:iam::${AWS_ACCOUNT_ID}:role/${CLUSTER_NAME}-karpenter"

Configure your target Karpenter version. You should select one of the following versions:

0.37.7
0.36.9
0.35.12
v0.34.13
v0.33.12

# Note: v0.33.x and v0.34.x include the v prefix, omit it for versions v0.35+
export KARPENTER_VERSION="0.37.7" # Replace with your minor version

Attach the v1beta1 policy from your target version to your existing NodeRole.

POLICY_DOCUMENT=$(mktemp)
curl -fsSL https://raw.githubusercontent.com/aws/karpenter-provider-aws/website/docs/v1.0/upgrading/get-controller-policy.sh | sh | envsubst > ${POLICY_DOCUMENT}
POLICY_NAME="KarpenterControllerPolicy-${CLUSTER_NAME}-${KARPENTER_VERSION}"
ROLE_NAME="${CLUSTER_NAME}-karpenter"
POLICY_ARN="$(aws iam create-policy --policy-name "${POLICY_NAME}" --policy-document "file://${POLICY_DOCUMENT}" | jq -r .Policy.Arn)"
aws iam attach-role-policy --role-name "${ROLE_NAME}" --policy-arn "${POLICY_ARN}"

Rollback the Karpenter Controller: Note that webhooks must be enabled to rollback. Without enabling the webhooks, Karpenter will be unable to correctly operate on v1 versions of the resources already stored in ETCD.

# Service account annotation can be dropped when using pod identity
helm upgrade --install karpenter oci://public.ecr.aws/karpenter/karpenter --version ${KARPENTER_VERSION} --namespace "${KARPENTER_NAMESPACE}" --create-namespace \
 --set serviceAccount.annotations."eks\.amazonaws\.com/role-arn"=${KARPENTER_IAM_ROLE_ARN} \
 --set settings.clusterName=${CLUSTER_NAME} \
 --set settings.interruptionQueue=${CLUSTER_NAME} \
 --set controller.resources.requests.cpu=1 \
 --set controller.resources.requests.memory=1Gi \
 --set controller.resources.limits.cpu=1 \
 --set controller.resources.limits.memory=1Gi \
 --set webhook.enabled=true \
 --set webhook.port=8443 \
 --wait

Rollback the CRDs.

helm upgrade --install karpenter-crd oci://public.ecr.aws/karpenter/karpenter-crd --version "${KARPENTER_VERSION}" --namespace "${KARPENTER_NAMESPACE}" --create-namespace \
 --set webhook.enabled=true \
 --set webhook.serviceName=karpenter \
 --set webhook.port=8443

Rollback your cloudformation stack and remove the temporary v1beta1 controller policy.

TEMPOUT=$(mktemp)
VERSION_TAG=$([[ ${KARPENTER_VERSION} == v* ]] && echo "${KARPENTER_VERSION}" || echo "v${KARPENTER_VERSION}")
curl -fsSL https://raw.githubusercontent.com/aws/karpenter-provider-aws/${VERSION_TAG}/website/content/en/preview/getting-started/getting-started-with-karpenter/cloudformation.yaml > "${TEMPOUT}"
aws cloudformation deploy \
 --stack-name "Karpenter-${CLUSTER_NAME}" \
 --template-file "${TEMPOUT}" \
 --capabilities CAPABILITY_NAMED_IAM \
 --parameter-overrides "ClusterName=${CLUSTER_NAME}"

ROLE_NAME="${CLUSTER_NAME}-karpenter"
POLICY_NAME="KarpenterControllerPolicy-${CLUSTER_NAME}-${KARPENTER_VERSION}"
POLICY_ARN=$(aws iam list-policies --query "Policies[?PolicyName=='${POLICY_NAME}'].Arn" --output text)
aws iam detach-role-policy --role-name "${ROLE_NAME}" --policy-arn "${POLICY_ARN}"
aws iam delete-policy --policy-arn "${POLICY_ARN}"

Before Upgrading to `v1.1.0`

You’ve successfully upgraded to v1.0, but more than likely your manifests are still v1beta1. You can continue to apply these v1beta1 manifests on v1.0, but support will be dropped in v1.1. Before upgrading to v1.1+, you will need to migrate your manifests over to v1.

Manifest Migration

You can manually migrate your manifests by referring to the changelog and the updated API docs (NodePool, EC2NodeClass). Alternatively, you can take advantage of the conversion webhooks. Performing a get using kubectl will return the v1 version of the resource, even if it was applied with a v1beta1 manifest.

For example, applying the following v1beta1 manifest and performing a get will return the v1 equivalent:

cat <<EOF | kubectl apply -f -
apiVersion: karpenter.sh/v1beta1
kind: NodePool
metadata:
 name: default
spec:
 template:
 spec:
 requirements:
 - key: kubernetes.io/arch
 operator: In
 values: ["amd64"]
 - key: kubernetes.io/os
 operator: In
 values: ["linux"]
 - key: karpenter.sh/capacity-type
 operator: In
 values: ["on-demand"]
 - key: karpenter.k8s.aws/instance-category
 operator: In
 values: ["c", "m", "r"]
 - key: karpenter.k8s.aws/instance-generation
 operator: Gt
 values: ["2"]
 nodeClassRef:
 apiVersion: karpenter.k8s.aws/v1beta1
 kind: EC2NodeClass
 name: default
 limits:
 cpu: 1000
 disruption:
 consolidationPolicy: WhenUnderutilized
 expireAfter: 720h # 30 * 24h = 720h
EOF
kubectl get nodepools default -o yaml > v1-nodepool.yaml

Note

Due to the many-to-one relation between NodePools and EC2NodeClasses, the kubelet field is not automtatically migrated by the conversion webhooks. When updating your manifests, make sure you are migrating the kubelet field from your NodePools to your EC2NodeClasses. For more information, refer to kubelet configuration migration.

Kubelet Configuration Migration

One of the changes made to the NodePool and EC2NodeClass schemas for v1 was the migration of the kubelet field from the NodePool to the EC2NodeClass. This change is difficult to properly handle with conversion webhooks due to the many-to-one relation between NodePools and EC2NodeClasses. To facilitate this, Karpenter adds the compatibility.karpenter.sh/v1beta1-kubelet-conversion annotation to converted NodePools. If this annotation is present, it will take precedence over the kubelet field in the EC2NodeClass.

This annotation is only meant to support migration, and support will be dropped in v1.1. Before upgrading to v1.1+, you must migrate your kubelet configuration to your EC2NodeClasses, and remove the compatibility annotation from your NodePools.

Warning

Do not remove the compatibility annotation until you have updated your EC2NodeClass with the matching kubelet field. Once the annotations is removed, the EC2NodeClass will be used as the source of truth for your kubelet configuration. If the field doesn’t match, this will result in Nodes drifting.

If you need to rollback to a pre-v1.0 version after removing the compatibility annotation, you must re-add it before rolling back.

If you have multiple NodePools that refer to the same EC2NodeClass, but have varying kubelet configurations, you will need to create a separate EC2NodeClass for unique set of kubelet configurations.

For example, consider the following v1beta1 manifests:

apiVersion: karpenter.sh/v1beta1
kind: NodePool
metadata:
 name: nodepool-a
spec:
 template:
 spec:
 kubelet:
 maxPods: 10
 nodeClassRef:
 apiVersion: karpenter.k8s.aws/v1beta1
 kind: EC2NodeClass
 name: nodeclass
---
apiVersion: karpenter.sh/v1beta1
kind: NodePool
metadata:
 name: nodepool-b
spec:
 template:
 spec:
 kubelet:
 maxPods: 20
 nodeClassRef:
 apiVersion: karpenter.k8s.aws/v1beta1
 kind: EC2NodeClass
 name: nodeclass
---
apiVersion: karpenter.k8s.aws/v1beta1
kind: EC2NodeClass
metadata:
 name: nodeclass

In this example, we have two NodePools with different kubelet values, but they refer to the same EC2NodeClass. The conversion webhook will annotate the NodePools with the compatibility.karpenter.sh/v1beta1-kubelet-conversion annotation. This is the result of that conversion:

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
 name: nodepool-a
 annotations:
 compatibility.karpenter.sh/v1beta1-kubelet-conversion: "{\"maxPods\": 10}"
spec:
 template:
 spec:
 nodeClassRef:
 group: karpenter.k8s.aws
 kind: EC2NodeClass
 name: nodeclass
---
apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
 name: nodepool-b
 annotations:
 compatibility.karpenter.sh/v1beta1-kubelet-conversion: "{\"maxPods\": 20}"
spec:
 template:
 spec:
 nodeClassRef:
 group: karpenter.k8s.aws
 kind: EC2NodeClass
 name: nodeclass
---
apiVersion: karpenter.k8s.aws/v1
kind: EC2NodeClass
metadata:
 name: nodeclass

Before upgrading to v1.1, you must update your NodePools to refer to separate EC2NodeClasses to retain this behavior. Note that this will drift the Nodes associated with these NodePools due to the updated nodeClassRef.

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
 name: nodepool-a
spec:
 template:
 spec:
 nodeClassRef:
 group: karpenter.k8s.aws
 kind: EC2NodeClass
 name: nodeclass-a
---
apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
 name: nodepool-b
spec:
 template:
 spec:
 nodeClassRef:
 group: karpenter.k8s.aws
 kind: EC2NodeClass
 name: nodeclass-b
---
apiVersion: karpenter.k8s.aws/v1
kind: EC2NodeClass
metadata:
 name: nodeclass-a
spec:
 kubelet:
 maxPods: 10
---
apiVersion: karpenter.k8s.aws/v1
kind: EC2NodeClass
metadata:
 name: nodeclass-b
spec:
 kubelet:
 maxPods: 20

NodeClassRef Requirements

Starting with Karpenter v1.1.0, nodeClassRef.group and nodeClassRef.kind are strictly required on both NodePools and NodeClaims. Ensure these values are set for all resources before upgrading Karpenter. Failing to do so will result in Karpenter being unable to operate against those resources. For the AWS provider, the group will always be karpenter.k8s.aws and the kind will always be EC2NodeClass.

Stored Version Migration

Once you have upgraded all of your manifests, you need to ensure that all existing resources are stored as v1 in ETCD. Karpenter v1.0.6+ includes a controller to automatically migrate all stored resources to v1. To validate that the migration was successful, you should check the stored versions for Karpenter’s CRDs:

for crd in "nodepools.karpenter.sh" "nodeclaims.karpenter.sh" "ec2nodeclasses.karpenter.k8s.aws"; do
 kubectl get crd ${crd} -ojsonpath="{.status.storedVersions}{'\n'}"
done

For more details on this migration process, refer to the kubernetes docs.

Note

If the v1beta1 stored version persists, ensure that you are on Karpenter v1.0.6+. Additionally, ensure that the storage version on the CRD in question is set to v1.

kubectl get crd ${crd} -ojsonpath="{.spec.versions[?(.storage==true)].name}{'\n'}"

If it is not, this indicates an issue upgrading the CRD when upgrading Karpenter to v1.0.x. Revisit step 9 of the upgrade procedure and ensure the CRD was updated correctly.

Changelog

Features:
- AMI Selector Terms has a new Alias field which can only be set by itself in EC2NodeClass.Spec.AMISelectorTerms
- Disruption Budgets by Reason was added to NodePool.Spec.Disruption.Budgets
- TerminationGracePeriod was added to NodePool.Spec.Template.Spec.
- LOG_OUTPUT_PATHS and LOG_ERROR_OUTPUT_PATHS environment variables added
API Rename: NodePool’s ConsolidationPolicy WhenUnderutilized is now renamed to WhenEmptyOrUnderutilized
Behavior Changes:
- Expiration is now forceful and begins draining as soon as it’s expired. Karpenter does not wait for replacement capacity to be available before draining, but will start provisioning a replacement as soon as the node is expired and begins draining.
- Karpenter’s generated NodeConfig now takes precedence when generating UserData with the AL2023 amiFamily. If you’re setting any values managed by Karpenter in your AL2023 UserData, configure these through Karpenter natively (e.g. kubelet configuration fields).
- Karpenter now adds a karpenter.sh/unregistered:NoExecute taint to nodes in injected UserData when using alias in AMISelectorTerms or non-Custom AMIFamily. When using amiFamily: Custom, users will need to add this taint into their UserData, where Karpenter will automatically remove it when provisioning nodes.
- Discovered standard AL2023 AMIs will no longer be considered compatible with GPU / accelerator workloads. If you’re using an AL2023 EC2NodeClass (without AMISelectorTerms) for these workloads, you will need to select your AMI via AMISelectorTerms (non-alias).
- Karpenter now waits for underlying instances to be completely terminated before removing the associated nodes. This means it may take longer for nodes to be deleted and for nodeclaims to get cleaned up.
- NodePools now have status conditions that indicate if they are ready. If not, then they will not be considered during scheduling.
- NodeClasses now have status conditions that indicate if they are ready. If they are not ready, NodePools that reference them through their nodeClassRef will not be considered during scheduling.
- Karpenter will no longer set associatePublicIPAddress to false in private subnets by default. Users with IAM policies / SCPs that require this field to be set explicitly should configure this through their EC2NodeClass (ref).
API Moves:
- ExpireAfter has moved from the NodePool.Spec.Disruption block to NodePool.Spec.Template.Spec, and is now a drift-able field.
- Kubelet was moved to the EC2NodeClass from the NodePool.
RBAC changes: added delete pods | added get, patch crds | added update nodes | removed create nodes
Breaking API (Manual Migration Needed):
- Ubuntu is dropped as a first class supported AMI Family
- karpenter.sh/do-not-consolidate (annotation), karpenter.sh/do-not-evict (annotation), and karpenter.sh/managed-by (tag) are all removed. karpenter.sh/managed-by, which currently stores the cluster name in its value, will be replaced by eks:eks-cluster-name. karpenter.sh/do-not-consolidate and karpenter.sh/do-not-evict are both replaced by karpenter.sh/do-not-disrupt.
- The taint used to mark nodes for disruption and termination changed from karpenter.sh/disruption=disrupting:NoSchedule to karpenter.sh/disrupted:NoSchedule. It is not recommended to tolerate this taint, however, if you were tolerating it in your applications, you’ll need to adjust your taints to reflect this.
Environment Variable Changes:
- Environment Variable Changes
- LOGGING_CONFIG, ASSUME_ROLE_ARN, ASSUME_ROLE_DURATION` Dropped
- LEADER_ELECT renamed to DISABLE_LEADER_ELECTION
- FEATURE_GATES.DRIFT=true was dropped and promoted to Stable, and cannot be disabled.
  - Users currently opting out of drift, disabling the drift feature flag will no longer be able to do so.
Defaults changed:
- API: Karpenter will drop support for IMDS access from containers by default on new EC2NodeClasses by updating the default of httpPutResponseHopLimit from 2 to 1.
- API: ConsolidateAfter is required. Users couldn’t set this before with ConsolidationPolicy: WhenUnderutilized, where this is now required. Users can set it to 0 to have the same behavior as in v1beta1.
- API: All NodeClassRef fields are now all required, and apiVersion has been renamed to group
- API: AMISelectorTerms are required. Setting an Alias cannot be done with any other type of term, and must match the AMI Family that’s set or be Custom.
- Helm: Deployment spec TopologySpreadConstraint to have required zonal spread over preferred. Users who had one node running their Karpenter deployments need to either:
  - Have two nodes in different zones to ensure both Karpenter replicas schedule
  - Scale down their Karpenter replicas from 2 to 1 in the helm chart
  - Edit and relax the topology spread constraint in their helm chart from DoNotSchedule to ScheduleAnyway
- Helm/Binary: controller.METRICS_PORT default changed back to 8080

Updated metrics

The following changes have been made to Karpenter’s metrics in v1.0.0.

Renamed Metrics

Type	Original Name	New Name
Node	karpenter_nodes_termination_time_seconds	karpenter_nodes_termination_duration_seconds
Node	karpenter_nodes_terminated	karpenter_nodes_terminated_total
Node	karpenter_nodes_leases_deleted	karpenter_nodes_leases_deleted_total
Node	karpenter_nodes_created	karpenter_nodes_created_total
Pod	karpenter_pods_startup_time_seconds	karpenter_pods_startup_duration_seconds
Disruption	karpenter_disruption_replacement_nodeclaim_failures_total	karpenter_voluntary_disruption_queue_failures_total
Disruption	karpenter_disruption_evaluation_duration_seconds	karpenter_voluntary_disruption_decision_evaluation_duration_seconds
Disruption	karpenter_disruption_eligible_nodes	karpenter_voluntary_disruption_eligible_nodes
Disruption	karpenter_disruption_consolidation_timeouts_total	karpenter_voluntary_disruption_consolidation_timeouts_total
Disruption	karpenter_disruption_budgets_allowed_disruptions	karpenter_nodepools_allowed_disruptions
Disruption	karpenter_disruption_actions_performed_total	karpenter_voluntary_disruption_decisions_total
Provisioner	karpenter_provisioner_scheduling_simulation_duration_seconds	karpenter_scheduler_scheduling_duration_seconds
Provisioner	karpenter_provisioner_scheduling_queue_depth	karpenter_scheduler_queue_depth
Interruption	karpenter_interruption_received_messages	karpenter_interruption_received_messages_total
Interruption	karpenter_interruption_deleted_messages	karpenter_interruption_deleted_messages_total
Interruption	karpenter_interruption_message_latency_time_seconds	karpenter_interruption_message_queue_duration_seconds
NodePool	karpenter_nodepool_usage	karpenter_nodepools_usage
NodePool	karpenter_nodepool_limit	karpenter_nodepools_limit
NodeClaim	karpenter_nodeclaims_terminated	karpenter_nodeclaims_terminated_total
NodeClaim	karpenter_nodeclaims_disrupted	karpenter_nodeclaims_disrupted_total
NodeClaim	karpenter_nodeclaims_created	karpenter_nodeclaims_created_total

Dropped Metrics

Type	Name
Disruption	karpenter_disruption_replacement_nodeclaim_initialized_seconds
Disruption	karpenter_disruption_queue_depth
Disruption	karpenter_disruption_pods_disrupted_total
	karpenter_consistency_errors
NodeClaim	karpenter_nodeclaims_registered
NodeClaim	karpenter_nodeclaims_launched
NodeClaim	karpenter_nodeclaims_initialized
NodeClaim	karpenter_nodeclaims_drifted
Provisioner	karpenter_provisioner_scheduling_duration_seconds
Interruption	karpenter_interruption_actions_performed

Note

Karpenter now waits for the underlying instance to be completely terminated before deleting a node and orchestrates this by emitting NodeClaimNotFoundError. With this change we expect to see an increase in the NodeClaimNotFoundError. Customers can filter out this error by label in order to get accurate values for karpenter_cloudprovider_errors_total metric. Use this Prometheus filter expression - ({controller!="node.termination"} or {controller!="nodeclaim.termination"}) and {error!="NodeClaimNotFoundError"}.

Karpenter – Upgrading

V1.0: Upgrade Guide

Warning

Pre-upgrade Validation

Staging Environment Setup

Staging Deployment

Production Approval and Deployment

Post-Deployment

CRD Upgrades

Note

Upgrading to 1.0.0+

Warning

Upgrading to 0.37.0+

Warning

Note

Upgrading to 0.36.0+

Warning

Warning

Note

Upgrading to 0.35.0+

Warning

Note

Upgrading to 0.34.0+

Warning

Warning

Note

Upgrading to 0.33.0+

Warning

Note

Upgrading to 0.32.0+

Warning

Upgrading to 0.31.0+

Upgrading to 0.30.0+

Note

Upgrading to 0.29.0+

Warning

Upgrading to 0.28.0+

Warning

Warning

Rolling Back

Upgrading to 0.27.3+

Upgrading to 0.27.0+

Upgrading to 0.26.0+

Upgrading to 0.25.0+

Upgrading to 0.24.0+

Upgrading to 0.22.0+

Upgrading to 0.20.0+

Upgrading to 0.19.0+

Upgrading to 0.18.0+

Upgrading to 0.17.0+

Upgrading to 0.16.2+

Upgrading to 0.16.0+

Upgrading to 0.15.0+

Upgrading to 0.14.0+

Upgrading to 0.13.0+

Upgrading to 0.12.0+

Upgrading to 0.11.0+

Upgrading to 0.10.0+

Upgrading to 0.6.2+

V1.0: Compatibility

Compatibility

Compatibility Matrix

Note

Note

Compatibility issues

How Do We Break Incompatibility?

How Do We Find Incompatibilities?

Security Patches

Release Types

Stable Releases

Release Candidates

Snapshot Releases

Note

V1.0: v1 Migration

Before You Start

Forceful Expiration

Deprecated Annotations, Labels, and Tags Removed

Zap Logging Config Removed

New MetadataOptions Defaults

Ubuntu AMIFamily Removed

Upgrading to `1.0.0`+

Upgrading to `0.37.0`+

Upgrading to `0.36.0`+

Upgrading to `0.35.0`+

Upgrading to `0.34.0`+

Upgrading to `0.33.0`+

Upgrading to `0.32.0`+

Upgrading to `0.31.0`+

Upgrading to `0.30.0`+

Upgrading to `0.29.0`+

Upgrading to `0.28.0`+

Upgrading to `0.27.3`+

Upgrading to `0.27.0`+

Upgrading to `0.26.0`+

Upgrading to `0.25.0`+

Upgrading to `0.24.0`+

Upgrading to `0.22.0`+

Upgrading to `0.20.0`+

Upgrading to `0.19.0`+

Upgrading to `0.18.0`+

Upgrading to `0.17.0`+

Upgrading to `0.16.2`+

Upgrading to `0.16.0`+

Upgrading to `0.15.0`+

Upgrading to `0.14.0`+

Upgrading to `0.13.0`+

Upgrading to `0.12.0`+

Upgrading to `0.11.0`+

Upgrading to `0.10.0`+

Upgrading to `0.6.2`+

Before Upgrading to `v1.1.0`