Karpenter – Contributing

V1.0: Community Meetings

Mon, 01 Jan 0001 00:00:00 +0000

Karpenter hosts two types of regular community meetings to foster collaboration and maintain project health:

The Working Group meetings focus on project direction, technical discussions, and feature development. These bi-weekly sessions bring together contributors, users, and maintainers to review designs, plan releases, and address architectural decisions that shape Karpenter’s future.

The Issue Triage meetings, held weekly, are dedicated to maintaining project health through focused issue management across our repositories. These sessions ensure proper prioritization, assignment coordination, and timely response to community contributions.

Meeting times alternate to accommodate our global community. All interested participants are welcome to join, contribute, and learn more about Karpenter.

Working Group Meetings

The Karpenter Working Group meetings serve as a collaborative forum for project development, technical discussions, and community engagement. These sessions bring together contributors, users, and maintainers to shape the project’s direction and ensure its continued success. Through these meetings, we align on architectural decisions, review critical features, and address community needs.

Key focus areas include:

Review and discussion of design proposals and technical implementations
Planning of upcoming releases and feature roadmaps
Coordination between contributors and cross-team collaboration
Technical deep-dives on specific components and challenges
Discussion of user feedback and community requirements

The working group provides a platform for both synchronous decision-making and asynchronous follow-ups, ensuring the project maintains its technical excellence while growing its community impact.

Working group meetings will be held every other Thursday, alternating between a 3:00 PM PST (Americas-friendly) and 9:00 AM PST (Europe-friendly) start time. Please refer to our calendar for an up-to-date schedule.

Issue Triage Meetings

Karpenter is expanding our community meetings to include dedicated Issue Triage sessions alongside our regular Working Group meetings. This addition aims to improve issue management across both kubernetes-sigs/karpenter and karpenter-provider-aws repositories.

These weekly meetings rotate between repositories and time zones to accommodate our global community. These dedicated triage sessions focus on:

Issue prioritization
Assignment coordination
Discussion of complex issues
Maintaining repository health

An issue triage meeting for each repository will be held monthly, alternating between a 3:00 PM PST (Americas-friendly) and 9:00 AM PST (Europe-friendly) start time. Please refer to our calendar for an up-to-date schedule.

Getting Involved

All community members are welcome to join these meetings. Meeting links and calendar invites are shared through our usual communication channels.

All invites are managed through our Calendar.
Alternatively, you can use our iCal Export to add the events to Outlook or other email providers.

Meeting Access

Join our meetings using the following credentials:

🔗 Zoom: https://zoom.us/j/95618088729 🆔 Meeting ID: 956 1808 8729 🔑 Passcode: 77777

Need help? Contact the maintainers on Slack.

Add future questions or read past discussions in our Working Group Log.

V1.0: Design Guide

Mon, 01 Jan 0001 00:00:00 +0000

Technical designs are essential to building robust, intuitive, and performant products that delight users. Writing a design can accelerate decision making and avoid wasting time on an implementation that never lands. But what makes a good design? These guidelines were authored with the Karpenter community in mind, but apply broadly to the development of Kubernetes Operators.

Designs don’t have to be long or formal, and should match the scope of the problem they’re trying to solve.

Are there multiple potential solutions?
Will users need to be aware of the changes?
Would it be painful to discard a rejected implementation?
When in doubt, write a 1 pager.

Tell a Story

A design is a story that connects a user need with a technical direction that solves the need. Designs come in all shapes and sizes, and this document intentionally avoids prescribing a one-size-fits-all template. There’s no substitute for an author thinking deeply about a problem space, and mapping that to a clear story that walks readers through the ideas and helps them reason about a solution space. Keep readers engaged with concise language and make every word count.

Your story should include,

[Context] Include some technical background that helps readers think about your idea in context
[Problem] Clearly identify the problem to be solved and some guiding principles to help think about the solutions
[Solutions] Talk through different potential solutions and their tradeoffs. Include diagrams to clarify concepts
[Recommendation] Make a recommendation, but don’t be overly invested in it

The best way to improve your story telling skills is to write and review designs. Seek inspiration from recent designs in the project as well as from other domains. Focus on your audience and continuously reread and refine your design with their perspective in mind.

Gather Broad Feedback

The bigger the change, the more likely your design will have broader implications than intended. Be vocal about design ideas as they’re explored and run them by engineering leaders in relevant systems. Surface your design ideas at the Karpenter working group, or asynchronously on the Kubernetes Slack channel for Karpenter.

The Kubernetes community is also a valuable source of feedback from both users and Kubernetes developers. Does your design touch scoped owned by any Kubernetes SIGs? Consider discussing the design ideas at the SIG or in their slack channel. Socializing high level ideas before the review gives your audience more time to think about possible interactions with existing and future systems.

It can be tempting to rush to solutions that unblock user adoption or ease user pain, but the wrong solution can have a greater negative impact on users than it solves. It’s impossible to know all future use cases and how your design choices may impact them, but the more thorough your investigation, the more likely your solution is to deliver long term value.

Simple Solutions to Complex Problems

The best solutions are invisible to users and “Just Work™”. It’s easy to forget that users have business problems to focus on and each parameter and behavior your design introduces increases user cognitive load. Pragmatically, it’s not always possible to meet the broad requirements of Kubernetes without providing options, but solution spaces typically include a spectrum of configuration complexity. Recognize that a solution for one user segment may be directly at odds with another or create long term technical debt for the project. Often, requirements only exist to workaround bugs or missing features in related systems. Deep dive requirements until you’re certain they’re necessary and ensure each bit of complexity justifies its existence.

Common Gotchas

Does your change introduce new APIs?

APIs are notoriously hard to get right and even harder to change. Kubernetes defines an api deprecation policy that helps systems make backwards incompatible changes to APIs before graduating to a stable API with compatibility guarantees. Once an API is stable, features are typically via feature gates, which allows for experimentation and deprecation.

Think about how your API changes impact existing parameters and their deprecation policies. Consider how the user interacts with the product as a whole and if the feature supersedes or overlaps with existing concepts. Weigh the costs of deprecating existing features to the benefit of simplifying the product for all future users. The answer will change depending on the maturity of the product and breadth of adoption.

Build minimal and maintainable APIs by:

Push back on requirements that introduces concepts for all users to solve problems for a few.
Identify an opinionated default that solves the majority of use cases.
Delay introducing a parameter into your API surface until users demand it; you can always add it later.
Rely on existing concepts and idioms from the Kubernetes ecosystem. Look to Kubernetes APIs and projects like Tekton, Knative, and ACK and find concepts that will be familiar to users.
Take advantage of opportunities to refine APIs while the impact of backwards incompatibility is small

Does your change behave differently with different cloud providers?

Kubernetes is an open standard that users rely on to work across vendors. Users care deeply about this, as it minimizes the technical complexity to operate in different environments. Identify whether or not your feature varies across cloud providers or are bespoke to a specific provider. For some features, it’s possible to rely on existing vendor neutral abstractions. For others, it’s possible to define a neutral abstraction that cloud providers can implement.

Achieving consensus for new neutral concepts is hard. Often, the best path is to demonstrate value on a single vendor, and work to achieve neutrality as a followup effort. Be cautious about introducing or changing vendor neutral interfaces, as it will require changes from all providers. Similarly, invest heavily in getting these interfaces right in the early stages. As projects mature, these interfaces are rarely changed.

Does your change expose details users may rely on?

Kubernetes based systems often use a layered architectural pattern that exposes underlying layers of abstraction. This approach enables broad extensibility and allows other systems to integrate at multiple layers of the stack. For example, Karpenter creates EC2 instances in your AWS account. This enables you to view logs or react to their creation with other automation without requiring any features from Karpenter. However, Karpenter also applies specific EC2 tags to the EC2 instances. Are the tags an implementation detail or an interface? What can you change without breaking compatibility?

Be intentional and explicit about the interface and implementation of your design and ensure that this is communicated to users. If implementation details are exposed through other APIs, expect users to rely on them as an interface unless told otherwise. In general, aim to minimize the project’s interface to maximize future flexibility.

Does your change have a risk of breaking an undocumented invariant?

Systems often contain mechanisms that are implicitly assumed as invariant, but may not be obvious, especially over time. Existing mechanisms may not be extensible enough to support your design, and may require them to be rewritten as part of the design scope. Be aware that regression tests never have complete coverage and well intentioned engineers thought carefully about how things were done before your requirements.

Identify the fundamental reason the existing mechanism is insufficient and be able to explain it in plain terms.
Separate the new mechanism from the new feature that relies on it.
Clean up after yourself and avoid getting stuck halfway between old and new mechanisms.

Does your change impact performance?

Users have high expectations for performance on Kubernetes. Karpenter is especially sensitive, as it has the potential to impact application availability during traffic spikes. Think about how your solution scales, and look for opportunities to improve performance at the design level. Often, good designs don’t require trading-off a great UX for performance. Make it work, make it fast, make it pretty.

Beware code that scales linearly with pods or nodes. Milliseconds in testing turn into seconds at scale.
Cloud provider read APIs can have surprisingly high latency and low limits, use caching to minimize calls.
Increases to memory and CPU usage increase capex cost for operators. Profile and optimize your implementations.

V1.0: Documentation Updates

Mon, 01 Jan 0001 00:00:00 +0000

Documentation for https://karpenter.sh/docs/ is built under website/content/en/preview/.
Documentation updates should be made to the “preview” directory. Your changes will be promoted to website/content/en/docs/ by an automated process after the change has been merged.
Previews for your changes are built and available a few minutes after you push. Look for the “netlify Deploy Preview” link in a comment in your PR.

V1.0: Development Guide

Mon, 01 Jan 0001 00:00:00 +0000

Dependencies

The following tools are required for contributing to the Karpenter project.

Package	Version	Install
go	v1.19+	Instructions
kubectl		`brew install kubectl`
helm		`brew install helm`
Other tools		`make toolchain`

Developing

Setup / Teardown

Based on how you are running your Kubernetes cluster, follow the Environment specific setup to configure your environment before you continue. You can choose to either run the Karpenter controller locally on your machine, pointing to the Kubernetes cluster specified in your ~/.kube/config or inside the Kubernetes cluster specified in your ~/.kube/config deployed with Helm.

Locally

Once you have your environment set up, run the following commands to run the Karpenter Go binary against the Kubernetes cluster specified in your ~/.kube/config

make run

Inside a Kubernetes Cluster

Once you have your environment set up, to install Karpenter in the Kubernetes cluster specified in your ~/.kube/config run the following commands.

make apply # Install Karpenter
make delete # Uninstall Karpenter

Developer Loop

Make sure dependencies are installed
- Run make codegen to make sure yaml manifests are generated (requires a working set of AWS credentials, see Specifying Credentials)
- Run make toolchain to install cli tools for building and testing the project
You will need a personal development image repository (e.g. ECR)
- Make sure you have valid credentials to your development repository.
- $KO_DOCKER_REPO must point to your development repository
- Your cluster must have permissions to read from the repository

Build and Deploy

Note: these commands do not rely on each other and may be executed independently

make apply # quickly deploy changes to your cluster
make presubmit # run codegen, lint, and tests

If you are only interested in building the Karpenter images and not deploying the updated release to your cluster immediately with Helm, you can run the following:

make image # build and push the karpenter images

*Note: that this will produce a build with the version of https://github.com/kubernetes-sigs/karpenter in your local filesystem.

You can test out changes made in https://github.com/kubernetes-sigs/karpenter by replacing the dependency of https://github.com/aws/karpenter-provider-aws/. For local changes, replace $PATH_TO_KUBERNETES_SIGS_KARPENTER with the relative or absolute path and run:

go mod edit -replace sigs.k8s.io/karpenter=$PATH_TO_KUBERNETES_SIGS_KARPENTER

Then you can build your image using the previous steps.

Publishing Images Only

If you only need to build and publish an image to a container registry, run the following:

make image # build and push the karpenter images

go mod edit -replace sigs.k8s.io/karpenter=$PATH_TO_KUBERNETES_SIGS_KARPENTER

Then you can build your image using the previous steps.

*Note: you need to commit the go.mod changes before running make image for the changes to be picked up.

Testing

make test # E2E correctness tests

Change Log Level

By default, make apply will set the log level to debug. You can change the log level by setting the log level in your Helm values.

--set logLevel=debug

Debugging Metrics

OSX:

open http://localhost:8080/metrics && kubectl port-forward service/karpenter -n kube-system 8080

Linux:

gio open http://localhost:8080/metrics && kubectl port-forward service/karpenter -n karpenter 8080

Tailing Logs

While you can tail Karpenter’s logs with kubectl, there’s a number of tools out there that enhance the experience. We recommend Stern:

stern -n karpenter -l app.kubernetes.io/name=karpenter

Environment specific setup

AWS

For local development on Karpenter you will need a Docker repo which can manage your images for Karpenter components. You can use the following command to provision an ECR repository. We recommend using a single “dev” repository for development across multiple projects, and to use specific image hashes instead of image tags.

aws ecr create-repository \
 --repository-name dev \
 --image-scanning-configuration scanOnPush=true \
 --region "${AWS_DEFAULT_REGION}"

Once you have your ECR repository provisioned, configure your Docker daemon to authenticate with your newly created repository.

export KO_DOCKER_REPO="${AWS_ACCOUNT_ID}.dkr.ecr.${AWS_DEFAULT_REGION}.amazonaws.com/dev"
aws ecr get-login-password --region "${AWS_DEFAULT_REGION}" | docker login --username AWS --password-stdin "${KO_DOCKER_REPO}"

Profiling

Karpenter exposes a pprof endpoint on its metrics port when profiling is enabled.

Learn about profiling with pprof: https://jvns.ca/blog/2017/09/24/profiling-go-with-pprof/

Prerequisites

brew install graphviz
go install github.com/google/pprof@latest

Get a profile

# Connect to the metrics endpoint
kubectl port-forward service/karpenter -n karpenter 8080
open http://localhost:8080/debug/pprof/
# Visualize the memory
go tool pprof -http 0.0.0.0:9000 localhost:8080/debug/pprof/heap
# Visualize CPU
go tool pprof -http 0.0.0.0:9000 "localhost:8080/debug/pprof/profile?seconds=60"