How to Configure Pod-Level Resource Managers in Kubernetes v1.36

Introduction

Kubernetes v1.36 introduces Pod-Level Resource Managers as an alpha feature, giving you a more flexible and powerful resource management model for performance-sensitive workloads. This enhancement extends the kubelet's Topology, CPU, and Memory Managers to support pod-level resource specifications (.spec.resources), evolving them from a strictly per-container allocation model to a pod-centric one.

How to Configure Pod-Level Resource Managers in Kubernetes v1.36

Before this feature, to achieve NUMA-aligned exclusive resources for your main application container, you had to allocate exclusive, integer-based CPU resources to every container in the pod — even lightweight sidecars. This wasted resources or forced you to sacrifice Guaranteed QoS class. Now, with pod-level resource managers, you can create hybrid allocation models that bring flexibility and efficiency without losing NUMA alignment.

What You Need

Kubernetes cluster running v1.36 or later (alpha features require explicit enablement)
kubelet configured with the following feature gates:
- PodLevelResourceManagers (alpha)
- PodLevelResources (alpha)
Topology Manager enabled and set to a scope that supports pod-level alignment (e.g., pod scope)
Performance-sensitive workloads that benefit from NUMA alignment, such as ML training, low-latency databases, or high-frequency trading apps
Sidecar or auxiliary containers that share resources without needing dedicated exclusive cores

Step-by-Step Guide

Step 1: Enable the Required Feature Gates

The first step is to activate the alpha features on your kubelet. Edit the kubelet configuration file (usually /var/lib/kubelet/config.yaml) or pass them as command-line arguments. Add the following:

featureGates:
  PodLevelResourceManagers: true
  PodLevelResources: true

If you use kubeadm, you can set these in the kubeletExtraArgs of the kubeadm configuration file. After updating, restart the kubelet on every node where you plan to run pod-level resource pods.

Step 2: Configure the Topology Manager Scope

Pod-level resource managers work closely with the Topology Manager. To perform NUMA alignment at the pod level, set the Topology Manager scope to pod. This is done in the kubelet configuration:

topologyManagerScope: pod

Alternatively, you can use container scope if you prefer per-container alignment, but the full benefits of pod-level resources come with pod scope. Ensure the Topology Manager policy is set to one of single-numa-node, restricted, or best-effort based on your strictness requirements.

Step 3: Define Pod-Level Resources in Your Pod Spec

Now that the cluster is ready, create a pod definition that uses spec.resources to declare the total pod budget. This replaces or augments per-container resource declarations. Below is an example for a tightly-coupled database pod with a main container, a metrics exporter sidecar, and a backup agent sidecar.

apiVersion: v1
kind: Pod
metadata:
  name: tightly-coupled-database
spec:
  # Pod-level resources establish the overall budget and NUMA alignment size.
  resources:
    requests:
      cpu: "8"
      memory: "16Gi"
    limits:
      cpu: "8"
      memory: "16Gi"
  containers:
  - name: database
    image: db:latest
    # This container gets exclusive resources from the pod budget.
  - name: metrics-exporter
    image: metrics-exporter:v1
    # This sidecar shares resources from the pod shared pool.
  - name: backup-agent
    image: backup-agent:v1
    # This sidecar also runs in the shared pool.

Notice that only the pod-level resources are defined. The kubelet will automatically allocate exclusive CPU and memory from the NUMA-aligned budget for the database container. The remaining resources form a pod shared pool that the metrics-export and backup-agent containers share among themselves, isolated from the database’s exclusive slices.

Step 4: Verify Resource Allocation

After the pod is running, you can inspect the actual resource allocation using kubectl describe pod or by checking the kubelet logs for topology decisions. Look for annotations or events indicating that the CPU and memory managers used pod-level scoping. For example:

kubectl describe pod tightly-coupled-database

Examine the Allocated Resources section. You should see that the database container got full exclusive cores from one NUMA node, while the sidecars have shared CPU/memory from the same node but not dedicated cores.

Step 5: Test with Real Workloads

Deploy your performance-critical workload with the configuration above. Monitor latency and throughput. Because both the main container and sidecars reside on the same NUMA node, inter-container communication remains local, reducing memory access latency. The exclusive resources for the main container guarantee predictable performance, while the shared pool eliminates waste for sidecars.

You can experiment with different Topology Manager policies and observe the impact on pod scheduling and resource allocation. Remember that this feature is alpha; test thoroughly in a non-production environment.

Tips for Success

Start with a single pod: Before rolling out to many workloads, test with one pod to understand the allocation behavior.
Monitor kubelet logs: Enable verbose logging for the resource managers to see how pod-level budgets are split. Look for logs containing PodLevelResources or pod shared pool.
Avoid mixing scopes: If you use pod-level resources, avoid also defining per-container resource requests/limits in the same pod unless you fully understand the hierarchy. The pod-level specification takes precedence for the overall budget.
Use pod-level resources only for pods with multiple containers: For single-container pods, the traditional per-container approach works fine and is simpler.
Check NUMA topology: Use tools like numactl --hardware on your nodes to verify that your hardware supports NUMA and that cores and memory are properly partitioned.
Watch for beta changes: As the feature moves to beta and stable, the configuration syntax and defaults may change. Keep an eye on the Kubernetes changelog.
Consider admission controllers: For production, you may want to enforce pod-level resource policies via a mutating admission webhook to ensure consistent usage.

Tags: