Skip to main content
  1. Tech Blog: AI, Security, Infrastructure & Open Source/

Kubernetes 1.37 Garhwal — Scale-to-Zero and Pod Identity

·1262 words·6 mins
Osmond van Hemert
Author
Osmond van Hemert
Kubernetes & Containers - This article is part of a series.
Part : This Article

Kubernetes 1.37 landed on August 26, and it’s named Garhwal, after the Himalayan region of Uttarakhand. Sixty-seven enhancements shipped: 16 graduating to stable, 23 to beta, 27 entering alpha, plus one formal deprecation. Most releases have one or two headline features and a long tail of housekeeping. This one has four changes worth stopping for, and they point in the same direction: Kubernetes is optimizing for workloads that don’t look like a stateless web service anymore — queue consumers that should cost nothing when idle, distributed training jobs that need every pod to start together, and services that need to prove their identity without handing over a bearer token.

HPA Can Finally Scale to Zero
#

The HorizontalPodAutoscaler has been able to scale a Deployment down to a single replica since Kubernetes 1.1. Getting it to zero has required an add-on, an external component, or an alpha feature gate — until now. KEP-2021 graduates to beta and ships enabled by default in 1.37, and the API server will accept minReplicas: 0 on any HPA backed by an object or external metric.

The reason it took this long is a real constraint, not bureaucracy: the HPA usually scales on CPU or memory, and both come from running pods. Once you’re at zero, there’s nothing left to measure. Object and external metrics don’t have that problem — a queue length exists whether or not a worker is running to drain it:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: queue-worker
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: queue-worker
  minReplicas: 0
  maxReplicas: 10
  metrics:
    - type: External
      external:
        metric:
          name: queue_consumer_lag
          selector:
            matchLabels:
              name: worker_tasks
        target:
          type: Value
          value: "30"

The subtlety worth knowing before you flip this on: a replica count of zero is ambiguous. It could mean the HPA scaled down, or an operator manually paused the workload. Kubernetes resolves it with a ScaledToZero status condition — the controller only wakes a workload it put to sleep itself, so a manual kubectl scale --replicas=0 still behaves like a pause. Google’s Dipesh Rawat, this release’s lead, put the value plainly: if you’re running high-processing workloads on GPUs, this is a direct cost lever, not a nice-to-have. That’s the same argument cloud FinOps engineers have been making about idle capacity for years — Kubernetes just gave them a built-in switch instead of a third-party controller.

Pod Certificates Go GA — a Second Identity Primitive
#

Kubernetes has had exactly one built-in way for a workload to prove who it is: the service account JWT, written to the pod’s filesystem and kept fresh by the kubelet. It works well, but it’s a bearer token — whoever holds the JWT is the identity it names, full stop. Kubernetes 1.37 adds a second, structurally different option: Pod Certificates (KEP-4317) and ClusterTrustBundles (KEP-3257), both stable.

X.509 certificates are proof-of-possession credentials — the private key never leaves the pod, and what gets shared is a certificate plus a signature, not the secret itself. Under the hood, the kubelet requests a PodCertificateRequest on the pod’s behalf, a signer controller issues it, and the kubelet writes the private key and certificate chain to the container filesystem, refreshing them automatically before they expire (a maximum 24 hours for any signer Kubernetes ships itself, up to 91 days for others). Applications pick up rotation via inotify or polling on a single credential-bundle file — the same low-friction pattern that made service account tokens easy to adopt in the first place.

This is a genuinely additive primitive, not a JWT replacement: Kubernetes now supports a pluggable signer interface, so a cluster can run a SPIFFE-flavored client-cert signer alongside a server-TLS signer for mTLS between services, without picking one identity model for the whole cluster. It’s the kind of unglamorous plumbing that container security hardening work depends on — the incidents worth worrying about are rarely the ones a bearer token was designed to stop.

Scheduling Learns What a Workload Actually Is
#

The most structurally interesting change in 1.37 isn’t a single KEP — it’s a pattern across several. Kubernetes has scheduled pods one at a time since day one, with no native concept of “these twelve pods are one job.” That’s fine for a stateless service; it’s a liability for distributed training, where a job that gets nine of ten required pods scheduled is worse than useless — it’s nine GPUs burning money while waiting for a tenth that may never come.

Gang scheduling (KEP-4671) gives Kubernetes native all-or-nothing placement for a PodGroup. Workload-aware preemption (KEP-5710), also beta in 1.37, extends the same idea to eviction: the scheduler weighs an entire PodGroup against a lower-priority one, instead of preempting individual pods and leaving a job half-scheduled. And DRA device taints and tolerations (KEP-5055) reach stable, letting a degraded GPU be marked off-limits using the same taint model that already isolates a bad node.

None of this is abstract. Red Hat’s Sascha Grunert called gang scheduling out specifically as the fix for distributed training jobs that need to start together or not at all. If you’re running anything adjacent to LLM agents or training pipelines in production, this is the release where Kubernetes stopped treating “AI workload” as a workaround and started building for it directly.

The Migrations You Can’t Skip
#

Two smaller stable graduations round out a quieter but real win: Storage Version Migration (KEP-4192) is GA and on by default, replacing years of manual kubectl get/kubectl replace scripts with a declarative StorageVersionMigration object that rewrites stale resources — the thing you actually need when rotating encryption-at-rest keys or dropping an old CRD version. And etcd RangeStream, paired with etcd 3.7, graduates to beta, streaming large list reads in bounded chunks instead of assembling a whole page in memory first — a direct fix for the OOM risk that shows up on clusters with many large objects.

None of it is optional to think about, though. cgroup v1 nodes will refuse to start the kubelet unless you explicitly opt out — a default that’s been coming since 1.35 and now bites for real. kube-proxy’s ipvs mode is formally deprecated, with removal scheduled for 1.43, as the ecosystem keeps moving toward nftables. And static pods can no longer reference Secrets or ConfigMaps through secretRef/configMapRef — if your node bootstrap tooling depended on that, it breaks on upgrade, not gracefully.

My Take
#

The theme across Garhwal is the same one running through Kubernetes’ broader maturity arc: the easy, general-purpose problems got solved years ago, so each release now goes after a narrower, gnarlier one. Scale-to-zero only works because someone was willing to build a status condition just to disambiguate “scaled down” from “paused.” Pod Certificates only work because someone built a pluggable signer interface instead of shipping one opinionated cert format. That’s a lot of design effort spent on cases that affect a minority of workloads at any given company — and it’s exactly the right amount, because in aggregate, “queue workers on GPUs” and “distributed training jobs” and “mTLS between services” describe most of what’s interesting to run on Kubernetes in 2026.

If you operate clusters, the actual to-do list from this release is short and concrete: audit your node images for cgroup v1 before you upgrade, check what kube-proxy mode you’re actually running, and if you have idle GPU-backed workloads sitting at minReplicas: 1 out of habit, that’s now a real cost line you can turn off. Nothing here demands a rewrite. It demands twenty minutes with a kubectl get configmap and an honest look at what’s costing you money while it does nothing.

Kubernetes & Containers - This article is part of a series.
Part : This Article

Related