Kubernetes Best Practices

Two settings decide whether losing a node is a blip or an outage: where your replicas are placed, and how many of them Kubernetes is allowed to evict at once. This page covers both, as Helm values.

Neither is on by default. A fresh install runs one replica of everything, and one replica cannot survive its node going away no matter how it's configured - start by giving the workloads you care about replicas: 2 or more.

INFO

Requires plane-enterprise 3.8.0 Commercial Edition. Earlier versions have neither key; you had to apply PodDisruptionBudget manifests by hand and write podAntiAffinity rules yourself. plane-ce does not currently render either.

Why you need both

They fix different halves of the same problem, and either one alone leaves a gap.

Spreading without a budget. Your three api replicas sit on three different nodes. A drain comes along, the eviction API has no reason to say no, and it evicts all three at once. The service is down until fresh pods pass their readiness probes.

A budget without spreading. Your budget says "never more than one api pod down at a time", but nothing ever told the scheduler to keep the replicas apart, so all three landed on one node. When that node drains, the budget can only stagger the evictions - one at a time, each waiting for the previous replacement to be Ready. Better than losing all three, but the service runs degraded through the whole window, and if the node is being reclaimed on a deadline (a spot interruption gives you two minutes) it dies with the remaining pods still on it.

Configure the spread so a single node failure can only ever touch one replica, then the budget guarantees the others keep serving while that one comes back.

Spreading pods across availability zones

Set topologySpreadConstraints on any workload. Each entry needs a topologyKey; everything else has a sensible default.

services:
  api:
    replicas: 3
    topologySpreadConstraints:
      # Spread evenly across AZs
      - topologyKey: topology.kubernetes.io/zone
      # ...and don't stack two replicas on one node
      - topologyKey: kubernetes.io/hostname

That's the whole configuration. The chart fills in the labelSelector with the workload's own pod label, so there is nothing to type and nothing to get wrong.

WARNING

This matters more than it looks. A topologySpreadConstraint whose selector matches no pods is satisfied by every possible placement - it fails open, silently, and you only find out when an outage doesn't behave the way you expected. Writing these by hand is where that mistake happens.

The options

Key	Default	What it does
topologyKey	required	The node label to spread over. topology.kubernetes.io/zone or kubernetes.io/hostname
maxSkew	1	How uneven the spread may get - the largest allowed difference between two domains
whenUnsatisfiable	ScheduleAnyway	What to do when the skew can't be met: ScheduleAnyway (soft) or DoNotSchedule (hard)
minDomains	unset	Minimum number of domains to spread over
nodeAffinityPolicy	unset	Whether node affinity/selectors are honoured when counting domains
nodeTaintsPolicy	unset	Whether node taints are honoured when counting domains
labelSelector	this workload	Override only to spread against something other than the workload itself

Soft or hard?

ScheduleAnyway is the default for a reason: it spreads pods when it can and schedules them anyway when it can't. DoNotSchedule guarantees the spread, and the price is that a pod with nowhere legal to go stays Pending rather than running somewhere imperfect.

Use DoNotSchedule only where you can guarantee the capacity - a hostname constraint with DoNotSchedule needs at least as many schedulable nodes as replicas, forever, including while a node is being replaced. On a cluster with an autoscaler that means waiting for a new node to boot and register (a minute or more) instead of running temporarily two-to-a-node.

A good default for most clusters:

topologySpreadConstraints:
  # Hard: never two replicas on one node - this is the one that protects you
  - topologyKey: kubernetes.io/hostname
    whenUnsatisfiable: DoNotSchedule
  # Soft: prefer an even spread across AZs
  - topologyKey: topology.kubernetes.io/zone
    whenUnsatisfiable: ScheduleAnyway

If you run a small or bursty cluster, make the hostname rule ScheduleAnyway too. A soft rule that usually spreads beats a hard rule that occasionally leaves pods Pending.

Prerequisites

Your nodes must carry topology.kubernetes.io/zone. Managed clusters (EKS, GKE, AKS) set it automatically; verify it on self-managed ones:

kubectl get nodes -L topology.kubernetes.io/zone

PodDisruptionBudgets

A PodDisruptionBudget tells Kubernetes how much of a workload may be voluntarily disrupted at once. Turn budgets on for every eligible workload with one key:

podDisruptionBudget:
  enabled: true
  maxUnavailable: 1
  unhealthyPodEvictionPolicy: AlwaysAllow

Override per workload where you want something different:

services:
  worker:
    replicas: 6
    podDisruptionBudget:
      maxUnavailable: 2 # a batch worker can lose two at a time
  api:
    replicas: 3
    podDisruptionBudget:
      minAvailable: 2 # never fewer than two API pods serving

What a budget does and does not do

It applies to voluntary disruption only:

Covered	Not covered
kubectl drain - node maintenance, cluster upgrades	A node crashing, losing power or being terminated
Autoscaler consolidation (Karpenter, Cluster Autoscaler)	A pod being OOM-killed
A cloud provider reclaim that drains gracefully	Rolling updates - those follow the Deployment's own maxUnavailable/maxSurge

The last one surprises people: a budget does not make helm upgrade roll your pods more gently. If an upgrade is taking a service down, change the Deployment strategy, not the budget.

maxUnavailable or minAvailable

Prefer maxUnavailable. It keeps its meaning when the replica count changes, and if someone scales a workload back to a single replica it degrades to a harmless no-op.

minAvailable: 1 on a workload that later drops to replicas: 1 does the opposite: it silently blocks every future drain of whatever node that pod lands on, and the drain hangs until someone works out why. Set one or the other - the Kubernetes API rejects both, and the chart refuses to render if you ask for both.

unhealthyPodEvictionPolicy

AlwaysAllow lets a drain evict pods that aren't Ready even while the budget is at its limit. The Kubernetes default (IfHealthyBudget) refuses - which means a crashlooping replica can hold a node hostage, exactly when you most need the drain to work. Requires Kubernetes 1.27 or later.

Which workloads can have one

Budgets are only rendered for the stateless, horizontally-scalable workloads:

api, web, space, admin, live, live_exporter, worker, worker_importers, silo, email_service, external_api, outbox_poller, automation_consumer, webhook_consumer, agent_consumer, pi, pi_worker, runner, iframely

Everything else is deliberately excluded, and asking for a budget on one fails the render with an explanation rather than being quietly ignored:

Excluded	Why
beatworker, pi_beat_worker, monitor, argus	Single replica by design - a second copy would run every scheduled job twice
postgres, redis, rabbitmq, opensearch, minio, garage	In-chart stateful services: one replica on a ReadWriteOnce volume. Use managed equivalents
migrator, pi-migrator, storage_migration	Run-once Jobs

The reasoning is the same in every case: a budget only decides whether an eviction is allowed. With one replica there is no second copy to keep serving, so it cannot protect anything - all it can do is refuse the eviction until the drain gives up or the node disappears under it, turning a routine node rotation into a stuck one.

For the same reason, an eligible workload still at replicas: 1 is skipped silently when you turn budgets on chart-wide. That keeps podDisruptionBudget.enabled: true a safe one-line change. Requesting one explicitly on a single-replica workload still fails, because that one is a mistake rather than a default.

A complete example

A three-AZ cluster, the traffic-facing workloads spread and protected:

podDisruptionBudget:
  enabled: true
  maxUnavailable: 1
  unhealthyPodEvictionPolicy: AlwaysAllow

services:
  api:
    replicas: 3
    topologySpreadConstraints: &spread
      - topologyKey: kubernetes.io/hostname
        whenUnsatisfiable: DoNotSchedule
      - topologyKey: topology.kubernetes.io/zone
        whenUnsatisfiable: ScheduleAnyway
  web:
    replicas: 3
    topologySpreadConstraints: *spread
  live:
    replicas: 3
    topologySpreadConstraints: *spread
  worker:
    replicas: 3
    topologySpreadConstraints: *spread
  silo:
    replicas: 2
    topologySpreadConstraints: *spread

The YAML anchor (&spread / *spread) is plain YAML, not a chart feature - it saves repeating the block for every workload.

Verifying it worked

Budgets exist and have room to act. ALLOWED DISRUPTIONS should be at least 1; a persistent 0 means the workload is already at its limit and the next drain will block.

kubectl get pdb -n <namespace>

NAME             MIN AVAILABLE   MAX UNAVAILABLE   ALLOWED DISRUPTIONS   AGE
plane-api-pdb    N/A             1                 1                     2m
plane-web-pdb    N/A             1                 1                     2m

Replicas are actually on different nodes. This is the check people skip, and it's the one that catches a selector that silently matches nothing:

kubectl get pods -n <namespace> -l app.name=<namespace>-<release>-api \
  -o custom-columns=POD:.metadata.name,NODE:.spec.nodeName,ZONE:.spec.nodeName

Every pod should report a different node. To check zones, join against the node labels:

kubectl get pods -n <namespace> -l app.name=<namespace>-<release>-api \
  -o jsonpath='{range .items[*]}{.spec.nodeName}{"\n"}{end}' \
  | xargs -I{} kubectl get node {} -o jsonpath='{.metadata.labels.topology\.kubernetes\.io/zone}{"\n"}'

Drill it. Drain a node that hosts one of the replicas and watch the workload stay up:

kubectl drain <node> --ignore-daemonsets --delete-emptydir-data

The drain should pause while each replacement becomes Ready rather than removing everything at once. To be sure about an AZ failure, drain every node in one zone.

Related

• High availability - the full HA design: workload tiers, managed external services, ingress, and Karpenter node pools
• Kubernetes installation - installing the chart in the first place