If your pods are not rescheduled when a node goes down and stay stuck Pending after node failure, run kubectl describe pod and read the FailedScheduling event. After a drain it reads 0/2 nodes are available: 1 node(s) didn't match Pod's node affinity/selector, 1 node(s) were unschedulable.; after a node dies, the second item is 1 node(s) had untolerated taint {node.kubernetes.io/unreachable: } instead (1 node(s) had untolerated taint(s) on Kubernetes 1.35 and later). Either way, the pod has been recreated, and no node satisfies its own placement rules. On our cluster 15 stateless app Deployments carried a required node affinity, role NotIn [observability], and the role=observability label is on the only other node, so each of them could run on exactly one node. Nine of them also had topology spread rules that kept two defaults: nodeTaintsPolicy (Ignore), which can leave the second replica Pending with didn't match pod topology spread constraints once a node is gone, and matchLabelKeys (empty), which during an ordinary rolling update moved seven single-replica apps to the other node. We changed the rule from requiredDuringSchedulingIgnoredDuringExecution to preferredDuringSchedulingIgnoredDuringExecution with weight 100 and added nodeTaintsPolicy: Honor and matchLabelKeys: [pod-template-hash] to the spread rules; below are the scheduler output, the Kubernetes documents that explain it, read-only checks with the value that means you have the same problem, and the drain test that confirmed the fix. The cluster is a 2-node k3s cluster (k3s v1.33.5+k3s1, one server and one agent); all times are UTC, 2026-11-13 unless a date is given.

What we saw: 0/2 nodes are available: 1 node(s) didn't match Pod's node affinity/selector

The two nodes, both Ready, no taints:

The finding came from a full run of kubegap, an open-source audit of Kubernetes cluster operations, on 2026-11-11 at 05:08:33. Its problem 5, translated from Korean: "The two nodes cannot stand in for each other. The apps are tied to agent-1, and the databases, the registry and the observability stack to server-1, by labels and local-path volumes. server-1 is the only k3s server and etcd, so losing server-1 also stops the API server." It counted 18 workloads that could run only on agent-1 and 24 only on server-1, CoreDNS with 1 replica on server-1 and no PodDisruptionBudget, and both ingress-nginx replicas on server-1.

On 2026-11-13 at 02:25 we listed the node rules of every Deployment and StatefulSet. Fifteen running app Deployments, in four production namespaces, had the same required rule (excerpt, names masked):

$ kubectl get deploy,sts -A -o json | jq …   # a variant of check 1's jq program that also prints replicas and PVC claims; check 1 prints the rule as nodeSelector=… required=…
Deployment   prod-f/signal-f     1  role NotIn observability
Deployment   prod-f/web-f        1  role NotIn observability
Deployment   prod-b/api-b        2  role NotIn observability
Deployment   prod-b/admin-b      1  role NotIn observability
...
Deployment   prod-d/api-d        2  role NotIn observability
Deployment   prod-e/web-e        1  role NotIn observability

The same rule written as a label selector:

$ kubectl get nodes -l 'role notin (observability)'
NAME      STATUS   ROLES    AGE    VERSION
agent-1   Ready    <none>   347d   v1.33.5+k3s1

One node. Nine of the 15 Deployments also had this spread rule, with no nodeTaintsPolicy and no matchLabelKeys:

topologySpreadConstraints:
  - maxSkew: 1
    topologyKey: kubernetes.io/hostname
    whenUnsatisfiable: DoNotSchedule
    labelSelector:
      matchLabels:
        app: api-d

Three of them ran 2 replicas at 02:27, and all six of those pods were on agent-1: api-b, api-d and an admin app that its HorizontalPodAutoscaler later scaled to 1. At 02:35 there were 38 running pods (DaemonSets excluded) on agent-1 and 59 on server-1.

We fixed the rules first (next section) and then drained agent-1 at 02:51:48 to test the fix. So the scheduler never wrote the message at the top of this page for the 15 app Deployments. The scheduler wrote it during the drain for pods that are tied to agent-1 on purpose: a TURN server on hostNetwork, a GPU embedding service and a vector database, each with a kubernetes.io/hostname selector. With the old rule the scheduler would have given the 15 apps' pods the same didn't match Pod's node affinity/selector reason for server-1, because the rule matches only agent-1 (inference from the rule and the selector output above; we did not run the drain before the fix).

During the drain at 02:54 UTC: the eight pods that had nowhere else to go, each with the scheduler's reason. No other pod was Pending.

At 02:54:12 these eight were the only Pending pods: four pods with local-path volumes on agent-1 (didn't match PersistentVolume's node affinity), the three pods above with hostname selectors, and the second GPU workload (Insufficient nvidia.com/gpu; server-1 has no GPU). server-1 was running 89 non-DaemonSet pods and agent-1 none. One of the eight lines, unmasked except for the pod name:

0/2 nodes are available: 1 node(s) didn't match Pod's node affinity/selector, 1 node(s) were unschedulable. preemption: 0/2 nodes are available: 2 Preemption is not helpful for scheduling.

Prometheus, running pods per node (DaemonSets excluded), 02:35–03:10 UTC. agent-1 drops to 0 at the drain and refills after uncordon and the rollout restarts.

Why

A required NotIn rule is satisfied every day and fails on the day the node is lost

The Kubernetes documentation for node affinity:

• requiredDuringSchedulingIgnoredDuringExecution: "The scheduler can't schedule the Pod unless the rule is met."
• preferredDuringSchedulingIgnoredDuringExecution: "If a matching node is not available, the scheduler still schedules the Pod." Each preferred term has a weight from 1 to 100.
• "NotIn and DoesNotExist allow you to define node anti-affinity behavior."

role NotIn [observability] reads as "anywhere except the observability node". With two nodes, it means "only agent-1". IgnoredDuringExecution means running pods are never checked again, so the rule does nothing while agent-1 is up. It is evaluated only when a pod is scheduled, so it fails only when a pod has to be scheduled while agent-1 cannot take it: when agent-1 has failed, is drained or cordoned, or has no room left.

The scheduler runs its filters in a fixed order (NodeUnschedulable, NodeName, TaintToleration, NodeAffinity, …, PodTopologySpread, from default_plugins.go in the Kubernetes source), and a node is rejected by the first filter that fails. A drained node therefore reports were unschedulable. On the surviving node only the node affinity failure is reported; a topology spread failure on the same node is never evaluated, so it does not appear in the message.

For a node that dies instead of being drained, the second item is a taint, not were unschedulable. Assembled from the Kubernetes source, not observed by us: up to Kubernetes 1.34 the line reads 1 node(s) had untolerated taint {node.kubernetes.io/unreachable: }, and from 1.35 it reads 1 node(s) had untolerated taint(s) without the key.

nodeTaintsPolicy defaults to Ignore

The Kubernetes documentation for topology spread: "If this value is null, the behavior is equivalent to the Ignore policy." With Honor, "nodes without taints, along with tainted nodes for which the incoming pod has a toleration, are included." The field is GA in 1.33. The Kubernetes blog describes the failure: "This may lead to a node with untolerated taint as the only candidate for spreading, and as a result, the pod will stuck in Pending if it doesn't tolerate the taint." and "For backwards compatibility, nodeTaintsPolicy defaults to Ignore."

What this means on two nodes once the node rule is preferred, as we reasoned it before the test (not reproduced; we added Honor before draining): with Ignore, an unreachable or cordoned agent-1 still counts as a spreading domain, and it holds 0 matching pods, because the spread count skips pods that are terminating (countPodsMatchSelector in the Kubernetes source). The first replacement replica goes to server-1. The second would make the skew 2 − 0 = 2, more than maxSkew: 1, so with DoNotSchedule it stays Pending with didn't match pod topology spread constraints. A 2-replica Deployment would run at half its replicas for as long as agent-1 is gone.

Without matchLabelKeys, every rolling update counts the old pod

This part we observed. Our first fix changed the node rule to preferred and added Honor. Argo CD had applied it by 02:39:23, which started a rolling update of all 15 Deployments. The new pod of seven single-replica apps, and one new replica each of api-b and api-d, went to server-1, although the new rule prefers agent-1 with weight 100.

The reason (inference from the spread rule and the documentation below): the labelSelector matches pods of every revision, so during the update the old pod on agent-1 is still counted. A new pod on agent-1 would make the skew 2 − 0 = 2; on server-1 it makes 1 − 1 = 0. DoNotSchedule is a filter and the preferred node affinity is only a score, so agent-1 is removed before scores are compared and the weight of 100 never applies to it.

After the first fix, an ordinary rollout put seven single-replica apps on server-1. Their images had never been there: 2 min 25 s to 5 min 49 s to pull.

Those images had never been pulled on server-1. There were nine pulls on server-1 from the in-cluster registry between 02:41:44 and 02:45:35, from 2m25.179s (843,669,709 bytes) to 6m13.859s (835,296,653 bytes); a 94,655,960-byte image took 5m49.087s.

The Kubernetes documentation for matchLabelKeys: with pod-template-hash, "which is added automatically by the Deployment controller", the rule can "distinguish between different revisions in a single Deployment". The field is beta and enabled by default since 1.27. With matchLabelKeys: [pod-template-hash] the old revision's pod is not counted, so a single-replica app's new pod can go back to the preferred node.

What to check in your cluster now

Checks 1 to 4 only read. They need kubectl and jq.

1. Workloads whose required rules leave exactly one node. A required rule (nodeSelector or requiredDuringSchedulingIgnoredDuringExecution) is checked against labels that exist today. List every required node rule with the first command, then rewrite each one as a selector for kubectl get nodes -l (key=value for nodeSelector, key in (a,b) for In, key notin (a) for NotIn, key for Exists, !key for DoesNotExist, a comma for AND, one command per OR term) and count the nodes it prints:

kubectl get deploy,sts -A -o json | jq -r '.items[] | .spec.template.spec as $s
  | select($s.nodeSelector or $s.affinity.nodeAffinity.requiredDuringSchedulingIgnoredDuringExecution)
  | "\(.kind) \(.metadata.namespace)/\(.metadata.name) nodeSelector=\($s.nodeSelector // {} | tojson) required=\($s.affinity.nodeAffinity.requiredDuringSchedulingIgnoredDuringExecution.nodeSelectorTerms // [] | map(.matchExpressions // [] | map("\(.key) \(.operator) \(.values // [] | join(","))") | join(" AND ")) | join(" OR "))"'
kubectl get nodes -l 'role notin (observability)'   # our rule; use yours

For our old rule, kubectl get nodes -l printed one node. Same problem if a stateless Deployment (no local volume, no GPU, no hostNetwork) matches one node: that Deployment cannot start anywhere while that node is down. Workloads pinned on purpose (a database on a local volume, a GPU job) show up here too; for them the answer is a restore, not a reschedule. After our fix, the 27 workloads that still match one node are databases, the registry, observability, a GPU job and a TURN server.

2. Spread rules that block the second replica, or push single replicas off their node.

kubectl get deploy -A -o json | jq -r '
  .items[] | (.metadata.namespace+"/"+.metadata.name) as $n | .spec.replicas as $r |
  .spec.template.spec.topologySpreadConstraints[]? | select(.whenUnsatisfiable=="DoNotSchedule") |
  "\($n) replicas=\($r) key=\(.topologyKey) nodeTaintsPolicy=\(.nodeTaintsPolicy // "Ignore(default)") matchLabelKeys=\((.matchLabelKeys // [])|join(","))"'

Same problem if:

• key=kubernetes.io/hostname, nodeTaintsPolicy=Ignore(default) and replicas 2 or more: when one node is cordoned or unreachable it still counts as a place with 0 replicas, so with maxSkew: 1 only one replica can start on the remaining node.
• the same rule with an empty matchLabelKeys on a Deployment that should stay on one preferred node: during every rolling update the old pod is still counted, and the new pod is forced onto the other node.

After our fix all 9 app rules print nodeTaintsPolicy=Honor matchLabelKeys=pod-template-hash. The k3s packaged CoreDNS Deployment still prints a hostname DoNotSchedule rule with Ignore(default). With 1 replica it does not meet the first condition, and we left it as it is (see "Not changed, and why").

3. Volumes that tie pods to one node.

kubectl get pv -o json | jq -r '.items[] | select(.spec.nodeAffinity.required) |
  "\(.spec.claimRef.namespace)/\(.spec.claimRef.name) \(.spec.storageClassName) \([.spec.nodeAffinity.required.nodeSelectorTerms[].matchExpressions[].values[]]|join(","))"'

Same problem if a production database's PV is listed with one node. Its pod will report didn't match PersistentVolume's node affinity on every other node. Moving it needs a restore from a copy kept elsewhere, or replicated storage. Our local-path PVs: 24 on server-1, 14 on agent-1.

4. Is there room for the pods that would move?

kubectl describe nodes | grep -E '^Name:|^Allocatable:|^Allocated resources:|^  (cpu|memory)[: ]'

Add the Allocated resources requests of the node you might lose to those of the other node, and compare the sum with the other node's Allocatable. This overstates the load that would move, because it includes DaemonSet pods and pods that cannot move; subtract them if the sum comes out close. Ours: the two Allocated resources lines add up to 3,340m + 5,140m = 8,480m CPU, under the 12,000m allocatable, and 15,314 + 17,366 = 32,680 MiB memory, over the 31,452 MiB allocatable. After we subtracted DaemonSet pods and the eight pinned pods, 1,550m CPU and 8,896 MiB would move from agent-1, and server-1 had 6,860m CPU and 14,086 MiB unrequested. Same problem if, after the subtraction, the requests that would move are larger than the other node's Allocatable minus its own requests: some pods will report Insufficient cpu or Insufficient memory.

5. Test it with a drain. This one changes the cluster.

kubectl drain <node> --ignore-daemonsets --delete-emptydir-data --timeout=300s
kubectl get pods -A --field-selector status.phase=Pending \
  -o custom-columns='NAMESPACE:.metadata.namespace,POD:.metadata.name,WHY:.status.conditions[0].message'
kubectl uncordon <node>
kubectl rollout restart deployment/<name> -n <namespace>   # pods do not move back on their own

Before draining, let running Jobs finish and look for pods without an owner:

kubectl get pods -A -o json | jq -r '.items[]|select(.metadata.ownerReferences==null)|.metadata.namespace+"/"+.metadata.name'

Drain deletes finished pods, and an ownerless one is gone for good. Exclude it with --pod-selector.

Pass: every Deployment that is not pinned on purpose is Available on the other node, and each Pending pod has a reason you expected (didn't match PersistentVolume's node affinity, Insufficient nvidia.com/gpu, a hostname selector). Fail: a stateless app Pending with didn't match Pod's node affinity/selector or didn't match pod topology spread constraints.

A drain evicts at once. A node that really dies is first marked unreachable, and Kubernetes adds a toleration "with tolerationSeconds=300" to every pod, so pods "remain bound to Nodes for 5 minutes after one of these problems is detected". Add at least five minutes to what the drain shows: the count starts when the node controller marks the node unreachable, not when the node fails.

If you do not want to repeat checks 1 and 3 by hand after every change, kubegap runs them: it reports each workload that can run on only one node, pods tied to one node by a local volume, hostNetwork or a GPU, replicas of one Deployment on the same node, and workloads that depend on a node label. It runs check 4 only for losing the largest node, which for us is server-1, so check 4 for losing agent-1 stays manual. Its report on our cluster had no finding for the spread defaults of check 2, so that one stays manual. Its run on 2026-11-11 is where this problem was found.

What we changed, and what we have not

Changed

Commit 1, pushed 02:38:33. In 16 Deployment manifests (15 deployed, one not deployed), requiredDuringSchedulingIgnoredDuringExecution became preferredDuringSchedulingIgnoredDuringExecution with weight 100 and the same NotIn expression, and nodeTaintsPolicy: Honor was added to the 10 spread rules. A server-side dry-run passed for all 16. Argo CD applied it by 02:39:23.

Commit 2, pushed 02:46:38. matchLabelKeys: [pod-template-hash] on the 10 spread rules. After its rollout settled at 02:50:25, every single-replica app was back on agent-1, and api-b and api-d had one replica on each node.

One Deployment after both commits:

      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: kubernetes.io/hostname
          whenUnsatisfiable: DoNotSchedule
          nodeTaintsPolicy: Honor
          matchLabelKeys:
            - pod-template-hash
          labelSelector:
            matchLabels:
              app: api-d
      affinity:
        nodeAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
            - weight: 100
              preference:
                matchExpressions:
                  - key: role
                    operator: NotIn
                    values:
                      - observability

We chose preferred with weight 100 rather than removing the rule, so that each app keeps its usual node every day and can still start on the other node when its usual node is gone.

The drain test. A backup Job running on agent-1 was allowed to finish first (02:50:47). A probe of 20 public addresses started at 02:50:47, one round about every 5 s. Then:

$ kubectl drain agent-1 --ignore-daemonsets --delete-emptydir-data --pod-selector=app!=<the ownerless pod's app label> --timeout=300s
node/agent-1 cordoned
Warning: ignoring DaemonSet-managed Pods: ...
evicting pod ...
...
node/agent-1 drained
# exit 0 at 2026-11-13T02:52:55Z

The drain started at 02:51:48 and printed 83 eviction lines. The selector excluded an ownerless pod that had been Failed since May, because drain would have deleted it and nothing would have recreated it.

• At 02:52:55 two of the 15 app Deployments were not fully available, admin-b (0 of 1) and api-d (1 of 2); from 02:53:12 to 02:53:51 only api-d was still short, and at 02:54:10 all 15 were fully available. That is 2 min 22 s after the drain started.
• The eight Pending pods were the ones shown in the first section, each with an expected reason.
• 24 containers on server-1 started from images already there and 14 images were pulled, taking from 1.349 s to 2m9.6s. Eight of the nine images pulled during the first rollout were still on server-1; for the ninth, one pod reported it already present at 02:52:00 and another reported a 1.349 s pull at 02:52:02, which we did not look into.

All 20 public addresses during the drain and the restore. The two apps with one replica per node never failed while agent-1 was drained; single-replica apps were down for about 5 to 85 seconds; the two tied to agent-1 stayed down until after uncordon.

From the probe (an outage window is the first to the last failing round, plus one interval):

• The two 2-replica APIs never left their normal response code during the drain.
• Single-replica apps were off baseline for about 5 to 85 s: 1 round for the search web UI, 4 rounds from 02:51:54 to 02:52:19 for one site, 11 rounds from 02:52:04 to 02:53:02 for one admin app, 15 rounds to 02:53:24 for an internal API that has no node rule.
• A search API returned 503 from 02:52:04 to 02:54:24, because its embedding service needs the GPU on agent-1.
• An API whose Postgres is on a local volume on agent-1 returned 502 and 503 from 02:51:48 to 02:54:51.

Restore. kubectl uncordon agent-1 at 02:54:29; the pinned pods were Ready and nothing was Pending at 02:54:57. Pods do not move back by themselves, so we ran kubectl rollout restart on the 15 app Deployments at 02:54:57 (settled 02:58:50, single-replica apps back on agent-1) and, at 02:59:19, on 11 Deployments without node rules that the drain had moved to server-1. At 03:04:37 there were 37 running pods on agent-1 and 63 on server-1. During the restarts one probe round each returned no answer for two addresses (02:55:18 and 02:56:00), and the search API was off baseline until 03:04:33 while it restarted. All 20 addresses were at their normal code from 03:04:43.

kubegap again. A full rerun at 03:04:58: of the 60 findings the 2026-11-11 report listed under problem 5, 19 are gone (the 15 one-eligible-node findings of the app Deployments and 4 replicas-on-one-node findings) and 41 remain.

Not changed, and why

These are our decisions for this cluster, with the reasons below. They are not advice for your cluster.

• The control plane stays one k3s server on server-1. The k3s documentation: "HA embedded etcd cluster must be comprised of an odd number of server nodes for etcd to maintain quorum", and HA needs "Three or more server nodes". Two machines cannot form an etcd quorum that survives one failure, and we will not add a third machine: the cluster runs a few personal projects and small contract work. Losing server-1 is handled as a restore, from the etcd snapshots, server token and database dumps copied to agent-1 twice a day or daily; a restore from those copies has been tested. kubegap's rerun also shows that without server-1 the CPU requests would not fit: 6.0 cores allocatable against 7.61 requested.
• Ingress controller replicas and CoreDNS were not spread. Both run only on server-1 (ingress-nginx: 2 replicas, MetalLB L2 address, externalTrafficPolicy: Local, PodDisruptionBudget minAvailable 1; CoreDNS: 1 replica, no PodDisruptionBudget). Both matter only when server-1 is lost, which is the restore case above. CoreDNS is a k3s packaged component; the k3s documentation says those manifests "should not be altered. The files are re-written to disk whenever K3s is started", so changing it would also need a .skip file on the server so that k3s stops managing it.
• Pods tied to agent-1 by local-path volumes, hostNetwork or GPUs stay tied. Moving them needs replicated storage, a network change or another GPU.

Where else this happened

The Incident Library, a searchable collection of public post-incident reports, has no case with this exact cause (a required node affinity or a topology spread default leaving replacement pods Pending after a node loss). These three show a cluster that could not survive losing nodes: replicas packed onto one node, a control plane node that could not be replaced, and all workers NotReady at once.

• Moonlight, 2019-01-18: on a three-node GKE cluster the scheduler sometimes put all three replicas of a web app on one node; CPU-heavy pods drove that node into kernel panics, and the crash moved from node to node until the site was down. Anti-affinity rules on all major Deployments restored it.
• Magic Pages, 2025-04-14: one of three control plane nodes of a hetzner-k3s cluster ran out of memory, and the cluster's own Hetzner API calls kept the rate limit at zero, so the node could not be replaced and every customer site went down.
• Saltside, 2017-04-26: parallel Helm installs saturated small gp2 root volumes and left all three workers of a kops cluster NotReady.

The three cases above are part of a set of six in which a NotReady node was a single point of failure: Node NotReady + single point of failure, 6 cases. Two of the cases under Kubernetes API server + single point of failure, 7 cases are clusters that went down with their only control plane node, as ours would if server-1 were lost: a homelab whose only master node kernel-panicked after a power outage (2025-03-03), and NRE Labs, whose single-master kubeadm cluster kept losing its API server (2018-12-04).

Timeline

FAQ

pods not rescheduled when node goes down

Pods are evicted only after the node is marked unreachable and the 300-second toleration has run out. If the replacement pods are then stuck Pending after the node failure, read the FailedScheduling event, which starts 0/2 nodes are available: on a two-node cluster: the dead node shows had untolerated taint {node.kubernetes.io/unreachable: } (had untolerated taint(s) on 1.35 and later), and didn't match Pod's node affinity/selector on the surviving node means a required node rule excludes it. Count the nodes it matches with kubectl get nodes -l.

required vs preferred node affinity

requiredDuringSchedulingIgnoredDuringExecution is a filter: no matching node, no scheduling. preferredDuringSchedulingIgnoredDuringExecution is a score with weight 1 to 100, added to matching nodes that passed every filter: the pod usually goes to a matching node and goes elsewhere when none is available, but a filter such as a DoNotSchedule spread rule can still remove the matching node. We use preferred with weight 100 so apps keep their usual node daily but can move when it fails.

didn't match pod topology spread constraints

With whenUnsatisfiable: DoNotSchedule and maxSkew: 1, a node that is cordoned or unreachable still counts as a domain with zero pods while nodeTaintsPolicy is unset (Ignore), so the second replica cannot join the first on the surviving node; the Kubernetes blog describes pods stuck Pending this way. Set nodeTaintsPolicy: Honor. For rolling updates, add matchLabelKeys: [pod-template-hash] so old pods are not counted.

k3s two node high availability

Not for the control plane. k3s embedded etcd needs an odd number of servers, and HA needs three or more; two servers cannot keep quorum when one fails. With two machines, Deployments can move between nodes after fixing their rules, but the single server is recovered from etcd snapshots and dumps kept on the other node.

test node failure with kubectl drain

Let running Jobs finish, and use --pod-selector to exclude finished pods that have no owner, because drain deletes finished pods and nothing recreates an ownerless one. Then run kubectl drain <node> --ignore-daemonsets --delete-emptydir-data and list Pending pods with their scheduling message. Pass if every stateless Deployment is Available elsewhere and each Pending pod has an expected reason. A drain evicts at once; after a real failure, eviction comes at least 300 seconds later, counted from when the node is marked unreachable. Uncordon, then rollout restart, because pods do not move back.

Originally published at benchlet.com.

Every backup Job completed. Five of the seven dumps were on the same disk as the databases they copy.

If your Kubernetes CronJob pg_dump backup writes to a PVC of the local-path StorageClass (the k3s default) and kubectl get jobs -A shows every backup Job as Complete 1/1, that output does not tell you which node's disk the dump is on. On our cluster all seven backup Jobs completed on 2026-11-11, but five of the dumps were on the same single disk as the databases they copy, and every k3s etcd snapshot and the server token needed to restore them were on that disk too. A local-path volume is created on the node where the first pod that uses it is scheduled, and in five of our seven backup CronJobs nothing placed that pod on a node other than the database's. To see yours, list each PV with its node (check 1 below): if a backup claim and the database's data claim show the same NODE, the dump is on the database's node, and on the same disk if that node has only one. The cluster runs k3s v1.33.5+k3s1 on two computers: one server, which holds etcd and all seven backed-up databases on a single SSD, and one agent. All times are UTC.

What we saw

server-1 is the only server and has one 500 GB SATA SSD, no RAID; agent-1 is the agent, on the same network. Storage is the k3s built-in local-path provisioner; applications are deployed by Argo CD. We captured this state on 2026-11-12 from 01:25 UTC; names are masked and columns are trimmed.

Seven backup CronJobs (six pg_dump, one mysqldump) run daily between 02:50 and 03:50. The latest run of each (the output had three Jobs per CronJob and one old Job created by hand; only the newest Job of each CronJob is kept here, and the CONTAINERS, IMAGES and SELECTOR columns are cut):

$ kubectl get jobs -A
NAMESPACE   NAME                               STATUS     COMPLETIONS   DURATION   AGE
prod-a      app-a-postgres-backup-29906115     Complete   1/1           9m7s       22h
prod-b      app-b-postgres-backup-29906130     Complete   1/1           4s         21h
prod-b      app-b-postgres-3-backup-29906130   Complete   1/1           4s         21h
prod-b      app-b-postgres-2-backup-29906130   Complete   1/1           4s         21h
prod-c      app-c-postgres-backup-29906150     Complete   1/1           4s         21h
prod-d      app-d-mysql-backup-29906090        Complete   1/1           11s        22h
prod-e      app-e-postgres-backup-29906100     Complete   1/1           4s         22h

Each job writes a gzip dump (<name>_<timestamp>.sql.gz; the mysqldump job <name>-<date>-<time>.sql.gz) to a local-path PVC mounted at /backup and deletes older files. No command in any of the seven sends anything outside the cluster: there is no s3, rclone, restic, scp or rsync in them.

kubectl get pv at 2026-11-12 01:25 UTC. Red: a dump volume on the same node as its database, 5 of 7.

The same volumes, next to the node of each source database pod:

All seven source databases are on server-1, whose one disk holds a single filesystem at / with the etcd database, the etcd snapshots and the local-path volumes. Every PV had reclaim policy Delete.

All five k3s etcd snapshots, file:// on the server's own disk.

The five snapshots are etcd-snapshot-server-1-<unix time>, 41,775,136 to 45,232,160 bytes each. The k3s unit has no --etcd-s3* argument and there is no /etc/rancher/k3s/config.yaml. The server token, /var/lib/rancher/k3s/server/token (mode 600, root), is on the same disk. A cleanup CronJob keeps only the 5 newest snapshot files, daily at 03:00.

One disk failure on server-1 would have removed the etcd database, all 5 snapshots and the token, every source database, and 5 of the 7 dumps (inference from the facts above). Only the app-a and app-b-postgres-3 dumps on agent-1 would have been left.

Why local-path created the dump volumes on the database's node

local-path creates the volume where the first pod runs. Both StorageClasses use WaitForFirstConsumer: the volume is created on the node where the first pod that uses the claim is scheduled. The second class is named after the server, but nothing in it pins a volume to that node: no allowedTopologies, only a path.

# kubectl get storageclass local-path-server-1 -o yaml (trimmed)
allowVolumeExpansion: false
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: local-path-server-1
parameters:
  path: /var/lib/rancher/k3s/storage
provisioner: rancher.io/local-path
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer

After that the PV is fixed to the node. From the local-path-provisioner README: PersistentVolumes created by the provisioner "use the kubernetes.io/hostname label for node affinity. This ensures the volume is only accessible on the node where it was provisioned."

Only one of the seven jobs was placed on another node on purpose.

• app-a-postgres-backup has nodeSelector: kubernetes.io/hostname: agent-1. It was created on 2026-07-02 with that selector on purpose, and it was the only backup placed away from its database by design.
• app-b-postgres-3-backup had no nodeSelector and used local-path-server-1. Its volume is on agent-1 only because its first pod was scheduled there (inference from the binding rule above).
• app-c-postgres-backup had nodeSelector: storage: server-1, which pinned its dump to the database's node.
• app-d-mysql-backup had nodeSelector: kubernetes.io/hostname: server-1 and wrote to a claim from the database StatefulSet's own volumeClaimTemplate, so its dump volume was created with the database pod, on its node.
• app-b-postgres-backup, app-b-postgres-2-backup and app-e-postgres-backup had no nodeSelector, and their dump volumes were created on server-1, where their databases run.

A Complete Job status does not record which disk the dump was written to.

k3s keeps etcd snapshots on the server unless told otherwise. From the k3s etcd-snapshot docs: "Scheduled snapshots are enabled by default, at 00:00 and 12:00 system time, with 5 snapshots retained per server node on local disk" and "K3s etcd snapshots are stored on the node file system, and may optionally be uploaded to an S3-compatible object store for disaster recovery scenarios." We had not set --etcd-s3, and server-1 is the only server, so all five snapshots were on one disk.

The snapshot is unusable without the token. From the k3s backup and restore docs: "you must also back up the server token file at /var/lib/rancher/k3s/server/token" and "If you do not use the same token value when restoring, the snapshot will be unusable". A copy of the snapshots alone would not have been enough.

The snapshot does not hold volume contents. It is "a complete copy of the etcd datastore, along with cluster certificate authority (CA) certificates and private keys" (k3s docs), so it has the API objects but not the data in volumes (inference from what etcd stores): the dumps and the snapshots have to be moved separately.

Check where your Kubernetes CronJob pg_dump backup is stored

These are read-only, except checks 5 and 6, which create throwaway objects. kubegap, an audit tool that reads the cluster of your current kube context, found this problem here on 2026-11-11 by reading the same three things as checks 1 to 3. It reports all three in one run, so you can run it instead of pasting those three commands. Checks 4 to 6 are run by hand.

1. Which node holds each dump, and which node holds the database it copies?

kubectl get pv --sort-by=.spec.claimRef.namespace -o custom-columns='NS:.spec.claimRef.namespace,PVC:.spec.claimRef.name,NODE:.spec.nodeAffinity.required.nodeSelectorTerms[0].matchExpressions[0].values[0],CLASS:.spec.storageClassName,RECLAIM:.spec.persistentVolumeReclaimPolicy'

Same problem if: the row of a backup PVC and the row of the database's data PVC show the same NODE, and that node has one disk (lsblk -d -o NAME,TYPE,SIZE on the node lists one row of TYPE disk; on server-1 it was sda disk 465.8G). Read NODE, not CLASS: with local-path a StorageClass name does not choose the node; the first pod that uses the claim does (WaitForFirstConsumer). RECLAIM Delete means deleting the claim deletes the directory on the node.

2. Does any backup job send its result to another machine?

kubectl get cronjob -A -o json | jq -r '.items[]
  | select(.metadata.name | test("backup|dump|snapshot"))
  | [.metadata.namespace, .metadata.name,
     (.spec.jobTemplate.spec.template.spec.nodeSelector // {} | tostring),
     ([.spec.jobTemplate.spec.template.spec.containers[] | ((.command // []) + (.args // [])) | join(" ")] | join(" ")
      | test("s3|rclone|restic|kopia|scp|rsync") | tostring)]
  | @tsv'

On our cluster, before the change:

prod-a	app-a-postgres-backup	{"kubernetes.io/hostname":"agent-1"}	false
prod-b	app-b-postgres-backup	{}	false
prod-b	app-b-postgres-3-backup	{}	false
prod-b	app-b-postgres-2-backup	{}	false
prod-c	app-c-postgres-backup	{"storage":"server-1"}	false
prod-d	app-d-mysql-backup	{"kubernetes.io/hostname":"server-1"}	false
prod-e	app-e-postgres-backup	{}	false

Same problem if: the last column is false and the nodeSelector column is empty or names the database's node. Ours had all seven false, four with {} and two naming server-1. The match reads only the command and args, so true means "look closer", and a script mounted from a ConfigMap shows false even if it uploads: open the script before you conclude.

3. Where are the etcd snapshots? (k3s with embedded etcd)

kubectl get etcdsnapshotfile

Same problem if: every LOCATION starts with file:// and nothing copies that directory to another machine (check 2). Those snapshots are on the server's own disk, next to the etcd database they copy.

4. Did you back up the server token anywhere but the server?

On the server: sudo ls -l /var/lib/rancher/k3s/server/token. Then look for the same file where your snapshot copies go.

Same problem if: the token exists only on the server. Without the same token value, a restored snapshot is unusable (k3s docs, backup and restore).

5. Does the newest dump restore from the other machine?

A Job in the dump's namespace that mounts the dump PVC read-only (it is scheduled on the node that holds the PVC), starts an empty server in an emptyDir, restores with ON_ERROR_STOP, and counts tables. The Job below is for PostgreSQL; use the image your backup uses (uid 70 is postgres in the alpine image):

apiVersion: batch/v1
kind: Job
metadata: {name: restore-test, namespace: <ns>}
spec:
  backoffLimit: 0
  ttlSecondsAfterFinished: 3600
  activeDeadlineSeconds: 900
  template:
    spec:
      restartPolicy: Never
      securityContext: {runAsUser: 70, runAsGroup: 70, fsGroup: 70}
      containers:
      - name: restore
        image: postgres:16-alpine
        command: ["/bin/sh", "-c"]
        args:
        - |
          set -eu
          F=$(ls -t /backup/*.sql.gz | head -1); echo "latest $F"
          gzip -t "$F"
          initdb -D /work/pg -U postgres >/dev/null
          pg_ctl -D /work/pg -o "-c listen_addresses='' -c unix_socket_directories=/work" -w start >/dev/null
          createdb -h /work -U postgres restored
          zcat "$F" | psql -h /work -U postgres -d restored -v ON_ERROR_STOP=1 -q
          psql -h /work -U postgres -d restored -Atc "select count(*) from pg_tables where schemaname not in ('pg_catalog','information_schema')"
        volumeMounts:
        - {name: backup, mountPath: /backup, readOnly: true}
        - {name: work, mountPath: /work}
      volumes:
      - {name: backup, persistentVolumeClaim: {claimName: <dump-pvc>, readOnly: true}}
      - {name: work, emptyDir: {}}

Then kubectl get pod -n <ns> -l job-name=restore-test -o wide shows the node it ran on, and the table count should equal the source's (select count(*) from pg_tables where schemaname not in ('pg_catalog','information_schema') in the source database).

Same problem if: the Job ran on the database's node, fails, or the count differs.

6. Does the etcd copy restore?

Copy the newest snapshot and the token from the other machine to a third one with Docker, check the sha256 against the copy, then restore with the same k3s version and no agent (no kubelet, so no workload from the snapshot can start):

docker run --rm --privileged -e K3S_TOKEN="$(cat token)" -v k3s-test:/var/lib/rancher/k3s -v "$PWD":/snap:ro \
  rancher/k3s:<your version, e.g. v1.33.5-k3s1> server --disable-agent \
  --cluster-reset --cluster-reset-restore-path=/snap/<snapshot file>
docker run -d --name k3s-test --privileged -e K3S_TOKEN="$(cat token)" -v k3s-test:/var/lib/rancher/k3s \
  rancher/k3s:<same version> server --disable-agent
docker exec k3s-test kubectl get ns,deploy,sts,cronjob -A --no-headers | wc -l
docker rm -f k3s-test; docker volume rm k3s-test

The first command prints "Managed etcd cluster membership has been reset, restart without --cluster-reset flag now" and exits. Same problem if: it exits with an error, or the object counts are far from the live cluster's (objects created after the snapshot account for small differences). Delete the copied snapshot afterwards: it contains every Secret of the cluster.

What we changed and what we have not

What was on each machine before and after. Red: lost together with the database if server-1's disk fails.

Changes 2 and 3 went through the manifests repository and Argo CD. Change 1 was made with kubectl patch and recorded in the repository afterwards. The node settings on server-1 are recorded in the README next to the CronJob.

1. Retain on 30 of the 32 PVs (2026-11-12 01:35). We set every PV outside the logging and monitoring namespaces to Retain with kubectl patch pv <name> -p '{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}', and no pod restarted. Retain keeps the node directory when a claim is deleted and does nothing when the disk fails, so it is not the fix for this problem.

2. Every dump Job runs on agent-1. All seven CronJobs now have:

spec:
  jobTemplate:
    spec:
      template:
        spec:
          nodeSelector:
            kubernetes.io/hostname: agent-1

The five that were on server-1 write to a new PVC of class local-path (app-b-postgres-backup-agent-1, app-b-postgres-2-backup-agent-1, app-c-postgres-backup-agent-1, app-d-mysql-backup-agent-1, app-e-postgres-backup-agent-1). Because of WaitForFirstConsumer, each is created on agent-1, where its first pod runs. Changing only the nodeSelector would not have moved the old volumes: an existing local-path PV keeps its node affinity, so a pod pinned elsewhere could not mount it (inference from the provisioner's node affinity rule). Retention is now 7 days for all seven (find -mtime +6 -delete).

3. The etcd snapshots and the token are pulled to agent-1. A CronJob kube-system/etcd-snapshot-backup on agent-1 runs at 30 0,12 * * *, 30 minutes after each k3s snapshot, and pulls /var/lib/rancher/k3s/server/db/snapshots/ and /var/lib/rancher/k3s/server/token from server-1 with rsync over SSH into the PVC etcd-snapshot-backup-agent-1, kept 7 days. Its name contains "backup", so the existing alert BackupCronJobStale (no success for 26 hours) covers it.

On server-1, root may log in over SSH only with a forced-command key and only from agent-1 and the pod network:

# /etc/ssh/sshd_config.d/10-etcd-snapshot-backup.conf
Match Address 10.0.0.11,10.42.0.0/16
    PermitRootLogin forced-commands-only

# /root/.ssh/authorized_keys (one line; key not shown)
restrict,from="10.0.0.11,10.42.0.0/16",command="/usr/bin/rrsync -ro /var/lib/rancher/k3s/server/" ssh-ed25519 <public key> etcd-snapshot-backup@agent-1

rrsync -ro limits the key to read-only rsync under /var/lib/rancher/k3s/server/. sshd -t passed. sshd -T -C user=root,host=x,addr=<address> was compared before and after for two addresses: 203.0.113.9, outside the Match, was unchanged, and 10.0.0.11 changed only in permitrootlogin (no to forced-commands-only). Addresses in 10.42.0.0/16 are inside the Match too. The private key is kept in the secret manager the cluster already uses and reaches the pod as a Secret through External Secrets.

The first copy failed. All three attempts of the Job, on agent-1, ended the same way:

ssh: connect to host 10.0.0.12 port 22: Operation timed out
rsync: connection unexpectedly closed (0 bytes received so far) [Receiver]
rsync error: unexplained error (code 255) at io.c(232) [Receiver=3.4.1]

The agent-1 node itself could not reach server-1:22 either. The ufw rows on server-1 before the change:

22/tcp                     ALLOW IN    100.64.0.0/10
Anywhere                   ALLOW IN    10.42.0.0/16
Default: deny (incoming), allow (outgoing), allow (routed)

That allows 22/tcp from the Tailscale range and every port from the pod network, but nothing from agent-1's own address, 10.0.0.11. The pod's connection timed out although the pod network was allowed, so it reached server-1 from 10.0.0.11 (inference). One rule fixed it:

ufw allow from 10.0.0.11 to any port 22 proto tcp comment "etcd-snapshot-backup from agent-1"

The next run copied 5 snapshots and the token to agent-1.

Verification (2026-11-12 17:24 to 17:55). Each backup CronJob was run once by hand after Argo CD applied the change: all seven ran on agent-1 and completed (app-a-postgres-backup, the largest, in 9 minutes). Every backup PVC in use is now on agent-1.

The newest dump of each of the five moved databases was then restored from its PVC on agent-1 into an empty server (psql with ON_ERROR_STOP=1, or mysql), and compared with counts taken inside the source pod at the same minute:

Five dumps restored from agent-1 at 17:42 UTC. Table and row counts equal the source; the MySQL source rows were not counted.

The etcd copy was restored the same way as check 6. The newest snapshot (12:00) and the token were copied from agent-1 with a matching sha256 and restored with the same k3s version (--cluster-reset --cluster-reset-restore-path, then a normal start) in a container on a third machine, with the agent disabled so nothing from the snapshot could run:

The 12:00 snapshot copy restored by k3s v1.33.5 with no agent.

Restored versus live: namespaces 28/28, deployments 68/68, statefulsets 22/22, cronjobs 29/30, secrets 145/147, configmaps 153/154, persistentvolumes 32/38. Every difference is an object created after the 12:00 snapshot: the new CronJob, two Secrets (the copy's SSH key and one from an unrelated change that day), the copy's known_hosts ConfigMap and the six new PVCs. The only processes in the container were k3s and its supervisor.

Three earlier attempts failed on the test script, not the copy; you may hit them with check 6:

• kubectl exec ... cat of the snapshot returned 43,974,656 bytes instead of 45,989,920; kubectl cp with a sha256 check worked.
• With --network none or an internal Docker network, k3s exited with no default routes found; the default bridge network worked.
• With --token-file, k3s exited with /var/lib/rancher/k3s/server/token does not exist, please pass --token; K3S_TOKEN worked.

What we have not changed:

• No off-site copy. Both copies are on a second computer in the same house: a disk failure on server-1 no longer takes the backups with it, but a fire, theft or power event that takes both machines would. This is a practice cluster, and I chose to stop at the second machine.
• The old dump PVCs on server-1 are kept. They hold the dumps taken before the change and are no longer written to.
• Two dumps were not restore-tested in this change. app-a and app-b-postgres-3 were already on agent-1; the restore test covered the five that moved.
• Eight PVs are Delete. Read at 18:21 after the change, 30 PVs are Retain and 8 are Delete: the six new backup claims, which got Delete from the StorageClass default, and one volume each in the logging and monitoring namespaces.
• The snapshot copy holds every Secret and the cluster CA keys, so access to agent-1's disk now matters as much as access to server-1's (inference).

Where else this happened

The Incident Library, a searchable collection of public incident reports, has three that show what happened elsewhere when a control plane or a data store depended on a single disk, node, zone or pod; the first is the disk failure that server-1 is exposed to:

• An individual engineer's blog, 2023-05-07 (IL-0185): a disk failure made the first control plane node of a Kubespray cluster read-only, and recovery needed the inventory reordered and a new control plane node joined.
• Devsisters, 2021-02-19 (IL-0172): an AWS Tokyo cooling failure took down 6 of 60 CockroachDB nodes on Kubernetes in 22 minutes; ranges lost quorum because no availability zone locality was configured, and a table backup taken from the surviving nodes did not work, which is the failure that checks 5 and 6 are there to catch before you need the backup.
• Palark, 2020-03-13 (IL-0196): one RedisFailover change restarted the only Redis pod that held data, and Sentinel promoted the empty new replica, erasing the data; as with the five dumps on server-1, the data was in one place.

The library lists 5 reports where etcd was a single point of failure, IL-0185 among them, and 2 where data was lost, the Devsisters and Palark cases above.

Timeline (UTC)

FAQ

kubernetes cron job for postgres db backup

A Kubernetes CronJob pg_dump backup, like a Kubernetes CronJob mysqldump backup, writes its dump to a PVC. With local-path, the claim is created on the first Job pod's node, possibly the database's. Pin the Job with nodeSelector to another node with a new claim: an existing local-path PV stays on its node. Or upload it off the cluster.

kubernetes rancher local path specify node

The StorageClass does not choose the node. With volumeBindingMode: WaitForFirstConsumer, local-path creates the volume on the node where the first pod using the claim is scheduled, and then pins the PV there with kubernetes.io/hostname node affinity. To choose the node, put nodeSelector or node affinity on that first pod before the claim is bound.

where does k3s local-path store data

In /var/lib/rancher/k3s/storage on the node that holds the volume (${HOME}/.rancher/k3s/storage in rootless mode). The node is recorded in the PV's node affinity. Read it with the kubectl get pv command in check 1 above. With reclaim policy Delete, deleting the claim deletes that directory on the node.

How do you verify that PostgreSQL backups on Kubernetes can actually be restored?

Run a Job that mounts the dump's claim read-only, starts an empty PostgreSQL in an emptyDir, restores with psql -v ON_ERROR_STOP=1, and counts tables. Compare with the same count in the source. Check which node the Job ran on: for a backup meant to survive a disk failure, it must not be the database's node.

restore k3s etcd snapshot new machine

Copy the snapshot and the server token file /var/lib/rancher/k3s/server/token to the new machine. Run k3s server --cluster-reset --cluster-reset-restore-path=<file> with the token in K3S_TOKEN (with --token-file, k3s exited with /var/lib/rancher/k3s/server/token does not exist, please pass --token), then start k3s without --cluster-reset. We used the same k3s version. Without the same token the snapshot is unusable.

Originally published at benchlet.com.

This is a two-node k3s cluster that I run alone for a few side projects. Times are UTC. Object names below are role names, not the real ones.

What it looked like

Three days later the failed pods were still there. The control plane garbage-collects terminated pods only past a threshold of 12,500, so on a small cluster they stay until something else deletes them:

$ kubectl get pods -A --field-selector=status.phase=Failed
NAMESPACE   NAME                        READY   STATUS    RESTARTS   AGE
prod-a      turn-server-…-k2m9q         0/1     Evicted   0          236d
prod-a      signaling-…-r7t4c           0/1     Evicted   0          3d
prod-b      api-server-…-p3w8d          0/1     Evicted   0          28d
prod-b      api-server-…-z6h2n          0/1     Evicted   0          28d
prod-b      api-server-…-b9v5f          0/1     Evicted   0          9d
prod-b      api-server-…-j4c7s          0/1     Evicted   0          3d
…                                       (23 Failed; the 16 Evicted all on the same node)

Thirteen of the sixteen messages had the same shape; the other three stop after the available value:

The node was low on resource: ephemeral-storage.
Threshold quantity: 5729877282, available: 15662516Ki.
Container app was using 17556Ki, request is 0, has larger consumption of ephemeral-storage.

The threshold is 5,729,877,282 bytes, 5.34 GiB. The node had 15,662,516 KiB available, 14.9 GiB. The container being evicted held 17 MiB. That is the message on a pod evicted at 14:03; the seven pods evicted between 16:13:49 and 16:14:16 carry "available" values between 12.1 and 15.4 GiB.

The previous two write-ups in this series start from the same afternoon: the 4.5 GiB image that filled the disk and the signaling server that could not come back because its image had been garbage-collected, then the TURN server that had been crash-looping since September. This one is about the evictions themselves.

The three waves

I had the one-minute free-space series from Prometheus and the eviction time of each pod from its last status condition. Put together:

Only the first pod of the first wave and the first two of the third were evicted with free space under the threshold.

Why thirteen pods were evicted above the threshold

The node's kubelet configuration:

evictionHard:           imagefs.available: 5%, nodefs.available: 5%
evictionMinimumReclaim: imagefs.available: 10%, nodefs.available: 10%
evictionSoft:           none

evictionMinimumReclaim is the setting I had never looked at. Once a hard threshold has been crossed, the kubelet does not stop evicting as soon as the signal is back above the threshold. It keeps evicting, one pod per loop, until the signal is above the threshold plus the minimum reclaim, both measured as a share of the filesystem. On a 106.7 GiB filesystem that is 5.34 GiB plus 10.67 GiB: eviction continues until 16.0 GiB are free.

Both later waves ended within a minute of free space jumping past 16.0 GiB: the last evictions are at 14:03:56 and 16:14:16, the jumps show in the 14:05 (15.3 to 27.3 GiB) and 16:15 (14.5 to 27.3 GiB) samples. The evictions themselves freed almost nothing; the thirteen messages that state a usage add up to 182 MiB. What freed 12 GiB in a minute each time is not in any record I have. The first write-up put it down to the kubelet reclaiming unused images, which is what it does before it evicts, and the shape of the series is consistent with that. The image garbage collector's own high-water mark, 85 % used, is the same 16.0 GiB on this disk. After 14:05 the threshold was cleared, and from 14:20 the node sat at 12.9 GiB, above 5.34 and below 16.0, with no evictions until the second dip.

The upstream default for evictionMinimumReclaim is empty, which means "stop at the threshold". k3s sets 5 % thresholds and a 10 % minimum reclaim in its agent defaults, and this node runs them unchanged; I had not set either. On a node where the image store and the root filesystem are the same disk, 10 % is 10.67 GiB, and the 42 pods on this node hold 0.2 GiB between them today, and the sixteen evicted containers held 182 MiB in total, so evicting pods was never going to reach it.

Nothing asked for disk

The last line of the thirteen messages that have one ends with request is 0. The audit counted what the scheduler and the kubelet actually knew about disk on this cluster:

running containers                           136
with an ephemeral-storage request              0
with an ephemeral-storage limit                0
LimitRange objects in any namespace            0

With no requests, the scheduler places pods without looking at disk at all. When the kubelet has to pick what to evict under disk pressure it sorts pods three ways: pods whose usage exceeds their request first, then by priority, then by how far usage exceeds the request. With every request at zero, every pod holding anything on disk is in the first group, and the order inside it is priority and then whoever holds the most. One more rule sits in front of all of that: a pod the kubelet has no stats for ranks ahead of every pod it can measure. The one pod in wave 1 and the two in wave 3 whose messages carry no usage line are, I infer from the ranking code, exactly those, which is why at 16:10 and 16:12, with two runners still on the node, the signaling and TURN servers went first. The evicted containers held between 0.7 MiB and 37 MiB. The things actually consuming the disk, a 4.5 GiB model image being unpacked and, later, two CI runners building at once, are not things pod eviction can free: the image is node-level storage, and the runners are the pods holding the most, which puts them first in the sort only if they are still there when the kubelet looks.

The CI runners run as a RunnerDeployment with a docker-in-docker sidecar, whose image layers go into the sidecar's writable layer and whose checkouts go into an emptyDir, both on the node's root disk. They had CPU and memory requests and limits and nothing for storage, and no node selector, so a runner lands on whichever node has room, including the one that serves production. The audit I used to find this listed the runners as the probable cause of the 11-08 dip. For the first dip the Prometheus series does not support that: no runner pod was on the node between 13:30 and 13:55, and the free-space curve follows the model image deploy described in the first write-up; that attribution is an inference from timing, not a measurement. For the second dip it points the other way: two runner pods sat on the node from 16:04 to 16:13, the only time that day two were there at once, and free space went from 10.3 GiB at 16:07 to 2.1 GiB at 16:12. What they wrote is not recorded; the runners' own records are gone with the pods.

What I could not see

• Per-container disk use on 11-08. container_fs_usage_bytes has no series on this k3s install, and the kubelet's eviction counters were reset by a node reboot two days later. The eviction messages are the only per-pod record.
• Why the kubelet waited from 13:56, when free space was back at 14.4 GiB, until 14:03 to evict the next two pods. The kubelet log on the node was not readable from my session. One consequence is measured: the replacement pods created at 13:46 did not start until 14:09, so the node admitted nothing for 23 minutes.
• What fills the disk today. The kubelet's stats summary says the root filesystem has 72.9 GiB used, of which the image store accounts for 35.0 GiB and all 42 pods' ephemeral storage for 0.2 GiB. The node image list adds up to 15.3 GiB, so about 20 GiB of the image store belongs to nothing in the image list, which I expect to be containerd snapshots and content I cannot read, and about 38 GiB more is outside kubelet accounting entirely. Breaking that down needs root on the node.

The fix

Three changes, all in git, applied by Argo CD.

The runner and its docker-in-docker sidecar now request and limit ephemeral storage. The limits mean the kubelet evicts the runner on its own once the sidecar's writable layer passes 20 GiB, the runner's passes 10 GiB, or the pod as a whole, emptyDir included, passes 30 GiB, before the node is under pressure. The request means that under pressure a runner that has grown past 5 GiB sorts ahead of pods that are a few MiB over zero:

resources:
  requests: { cpu: 500m, memory: 1Gi, ephemeral-storage: 2Gi }
  limits:   { cpu: "2",  memory: 4Gi, ephemeral-storage: 10Gi }
dockerdContainerResources:
  requests: { ephemeral-storage: 5Gi }
  limits:   { ephemeral-storage: 20Gi }

The two GPU workloads request 2 Gi each. With a 2 Gi request and a few MiB of usage they no longer exceed their request, so the kubelet sorts them after every pod that does; while everything else requests zero, they are the last to go.

And an alert that fires before the kubelet acts:

- alert: NodeRootDiskLow        # < 15 % for 10m, warning
- alert: NodeRootDiskCritical   # <  8 % for  5m, critical → the route that mentions everyone

After the sync:

$ kubectl get pods -A -o json | … ephemeral-storage requests
ci            runner-…-d8k3p   runner   2Gi / 10Gi
ci            runner-…-d8k3p   docker   5Gi / 20Gi
ci            runner-…-m2r6w   runner   2Gi / 10Gi
ci            runner-…-m2r6w   docker   5Gi / 20Gi
prod-b        embedding-…      app      2Gi / –
prod-c        ai-…             ai       2Gi / –

Timeline

What I am changing

No disk alert fired that day. The kube-prometheus-stack chart's NodeFilesystemAlmostOutOfSpace rule wants free space under 5 % for 30 minutes; on 11-08 it was under 5 % for eight minutes at the longest stretch and thirteen in the day. The kubelet evicts at 5 % immediately; it had finished before the rule's 30 minutes were up.

Done:

• ephemeral-storage requests and limits on the CI runner and its docker-in-docker sidecar; requests on the two GPU workloads.
• NodeRootDiskLow at 15 % for 10 minutes, NodeRootDiskCritical at 8 % for 5 minutes, the latter on the route that mentions everyone.

Not done yet:

• evictionMinimumReclaim. It lives in the k3s agent configuration on the node, outside git. With the upstream default, no minimum reclaim, the day would have been three evictions: the first pod of wave 1 and the first two of wave 3. At 2–3 % it would have been seven: all five of wave 1, none of wave 2, two of wave 3. It needs a node-level change and a kubelet restart.
• A node for CI. The runners should not share a disk with production at all. That needs hardware I do not have this month.
• Root on the node to break down the 38 GiB that no pod or image accounts for.
• A default ephemeral-storage request through a LimitRange in every production namespace, small enough not to change scheduling, so a new Deployment cannot arrive with zero again; the two cluster components stay outside it.

If your eviction messages show "available" values far above the threshold, read the kubelet's evictionMinimumReclaim first. kubectl get --raw /api/v1/nodes/<node>/proxy/configz | jq .kubeletconfig.evictionMinimumReclaim shows it without logging in to the node.

Public post-incident reports where a node ran out of disk and took unrelated workloads with it are collected under the node disk full and pods evicted symptoms in Incident Library. The audit that surfaced this is kubegap.

Originally published at benchlet.com.

This is a two-node k3s cluster that I run alone for a few side projects. Times are UTC.

What it looked like

A read-only audit of the cluster (kubegap) that I ran on 2026-11-11 at 05:08 for a different problem listed this one next to it: a Deployment in a production namespace with 0/1 ready and a pod in CrashLoopBackOff with 718 restarts. By 11:40, when I got to it, the number was 794.

$ kubectl -n dam-prod get pod -l app=dam-coturn
NAME                          READY   STATUS                   RESTARTS            AGE
dam-coturn-549854dfcb-bchx5   0/1     CrashLoopBackOff         794 (2m11s ago)     2d19h
dam-coturn-549854dfcb-c8mfp   0/1     ContainerStatusUnknown   28 (2d19h ago)      2d21h
dam-coturn-549854dfcb-jgp4h   0/1     ContainerStatusUnknown   11621 (2d21h ago)   235d

Three pods. The one created in March with 11,621 restarts, a replacement from 2026-11-08 with 28, and the current one with 794. The two older ones had been evicted on 11-08 when the node ran out of disk (that is the previous write-up); the eviction did not cause this, it only produced a new pod that failed the same way.

The previous container's log had the answer in its first line, followed by the program's entire usage text:

$ kubectl -n dam-prod logs dam-coturn-549854dfcb-bchx5 --previous | head -4
turnserver: unrecognized option '--no-dtls'
2026-11-11T11:38:04.215+0000 WARNING Cannot find config file: turnserver.conf. ...
2026-11-11T11:38:04.215+0000 WARNING Cannot find config file: turnserver.conf. ...
turnserver: unrecognized option '--no-dtls'

Started 11:38:04, finished 11:38:04, exit code 255. The binary did not know a flag that was in the Deployment's args, so it printed help and quit. A few hundred lines further down, the same usage text showed what had replaced it:

 --no-tls          Do not start TLS client listeners.
 --dtls            Start DTLS client listeners. DTLS is not started by default.

The alert had been firing since before the retention window. The Alertmanager list that morning had six groups, five of them routed to the Discord receiver and two of those for this namespace:

Which binary was this?

The Deployment said image: coturn/coturn:latest with imagePullPolicy: Always. The pod's status said what that had resolved to:

Image ID:  docker.io/coturn/coturn@sha256:bbefd3e1fdfd...

Docker Hub's tag API maps that digest to 4.18.0, pushed 2026-09-08 11:41, and latest has pointed at it since 11:42, five seconds later. So the running binary was coturn 4.18.0, and the flag it rejected had to have been removed somewhere between whatever version the pod had pulled in March and 4.18.0.

The coturn ChangeLog has one line about DTLS in 4.17.0: "Start the DTLS listeners only when --dtls is given (#2026)". That pull request, merged 2026-08-02, made DTLS opt-in but kept --no-dtls as a deprecated alias that logs a warning. The source at the 4.17.0 tag still has the entry:

/* deprecated: */ {"no-dtls", optional_argument, NULL, NO_DTLS_OPT},

At the 4.18.0 tag the line is gone. The commit that removed it is coturn/coturn#2079, merged 2026-09-07 22:43, whose title and ChangeLog entry read "Update help with missing arguments". The body says what it does: "Remove the deprecated --no-dtls and --no-cli turnserver flags — both were already no-ops". Thirteen hours later Docker Hub's latest moved to the image built from it. (The PR says it removed --no-cli too, but its diff only drops that flag from the help text and the test scripts, not from the option table; the 4.18.0 binary still accepts it and logs an ERROR line about it. More on that below.)

When did it start?

Prometheus on this cluster keeps 30 days. The oldest sample for the March pod's restart counter is from 10-12 11:41, and it already reads 4,014. From there the counter climbs by 280 or 281 a day, 284 or 285 on two days, until the eviction on 11-08. Condensed from a one-day-step range query:

kube_pod_container_status_restarts_total{pod="dam-coturn-...-jgp4h"}
10-12   4,014
10-13   4,294     +280
10-14   4,574     +280
...
11-07  11,314
11-08  11,594
11-09  11,621     evicted 11-08 13:46; counter frozen

280 a day is one cycle every 308 s (86,400 ÷ 308 = 280.5): the five-minute back-off cap, plus about eight seconds for the kubelet to check the tag with the registry, create the container, watch it exit within a second, and notice. If the slope was the same before the retention window and the counter was near zero before that, 4,014 restarts take 14.3 days, which puts the first failure at about 2026-09-28, three weeks after latest changed. A few restarts could predate it; the date is an estimate. The hourly series in the chart below starts an hour later and reads 4,018; its last point for the current pod is 786, the last scrape before kubectl read 794.

I cannot say why the container restarted on that day after running since 2026-03-20 with, as far as the counter arithmetic shows, no restarts before then. The node has rebooted since (current boot 11-10) and nothing older than 30 days is in Prometheus. One restart was enough: with imagePullPolicy: Always the kubelet asks the registry what the tag points to on every container start, restarts included, and downloads only if the digest changed. The events for the current pod show it, Pulling image "coturn/coturn:latest" 351 times in 29 hours, one per restart. From 09-08 on the answer to that pull was a binary that would not accept the Deployment's arguments.

There was nothing to roll back to, either. The ReplicaSet is the one from March, so no imageID for the March container exists anywhere, and Prometheus does not keep container info that long. Which version had been running for six months is unknown.

The fix

One commit, which Argo CD applied within the minute:

spec:
  strategy:
    type: Recreate
  template:
    spec:
      containers:
        - name: coturn
          image: coturn/coturn:4.18.0@sha256:bbefd3e1fdfd...
          imagePullPolicy: IfNotPresent
          args:
            # --no-dtls removed
            - --no-multicast-peers
            - --denied-peer-ip=10.0.0.0-10.255.255.255
            - --denied-peer-ip=172.16.0.0-172.31.255.255
            - --denied-peer-ip=192.168.0.0-192.168.255.255
            - --denied-peer-ip=127.0.0.0-127.255.255.255

The pin is to the digest that was already on the node, so nothing new had to be pulled. The denied-peer-ip ranges were overdue regardless: this relay was coming back onto the internet after six weeks off, and a TURN server that will relay to private addresses, including the relay's own LAN, is a way into the network behind it.

Recreate is there because of the host port. This Deployment uses hostNetwork and declares hostPort: 3478 for UDP and TCP, and the pod is pinned to one node by a nodeSelector. I did not try a RollingUpdate, but the arithmetic says it would not have finished: with one replica, maxUnavailable: 25% rounds down to 0 and maxSurge: 25% rounds up to 1, so the controller has to create the new pod before it may delete the old one; the scheduler will not place a second pod that needs port 3478 on a node where a pod already holds it, crashing or not, and there is no other node it may use. The new pod would sit in Pending and the old one would keep crashing. Recreate deletes first. Every future rollout will have a gap of about 15 s (measured on the rollouts today); with one replica there was never going to be a rollout without one.

$ kubectl -n dam-prod get events --sort-by=.lastTimestamp | grep coturn   (condensed)
Scaled down replica set dam-coturn-549854dfcb from 1 to 0
Container image "coturn/coturn:4.18.0@sha256:bbefd3e1fdfd..." already present on machine
Started container coturn
Scaled up replica set dam-coturn-6f99fd45d5 from 0 to 1

$ kubectl -n dam-prod logs deploy/dam-coturn | grep -E "Version|Black|Relay address"
INFO Black listing: 10.0.0.0-10.255.255.255
INFO Black listing: 172.16.0.0-172.31.255.255
INFO Black listing: 192.168.0.0-192.168.255.255
INFO Black listing: 127.0.0.0-127.255.255.255
INFO Relay address to use: 192.168.0.101
INFO Coturn Version Coturn-4.18.0 'Gorst'

A STUN Binding request to port 3478 got no reply at 11:42 and a Binding Success at 11:58:21, both from the LAN and through the router's public address. That checks that the server answers, not that it relays; I did not run an Allocate with the credentials.

Two lines in the new log needed a second commit. ERROR no-cli option is deprecated, see --cli means that flag will go the way --no-dtls went. And WARNING NO EXPLICIT LISTENER ADDRESS(ES) ARE CONFIGURED was followed by the server opening 3478 on every interface the host has, including the cluster's and the VPN's. The second commit dropped --no-cli and added --listening-ip for the one address the relay should bind; the next pod's log lists listeners on that address only. A third added the link-local, 100.64/10 and 0/8 ranges to the denied list, since the VPN gives this node an address in 100.64/10. One line, ERROR CONFIG: Unknown argument: with nothing after the colon, is still there and I have not found what produces it.

Fifteen minutes later neither KubePodCrashLooping nor KubeDeploymentReplicasMismatch was active for the namespace, in Prometheus or in Alertmanager:

Timeline

What I am changing

The alert fired for the whole window. KubePodCrashLooping for this Deployment's pods is in every sample Prometheus has. In Prometheus the firing set on 10-12 and on 11-11 was the same list: five CPUThrottlingHigh, KubeCPUOvercommit, KubeDeploymentReplicasMismatch, two KubeJobFailed, KubePodCrashLooping, KubeQuotaFullyUsed, plus KubeMemoryOvercommit from 11-09. Alertmanager inhibits the throttling and quota ones, so what reached Discord was the overcommit alerts, the job failures and this crash loop, re-sent on the route's repeat interval. The crash-loop alert even arrived as a fresh notification three times, once per pod name, and was still not acted on. The previous incident's alert was new and went unnoticed among these standing ones. This one was one of the standing ones.

Done:

• Image pinned by digest, imagePullPolicy: IfNotPresent, strategy: Recreate for the host-network workload.
• --no-dtls and --no-cli gone; --denied-peer-ip for RFC 1918, loopback, link-local, 100.64/10 and 0/8, --no-multicast-peers, --listening-ip.

Not done yet:

• The other 14 images on this cluster referenced by a mutable tag. Each gets the same treatment: digest, IfNotPresent, and a deliberate upgrade step instead of an accidental one.
• The TURN credentials. The relay still authenticates with a static username and password that are compiled into the web client. The standard answer is --use-auth-secret with short-lived credentials minted by the signaling server, which is an application change, not a manifest change.
• The standing alerts, again. Fix or silence each one so that a firing alert means something; page on available == 0 for anything with a public hostname.
• A startup-log check: grep the first lines of a new container's log for ERROR and deprecated and fail the rollout on a match.

For any Deployment on a mutable tag that has been up for months, record what is actually running before anything restarts it: kubectl get pod -o jsonpath='{.status.containerStatuses[*].imageID}'. The tag in the manifest does not tell you.

Public post-incident reports that start the same way, a container restarting until something upstream of it is changed, are collected under the crash loop symptom in Incident Library; the ones where an upgrade removed something the configuration relied on are under breaking change in an upgrade.

Originally published at benchlet.com.

This is a two-node k3s cluster that I run alone for a few side projects. Times are UTC.

What it looked like

By the morning of 2026-11-11 the Deployment's ReplicaSet had three pods. The original, created in March, had ended on 2026-11-08 at 13:46:42 with exit code 0 and phase Succeeded. A replacement created one second later had never run and ended with ContainerStatusUnknown, exit code 137. A second replacement, created at 16:10:52, was Pending with this event repeating every five minutes:

Failed to pull image "registry.example.com/dam-signaling:prod-<sha>":
  failed to pull and unpack image ...:
  .../manifests/sha256:211614369c9d... : not found

The audit tool I mention below counted 1,565 of these events in 24 hours on that node. On the registry side, the server log had 1,068 requests for that manifest digest since 2026-11-09 23:58, every one answered 404 manifest unknown.

The usual causes of manifest unknown did not fit. The tag existed in the registry's filesystem. The same node had pulled other images from the same registry that afternoon. The Deployment had not changed since March.

Finding it

I found the outage on 2026-11-11 at 05:08, from a read-only audit of the cluster (kubegap) that I had run for another reason. It listed the Deployment with zero available replicas and the pull failures next to it. The investigation then went through the disk, the registry, and the node cache, in that order.

The disk

The node's root filesystem is 106.7 GiB. On 2026-11-08 free space went from 19.4 GiB at 12:45 to 2.6 GiB at 13:48. The node reported DiskPressure from 13:47 to 14:10 and again from 16:10 to 16:20.

node_filesystem_avail_bytes{mountpoint="/"}      GiB, one-minute samples
12:45  19.4
12:58   8.1
13:07  11.8
13:38  11.4
13:48   2.6     DiskPressure 13:47–14:10
13:56  14.4
14:05  27.3
14:18  13.3

I caused this. That afternoon I pushed 25 deploy commits for a new service, and one of them shipped a 4.5 GiB image with the service's ML models inside. The Deployment picked it up at 13:37 and the pod started at 13:40. The kubelet's evictionHard is imagefs.available<5% and nodefs.available<5%; on k3s with a single root filesystem those are the same disk. When a node crosses that threshold the kubelet first reclaims node-level resources, which for an image filesystem means deleting every image no container is using, and then evicts pods if it is still under. The jump from 2.6 GiB to 27.3 GiB free between 13:48 and 14:05 is consistent with that. By 16:15 the node's eviction counters read 23 for imagefs and 27 for nodefs, and 16 pods across the cluster carried the Evicted reason.

The signaling pod is not among those 16. It ended at 13:46:42, one minute before the DiskPressure condition was recorded, with exit code 0. That is not what an eviction looks like, and I could not read the kubelet journal on that node that day, so I do not know what stopped it. The sequence I can show is: the container stopped inside the disk-pressure window; the first replacement died during the second window; the second replacement needed an image that was no longer in the node's store.

The models-in-image problem got the obvious fix the same evening: models on a volume, 1.6 GiB image.

The registry

The pull was failing on a tag that had been pulled without trouble in March. The registry is docker/distribution 2.8.3 with filesystem storage, so I looked at the files:

repositories/dam-signaling/_manifests/tags/prod-<sha>/current/link
  -> sha256:447fa22a...      OCI image index, 2 children

children of the index
  sha256:211614369c9d...     linux/amd64 image manifest   (what containerd asked for)
  sha256:203e8548...         annotation vnd.docker.reference.type: attestation-manifest

repositories/dam-signaling/_manifests/revisions/sha256/
  447fa22a...                present
  211614369c9d...            missing
  203e8548...                missing
  (directory mtime 2026-03-24 04:00:01)

blobs referenced by the image manifest: 10
  present     4   (base layers shared with other images)
  missing     6   (config + 5 layers)

The tag pointed at an image index. The index was there. Both manifests the index pointed at had been unlinked from the repository, and 6 of the 10 blobs behind the platform manifest were gone. The revisions directory had last been written at 04:00:01 on 2026-03-24, the night after the push.

04:00 is when the registry's garbage-collection CronJob runs. From mid-March until 2026-06-02 that job ran registry garbage-collect --delete-untagged=true. I had turned the flag on to reclaim disk. I turned it off after another image broke, and did not go back to check what else the flag had removed.

What happens, in three parts:

• Since buildx 0.10 (January 2023; docker/build-push-action v4 picked it up), a push attaches a provenance attestation by default. Even for one platform the result is an OCI image index with two children, the platform manifest and the attestation manifest. The tag points at the index. The children have no tag of their own.
• distribution 2.x's garbage collector with --delete-untagged enumerates manifests, keeps those a tag resolves to, and deletes the rest. It does not descend into an index. It therefore kept the index, deleted both children, and then deleted every blob that only the children referenced.
• Afterwards the tag list is intact and the tag resolves to the index. A client that follows the index to the platform manifest gets manifest unknown.

This was reported as distribution/distribution#3004 in 2019 and again as #3178 in 2020. The fix, #4285, was merged in April 2024 and is only in the 3.x releases. 2.8.3, the last 2.x release, predates it.

The node cache

containerd on the node had the image, and with the default imagePullPolicy: IfNotPresent for a non-latest tag nothing asks the registry for an image that is already present. The pod restarted five times between March and November, most recently on 2026-10-28, each time from the local store. The first client to ask the registry for the platform manifest after March was the replacement pod on 2026-11-08, after the cached copy had been removed.

Knowing what to look for, I checked every image tag referenced by a running workload against the registry filesystem:

images referenced by workloads                            40
complete                                                  34
broken (index kept, children unlinked, blobs missing)      6

All six had been pushed in March 2026 while the flag was on: four on the 14th between 19:02 and 19:14, two on the 23rd. Every image pushed after May was either a single manifest or an intact index. Five of the six broken ones were still running from the node's cache, with no pullable copy anywhere.

The fix

Rebuild from the same commit with two lines added to the workflow:

- uses: docker/build-push-action@v6
  with:
    provenance: false
    platforms: linux/amd64

With no attestation and one platform, buildx pushes a single application/vnd.oci.image.manifest.v1+json, and there is no index child for this collector to orphan. The new tag resolved to that manifest with every referenced blob present. The pod was ready at 07:41:37 on 2026-11-11.

Then the other five. Four of those repositories had CI that pushed to GitHub's container registry and edited manifest paths that no longer existed, so whatever had put their images into our registry in March, it was not the workflow in the repo. Three of the four built successfully and failed at the deploy step before I noticed. I pointed the four workflows at the real registry and the real manifest paths, rebuilt, and by 09:20 all six workloads were running from complete single-manifest images.

Timeline

What I am changing

The alert rule existed and fired. KubeDeploymentReplicasMismatch went pending at 14:00 and firing at 14:30 on 2026-11-08 and stayed firing until the fix. Alertmanager's route sends everything above info to a Slack channel and repeats every four hours. I have not gone back through the channel history, and I did not act on it. One week earlier, on 2026-11-01, that channel already had nine alerts firing at the same time, including a TURN server that the audit dates as crash-looping since September. The rule was there. The channel was not being read.

Done:

• provenance: false and platforms: linux/amd64 in the six workflows that push to this registry.
• Models out of the image and onto a volume.
• Four CI pipelines reconnected to the manifests they are meant to update.

Not done yet:

• The remaining workflows in the organisation, or upgrade the registry to distribution 3.x, which handles index children in GC.
• A daily job that walks every image tag referenced by a workload: resolve the tag, fetch the index, fetch each child, HEAD each blob. On a broken image the tag list and the index both look normal, so the check has to go down to the blobs.
• Fix or silence the nine standing alerts. A page, not a Slack line, for available == 0 lasting more than ten minutes on anything with a public hostname.
• ephemeral-storage requests on the one workload allowed to be large, and an alert on node disk under 15 %.
• The registry's manifests under the same GitOps reconciliation as everything else. They are in git, but nothing applies them: the live CronJob was patched by hand on 2 June, two days after the commit that removed the flag.

If you run, or have ever run, docker/distribution 2.x with --delete-untagged, check every multi-manifest image pushed in that window by asking for the platform manifest, not the tag. A running pod only proves that the node has the image.

Public post-incident reports where pods could not come back because the registry could not serve their image are collected under the image pull failures symptom in Incident Library, searchable by the error text you saw.

Originally published at benchlet.com.