Disk Space — Diagnose & Reclaim

Measure → find the consumer → reclaim → fix the cause. Prefer the kubelet stats/summary API: no debug pods, no stderr noise.

node=<node-name>

---

1. Measure

Capacity vs allocatable

{ echo "NODE CAPACITY ALLOCATABLE RESERVED"; oc get nodes -o json | jq -r 'def gib: if test("Ki$") then (sub("Ki$";"")|tonumber/1048576) else (tonumber/1073741824) end; .items[] | (.status.capacity["ephemeral-storage"]|gib) as $c | (.status.allocatable["ephemeral-storage"]|gib) as $a | [.metadata.name, "\($c|floor)G", "\($a|floor)G", "\(($c-$a)|floor)G"] | @tsv'; } | column -t

Quick form, raw units (capacity in Ki, allocatable in bytes):

oc get nodes -o custom-columns='NODE:.metadata.name,CAPACITY:.status.capacity.ephemeral-storage,ALLOCATABLE:.status.allocatable.ephemeral-storage'

Real usage per node

{ echo "NODE SIZE USED AVAIL USE%"; for n in $(oc get nodes -o name | cut -d/ -f2); do oc get --raw /api/v1/nodes/$n/proxy/stats/summary | jq -r --arg n "$n" '.node.fs | [$n, "\(.capacityBytes/1073741824|floor)G", "\(.usedBytes/1073741824|floor)G", "\(.availableBytes/1073741824|floor)G", "\(100-(.availableBytes/.capacityBytes*100)|floor)%"] | @tsv'; done | sort -k5 -rn; } | column -t

Sum of pod usage per node

If node used ≫ this sum, the space is outside pods: images, journal, orphaned files.

{ echo "NODE PODS_EPHEMERAL"; for n in $(oc get nodes -o name | cut -d/ -f2); do oc get --raw /api/v1/nodes/$n/proxy/stats/summary | jq -r --arg n "$n" '[$n, "\(([.pods[]["ephemeral-storage"].usedBytes // 0] | add)/1073741824|floor)G"] | @tsv'; done | sort -k2 -rn; } | column -t

Ephemeral requests/limits per node

oc describe node ${node} | sed -n '/Allocated resources/,/Events/p'

Note

Requests near zero mean the scheduler is blind to ephemeral storage: heavy emptyDir pods land on already-full nodes.

---

2. Find the consumer

Top pods, one node

oc get --raw /api/v1/nodes/${node}/proxy/stats/summary | jq -r '.pods[] | "\((.["ephemeral-storage"].usedBytes // 0)/1073741824|floor)G \(.podRef.namespace)/\(.podRef.name)"' | sort -rn | head -20

Top pods, whole cluster

for n in $(oc get nodes -o name | cut -d/ -f2); do oc get --raw /api/v1/nodes/$n/proxy/stats/summary | jq -r --arg n "$n" '.pods[] | "\((.["ephemeral-storage"].usedBytes // 0)/1073741824|floor)G \($n) \(.podRef.namespace)/\(.podRef.name)"'; done | sort -rn | head -30

Outside pods: on the node

oc debug node/${node} -q -- chroot /host sh -c 'du -sh /var/lib/containers/storage /var/lib/kubelet/pods /var/log/pods /var/log/journal 2>/dev/null; df -h /var | tail -1'

---

3. Evictions

Who got evicted, where, why

oc get pods -A --field-selector=status.phase=Failed -o json | jq -r '.items[] | select(.status.reason=="Evicted") | [.spec.nodeName, .metadata.namespace+"/"+.metadata.name, .status.message] | @tsv' | column -t -s$'\t'

Effective kubelet thresholds

oc get --raw /api/v1/nodes/${node}/proxy/configz | jq '.kubeletconfig | {evictionHard, evictionSoft, systemReserved, imageGCHighThresholdPercent, imageGCLowThresholdPercent}'

Reading the numbers

Message: Threshold quantity: 24068434671, available: 23143328Ki on a 149G node.

Value	Meaning
24068434671 B ≈ 22.4 GiB	15% of 149G → imagefs.available<15%
Allocatable 133G	capacity − nodefs.available 10% − system-reserved
Eviction point	~85% used ≈ 126.6G
Image GC start	imageGCHighThresholdPercent: 85 → same point as eviction

Caution

When imagefs and nodefs share /var (IMAGEFS = 0 in stats/summary), the 15% imagefs threshold wins. Default image GC starts at 85% — too late: the node evicts before it cleans.

---

4. Reclaim

Single node

oc debug node/${node} -q -- chroot /host crictl rmi --prune

All workers, before/after

--prune removes only images not referenced by any container (running or exited). Safe.

for n in $(oc get nodes -l node-role.kubernetes.io/worker -o name | cut -d/ -f2); do echo "== $n"; oc debug node/$n -q -- chroot /host sh -c 'df -h /var | tail -1; crictl rmi --prune >/dev/null 2>&1; df -h /var | tail -1' 2>/dev/null; done

Air-gapped: add --image=<mirror>/rhel9/support-tools:latest to oc debug.

Danger

Masters: one at a time, oc wait --for=condition=Ready node/$n in between. Never in parallel.

Journal

oc debug node/${node} -q -- chroot /host journalctl --vacuum-size=500M

Evicted pod objects

oc delete pods -A --field-selector=status.phase=Failed

---

5. Fix the cause

Platform Prometheus on emptyDir

Most common offender: prometheus-k8s-* with no PVC grows until the node evicts.

oc -n openshift-monitoring get pod prometheus-k8s-0 -o jsonpath='{.spec.volumes[?(@.name=="prometheus-k8s-db")]}{"\n"}'

oc -n openshift-user-workload-monitoring get pod prometheus-user-workload-0 -o jsonpath='{.spec.volumes[?(@.name=="prometheus-user-workload-db")]}{"\n"}'

{"emptyDir":{}} → move to PVC. Merge into the existing config.yaml, do not overwrite it:

oc -n openshift-monitoring get cm cluster-monitoring-config -o jsonpath='{.data.config\.yaml}{"\n"}'

oc get sc

apiVersion: v1
kind: ConfigMap
metadata:
  name: cluster-monitoring-config
  namespace: openshift-monitoring
data:
  config.yaml: |
    enableUserWorkload: true
    prometheusK8s:
      retention: 15d
      retentionSize: 80GB
      volumeClaimTemplate:
        metadata:
          name: prometheus-db
        spec:
          storageClassName: <SC>
          resources:
            requests:
              storage: 100Gi

oc -n openshift-monitoring get pvc,pod -l app.kubernetes.io/name=prometheus -w

retentionSize ≈ 80% of the PVC. Switching from emptyDir restarts the pods and drops existing TSDB data.

Other emptyDir consumers

Set resources.limits.ephemeral-storage and emptyDir.sizeLimit: the kubelet evicts the offending pod, not its neighbours.

Image GC before eviction

One KubeletConfig per pool: edit the existing one if present. Rolls the pool (drain + reboot).

oc get kubeletconfig

apiVersion: machineconfiguration.openshift.io/v1
kind: KubeletConfig
metadata:
  name: worker-image-gc
spec:
  machineConfigPoolSelector:
    matchLabels:
      pools.operator.machineconfiguration.openshift.io/worker: ""
  kubeletConfig:
    imageGCHighThresholdPercent: 75
    imageGCLowThresholdPercent: 65

oc get mcp worker -w

---

6. Integrated registry

oc adm prune images cleans registry storage, not node disks. prune deployments/builds removes API objects only.

Run the operator pruner now

oc get configs.imageregistry.operator.openshift.io cluster -o jsonpath='{.spec.managementState}{"\n"}'

oc get imagepruner cluster -o jsonpath='{.spec.suspend} {.spec.schedule} {.spec.keepTagRevisions}{"\n"}'

oc -n openshift-image-registry create job --from=cronjob/image-pruner image-pruner-manual-$(date +%s)

managementState: Removed → nothing to prune.

error: you must use a client config with a token

Cert-based kubeconfig (system:admin) has no token. Use a SA:

oc -n openshift-image-registry create sa image-pruner

oc adm policy add-cluster-role-to-user system:image-pruner -z image-pruner -n openshift-image-registry

oc adm prune images --keep-tag-revisions=3 --keep-younger-than=60m --token=$(oc -n openshift-image-registry create token image-pruner --duration=1h) --confirm

Also needs a reachable registry route from the bastion; prefer the in-cluster job above.

---

7. Troubleshooting

Symptom	Cause	Fix
NotFound on stats/summary	${node} empty	set node=
Evictions at ~85% used	imagefs.available<15% on shared /var	§5 image GC, §5 Prometheus
One/two nodes much fuller than others	single emptyDir pod (Prometheus, scanner-db)	§2, §5
Node used ≫ sum of pods	images, journal, deleted-but-open files	§2 on-node du, §4
client config with a token	cert-based login	§6 SA token or pruner job
MCO rejects KubeletConfig	second CR on pool, or low ≥ high	one CR per pool