Disk Space — Diagnose & Reclaim

Nodes fill up. This page is the order of operations: measure first, clean second, tune third — so the same nodes do not fill up again next month.

Measure with the API, not with `oc debug` loops

oc debug writes its progress banner to stderr while your command writes to stdout. In a for loop the two streams interleave and rows get misattributed to the wrong node. Always redirect: oc debug node/$N -- ... 2>/dev/null, or use the kubelet stats/summary endpoint (§1), which needs no debug pod at all.

---

1. Quick assessment

1.1 Fleet-wide df equivalent

No debug pods, ~1 s per node. This is the measurement to trust.

{ echo "NODE SIZE USED AVAIL USE% IMAGEFS"
  for n in $(oc get nodes -o name | cut -d/ -f2); do
    oc get --raw /api/v1/nodes/$n/proxy/stats/summary 2>/dev/null \
    | jq -r --arg n "$n" '.node | [
        $n,
        "\(.fs.capacityBytes/1073741824|floor)G",
        "\(.fs.usedBytes/1073741824|floor)G",
        "\(.fs.availableBytes/1073741824|floor)G",
        "\(100-(.fs.availableBytes/.fs.capacityBytes*100)|floor)%",
        "\((.runtime.imageFs.usedBytes//0)/1073741824|floor)G"
      ] | @tsv'
  done | sort -k5 -rn
} | column -t

Tip

A node missing from the output means its kubelet is not answering on :10250. That is a bigger problem than disk usage — check oc get nodes and systemctl status kubelet before anything else.

1.2 Single node, raw numbers

The form used in the Red Hat solution article, useful when you only have curl:

node=<NodeName>
curl -k -s -H "Authorization: Bearer $(oc whoami -t)" \
  "$(oc whoami --show-server)/api/v1/nodes/${node}/proxy/stats/summary" \
| jq -r '"Used:\(.node.fs.usedBytes) Capacity:\(.node.fs.capacityBytes) Available:\(.node.fs.availableBytes)", "imagefs:\(.node.runtime.imageFs.usedBytes)"'

1.3 Which pods consume ephemeral storage

oc get --raw /api/v1/nodes/${node}/proxy/stats/summary \
| jq -r '.pods[] | "\(.["ephemeral-storage"].usedBytes) \(.podRef.namespace)/\(.podRef.name)"' \
| sort -nr | head -20

In GB, including container log usage:

oc get --raw /api/v1/nodes/${node}/proxy/stats/summary \
| jq -r '.pods[] | [
    ((((."ephemeral-storage".usedBytes)//0) + ((.containers//[]|map(.logs.usedBytes//0)|add)//0))/1048576|floor),
    .podRef.namespace, .podRef.name
  ] | @tsv' \
| sort -rn | head -15 | column -t

1.4 Seven-day peak (did it already cycle?)

A node sitting at 40% today may have hit 90% last night and been rescued by image GC. Prometheus knows; df does not.

oc -n openshift-monitoring exec -c prometheus sts/prometheus-k8s -- \
  curl -sG --data-urlencode \
  'query=max_over_time((100 - node_filesystem_avail_bytes{mountpoint="/var"} / node_filesystem_size_bytes{mountpoint="/var"} * 100)[7d:1h])' \
  http://localhost:9090/api/v1/query \
| jq -r '.data.result[] | "\(.metric.instance) peak_7d=\(.value[1]|tonumber|floor)%"' | sort -k2 -rn

Projection — negative values mean the node runs out within the window:

oc -n openshift-monitoring exec -c prometheus sts/prometheus-k8s -- \
  curl -sG --data-urlencode \
  'query=predict_linear(node_filesystem_avail_bytes{mountpoint="/var"}[24h], 14*24*3600) / 1073741824' \
  http://localhost:9090/api/v1/query \
| jq -r '.data.result[] | "\(.metric.instance) in_14d=\(.value[1]|tonumber|floor)GB"' | sort -k2 -n

---

2. Where the space actually goes

Path	Typical content	Reclaimed by
/var/lib/containers/storage	image layers + container rootfs	image GC, crictl rmi
/var/lib/kubelet/pods	emptyDir, volume subpaths	pod deletion, ephemeral-storage limits
/var/log/pods	container stdout/stderr	containerLogMaxSize / containerLogMaxFiles
/var/log/journal	systemd journal	journalctl --vacuum-*
/var/lib/systemd/coredump	crash dumps	manual deletion
/var/lib/etcd	etcd DB (control plane)	etcdctl defrag
/sysroot	ostree deployments	rpm-ostree cleanup -bm

Reading the arithmetic

If du -sx /var/lib is much smaller than df reports as used, the difference is deleted-but-open files — space held by a running process. See §3.4.

---

3. Structured inventory

3.1 One table for the whole cluster

One debug pod per node instead of five. Slow on large filesystems (the du walk dominates).

{ echo "NODE ROLE VAR% VARLIB_G CONTAINERS_G KUBELET_G LOG_G IMGS CTRS"
  for n in $(oc get nodes -o name | cut -d/ -f2); do
    oc get node $n -o jsonpath='{.metadata.labels}' | grep -q 'node-role.kubernetes.io/master' && r=master || r=worker
    v=$(oc debug node/$n -- chroot /host bash -c '
      p=$(df --output=pcent /var | tail -1 | tr -dc 0-9)
      a=$(du -sx --block-size=1G /var/lib 2>/dev/null | cut -f1)
      b=$(du -sx --block-size=1G /var/lib/containers 2>/dev/null | cut -f1)
      k=$(du -sx --block-size=1G /var/lib/kubelet 2>/dev/null | cut -f1)
      l=$(du -sx --block-size=1G /var/log 2>/dev/null | cut -f1)
      i=$(crictl images -q 2>/dev/null | wc -l)
      c=$(crictl ps -a -q 2>/dev/null | wc -l)
      echo "$p $a $b $k $l $i $c"' 2>/dev/null)
    echo "$n $r $v"
  done
} | column -t

3.2 Breakdown per role

Control plane

MASTERS=$(oc get nodes -l node-role.kubernetes.io/master -o name | cut -d/ -f2)

for n in $MASTERS; do
  echo "===== $n"
  oc debug node/$n -- chroot /host du -xh --max-depth=1 /var/lib 2>/dev/null | sort -h | tail -8
done

Workers

WORKERS=$(oc get nodes -l '!node-role.kubernetes.io/master' -o name | cut -d/ -f2)

for n in $WORKERS; do
  echo "===== $n"
  oc debug node/$n -- chroot /host du -xh --max-depth=1 /var/lib 2>/dev/null | sort -h | tail -8
done

Drill into the container store — overlay is image layers, overlay-containers is writable container layers:

oc debug node/$N -- chroot /host du -xh --max-depth=1 /var/lib/containers/storage 2>/dev/null | sort -h | tail -6

3.3 Image accounting

Largest images (jq runs on your workstation, not on RHCOS):

oc debug node/$N -- chroot /host crictl images -o json 2>/dev/null \
| jq -r '.images[] | [((.size|tonumber)/1048576|floor), (.repoTags[0] // .repoDigests[0] // .id)] | @tsv' \
| sort -rn | head -20 | column -t

Aggregated per repository — more than 3–4 hits for the same repo means successive versions never removed:

oc debug node/$N -- chroot /host crictl images -o json 2>/dev/null \
| jq -r '.images[] | (.repoTags[0] // .repoDigests[0] // "<none>") | sub("@sha256:.*$";"") | sub(":[^:/]+$";"")' \
| sort | uniq -c | sort -rn | head -20

Images present on only one control-plane node — almost always leftovers from old rollouts:

for n in $MASTERS; do
  oc debug node/$n -- chroot /host crictl images -o json 2>/dev/null | jq -r '.images[].repoTags[0] // "<none>"'
done | sort | uniq -c | sort -n | awk '$1==1' | head -30

Total image bytes per node — compare against CONTAINERS_G from §3.1; a large gap means writable layers or orphaned storage:

for n in $MASTERS; do
  echo -n "$n images_total="
  oc debug node/$n -- chroot /host crictl images -o json 2>/dev/null | jq -r '[.images[].size|tonumber]|add/1073741824|floor'
done

3.4 Deleted-but-open files

for n in $MASTERS; do
  echo -n "$n deleted_fds="
  oc debug node/$n -- chroot /host bash -c 'ls -l /proc/*/fd 2>/dev/null | grep -c deleted' 2>/dev/null
done

Large offenders on one node:

oc debug node/$N -- chroot /host bash -c '
for p in /proc/[0-9]*; do
  for f in $p/fd/*; do
    t=$(readlink $f 2>/dev/null)
    case "$t" in *deleted*)
      s=$(stat -Lc %s $f 2>/dev/null)
      [ "${s:-0}" -gt 104857600 ] && echo "$((s/1048576))MB $(cat $p/comm) $t";;
    esac
  done
done 2>/dev/null | sort -rn | head' 2>/dev/null

Only a restart of the holding process releases that space — usually oc delete pod.

---

4. Manual cleanup on a node

Per the Red Hat solution Disk pressure alarm remains on a drained node: when a node reaches 100%, draining it does not release the space. Stale CRI-O objects survive the drain and must be removed by hand, after which garbage collection has to be configured so it does not recur.

1. Remove exited containers

   oc debug node/$N -- chroot /host bash -c 'crictl rm $(crictl ps -q --state Exited)' 2>/dev/null

2. Remove pods in NotReady state

   oc debug node/$N -- chroot /host bash -c 'crictl rmp $(crictl pods -q -s NotReady)' 2>/dev/null

3. Remove unused images

   oc debug node/$N -- chroot /host crictl rmi --prune 2>/dev/null

   Caution

   crictl rmi -q (as written in the KCS) removes all images not currently held by a container, including ones you will pull again minutes later. crictl rmi --prune is the conservative form and is what you want on a running cluster.

4. Journal, coredumps, ostree rollback

   oc debug node/$N -- chroot /host journalctl --disk-usage 2>/dev/null
   oc debug node/$N -- chroot /host journalctl --vacuum-size=500M 2>/dev/null
   oc debug node/$N -- chroot /host find /var/lib/systemd/coredump -type f -delete 2>/dev/null
   oc debug node/$N -- chroot /host rpm-ostree cleanup -bm 2>/dev/null

5. Verify

   oc debug node/$N -- chroot /host df -h /var 2>/dev/null

Cleanup with before/after measurement

for n in $MASTERS; do
  echo "===== $n"
  oc debug node/$n -- chroot /host df -h --output=used,avail,pcent /var 2>/dev/null | tail -1
  oc debug node/$n -- chroot /host crictl rmi --prune 2>/dev/null | tail -2
  oc debug node/$n -- chroot /host df -h --output=used,avail,pcent /var 2>/dev/null | tail -1
  oc wait --for=condition=Ready node/$n --timeout=120s
done

Control plane

Never act on two control-plane nodes at once. The oc wait between iterations is not optional.

---

Fleet-wide safe prune

Iterates every schedulable node, prunes unused images with crictl rmi --prune, and reports the reclaimed space via the kubelet stats/summary endpoint. Control plane is processed one node at a time with oc wait between iterations. Cordoned or NotReady nodes are skipped, not silently retried.

Terminal window

# ------------------------------------------------------------------
# Scope selection — pick ONE of the four
# ------------------------------------------------------------------
NODES=$(oc get nodes -o name | cut -d/ -f2)                                              # all
# NODES=$(oc get nodes -l node-role.kubernetes.io/worker -o name | cut -d/ -f2)          # workers only
# NODES=$(oc get nodes -l node-role.kubernetes.io/master -o name | cut -d/ -f2)          # masters only
# NODES=$(oc get nodes -l node-role.kubernetes.io/infra   -o name | cut -d/ -f2)         # infra only

# Optional: only nodes above N% usage — cheap pre-filter
THRESHOLD=70
NODES=$(for n in $NODES; do
  u=$(oc get --raw /api/v1/nodes/$n/proxy/stats/summary 2>/dev/null \
      | jq -r '.node | (100 - (.fs.availableBytes/.fs.capacityBytes*100)) | floor')
  [ -n "$u" ] && [ "$u" -ge "$THRESHOLD" ] && echo "$n"
done)

echo "NODE USED_BEFORE AVAIL_BEFORE USED_AFTER AVAIL_AFTER FREED_GB STATE"

for n in $NODES; do
  # Skip NotReady
  ready=$(oc get node "$n" -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}')
  if [ "$ready" != "True" ]; then
    echo "$n - - - - - SKIP_NOT_READY"
    continue
  fi

  # Skip cordoned
  sched=$(oc get node "$n" -o jsonpath='{.spec.unschedulable}')
  if [ "$sched" = "true" ]; then
    echo "$n - - - - - SKIP_CORDONED"
    continue
  fi

  # Detect role (used only to decide the wait strategy)
  role=$(oc get node "$n" -o jsonpath='{.metadata.labels}' \
          | grep -q '"node-role.kubernetes.io/master"' && echo master || echo worker)

  # BEFORE — via kubelet, no debug pod
  read used_b avail_b <<<$(oc get --raw /api/v1/nodes/$n/proxy/stats/summary 2>/dev/null \
    | jq -r '.node.fs | "\(.usedBytes/1073741824|floor) \(.availableBytes/1073741824|floor)"')

  # Prune — --prune, never -q, redirect stderr to keep the table clean
  oc debug node/$n -- chroot /host crictl rmi --prune >/dev/null 2>&1

  # For masters, wait Ready between iterations. Never touch two control-plane nodes at once.
  if [ "$role" = "master" ]; then
    oc wait --for=condition=Ready node/$n --timeout=180s >/dev/null 2>&1
  fi

  # AFTER
  read used_a avail_a <<<$(oc get --raw /api/v1/nodes/$n/proxy/stats/summary 2>/dev/null \
    | jq -r '.node.fs | "\(.usedBytes/1073741824|floor) \(.availableBytes/1073741824|floor)"')

  freed=$(( used_b - used_a ))
  echo "$n ${used_b}G ${avail_b}G ${used_a}G ${avail_a}G ${freed}G OK"
done | column -t

Control plane

Do not remove the oc wait inside the master branch, and do not parallelise this loop. crictl rmi --prune on a master triggers a rebuild of image references that briefly loads the kubelet — running it concurrently on two masters can push the API server into a partial outage.

Full sweep

For a heavier cleanup, replace the single crictl rmi --prune line with the four-step block from §4 (exited containers, NotReady pods, image prune, journal + coredump + ostree). Keep the oc wait gate on masters.

5. Garbage collection (the supported fix)

Configuring nodes by logging into them is not supported. Node behaviour is changed through a KubeletConfig object per machine config pool — and OpenShift supports only one KubeletConfig per pool, so edit the existing one rather than adding a second.

5.1 Read the effective configuration

oc get --raw /api/v1/nodes/$N/proxy/configz \
| jq '.kubeletconfig | {imageMinimumGCAge, imageGCHighThresholdPercent, imageGCLowThresholdPercent,
                        evictionHard, evictionSoft, containerLogMaxSize, containerLogMaxFiles}'

oc get kubeletconfig
oc get machineconfigpool

Defaults worth knowing: image GC starts at 85% and frees down to 80%; imageMinimumGCAge is 2m; evictionHard.imagefs.available is 15%. A node parked at 82% is below every threshold and will accumulate forever.

5.2 Pool selector

Current releases use the built-in pool label — no custom-kubelet label is needed:

oc get machineconfigpool worker -o jsonpath='{.metadata.labels}' | jq
# pools.operator.machineconfiguration.openshift.io/worker: ""

Older KCS articles

Solutions written for OCP 4.6–4.10 tell you to run oc label machineconfigpool worker custom-kubelet=small-pods first. On current versions that step is unnecessary — match on pools.operator.machineconfiguration.openshift.io/<pool> instead. Adding the label is harmless but triggers an extra pool reconciliation.

5.3 The CR

Workers

apiVersion: machineconfiguration.openshift.io/v1
kind: KubeletConfig
metadata:
  name: worker-gc-tuning
spec:
  machineConfigPoolSelector:
    matchLabels:
      pools.operator.machineconfiguration.openshift.io/worker: ""
  kubeletConfig:
    imageMinimumGCAge: 5m
    imageGCHighThresholdPercent: 75
    imageGCLowThresholdPercent: 65
    evictionSoft:
      memory.available: "500Mi"
      nodefs.available: "10%"
      nodefs.inodesFree: "5%"
      imagefs.available: "15%"
      imagefs.inodesFree: "10%"
    evictionSoftGracePeriod:
      memory.available: "1m30s"
      nodefs.available: "1m30s"
      nodefs.inodesFree: "1m30s"
      imagefs.available: "1m30s"
      imagefs.inodesFree: "1m30s"
    evictionHard:
      memory.available: "200Mi"
      nodefs.available: "5%"
      nodefs.inodesFree: "4%"
      imagefs.available: "10%"
      imagefs.inodesFree: "5%"
    evictionPressureTransitionPeriod: 3m
    containerLogMaxSize: 50Mi
    containerLogMaxFiles: 3

Control plane

Control-plane nodes are often provisioned with a smaller disk than workers while running the same platform images. Lower thresholds there.

apiVersion: machineconfiguration.openshift.io/v1
kind: KubeletConfig
metadata:
  name: master-gc-tuning
spec:
  machineConfigPoolSelector:
    matchLabels:
      pools.operator.machineconfiguration.openshift.io/master: ""
  kubeletConfig:
    imageMinimumGCAge: 10m
    imageGCHighThresholdPercent: 65
    imageGCLowThresholdPercent: 55

oc create -f garbage-collector.yaml
oc get machineconfigpool          # UPDATING=True until the roll completes
oc get mcp worker -w

This reboots nodes

Any KubeletConfig change rolls the whole pool, one node at a time. On the master pool that is a rolling control-plane reboot. Schedule it, and never run it concurrently with manual crictl cleanup.

Constraint: imageGCLowThresholdPercent must be lower than imageGCHighThresholdPercent, or the MCO rejects the object.

5.4 imageMaximumGCAge

From Kubernetes 1.30 (OCP 4.17+) the kubelet can remove images unused for longer than a given duration regardless of disk pressure. The tracked age resets when the kubelet restarts. Verify the field is accepted by the MCO on your release before relying on it.

    imageMaximumGCAge: 168h

---

6. Pruning cluster objects

Garbage collection cleans nodes. Pruning cleans etcd and the integrated registry. Different problems.

Every oc adm prune command is a dry run until you add --confirm.

# deployments (ReplicationControllers owned by DeploymentConfig)
oc adm prune deployments --orphans --keep-complete=5 --keep-failed=1 --keep-younger-than=60m
oc adm prune deployments --orphans --keep-complete=5 --keep-failed=1 --keep-younger-than=60m --confirm

# builds (build.openshift.io only — NOT Tekton)
oc adm prune builds --orphans --keep-complete=5 --keep-failed=1 --keep-younger-than=60m
oc adm prune builds --orphans --keep-complete=5 --keep-failed=1 --keep-younger-than=60m --confirm

# images
oc adm prune images --keep-tag-revisions=3 --keep-younger-than=60m
oc adm prune images --keep-tag-revisions=3 --keep-younger-than=60m --confirm

`error: you must use a client config with a token`

oc adm prune images refuses cert-based kubeconfig. Authenticate with a token bound to system:image-pruner:

oc create serviceaccount image-pruner -n openshift-image-registry
oc adm policy add-cluster-role-to-user system:image-pruner -z image-pruner -n openshift-image-registry
oc login --token=$(oc create token image-pruner -n openshift-image-registry --duration=2h) \
         --server=https://api.<cluster>.<domain>:6443

It also needs a route to the registry, which does not exist by default:

oc patch configs.imageregistry.operator.openshift.io/cluster --type=merge -p '{"spec":{"defaultRoute":true}}'

And afterwards the registry cache must be invalidated, or images referencing pruned layers stay broken:

oc rollout restart deployment/image-registry -n openshift-image-registry

Automatic pruning is driven by the ImagePruner singleton:

oc get imagepruner.imageregistry.operator.openshift.io/cluster -o yaml
oc patch imagepruner.imageregistry.operator.openshift.io/cluster --type=merge -p '{"spec":{
  "suspend": false, "schedule": "0 2 * * *",
  "keepTagRevisions": 3, "keepYoungerThanDuration": "60m"}}'

If the Image Registry Operator managementState is Removed, the pruner job runs with --prune-registry=false and touches only etcd metadata — registry storage is left alone.

Tekton / OpenShift Pipelines

oc adm prune builds does not see PipelineRun or TaskRun. Use the operator’s own pruner:

oc get tektonconfig config -o jsonpath='{.spec.pruner}' | jq

oc patch tektonconfig config --type=merge -p '{"spec":{"pruner":{
  "disabled": false, "schedule": "0 2 * * *",
  "resources": ["pipelinerun","taskrun"],
  "keep": 20, "keep-since": 10080}}}'

keep-since is in minutes. Listing pipelinerun only leaves standalone TaskRun objects untouched forever. Per-namespace overrides live in annotations — operator.tekton.dev/prune.keep, .keep-since, .schedule, .skip:

oc get ns -o json | jq -r '.items[]
  | select(.metadata.annotations // {} | keys[] | startswith("operator.tekton.dev/prune"))
  | .metadata.name'

---

7. Control plane: etcd

E=$(oc -n openshift-etcd get pods -l app=etcd -o name | head -1 | cut -d/ -f2)
oc -n openshift-etcd rsh -c etcdctl $E etcdctl endpoint status -w table --cluster
oc -n openshift-etcd rsh -c etcdctl $E etcdctl alarm list

Defrag only when DB SIZE is genuinely large and much bigger than the in-use size. One member at a time, checking health in between:

oc -n openshift-etcd rsh -c etcdctl $E etcdctl defrag --command-timeout=60s --cluster
oc -n openshift-etcd rsh -c etcdctl $E etcdctl endpoint health --cluster
oc -n openshift-etcd rsh -c etcdctl $E etcdctl alarm disarm

Note

A few hundred MB of etcd is normal and is not a disk-space problem. On a full control-plane node, check /var/lib/containers before blaming etcd — platform images accumulate across upgrades and dwarf the database.

---

8. Returning space to the hypervisor

fstrim frees nothing inside the guest — df will not change. It issues TRIM/UNMAP so the datastore can reclaim thin-provisioned blocks. Run it after image cleanup.

oc debug node/$N -- chroot /host fstrim -av --dry-run 2>/dev/null
oc debug node/$N -- chroot /host systemctl status fstrim.timer 2>/dev/null

for n in $WORKERS; do
  echo "== $n"
  oc debug node/$n -- chroot /host fstrim -av 2>/dev/null
  sleep 60
done

Requires thin-provisioned VMDK on VMFS6 or vSAN. On VMFS5 guest UNMAP is not propagated. The first run on a large never-trimmed volume produces an I/O burst — do it off-peak.

---

9. Troubleshooting

Symptom	Cause	Fix
Node missing from stats/summary loop	kubelet down on :10250	systemctl status kubelet, journal
df used ≫ du -sx /var	deleted-but-open files	§3.4, restart the holding pod
Drain did not free anything	stale CRI-O objects survive the drain	§4
Node stuck just below 85% forever	GC threshold never reached	§5, lower imageGCHighThresholdPercent
you must use a client config with a token	cert-based kubeconfig	§6, SA token
Images broken after prune	registry metadata cache not invalidated	oc rollout restart deployment/image-registry
MCO rejects the KubeletConfig	low ≥ high threshold, or a second CR on the pool	one CR per pool, low < high
Rows misattributed in a node loop	stderr/stdout interleaving	2>/dev/null on oc debug

---

10. Runbook order

1. Measure — §1.1 and §1.4. Do not act on a df loop.
2. Locate — §3.1, then §3.2 on the worst nodes.
3. Relieve — §4 on the affected nodes, control plane one at a time.
4. Prune — §6, dry run first.
5. Tune — §5, scheduled, pool by pool.
6. Reclaim at the hypervisor — §8.
7. Re-measure — §1.1.

---

References

Pruning objects to reclaim resources OCP 4.18 — Building applications, ch. 13

Freeing node resources using garbage collection OCP 4.18 — Nodes, working with nodes

Disk pressure alarm remains on a drained node Red Hat Solution — cri-o / RHCOS

Image Registry Operator Creating the registry route required by image pruning