Keeping disk pressure under control is a day-to-day chore for cluster admins. Left unchecked, old container images, completed builds, logs and stray volumes will quietly eat through node storage and the internal registry PVC, eventually triggering pod evictions or broken pushes. This post walks through a layered strategy for reclaiming and preventing waste—starting with the integrated image registry and ending with node-level log rotation.

---

1. Map the main storage consumers

1. Internal image registry – images, layers and cache.
2. etcd / API objects – build and deployment history.
3. Node root & runtime partitions – container layers, EmptyDir, logs.
4. Persistent volumes (PVs) – forgotten PVCs, failed databases, etc.

Prometheus + Alertmanager (the default monitoring stack) surfaces percent-full alerts for the registry PVC and node filesystems. Combine that with oc adm top nodes, oc adm top images, and the “Image Manifest V2 blobs cached” dashboard in the web console to decide where to focus first.

---

2. Prune the integrated image registry

How pruning decides what can go

oc adm prune images keeps anything that is:

• Newer than --keep-younger-than OR
• Within the last --keep-tag-revisions tag revisions OR
• Currently referenced by a pod, build, DeploymentConfig, etc.

Anything else—plus unreferenced layers—is queued for deletion.

Manual prune in two lines

# Dry-run
oc adm prune images \
  --keep-tag-revisions=3 \
  --keep-younger-than=60m

# Do it for real
oc adm prune images \
  --keep-tag-revisions=3 \
  --keep-younger-than=60m \
  --confirm

Run the dry-run first; add --confirm only after reviewing the candidate list. After a prune, redeploy the registry to flush its blob-metadata cache:

oc rollout restart deployment/image-registry -n openshift-image-registry

Schedule it automatically (ImagePruner CR)

Out of the box, OpenShift 4+ installs an imagepruner.imageregistry.operator.openshift.io custom resource that runs daily at midnight and keeps 3 tag revisions / 1 h of “fresh” images by default — values you can raise or lower:

spec:
  schedule: "0 3 * * *"           # run at 03:00
  keepTagRevisions: 10
  keepYoungerThanDuration: 24h
  suspend: false

Custom CronJob pruner (full control)

If you want a different cadence, memory/CPU requests, or flags such as --prune-over-size-limit, drop a CronJob similar to the Red Hat example below:

apiVersion: batch/v1
kind: CronJob
metadata:
  name: image-pruner
  namespace: openshift-image-registry
spec:
  schedule: "0 */6 * * *"         # every 6 h
  concurrencyPolicy: Forbid
  jobTemplate:
    spec:
      template:
        spec:
          serviceAccountName: pruner
          restartPolicy: OnFailure
          containers:
          - name: pruner
            image: quay.io/openshift/origin-cli:4.1
            command: ["oc"]
            args:
              - adm
              - prune
              - images
              - --keep-tag-revisions=5
              - --keep-younger-than=96h
              - --confirm

“Hard” prune for orphaned blobs

If disk usage is still high, perform a hard prune (registry garbage-collect). Run it during a maintenance window—it pauses pushes/pulls and can take time on large clusters.

---

3. Trim build and deployment history

Old replication controllers, builds and image streams also bloat etcd and chew through node storage. The same oc adm prune family handles these objects:

| Object type | Typical flags                                                                                            |
| ----------- | -------------------------------------------------------------------------------------------------------- |
| Deployments | `oc adm prune deployments --orphans --keep-complete=5 --keep-failed=1 --keep-younger-than=60m --confirm` |
| Builds      | `oc adm prune builds --orphans --keep-complete=5 --keep-failed=1 --keep-younger-than=60m --confirm`      |

Both support the familiar --keep-* and --keep-younger-than safeguards.

Developers can opt-in to automatic build pruning by adding successfulBuildsHistoryLimit and failedBuildsHistoryLimit in their BuildConfig, off-loading that work from cluster admins.

---

4. Set quotas for ephemeral storage

The kubelet can evict pods that exceed node disk thresholds, but it is better to prevent abuse up front. OpenShift lets you treat local disk like CPU / RAM:

resources:
  requests:
    ephemeral-storage: "2Gi"
  limits:
    ephemeral-storage: "4Gi"

Project-wide ResourceQuotas can cap the sum of all pod requests (requests.ephemeral-storage).

---

5. Rotate node logs with journald

On RHCOS the system journal defaults to 8 GiB and keeps 20 % free space before trimming old entries. Tune these in a MachineConfig (/etc/systemd/journald.conf) if your nodes have smaller disks, or raise them if you run verbose workloads:

SystemMaxUse=4G
SystemKeepFree=30%

Make the change, apply the MachineConfig, and wait for each node to pivot.

---

6. House-keep persistent volumes

• Delete PVCs when an application is decommissioned.
• Prefer ReadWriteOnce + reclaim policy Delete for scratch data that should vanish with the app.
• Audit large, long-lived volumes with oc get pvc -A -o=custom-columns=NAME:.metadata.name,SIZE:.spec.resources.requests.storage,PHASE:.status.phase.

---

7. Monitor & alert

• Registry: image_registry_pvc_capacity_remaining_percentage (custom alert).
• Nodes: node_filesystem_usage > 90 % / KubeNodeFileDescriptorLimit.
• etcd: etcd_debugging_mvcc_db_total_size_in_bytes.
• Ephemeral storage: kubelet_volume_stats_used_bytes{volumename="kubelet"}.

Grafana dashboards under Observe → Dashboards cover most of these out of the box.

---

8. Quick daily checklist

| Action                    | Command                                                                   |
| ------------------------- | ------------------------------------------------------------------------- |
| Show biggest image layers | `oc adm top images --all`                                                 |
| Dry-run image prune       | see §2                                                                    |
| Dry-run build prune       | see §3                                                                    |
| Check node root usage     | `oc adm top nodes --containers`                                           |
| Verify image pruner job   | `oc get imagepruners.imageregistry.operator.openshift.io cluster -o yaml` |

---

Conclusion

Space exhaustion is usually death by a thousand cuts rather than one dramatic incident. Automate what you can (image pruner CronJob, BuildConfig history limits), enforce quotas to keep workloads honest, and review registry/node metrics as part of your daily ops. With a few scheduled prunes and guard-rails in place, your OpenShift cluster will stay healthy—and your on-call rotation a little quieter.