Incidents

Problems I found and fixed

Each one was traced to its root cause, fixed in code, and documented. Most were silent: nothing failed loudly, the data was simply missing.

Kubernetes

Nodes that could never be replaced

Symptom: Karpenter kept an old node running for an hour, so two servers were billed for one workload.
Cause: A disruption budget of minAvailable 1 on a single replica allows zero evictions, so no node holding that pod could be drained.
Fix: Use maxUnavailable 1 in DEV, where a few seconds of downtime during a node move is acceptable.

Check every disruption budget against the replica count it protects.

Cost

A surprising NAT bill

Symptom: Gigabytes went through the NAT gateway while the cluster had almost no users.
Cause: Trivy downloaded its vulnerability database from the internet for every single scan job.
Fix: Run Trivy with its built-in server, so the database is downloaded once, and scan only the application namespace.

Per-gigabyte charges hide in background jobs; read the flow logs before guessing.

Cost

Eleven gigabytes of logs a day

Symptom: CloudWatch log ingestion cost more than the servers it described.
Cause: EKS audit logging recorded every API request, most of them Argo CD's own reconciliation.
Fix: Turn off audit logs in DEV and keep them in staging and production.

Log volume is a cost decision to make per environment.

Autoscaling

A worker that never scaled down

Symptom: Two worker pods kept running with an empty queue.
Cause: The worker's idle memory sat above the 80 percent memory trigger, and the autoscaler's formula keeps two replicas once memory never falls.
Fix: Size the memory request above the idle footprint and scale on queue length and CPU only.

Scale on a signal that drops when a pod is added.

Observability

Application metrics that never arrived

Symptom: Dashboards for the API were empty and no Nexus metric existed in Prometheus.
Cause: Prometheus discovers only declared container ports, and the Istio merged-metrics port 15020 was not declared, so every target was dropped silently.
Fix: Discover the declared HTTP port and rewrite the scrape address to port 15020 on the same pod.

Check that targets are up, not only that the monitor exists.

Delivery

A canary check that always passed

Symptom: Every release would have been promoted, even a broken one.
Cause: With no metrics, the success-rate query fell back to 'or vector(1)', which reads as one hundred percent success.
Fix: Restore the metrics, so the check measures real traffic again.

A safety check must fail loudly when its data is missing.

Observability

Logs that never reached Loki

Symptom: Loki held only its own canary logs, labelled unknown_service.
Cause: Container log files are readable only by root's user or group, and the collector ran as an unprivileged user, so it matched no files and reported no error.
Fix: Give the collector the root group as a supplemental group: read access without running as root.

A collector that reads nothing needs metrics of its own, or nobody notices.

Edge

Login pages refused by Cloudflare

Symptom: Creating the Access applications failed with 'domain does not belong to zone'.
Cause: Cloudflare Access accepts only names in an active zone, and the zone stays pending until the domain points at Cloudflare.
Fix: Read the zone status in Terraform and create the Access applications on the first apply after activation.

Let the code wait for the real world instead of a manual flag.

Infrastructure as code

A rebuild that stopped at the last step

Symptom: Rebuilding DEV created the network, cluster, database, and platform, then failed at the edge: the global unit had no attribute named zone_id.
Cause: Removing the unused Route 53 zone made its output null, and Terraform leaves null outputs out of the state, so the attribute the edge read no longer existed.
Fix: The edge now reads the zone as optional, as it already did for the Cloudflare zone, and serves the domain through Cloudflare alone.

A rebuild from empty finds what an in-place change silently broke.

Security

A release blocked by a new CVE

Symptom: A small API change failed CI at the image scan, although it touched no dependency.
Cause: golang.org/x/net v0.61.0, already in the image, had a high-severity denial-of-service flaw, CVE-2026-78661, published after the previous build.
Fix: Updated the module to v0.62.0 and scanned both images locally the way CI does before pushing. Nothing vulnerable reached DEV.

A scan gate fails on the day a flaw is published, not the day the code changes, so dependencies have to stay current.

Reliability

A Spot node reclaimed during a canary

Symptom: The public address returned 503 for about ninety seconds, just as a new API version started its canary.
Cause: AWS reclaimed the Spot node that ran the only API pod. DEV runs a single replica to save money.
Fix: Karpenter received the two-minute warning and started a replacement node, and the API recovered without help. The canary then passed its analysis and was promoted. Staging and production run two and three replicas with a disruption budget.

Spot is cheap only when a service can lose a node; a single replica turns every reclaim into an outage.

Security

Signature checks that could not reach the registry

Symptom: Every background result for the signed-image policy was an error, a 401 from ECR, while admission checks passed.
Cause: Only Kyverno's admission controller had a Pod Identity association. The reports controller, which re-checks running pods, had no AWS credentials.
Fix: Gave the reports controller the same read-only role. All seven results turned to pass within two minutes.

Every component that calls AWS needs its own identity; one that works proves nothing about the others.

Observability

Alerts that never arrived

Symptom: Alerts fired in Prometheus, but no email ever reached the inbox.
Cause: SNS rejected every notification, because Alertmanager's default subject lists all common labels and grows past the 100 characters SNS accepts. The email subscription had also never been confirmed.
Fix: A short subject with the status and alert name, a confirmed subscription, and a test alert that arrived as an email within a minute.

An alert path is proven only by an alert that reaches a person.

Observability

A collector that looked down

Symptom: Prometheus reported both trace collectors as down, although traces kept flowing into Tempo.
Cause: The collector's configuration replaced the chart's defaults without a telemetry reader, so its own metrics listened on localhost only.
Fix: Served the metrics on the pod address, as the log collector already did. Both targets came up.

Replacing a default also removes it; check what the chart was doing for you.

Observability

A dashboard with every panel empty

Symptom: All six panels of the AWS overview showed No data while the database, cache, and queue were in use.
Cause: The CloudWatch queries named the metric but not the query type and editor mode, so Grafana dropped them without sending a request.
Fix: Added both fields to every query, and the panels filled with RDS, Valkey, and SQS data at once.

Test a dashboard at a moment when it must show data; No data looks the same as no traffic.