One thing I’ve learned working in DevOps/SRE is that cloud problems rarely begin as real emergencies.

Most of them start with: “We’ll fix it later.”

Later becomes:

• hundreds of unused CloudWatch alarms
• old ECS task definitions nobody touches
• patching that keeps getting postponed
• IAM roles nobody reviews anymore
• “temporary” resources that somehow survive for years

None of these things breaks production immediately. That’s exactly why they become dangerous.

I once worked on cleaning up stale infrastructure resources across multiple AWS environments. At first glance, nothing looked critical. Systems were healthy, deployments were working, and the platform seemed stable.

But over time, the effects started showing up everywhere:

• Engineers ignored alerts because there were too many
• troubleshooting became slower
• dashboards became noisy
• Cloud costs slowly increased month after month

The hardest part wasn’t deleting old resources. It was figuring out what still mattered, what had dependencies, and what nobody actually owned anymore.

That experience changed how I think about infrastructure engineering. Technical debt in cloud environments compounds silently. Small operational shortcuts eventually become reliability problems.

Reliability isn’t only about uptime. It’s also about keeping systems understandable before complexity becomes production risk.

#SRE #DevOps #Cloud #AWS

Last week’s Amazon Web Services (AWS) US-EAST-1 outage exposed something a lot of DevOps and SRE teams already know: High availability diagrams are easy. Recoverability is hard.

A thermal event inside an AWS data center in Northern Virginia disrupted EC2 and EBS services in one of the world’s most heavily used cloud regions. Companies relying on single-region assumptions suddenly had to test whether their “resilient” architecture actually worked under real pressure.

What stood out to me wasn’t the outage itself. It was how many systems were technically redundant, but operationally unprepared.

I learned this during one of our AWS disaster recovery exercises, where we tested Route53 failover and cross-region recovery procedures.

On paper, everything looked ready:

• Secondary region infrastructure
• DNS failover policies
• Monitoring and synthetic checks
• Recovery runbooks

The failover worked. The difficult part came after traffic shifted.

That’s when hidden operational problems started appearing:

• Applications still tied to regional dependencies
• Dashboards no longer reflect reality
• Authentication behaves differently after a failover
• Downstream integrations are recovering inconsistently
• Manual validation steps are slowing recovery confidence

That experience changed how I think about cloud resilience.

In modern cloud engineering, redundancy alone is no longer enough. Multi-region AWS infrastructure, Kubernetes clusters, and distributed systems all increase operational complexity.

Reliability engineering is no longer just about preventing outages. It’s about proving systems can recover safely when outages inevitably happen. A disaster recovery plan that has never been realistically tested is still theoretical infrastructure.

Curious how other DevOps/SRE teams approach this: What’s usually the weakest part of DR testing in your environment today — observability, automation, or dependency validation?

#DevOps #SRE #AWS #CloudEngineering #DisasterRecovery #Route53 #PlatformEngineering #ReliabilityEngineering #Kubernetes #CloudInfrastructure

No CVE. No zero-day. Just a default that nobody changed.

IMDSv1 is enabled on EC2 instances by default. No authentication, no headers, no token. Any process on the instance — including any container running on it — can query the metadata endpoint and walk away with the IAM credentials attached to that instance. Whatever that role can do in your AWS account, so can they.

The reason this matters more than most default misconfigurations is the blast radius. It's not a single resource at risk. It's every S3 bucket, every RDS instance, every secret, every service that IAM role has access to. One vulnerable application on one instance is the entire surface.

This is exactly how the Capital One breach worked in 2019. An SSRF vulnerability in an application forced it to fetch credentials from the metadata endpoint. IMDSv1 handed them over without question. Over 100 million records followed.

IMDSv2 requires a session token before returning anything. That token has to be requested through a PUT call with a specific header — something a basic SSRF cannot do. The metadata endpoint becomes unreachable to anything that doesn't explicitly know how to ask for it correctly.

aws ec2 modify-instance-metadata-defaults \
  --http-tokens required

That one command enforces IMDSv2 across every new instance in the account. Nothing visible breaks. Applications that call the metadata endpoint correctly keep working. The attack surface disappears.

IMDSv1 doesn't announce itself. It doesn't generate alerts. It just sits there, enabled by default, waiting. The only way to know your exposure is to check.

Every AWS security guide says the same thing. Grant only the permissions a service needs. Nothing more. It's one of the first principles you learn and one of the hardest to actually maintain as an organization grows.

Here's what least privilege looks like in theory:

{
  "Effect": "Allow",
  "Action": ["s3:GetObject"],
  "Resource": "arn:aws:s3:::my-bucket/*"
}

Clean. Scoped. Intentional.

Here's what it looks like six months into a growing engineering org:

{
  "Effect": "Allow",
  "Action": ["s3:*"],
  "Resource": "*"
}

Nobody made that change maliciously. It happened because a developer needed access to a new bucket at 4 PM on a Friday before a release. The path of least resistance was a wildcard. The ticket got closed. The permission stayed.

This is the core problem with least privilege at scale. It isn't a technical failure. It's an organizational one.

Every time a team hits a permissions boundary during development, they have two options. File a request, wait for it to be reviewed, get the specific permission added, and continue. Or ask for something broader that covers this case and probably the next three. One of those options ships the feature today. The other one ships it next week.

IAM Access Analyzer tells you what permissions a role actually used over a trailing period:

aws accessanalyzer list-findings \
  --analyzer-arn arn:aws:access-analyzer:us-east-1:123456789012:analyzer/my-analyzer

That surfaces unused permissions across your account automatically. Not what you think a role needs — what it demonstrably used. That's the difference between an IAM policy written by intuition and one written by evidence.

Service Control Policies at the AWS Organizations level put a ceiling on what any role in an account can do regardless of what's in the role policy itself. Even if a wildcard slips through, an SCP can prevent the most dangerous actions from ever being callable in that account:

{
  "Effect": "Deny",
  "Action": [
    "iam:CreateUser",
    "iam:AttachRolePolicy"
  ],
  "Resource": "*"
}

The SCP doesn't replace least privilege. It contains the blast radius when least privilege fails, which it will.

At scale, least privilege is an engineering problem disguised as a security principle.

#AWS #IAM #CloudSecurity #DevOps #SRE #PlatformEngineering #CloudEngineering

We needed existing S3 buckets brought under management by a bucket manager CDK library, standardizing lifecycle policies, encryption, and access controls org-wide. The library expected to manage buckets it created, not ones already holding billions of objects. The path forward was new buckets, with everything moved over using S3 replication.

At that scale, the risk wasn't whether replication would work — it's mature and well documented. The risk was knowing when it was actually done, and what stayed dangerous even after it was.

aws s3control get-bucket-replication \
  --account-id 123456789012 \
  --bucket source-bucket-name

That command confirms replication is configured. It doesn't tell you how many of 1.8 billion objects actually landed on the other side. For that, we relied on daily S3 Inventory reports on both buckets, diffing object counts and total size until they converged. aws s3 ls won't get you there — it times out long before it counts that high.

The real danger wasn't cutover itself — it was what kept running after it. Once apps started writing to the new bucket, the old replication rule was still active unless removed. Anything still touching the old bucket, intentionally or not, would get pushed into the new bucket and silently overwrite what the app had just written. You could cut over cleanly and still get corrupted afterward, because the pipe was never closed.

The fix was making replication removal part of the cutover sequence itself, not cleanup done later once things looked stable.

The hardest part wasn't technical. A bucket at this scale is load-bearing for more systems than anyone has fully mapped, and the person running the migration owns every one of those dependencies — including the one that quietly outlives its purpose.

The real question isn't how to replicate the data. It's how you'll know every object made it, and what stops the old bucket from reaching back into the new one after you've moved on.

The application team kept reporting intermittent crashes. Logs showed the process terminating cleanly. No exception, no stack trace, just a clean exit. Restarts were automatic, so most users never noticed. But the pattern was consistent enough that it couldn't be ignored.

We started in the application code. Nothing. Checked dependencies. Nothing. The app looked completely healthy right up until it didn't exist anymore.

The actual problem was two lines in the task definition:

"memory": 512,
"memoryReservation": 512

Both set to the same value. That's the trap.

memoryReservation is a soft limit. The container is guaranteed that much but can burst above it if the host has headroom. memory is a hard ceiling. The moment the container crosses it, ECS kills the process. No warning. No OOM error in application logs — because the kill comes from outside the process entirely. From the app's perspective, nothing went wrong. It just stopped.

The way to confirm it isn't a code issue:

aws logs filter-log-events \
  --log-group-name /ecs/your-service \
  --filter-pattern "OutOfMemoryError OR signal: killed"

Then check CloudWatch Container Insights. If MemoryUtilized is consistently bumping the hard limit during peak load, that's your answer.

We set memory to 1024 and kept memoryReservation at 512. Clean exits stopped immediately.

The broader point: when a container restarts with no application-level error, check memory before you check code. ECS will kill a container without leaving any evidence inside the container itself — which is exactly why it looks like the app is crashing when it isn't.

We pushed a config change. I updated the ConfigMap, verified it in the cluster, and moved on. Fifteen minutes later, the app was still running on the old value.

No error. No warning. The ConfigMap was correct. The app was wrong.

When you source a ConfigMap into a pod as environment variables, those values are injected at pod startup. Kubernetes does not watch for changes and push updates into running containers. The pod has no idea the ConfigMap changed.

What makes this worse: if you mount the same ConfigMap as a file volume, it does update automatically after a short delay. Same ConfigMap, two different ways to consume it, completely different behavior. Nothing in your manifest warns you about this.

The fix is simple:

kubectl rollout restart deployment/your-app -n your-namespace

After every ConfigMap change. That's it.

The broader lesson: Kubernetes being declarative about desired state doesn't mean changes propagate the way you expect. Two things can both look correct in the cluster while your app runs on stale config.

If you changed a ConfigMap and didn't restart the pods, you didn't deploy the change. You scheduled it for the next restart.

A deploy pipeline got cancelled mid-run. Somebody pushed a hotfix, the CI job kicked off a Terraform apply, and before it finished, the pipeline was manually cancelled to push a different change instead. Routine enough. Nobody thought twice about it.

Twenty minutes later, every Terraform apply across every environment started failing with the same error:

Error: Error acquiring the state lock
Lock Info:
  ID:        a3f29d61-8e22-4f91-b8d3-1c9e7f4a2b55
  Path:      env:/prod/terraform.tfstate
  Operation: OperationTypeApply
  Who:       gitlab-runner@ci-node-03
  Created:   2026-01-14 09:42:11 UTC

The cancelled job never released its DynamoDB state lock. Terraform locks state before any write to prevent concurrent modifications from corrupting it — that part worked exactly as designed. The problem was that "cancelled" isn't the same as "finished," and nothing forced the lock release on interrupt.

For the next twenty minutes, nobody could deploy anything. Not the hotfix. Not the rollback. Not any unrelated change to any other service sharing that backend. A single interrupted pipeline run had taken down deploys for the entire team.

The immediate fix was manual:

terraform force-unlock a3f29d61-8e22-4f91-b8d3-1c9e7f4a2b55

But force-unlocking blind is its own risk. If the original apply was actually still running somewhere — in a half-finished state — force-unlocking and reapplying on top of it could corrupt state for real. Before running it, we had to confirm the original CI job was actually dead, not just cancelled in GitLab's UI while still executing on the runner:

ps aux | grep terraform

No process was alive on the runner node. Only then did we force-unlock.

The longer-term fix was two changes to the pipeline. First, a guard step before every apply that checks for an existing lock and fails fast with a clear message instead of hanging:

before_script:
  - terraform state pull > /dev/null || (echo "State locked or unreachable" && exit 1)

Second, and more importantly, removing the ability to manually cancel an apply job mid-run from the CI interface entirely. If a job needs to be stopped, it should run to completion or fail cleanly — not get killed in a state where it can't clean up after itself.

The incident wasn't really about DynamoDB or Terraform internals. It was about a pipeline that let a person interrupt an operation that wasn't safe to interrupt, and no safeguard caught that gap until it took down the whole team's ability to ship.

A critical CVE showed up in Prisma Cloud against one of our services. We patched it, rebuilt it, and redeployed it. The next scan: same CVE, same severity, same service. We patched it again. It came back again.

Three cycles of this before anyone questioned whether we were actually patching what was running in production.

The Dockerfile looked completely reasonable:

FROM node:18-alpine

node:18-alpine is a floating tag. It doesn't point to one fixed image — it points to whatever the maintainers most recently pushed under that tag. Every time we rebuilt, Docker pulled whatever 18-alpine resolved to at that moment, which might or might not be the same underlying image as last week's build, or the one currently running in production.

Here's the part that caused the loop. Prisma Cloud was scanning the image in our registry by tag. But the image actually deployed and running in the cluster had been pulled and cached on nodes days earlier, before the base image had shifted again upstream. The scan result and the running workload were, technically, two different images that happened to share the same tag.

docker inspect node:18-alpine --format='{{.RepoDigests}}'

Running that on the build host versus a running node showed two different digests under the identical tag. We were patching the version Prisma scanned, while a slightly different version — pulled earlier — kept running and kept getting flagged.

The fix was pinning the base image to a digest instead of a tag:

FROM node:18-alpine@sha256:4f2c9b7a1e8d...

A digest points to one exact, immutable image. No drift, no ambiguity about what's actually deployed versus what got scanned. When the base image needed a real update, that became a deliberate change to the Dockerfile — reviewed like any other dependency bump — not something that happened silently on every rebuild.

We also moved the CI pipeline to scan the image by digest after build, then deploy that exact digest, instead of deploying by tag and letting Kubernetes resolve it again at pull time on each node.

The underlying lesson: a floating tag isn't a version, it's a moving target. Vulnerability scanning, build, and deployment all need to agree on the exact same artifact — not the same label pointing at three different things depending on when you ask.

If a CVE keeps reopening after being patched, check whether the tag scanned and the image running are actually the same bytes. They might just be sharing a name.

Routine deployment. Small change, tested in staging, pipeline green. We kicked it off and walked away.

Five minutes later the service was completely down. Not degraded — gone. Every existing pod had been terminated before a single new one passed its health check.

The culprit was two fields nobody had explicitly set. Our deployment had never defined a rollout strategy, which meant Kubernetes was running on defaults nobody had reviewed. At our replica count, those defaults allowed every pod to terminate simultaneously while replacements were still initializing.

The fields that should have been there:

strategy:
  type: RollingUpdate
  rollingUpdate:
    maxUnavailable: 1
    maxSurge: 1

One pod down at a time. Each replacement has to pass its readiness probe before the next old one gets terminated. Slow, deliberate, zero surprise.

A RollingUpdate strategy only rolls safely if the numbers enforce it. The name implies safety. The defaults don't guarantee it.

Check every deployment in your cluster:

kubectl get deployments -A -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.strategy.rollingUpdate}{"\n"}{end}'

Any deployment returning null is running on defaults. Worth knowing before the next deploy, not after.

Patching broke across our RHEL fleet. Not with an explosion but with yum silently failing and /boot partitions sitting at 94-96% full.

Routine kernel patch cycle. Rapid7 flagged a high-severity CVE; the SLA clock was ticking. Kicked off the Ansible playbook, came back to failed hosts across multiple environments.

Buried in logs: “Installing package kernel-4.18.x would result in filling disk on /boot”

RHEL’s default installonly_limit keeps up to 5 kernels. Our /boot partitions were 500MB. We ran four environment lifecycles — dev, QA, staging, and prod — patching consistently across all of them. Every cycle was quietly stacking another 100-200MB kernel image with nowhere to go.

The fix was simple: installonly_limit=2. We rolled it out via Ansible across every RHEL host, cleaned old kernels with package-cleanup --oldkernels --count=2, and reran the patch cycle.

But what mattered most: we caught it in dev and staging first — before prod. That’s why lower environments must run the same patch cadence and OS workflow as production.

After resolving it, we added a pre-patch assertion: if /boot has less than 200MB free, clean old kernels first, then patch. Silent mid-cycle failures stopped completely.

The irony: disciplined patching created the very condition blocking future patching because installonly_limit wasn’t explicitly managed.

If you manage RHEL at scale and haven’t set this, check /boot utilization across all environments now — not just prod.

#RHEL #Linux #OS #DevOps #SRE #Patching #Kernel #AWS