AWS Builder Center

Builder Center

12 DevOps Mistakes I See Teams Still Making in 2026 (and How to Fix Each One)

12 DevOps Mistakes I See Teams Still Making in 2026 (and How to Fix Each One)

Hard-won, production-focused DevOps best practices: CI/CD security, Terraform state, Kubernetes limits, secrets, observability, cost, and incident habits, each with the fix.

Tidding Ramsey Binda

Devops Engineer

Tools change every year. The mistakes mostly do not. Below are twelve problems that keep showing up in real pipelines, clusters and cloud accounts, with a concrete fix for each. Use it as a checklist against your own setup.

1. Long-lived cloud credentials in CI

The mistake: AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY stored as CI secrets, never rotated, often with AdministratorAccess.
Why it hurts: one leaked log, one compromised third-party action, and an attacker has permanent access to your account.
The fix: use OIDC federation so each job gets short-lived credentials, and scope the trust policy tightly.
1
2
3
4
5
6
"Condition": {
"StringEquals": {
"token.actions.githubusercontent.com:aud": "sts.amazonaws.com",
"token.actions.githubusercontent.com:sub": "repo:my-org/my-app:ref:refs/heads/main"
}
}
Never trust only the aud claim. Always pin sub to the exact repo and branch or environment. (Full walkthrough in my previous article on ECS + GitHub Actions + OIDC.)

2. Trusting third-party CI actions by tag

The mistake:
1
- uses: some-org/some-action@v3
Tags are mutable. The maintainer (or an attacker who takes over their account) can move v3 to malicious code. Supply-chain attacks against popular GitHub Actions have already happened, including a widely reported 2025 compromise that leaked secrets from CI logs.
The fix: pin to a full commit SHA and let a bot update it.
1
- uses: actions/checkout@<full-40-char-commit-sha> # v4.x.x
Also:
  • Set the default GITHUB_TOKEN permissions to read-only and grant more per job.
  • Use Dependabot or Renovate to bump pinned SHAs through reviewed pull requests.
  • Limit which actions the organisation may run (allow-list in org settings).
1
2
permissions:
contents: read # start here, add only what a job needs

3. Terraform state treated casually

The mistakes:
  • State stored locally or committed to Git.
  • No locking, so two applies run at once and corrupt state.
  • One giant state file for the whole company.
The fix:
  • Remote state in an encrypted, versioned S3 bucket with locking. Recent Terraform versions support native S3 locking, so a separate DynamoDB table is no longer required for new setups (verify your Terraform version supports use_lockfile).
1
2
3
4
5
6
7
8
9
terraform {
backend "s3" {
bucket = "my-org-tf-state"
key = "prod/network/terraform.tfstate"
region = "eu-west-1"
encrypt = true
use_lockfile = true
}
}
  • Enable bucket versioning so you can recover from a bad apply.
  • Split state by blast radius: network, data, and apps in separate states. A mistake in an app change should never be able to touch your VPC or database.
  • Block public access on the state bucket and restrict who can read it. State files contain secrets in plain text.

4. Applying infrastructure changes without reading the plan

The mistake: terraform apply -auto-approve from a laptop, or a pipeline that applies on merge with nobody reviewing what will be destroyed.
The fix:
  • CI runs terraform plan on every pull request and posts the output.
  • Apply runs from CI only, after merge, with a protected environment.
  • Add policy checks so dangerous changes fail automatically:
1
2
3
terraform plan -out=tfplan
terraform show -json tfplan > plan.json
checkov -f plan.json # or tfsec / trivy config / OPA conftest
  • Use lifecycle { prevent_destroy = true } on databases and critical storage.
  • Watch for -/+ (destroy and recreate) in plans. That line is where outages come from.

5. Kubernetes pods with no resource requests and limits

The mistake: deployments with no resources block. One noisy pod eats a node, the scheduler packs badly, and you get random OOMKilled and evictions.
The fix: always set requests, and be deliberate about limits.
1
2
3
4
5
6
7
resources:
requests:
cpu: "100m"
memory: "256Mi"
limits:
memory: "512Mi" # memory limit protects the node
# many teams intentionally omit CPU limits to avoid throttling; measure before deciding
Then:
  • Right-size using real metrics (Prometheus, VPA recommendations), not guesses.
  • Add a PodDisruptionBudget so node drains do not take all replicas at once.
  • Add readiness and liveness probes, and use a startupProbe for slow-starting apps so liveness does not kill them during boot.
1
2
3
4
5
6
7
readinessProbe:
httpGet: { path: /health, port: 3000 }
periodSeconds: 5
livenessProbe:
httpGet: { path: /health, port: 3000 }
periodSeconds: 10
failureThreshold: 3

6. Running containers as root with a fat base image

The mistake: FROM ubuntu:latest, running as root, hundreds of packages you never use, and an unpinned tag.
The fix:
1
2
3
4
FROM node:20-alpine@sha256:<digest> # pin by digest for reproducibility
WORKDIR /app
COPY --chown=node:node . .
USER node
And in Kubernetes enforce it:
1
2
3
4
5
6
securityContext:
runAsNonRoot: true
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
capabilities:
drop: ["ALL"]
  • Use small or distroless images to reduce attack surface.
  • Scan images in CI and fail on critical vulnerabilities:
1
trivy image --severity CRITICAL --exit-code 1 my-app:${GITHUB_SHA}
  • Never use latest in production. Tag with the commit SHA.

7. Secrets in Git, environment variables and Docker images

The mistakes:
  • .env committed once "by accident" (it stays in Git history forever).
  • Secrets baked into images with ENV or COPY.
  • Kubernetes Secret objects assumed to be encrypted (they are only base64-encoded by default).
The fix:
  • Store secrets in AWS Secrets Manager, SSM Parameter Store or Vault, and inject at runtime.
  • On Kubernetes, use External Secrets Operator or the Secrets Store CSI driver, and enable encryption at rest for etcd.
  • Add secret scanning as a pre-commit hook and a CI gate:
1
gitleaks detect --source . --no-banner
  • If a secret ever hits Git, rotate it immediately. Deleting the commit is not enough.

8. Over-permissive IAM (the "Action": "*" habit)

The mistake:
1
{ "Effect": "Allow", "Action": "*", "Resource": "*" }
"We will tighten it later" never happens.
The fix:
  • Start with zero and add permissions as needed. Use IAM Access Analyzer to generate a policy from actual CloudTrail activity.
  • Use permission boundaries or SCPs in AWS Organizations to cap what any role can ever do.
  • Use iam:PassRole with a PassedToService condition, never unrestricted.
  • Prefer roles over IAM users. Humans should use IAM Identity Center (SSO) with MFA.
  • On EKS, use IRSA or EKS Pod Identity so each workload gets its own narrow role, not the node's role.
  • Regularly review unused roles and access keys.

9. Alerts that nobody trusts

The mistake: hundreds of alerts on CPU at 80%, all firing constantly, so the on-call person mutes the channel. Then a real outage goes unnoticed.
The fix:
  • Alert on symptoms users feel (error rate, latency, availability), not every internal metric.
  • Define SLOs and alert on error-budget burn rate.
  • Every alert must have: an owner, a severity, and a runbook link. If nobody can act on it, delete it.
  • Use the golden signals: latency, traffic, errors, saturation.
Example Prometheus alert for user-facing errors:
1
2
3
4
5
6
7
8
9
10
- alert: HighErrorRate
expr: |
sum(rate(http_requests_total{status=~"5.."}[5m]))
/ sum(rate(http_requests_total[5m])) > 0.05
 for: 5m
labels:
severity: page
annotations:
summary: "More than 5% of requests are failing"
runbook: "https://wiki.example.com/runbooks/high-error-rate"
Review alert noise monthly. Fewer, better alerts beat many ignored ones.

10. No tested backups or restore plan

The mistake: "We have backups" with nobody ever having restored one. Backups that are untested are only hopes.
The fix:
  • Automate backups (RDS automated backups and snapshots, S3 versioning, Velero for Kubernetes resources and volumes).
  • Copy critical backups to another account and another region so one compromised account cannot delete everything.
  • Schedule a restore drill every quarter and record how long it took (your real RTO).
  • Define RPO and RTO per system. Not everything needs the same protection, but everything needs a decision.
  • Protect against deletion: S3 Object Lock, RDS deletion protection, prevent_destroy in Terraform.

11. Cloud bills nobody owns

The mistakes: idle resources, oversized instances, NAT Gateway data charges, forgotten dev environments, unattached volumes, logs retained forever.
The fix:
  • Tag everything (team, service, env, owner) and enforce it with policy. Untagged cost is unowned cost.
  • Set AWS Budgets with alerts before you need them.
  • Auto-shutdown non-production environments at night and weekends.
  • Set log retention on every CloudWatch log group. The default is "never expire".
1
2
3
4
resource "aws_cloudwatch_log_group" "app" {
name = "/ecs/my-app"
retention_in_days = 14
}
  • Review the big three monthly: compute, data transfer (especially NAT and cross-AZ), storage.
  • Use Spot instances for stateless, interruption-tolerant workloads, and Savings Plans for steady baseline usage.
  • Delete unattached EBS volumes, old snapshots and unused Elastic IPs on a schedule.

12. Treating incidents as blame instead of learning

The mistake: an outage happens, someone gets blamed, and the real cause (a missing guardrail, a confusing process, a risky default) stays in place until it happens again.
The fix:
  • Write blameless postmortems for every significant incident. Ask "what let this happen?" not "who did this?"
  • Use a simple template: impact, timeline, root cause, what went well, what went badly, action items with owners and dates.
  • Track action items to completion. A postmortem without follow-through is just a story.
  • Practice: run game days, chaos experiments and failover tests in a safe environment.
  • Keep runbooks next to the alerts and update them after every incident.

Bonus: Things to double check for yourself in 2026

Ecosystems move fast, so verify the current state of anything you depend on before you build around it:
  • Kubernetes ingress: confirm the maintenance status of your ingress controller and consider the Gateway API as the long-term direction.
  • Terraform vs OpenTofu: check licensing and provider compatibility for your organisation's needs.
  • Container image sources: free public images and registries change their policies. Mirror critical base images into your own registry so a policy change or outage upstream cannot break your builds.
  • Dependency supply chain: package registries have seen worm-style attacks. Use lockfiles, pin versions, and review new dependencies.

A quick self-audit (copy this)

  • [ ] No static cloud keys in CI (OIDC everywhere)
  • [ ] Third-party actions pinned to SHAs
  • [ ] Terraform state remote, encrypted, versioned, locked
  • [ ] Plans reviewed before every apply
  • [ ] Every pod has requests, probes and a non-root security context
  • [ ] Secrets come from a secret manager, never from Git
  • [ ] IAM policies are least-privilege and reviewed
  • [ ] Every alert has an owner and a runbook
  • [ ] Backups restored successfully in the last 90 days
  • [ ] Resources tagged and budgets set
  • [ ] Last incident has a postmortem with tracked actions
Score yourself. Anything unchecked is your roadmap for next sprint.
Comment challenge: reply with your score out of 11 (for example, "7/11") and the one item you'll fix first. I'll reply with practical tips for whatever you pick.

Conclusion

DevOps maturity is rarely about the newest tool. It comes from boring, consistent habits: least privilege, automation with review, observable systems, tested recovery, and learning from failure. Fix these twelve and you will be ahead of most teams.
Now it's your turn.
{% poll Which of these mistakes has bitten you the hardest? %} Static credentials / over-permissive IAM Terraform state or unreviewed applies Kubernetes limits, probes and security context Alert noise or untested backups {% endpoll %}
Join the discussion in the comments:
  1. Share your worst DevOps incident (keep it blameless, no company names needed). What was the root cause and what guardrail did you add afterwards?
  2. What mistake is missing from this list? The best suggestions will go into a Part 3, with credit to you.
  3. Disagree with something? Say so. Good engineering comes from healthy debate, for example the CPU limits question in section 5.
  4. Student or beginner? Ask your DevOps question below. No question is too basic, and I answer every one.
The most-requested topic becomes the next deep-dive article. Vote with a comment: A) EKS + IRSA + Karpenter, B) Terraform modules and remote state, C) DevSecOps pipeline, D) Building a DevOps lab on a small budget.
# devops# kubernetes# terraform
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article