AWS Builder Center
Monitoring Kubernetes with CloudWatch Container Insights on Amazon EKS

Monitoring Kubernetes with CloudWatch Container Insights on Amazon EKS

Pod-level metrics, container logs, and a live EKS operations dashboard

Head of DevOps
Running a Kubernetes cluster without proper observability is like driving with your eyes closed. You know pods are running, but you have no idea which ones are consuming the most memory, which nodes are under pressure, or where that intermittent crash is coming from.
CloudWatch Container Insights solves this for Amazon EKS — it gives you pod-level metrics, container logs, and a live performance dashboard, all integrated into the same CloudWatch you already use for your other AWS resources.
In this demo, I set up Container Insights on the EKS cluster from the previous demos and explored the data it collects.



What Container Insights Provides

Standard CloudWatch:
EC2 metrics → CPU, Memory at instance level

Container Insights adds:
Cluster level → Node count, overall CPU/Memory
Node level → Per-node resource utilization
Pod level → CPU and Memory per pod
Container → Individual container metrics
Logs → Pod application logs
Node host logs
Kubernetes dataplane logs



Architecture

EKS Cluster (aiops-eks)
│
├── CloudWatch Agent (DaemonSet)
│ Collects: CPU, Memory, Network, Disk metrics
│ Destination: /aws/containerinsights/aiops-eks/performance
│
└── FluentBit (DaemonSet)
Collects: Application logs, Host logs, Dataplane logs
Destinations:
/aws/containerinsights/aiops-eks/application
/aws/containerinsights/aiops-eks/host
/aws/containerinsights/aiops-eks/dataplane
Two DaemonSets run on every node — CloudWatch Agent for metrics, FluentBit for logs. A DaemonSet ensures exactly one pod per node regardless of how many nodes exist.



Step 1 — IAM Permissions

The node group IAM role needs permission to write metrics and logs to CloudWatch:
IAM → Roles → (node group role) → Attach policy:
CloudWatchAgentServerPolicy
Without this, the CloudWatch Agent pods will run but silently fail to publish any data.



Step 2 — CloudWatch Agent (Metrics)

The CloudWatch Agent was deployed as a DaemonSet using AWS’s official manifest. A ConfigMap defines what to collect:
{
"logs": {
"metrics_collected": {
"kubernetes": {
"cluster_name": "aiops-eks",
"metrics_collection_interval": 60
}
},
"force_flush_interval": 5
}
}
kubectl get daemonset -n amazon-cloudwatch
# NAME DESIRED CURRENT READY
# cloudwatch-agent 1 1 1 ✅

Step 3 — FluentBit (Logs)

FluentBit requires a ConfigMap named  fluent-bit-cluster-info  with cluster details before it will start. This is a common setup mistake — deploying the DaemonSet before creating the ConfigMap causes CrashLoopBackOff:
Error: configmap "fluent-bit-cluster-info" not found
The fix is straightforward — create the ConfigMap first:
kubectl create configmap fluent-bit-cluster-info \
--from-literal=cluster.name=aiops-eks \
--from-literal=http.server=On \
--from-literal=http.port=2020 \
--from-literal=read.head=Off \
--from-literal=read.tail=On \
--from-literal=logs.region=eu-central-1 \
-n amazon-cloudwatch
After the ConfigMap was created and the DaemonSet restarted, both pods came up healthy:
kubectl get pods -n amazon-cloudwatch
# NAME READY STATUS
# cloudwatch-agent-xxxxx 1/1 Running ✅
# fluent-bit-xxxxx 1/1 Running ✅

Step 4 — Container Insights Dashboard

Once both DaemonSets were running, metrics started flowing into CloudWatch within 2–3 minutes.
CloudWatch → Container Insights → Performance monitoring → aiops-eks
The dashboard provides multiple views:
Cluster view:
Nodes status : 2 Ready
Node CPU util : 4.84%
Node Memory util : 27.8%
Container restarts : 0
Pod CPU and Memory over pod limit: Both charts show live utilization relative to the resource limits defined in the pod spec — this is what makes Container Insights more useful than raw CloudWatch metrics. You see utilization in context of what the pod is actually allowed to use.
Additional views available:
  • Namespaces — resource breakdown per namespace
  • Nodes — per-node CPU and memory
  • Services — network metrics per service
  • Workloads — deployment-level view
  • Pods — individual pod metrics
  • Containers — container-level granularity






Step 5 — Log Groups Created

After FluentBit started, four log groups appeared automatically:
/aws/containerinsights/aiops-eks/application ← Nginx and app container logs
/aws/containerinsights/aiops-eks/host ← Node OS logs
/aws/containerinsights/aiops-eks/performance ← CloudWatch Agent metrics
/aws/containerinsights/aiops-eks/dataplane ← Kubernetes control plane logs
Step 6 — Querying Logs with CloudWatch Logs Insights
CloudWatch → Logs Insights → /aws/containerinsights/aiops-eks/application

Query:
fields @timestamp, kubernetes.pod_name, log
| filter kubernetes.namespace_name = "aiops-demo"
| sort @timestamp desc
| limit 20
This returns the last 20 log lines from every container running in the  aiops-demo  namespace — no SSH, no kubectl exec required.




Key Lessons

The FluentBit ConfigMap must exist before the DaemonSet is deployed. If you deploy the DaemonSet first, it enters CrashLoopBackOff and keeps pulling the image every 30 seconds. Create the ConfigMap first, then apply the DaemonSet — or restart it after the ConfigMap is created.
IAM permissions are always the first thing to verify. The CloudWatch Agent ran without errors but published no data until  CloudWatchAgentServerPolicy  was attached to the node role. If your metrics are missing, check IAM before debugging the agent configuration.
Container Insights gives you context that raw metrics cannot. Knowing a pod uses 27.8% memory is useful — knowing it uses 27.8% of its 256Mi limit is actionable. Container Insights shows utilization relative to resource limits, which is what actually matters for capacity planning.
DaemonSets automatically scale with your cluster. When a new node is added by Cluster Autoscaler, the CloudWatch Agent and FluentBit pods are automatically scheduled on it. You do not need to do anything — monitoring scales with your cluster.



Container Insights vs Standard CloudWatch

Capability Standard CloudWatch Container Insights
EC2 instance metrics ✅ ✅
Pod-level metrics ❌ ✅
Container-level metrics ❌ ✅
Application logs❌ ✅ (FluentBit)
K8s-aware dashboard ❌ ✅
Auto-scales with cluster N/A ✅ (DaemonSet)



Full Source Code

All manifests and setup scripts used in this demo are available on GitHub:
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article