AWS Builder Center

Builder Center

Mastering AWS Network Observability: A Practical Guide

A practical guide to building end-to-end network observability on AWS — covering the key metrics, alarms, and tools across EC2, VPC Flow Logs, NAT Gateway, Network Firewall, PrivateLink, and the CloudWatch Network Monitoring suite to help you detect, diagnose, and resolve network issues before they impact your customers.

Dexter Moyo


As AWS CTO Werner Vogels famously reminds us: "Everything fails, all the time." When it comes to networking on AWS, the question isn't if something will go wrong — it's whether you'll have the visibility to detect it, diagnose it, and resolve it before your customers notice.
One of the most common operational challenges we see across enterprise customers is gaining meaningful observability across complex AWS network architectures. This post provides a practical, tool-by-tool guide to building that observability.

The Challenge: Network Monitoring at Scale

Modern AWS environments are rarely simple. A typical enterprise architecture might span multiple VPCs, regions, and accounts — interconnected through Transit Gateways, Cloud WAN, VPC peering, PrivateLink, Site-to-Site VPN, and Direct Connect. Add services like Amazon EKS, Application Load Balancers, NAT Gateways, and AWS Network Firewall into the mix, and you have a complex web of network paths where issues can hide.
The most common challenges we see:
  • Blind spots across hybrid and multi-VPC environments
  • Difficulty correlating network metrics with application-level impact
  • Alert fatigue from poorly tuned thresholds
  • Slow mean time to resolution (MTTR) due to manual, console-hopping troubleshooting

Architecture Overview



The Toolbox: A Layered Approach to Network Observability

Effective network observability is built on three pillars: Telemetry, Analysis & Visualization, and Diagnostic Tools.

How Telemetry Flows Through the Stack



1. Telemetry — Collecting the Right Data

EC2 Network Performance Metrics

Your EC2 instances have built-in network performance counters exposed through the ENA driver. Install the Amazon CloudWatch Agent and monitor these critical metrics
Metric
What It Tells You
 bw_in_allowance_exceeded 
Inbound bandwidth limit hit — packets queued or dropped
 bw_out_allowance_exceeded 
Outbound bandwidth limit hit
 pps_allowance_exceeded 
Packets-per-second limit exceeded
 conntrack_allowance_exceeded 
Connection tracking table full — new connections failing
 conntrack_allowance_available 
Remaining connection tracking capacity (Nitro only)
 linklocal_allowance_exceeded 
DNS/IMDS/NTP request rate exceeded (1,024 PPS per ENI)
Action: Create CloudWatch dashboards and alarms for these metrics across your Nitro fleet. A spike in  conntrack_allowance_exceeded  is often the hidden cause of intermittent connectivity failures.
 VPC Flow Logs  — VPC Flow Logs capture metadata about IP traffic to/from network interfaces in your VPC. Logs can be sent to CloudWatch Logs, S3, or Data Firehose. Use them to diagnose overly restrictive security group rules, monitor instance traffic, and determine traffic direction. Collection happens outside the network path, so there's zero impact on throughput or latency.
Some of the fields available:
  •  vpc-id  and  subnet-id  — for aggregation
  •  pkt-srcaddr  and  pkt-dstaddr  — essential for container workloads
  •  az-id  (v4) — for cross-AZ traffic analysis
  •  flow-direction  and  traffic-path  (v5) — to understand which networking constructs traffic traverses

Service-Specific Metrics

Below is monitoring guidance for key networking services, each with some recommended CloudWatch metrics and recommended alarm thresholds to add to your existing resources:
  •  NAT Gateway  — Alarm on  PacketsDropCount > 0.1%/sec  and  ErrorPortAllocation > 2/sec . Monitor aggregate throughput approaching the 100 Gbps / 10 Mpps scaling limits.
  •  AWS PrivateLink  — Watch  RstPacketsSent  (service side) and  PacketsDropped  /  RstPacketsReceived  (endpoint side).
  •  Site-to-Site VPN  — Alarm when  TunnelState = 0  (down) or  TunnelDataIn + TunnelDataOut > 1.0 Gbps  (approaching the 1.25 Gbps per-tunnel limit).
  •  Route 53 Resolver  — Alarm on  InboundQueryVolume  or  OutboundQueryAggregateVolume > 8,000 QPS  and  EndpointUnHealthyENICount > 0  for more than 10 minutes.
  •  VPC Lattice  — Monitor  RequestTime ,  HTTPCode_4XX_Count ,  HTTPCode_5XX_Count , and  ConnectionErrorCount  per service and target group.
  •  AWS Network Firewall  — Alarm on  InvalidDroppedPackets  or  OtherDroppedPackets  increasing significantly. Export stateful engine logs (Alert, Flow, and TLS logs) for deep inspection.

2. Amazon CloudWatch Network Monitoring Suite

Three CloudWatch Network Performance Monitoring services together provide end-to-end visibility:

 Internet Monitor  

Provides visibility into how internet issues impact your end users. It triangulates AWS monitoring data with your resource access patterns to surface performance and availability problems by city-network. Supports VPCs, NLBs, CloudFront distributions, and WorkSpaces.

 Network Synthetic Monitor  

Simplifies hybrid network performance monitoring. Specify a VPC subnet and an on-premises IP address — no agents to deploy. It actively probes using TCP or ICMP and publishes round-trip time and packet loss metrics to CloudWatch. Deploy probes in each AZ where mission-critical workloads run.

 Network Flow Monitor 

Uses eBPF-technology to passively monitor TCP traffic. It correlates your workload telemetry with AWS network infrastructure health indicators to help you rapidly attribute performance degradation to either your application or the underlying AWS network. Key metrics: round-trip time (microseconds), packet loss (%), and a network health indicator.

3. Analysis & Visualization

Raw telemetry is only useful if you can query and visualize it effectively:
  •  CloudWatch Contributor Insights  — Continuously analyze time-series data to surface top contributors impacting system performance — no manual intervention needed.
  •  AWS Network Manager  — Topology visualization, metrics, and events for Transit Gateway and Cloud WAN environments.

4. Diagnostic Tools

When you need to go deeper:
  •  VPC Traffic Mirroring  — Mirror ENI traffic to an NLB or GWLB endpoint for packet-level investigation. Be mindful that mirrored traffic counts toward ENI bandwidth.
  • Open-source tools —  iperf  for throughput testing,  dig +trace  for DNS resolution tracing,  mtr  for path analysis,  tcpdump  for packet capture, and  netcat  for connectivity testing.

AI-Assisted Network Operations with Kiro

Beyond traditional tooling,  Kiro  CLI brings agentic AI to network operations — translating natural language into AWS CLI commands, executing them, and explaining the results. We cover this in depth in a dedicated companion article:  How to Use Kiro for Network Operations  . 

Getting Started: Your Network Observability Checklist

Here's a prioritized action plan:
  1. Enable VPC Flow Logs with custom format (v3+ fields) across all VPCs
  1. Install the CloudWatch Agent on EC2 instances and configure ENA driver metric collection
  1. Set up CloudWatch Alarms for the critical metrics listed above for each networking service you use
  1. Enable CloudWatch Internet Monitor for internet-facing workloads
  1. Deploy Network Synthetic Monitor probes for hybrid connectivity
  1. Enable Network Flow Monitor for workload-level network performance visibility
  1. Build a unified CloudWatch Dashboard combining your network metrics, logs, and alarms
  1. Explore Kiro CLI for AI-assisted network troubleshooting and operational health checks

Conclusion

Network observability isn't a single tool — it's a practice built on layered telemetry, smart alerting, effective visualization, and increasingly, AI-assisted operations. The AWS networking stack provides deep instrumentation at every layer, but the key is knowing which metrics matter, where to set thresholds, and how to correlate signals across services.
We hope this guide gives you a concrete starting point to strengthen your network observability posture.
# aws-networking# networking# amazon-vpc# aws-network-firewall# monitoring-observability
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article