Security Monitoring in AWS: CloudWatch Logs, Login Failure Alerts, and NAT Gateway Traffic
Effective cloud security requires more than configuring firewalls and restricting access. It also requires visibility into what is happening across the environment, the ability to identify suspicious activity, and a reliable way to alert the right people when something unusual occurs.
Effective cloud security requires more than configuring firewalls and restricting access. It also requires visibility into what is happening across the environment, the ability to identify suspicious activity, and a reliable way to alert the right people when something unusual occurs.
In a recent AWS Builder Lab, Security Monitoring, I worked through a practical scenario involving a company that needed to monitor its network and Amazon EC2 instances for abnormal activity.
The exercise focused on three key areas: collecting security logs from an EC2 instance, creating CloudWatch alarms to detect repeated authentication failures, and monitoring outgoing network traffic through a NAT gateway.
This hands-on experience demonstrated how Amazon CloudWatch, CloudWatch Logs, metric filters, Amazon Simple Notification Service (Amazon SNS), and Amazon VPC monitoring metrics can work together to improve security visibility in an AWS environment.
Understanding the lab architecture
The lab used a simulated company environment with a web application and a database server deployed inside an Amazon Virtual Private Cloud (Amazon VPC).
The architecture included:
- A public subnet containing the internet-facing network infrastructure.
- A Network Load Balancer deployed across two public subnets.
- An EC2 instance running the web server in a private subnet.
- A separate EC2 instance acting as the database server in another subnet.
- Security groups assigned to the instances according to their roles.
External traffic passed through an internet gateway and the Network Load Balancer before reaching the web server. Requests to the WordPress application could also involve communication with the database server.
Because the database server contained sensitive information, monitoring its security logs was particularly important. I also needed to detect repeated login failures and unusual outbound network traffic that could indicate unexpected activity.
1. Collecting EC2 security logs with Amazon CloudWatch
The first task was to configure the database server to send local security logs to Amazon CloudWatch.
I connected to the Database Server EC2 instance using AWS Systems Manager Session Manager. This allowed me to access the Linux command line through the AWS console without opening a separate SSH connection.
The instance was running Amazon Linux 2023.
Installing the CloudWatch agent
I installed the Amazon CloudWatch agent using the package manager:
1
sudo dnf install -y amazon-cloudwatch-agent
The agent collects configured log files and sends their contents to CloudWatch Logs. This makes operating-system events available for centralized monitoring instead of requiring an administrator to inspect the instance's local files manually.
After installation, I launched the configuration wizard:
1
sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-config-wizard
During configuration, I selected Linux as the operating system, identified the host as an EC2 instance, and configured the agent to run as root.
I then specified the following settings:
| Configuration | Value |
|---|---|
| Log file | /var/log/secure |
| Log group | database_server_security_logs |
| Log stream | {instance_id} |
| Log retention | 1 day |
The
/var/log/secure file was the source of the authentication-related events monitored in this lab.Once the configuration was generated, I started the agent and loaded the configuration file:
1
sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -s -c file:/opt/aws/amazon-cloudwatch-agent/bin/config.json
To verify that the agent was running, I executed:
1
sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -m ec2 -a status
The expected status output included:
1
2
3
4
{
"status": "running",
"configstatus": "configured"
}
This confirmed that the agent was running with a loaded configuration.
Generating and verifying authentication failures
To generate test events, I attempted to switch to the
dbdev account using:1
su dbdev
I entered an incorrect password and repeated the attempt until I had generated four failed login attempts.
Next, I opened CloudWatch Logs and navigated to the
database_server_security_logs log group. I selected the log stream associated with the database server's EC2 instance ID and inspected the recorded events.This verified that the local security logs were being forwarded to CloudWatch.
Key takeaway: Centralized log collection provides a foundation for security monitoring. Before creating alerts, the relevant events must be collected and made available for analysis.
2. Creating a metric filter for failed logins
Collecting logs is useful, but manually reviewing every event is not an efficient way to detect suspicious activity.
The next step was to create a CloudWatch Logs metric filter that could identify authentication failures automatically.
Within the
database_server_security_logs log group, I created a metric filter using this filter pattern:1
"authentication failure"
The filter searched incoming log events for that phrase. I tested the pattern against the database server's log stream and checked that the results corresponded to the authentication failures generated earlier.
I then configured the metric filter with the following values:
| Setting | Value |
|---|---|
| Filter name | database server authentication failures |
| Metric namespace | authentication failures |
| Metric name | database server authentication failures |
| Metric value | 1 |
| Default value | 0 |
Each matching log event contributed a value of
1 to the custom metric.This converted text-based log events into numerical data that CloudWatch could evaluate over a period of time.
The distinction is important: the metric filter identifies matching log messages, while the alarm evaluates the resulting metric against a threshold.
3. Creating a CloudWatch alarm for repeated authentication failures
With the metric filter in place, I created a CloudWatch alarm to detect repeated failed login attempts.
The alarm used the custom metric generated by the filter and the following configuration:
| Setting | Value |
|---|---|
| Metric | database server authentication failures |
| Statistic | Sum |
| Evaluation period | 5 minutes |
| Threshold type | Static |
| Condition | Greater than 2 |
The alarm was configured to enter the
ALARM state when the sum of matching authentication failures exceeded two during a five-minute period.In practical terms, this meant that three or more matching failures within the configured period could trigger the alarm.
I also configured the missing-data treatment as Treat missing data as good (not breaching threshold). This setting prevents missing metric data from automatically being treated as a threshold breach, although it does not prove that the system is healthy when data is absent.
Sending notifications with Amazon SNS
An alert is more useful when it reaches someone who can investigate it.
To configure notifications, I created an Amazon SNS topic named:
1
CloudWatch_alarm_notifications
I supplied an email address as the notification endpoint and confirmed the subscription using the confirmation email sent by AWS Notifications.
I then named the alarm:
1
database server authentication failures alarm
After creating the alarm, I checked its state in the CloudWatch console. If the threshold was exceeded, the alarm could enter the
ALARM state and send an email notification through SNS.CloudWatch alarm state changes can take a short time to appear, so I refreshed the console after generating additional failed login attempts.
This exercise showed how logs, metric filters, alarms, and notifications form a practical monitoring workflow.
Security lesson: Repeated authentication failures can indicate anything from a user repeatedly entering an incorrect password to a more concerning access attempt. An alert provides a signal for investigation; it does not, by itself, establish that an attack has occurred.
4. Monitoring outgoing network traffic through a NAT gateway
The final task focused on network monitoring rather than authentication activity.
A NAT gateway allows resources in private subnets to initiate connections to external destinations without requiring those resources to have public IP addresses. Monitoring its traffic can help teams understand outbound network activity and identify unexpected changes in traffic volume.
For this exercise, I created a CloudWatch alarm using the NAT gateway metric:
1
BytesOutToDestination
I selected the metric associated with the lab's NAT gateway and configured the alarm as follows:
| Setting | Value |
|---|---|
| Metric | BytesOutToDestination |
| Statistic | Sum |
| Period | 15 minutes |
| Threshold type | Static |
| Condition | Greater than 2,000,000 bytes |
| Notification topic | CloudWatch_alarm_notifications |
| Alarm name | NAT gateway outgoing traffic alarm |
The alarm was designed to detect when the total value of the selected outbound-traffic metric exceeded two million bytes during a 15-minute period.
Testing the alarm
To generate outbound traffic, I returned to the database server's Session Manager terminal and set an environment variable to the S3 test bucket name supplied by the lab:
1
TestBucket=TEST_BUCKET
I replaced
TEST_BUCKET with the actual bucket name provided in the lab environment.Next, I uploaded the sample file:
1
aws s3 cp ~/SampleData.zip s3://$TestBucket/
The file was approximately 2.3 MB. After the upload, I returned to CloudWatch and refreshed the alarms page.
The lab instructions indicated that the
NAT gateway outgoing traffic alarm should enter the ALARM state after the metric crossed the configured threshold, with an email notification sent through the existing SNS topic.This demonstrated how network traffic metrics can complement operating-system logs. Authentication monitoring focuses on access-related events, while NAT gateway monitoring provides visibility into a different part of the environment: outbound network activity.
It is worth noting that high outbound traffic is not automatically malicious. Software updates, legitimate data transfers, backups, and application workloads can all generate significant traffic. The threshold should therefore be tuned to the environment's normal behavior.
How the monitoring workflow fits together
The lab demonstrated a straightforward monitoring pipeline:
- Collect: The CloudWatch agent forwards the database server's local security logs to CloudWatch Logs.
- Filter: A metric filter identifies log events containing
authentication failure. - Measure: Each matching event contributes to a custom CloudWatch metric.
- Detect: A CloudWatch alarm evaluates the metric against the configured threshold.
- Notify: Amazon SNS sends an email notification when the alarm enters the configured state.
- Monitor network activity: A separate alarm evaluates the NAT gateway's outbound-traffic metric.
These components address different stages of monitoring. Log collection creates visibility, metric filters turn selected events into measurable data, alarms evaluate that data, and notifications bring potential issues to an administrator's attention.
Practical lessons from the lab
Several lessons stood out from this exercise.
1. Centralized logs make investigation easier
Keeping security events available in CloudWatch Logs makes them easier to review and use for automated monitoring. It also reduces dependence on manually accessing each instance to inspect local log files.
2. Log collection must be configured correctly
Installing the agent is only part of the process. The configuration must specify the correct log file, log group, and stream settings, and the agent must be running successfully. In addition, the instance needs the appropriate IAM permissions and network connectivity to publish logs.
3. Metric filters connect log events to alarms
A metric filter turns selected log patterns into numerical measurements. Those measurements can then be evaluated by CloudWatch alarms, making it possible to detect specific events without continuously reviewing raw log messages.
4. Thresholds should reflect expected behavior
The authentication alarm and NAT gateway alarm used fixed thresholds for the lab. In a production environment, thresholds should be informed by normal operating patterns, application requirements, and the consequences of missed or excessive alerts.
5. Notifications need an operational response
An email alert is only the beginning of an incident response process. Teams should know who receives alerts, who investigates them, how they are escalated, and what evidence should be collected before taking action.
6. Missing data requires careful interpretation
Treating missing data as non-breaching may be appropriate for certain metrics, but it can also hide a monitoring gap if the metric stops reporting. Important monitoring systems should have separate checks for telemetry availability and agent health.
Conclusion
This AWS Builder Lab provided hands-on experience with monitoring security events and network activity in a cloud environment.
I configured an Amazon Linux 2023 EC2 instance to send security logs to CloudWatch, created a metric filter to detect authentication failures, configured a CloudWatch alarm with Amazon SNS email notifications, and created a second alarm to monitor outbound traffic through a NAT gateway.
The most important lesson was that effective security monitoring requires more than collecting data. Logs and metrics need to be connected to meaningful detection rules, appropriate thresholds, and a notification process that supports timely investigation.
For anyone learning AWS security or preparing for a cloud security role, understanding this workflow is a useful foundation for building better visibility into cloud workloads and responding to potentially abnormal activity.
AWS services and technologies covered
- Amazon CloudWatch
- Amazon CloudWatch Logs
- CloudWatch agent
- CloudWatch metric filters and alarms
- Amazon Simple Notification Service (SNS)
- Amazon EC2
- Amazon VPC and NAT gateways
- AWS Systems Manager Session Manager
- Amazon S3
- Amazon Linux 2023
- Linux command-line interface
Tags: AWS, Cloud Security, Security Monitoring, Amazon CloudWatch, CloudWatch Logs, Amazon SNS, EC2, NAT Gateway, Incident Detection, Cybersecurity
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article