Security Engineering on AWS: Investigating EC2 Compromise and Automating Incident Response
Cloud security involves more than preventing unauthorized access. When a potential security incident occurs, teams must preserve evidence, investigate suspicious activity, contain the affected resources, remediate the underlying weakness, and prepare for similar incidents in the future.
Cloud security involves more than preventing unauthorized access. When a potential security incident occurs, teams must preserve evidence, investigate suspicious activity, contain the affected resources, remediate the underlying weakness, and prepare for similar incidents in the future.
In this hands-on AWS Builder Lab, Security Engineering on AWS – Incident Response, I worked through a simulated EC2 security incident involving repeated SSH authentication failures. The exercise demonstrated how AWS services can support evidence collection, incident investigation, instance isolation, secure administrative access, and automated response.
The main objective was to move beyond identifying suspicious activity and implement a repeatable process for responding to it.
Understanding the Incident Response Environment
The lab used a simulated AWS environment containing an EC2 instance identified as potentially compromised, a jump server, a second application server for testing, and preconfigured logging and automation components.
The incident response workflow involved these AWS services:
- Amazon EC2: Identifying, protecting, snapshotting, and isolating affected instances.
- Amazon EBS: Capturing persistent storage through volume snapshots.
- Amazon S3: Storing incident metadata and collected artifacts.
- Amazon CloudWatch Logs: Reviewing authentication logs to investigate suspicious activity.
- AWS Identity and Access Management (IAM): Controlling the permissions attached to EC2 instances.
- AWS Systems Manager Session Manager: Providing administrative access without requiring inbound SSH connectivity.
- Amazon EventBridge: Matching events and initiating automated response actions.
- AWS Lambda: Executing the predefined incident response automation.
The workflow followed four broad stages: evidence collection, investigation, remediation, and automation.
Capturing EC2 Metadata and Preserving Evidence
The first stage was to collect information about the potentially compromised EC2 instance before proceeding with containment.
I accessed the jump server through the provided Systems Manager Session Manager URL and defined shell variables for the compromised instance ID, AWS Region, incident response bucket, and metadata filename.
1
2
3
4
compromised_instance_id=REPLACE_WITH_INSTANCE_ID
lab_region=REPLACE_WITH_REGION
ir_bucket=REPLACE_WITH_IR_BUCKET
metadata_file=metadata_$compromised_instance_id.jsonThe placeholders must be replaced with the values provided by the active lab environment.
I then used the AWS CLI to retrieve the instance description and save it locally:
1
2
3
4
aws ec2 describe-instances \
--instance-ids "$compromised_instance_id" \
--region "$lab_region" \
> "$metadata_file"This command collected the EC2 instance description in JSON format, providing a record of its configuration and associated metadata.
Next, I uploaded the file to the incident response S3 bucket:
1
2
3
4
aws s3api put-object \
--bucket "$ir_bucket" \
--body "$metadata_file" \
--key "$metadata_file"I could then verify that the object appeared in the S3 bucket.
This step demonstrated the value of collecting incident artifacts in a separate location. The metadata provides useful context for investigation and can support later analysis of the affected resource.
Enabling EC2 termination protection
Before taking further action, I enabled termination protection on the compromised instance.
This setting helps prevent accidental termination while evidence collection is in progress. It does not prevent an attacker from accessing the instance, nor does it replace network isolation.
The lab also highlighted an important storage consideration: the
DeleteOnTermination setting determines whether an attached EBS volume is deleted when its instance is terminated. Preserving the required storage artifacts is an important part of planning an incident response.Creating an EBS Snapshot for Forensic Investigation
The next step was to identify the EBS volume attached to the compromised instance and create a snapshot.
In the EC2 console, I navigated to Elastic Block Store → Volumes, identified the volume associated with the affected instance, and selected Create snapshot.
I used the following snapshot details:
- Description: A snapshot of the compromised ec2 volume
- Name tag:
compromised_ec2_snapshot
After creating the snapshot, I checked its status and waited for it to reach
Completed.A snapshot provides a point-in-time copy of persistent block storage that can support subsequent investigation. The lab also demonstrated how an Amazon Machine Image (AMI) can be created from the snapshot to support launching a separate instance for forensic analysis.
However, it is important to distinguish persistent disk capture from memory acquisition. The lab did not perform a live memory capture. It explained that tools such as the Linux Memory Extractor (LiME) can be used for memory acquisition, with Systems Manager Run Command providing a way to execute scripts without opening an interactive session on the compromised host.
The optional forensic-instance deployment was also described as a possible next step rather than a mandatory part of the exercise.
Quarantining the Compromised Instance
Once the initial artifacts had been collected, I proceeded to contain the affected instance.
First, I applied a tag to make its status explicit:
| Tag key | Tag value |
|---|---|
| Status | Quarantine |
Tags provide a visible indicator of resource status and can also be used as inputs for automation workflows.
Next, I detached the existing IAM role from the instance. This reduced the instance's access to AWS resources through its attached instance profile while containment was in progress.
I then replaced the original application security group,
APP-SG, with the lab's Quarantine-SG, which was configured to block inbound and outbound traffic.The objective was to restrict the instance's network communication while preserving the evidence already collected.
Why isolation requires careful handling
Security group changes are useful for containment, but they should not be treated as a guarantee that every existing connection has been terminated. The lab specifically highlights the role of security group connection tracking: modifying security group rules does not necessarily terminate already tracked connections immediately.
In a production environment, the response team must evaluate the wider network architecture and take any additional containment steps required by the incident.
The lab's response sequence therefore emphasized three separate actions: preserve the evidence, reduce the instance's permissions, and restrict its network access.
Investigating SSH Authentication Failures with CloudWatch Logs
After collecting the initial artifacts, I investigated the suspicious activity through Amazon CloudWatch Logs.
The lab environment already sent the relevant SSH logs to a CloudWatch log group named:
ec2-ssh-log-groupI opened the log group, selected the stream corresponding to the compromised instance ID, and searched the log events for authentication failures.
The records showed repeated failed SSH login attempts. These attempts could indicate a brute-force attack, although authentication failures alone do not conclusively prove that an instance has been compromised.
This distinction matters in incident response. Investigators need to correlate log evidence with other findings before determining the scope and cause of an incident.
Based on the lab scenario, the recommended remediation was to remove the need for inbound SSH access and use Systems Manager Session Manager for administrative access instead.
This approach reduces the need to expose SSH through an instance's security group and provides access control through IAM. Session Manager activity can also be audited through AWS logging capabilities, including CloudTrail for relevant API activity.
Replacing SSH Access with Systems Manager Session Manager
The next stage was to implement the recommended remediation.
Step 1: Grant the required Systems Manager permissions
I navigated to IAM, opened the
app-server-role, and attached the AWS managed policy:AmazonSSMManagedInstanceCoreThis policy provides permissions used by an EC2 instance to participate in Systems Manager management.
Because the role had previously been detached during containment, I reattached the instance profile to the EC2 instance.
The lab noted that a reboot might be required if the SSM Agent needed to refresh its access to the instance profile.
Step 2: Remove the inbound SSH rule
I opened the original
APP-SG security group and removed its inbound SSH rule.I then replaced
Quarantine-SG with APP-SG on the instance. The original security group had been updated, so restoring it no longer restored the SSH inbound rule.This is an important remediation principle: removing a temporary containment measure should not reintroduce the original vulnerability.
Step 3: Validate access
Finally, I opened the EC2 instance's connection options and selected the Session Manager tab.
After confirming that a session could be established, I ended the session and removed the
Status=Quarantine tag.The instance was returned to service with the lab's remediation applied: administrative access through Session Manager and no inbound SSH rule in the application security group.
The broader lesson is that remediation should be verified before a resource is considered ready for normal operation.
Automating Incident Response with EventBridge and Lambda
Manual incident response is valuable for understanding the process, but repeating every action by hand can delay containment when multiple incidents occur.
The final task introduced an automated workflow using Amazon EventBridge and AWS Lambda.
The lab supplied most of the supporting components in advance. These included a CloudWatch Logs subscription filter, a Log Enrichment Lambda function, and an Automation Lambda function. The task was to configure the EventBridge rule that connected the incident event to the automated response.
How the workflow operates
The event-processing sequence was:
- Failed SSH authentication attempts generate log events on an application server.
- CloudWatch Logs receives the authentication logs.
- A preconfigured subscription filter forwards matching events to the Log Enrichment Lambda function.
- The enrichment function identifies the application's internal IP address and retrieves the associated EC2 instance ID.
- The function publishes an event to EventBridge with the instance information.
- An EventBridge rule matches the event and invokes the Automation Lambda function.
- The automation performs the initial incident response actions on the affected instance.
This design separates log processing and event enrichment from the response actions themselves.
Creating the EventBridge rule
I created an EventBridge rule with the following configuration:
| Setting | Value |
|---|---|
| Rule name | EC2-IR-RULE |
| Description | A rule that will automate initial IR actions on EC2 |
| Event bus | default |
| Target | Lambda function |
| Target function | The lab-provided Automation Lambda ARN |
The event pattern specified in the lab was:
1
2
3
{
"source": ["log.lambda"]
}This pattern matches events whose
source field is log.lambda. The rule was configured to invoke the Automation Lambda function using the ARN supplied by the lab environment.Testing the automated response
To test the configuration, I initiated a failed SSH authentication attempt against the second application server from the jump server, using the lab-provided command and hostname.
The test was designed to verify that the automated response performed the expected actions:
- Captured the instance metadata and uploaded it to the incident response S3 bucket.
- Enabled EC2 termination protection.
- Created a snapshot of the instance volume.
- Applied the
Status=Quarantinetag. - Detached the instance's current IAM role.
- Replaced the application security group with
Quarantine-SG.
I then checked the second application's EC2 instance settings and the incident response bucket to verify the expected changes.
The key outcome was that the workflow could initiate the same initial response actions demonstrated earlier, without requiring an operator to repeat each step manually.
Lessons Learned
This lab reinforced several practical lessons about cloud incident response.
1. Preserve evidence before making unnecessary changes. Collecting metadata and creating storage snapshots provides useful material for investigation. The response sequence should be planned to preserve the evidence needed to understand the incident.
2. Treat authentication failures as indicators to investigate. Repeated failed logins may suggest brute-force activity, but the evidence should be assessed carefully rather than treated as definitive proof of a successful compromise.
3. Containment involves more than changing a security group. Network restrictions, IAM role removal, termination protection, tagging, and evidence preservation address different aspects of the response.
4. Reduce unnecessary inbound administrative access. Using Systems Manager Session Manager can reduce reliance on inbound SSH connectivity while providing IAM-based access control and opportunities for audit logging.
5. Automate repeatable response actions. EventBridge and Lambda can connect detected events to predefined response procedures, helping reduce repetitive manual work.
6. Validate the recovery process. Before removing quarantine controls and returning an instance to service, verify that the intended remediation is in place and that the required administrative access works.
7. Test automation carefully. Automated containment can affect legitimate workloads if an event is misclassified or a rule is too broad. Response rules and permissions should be tested in a controlled environment before being used for production systems.
The Security Engineering on AWS – Incident Response lab demonstrated how to approach a simulated EC2 security incident from initial evidence collection through investigation, containment, remediation, and automation.
I used the EC2 and EBS workflows to preserve instance information and disk data, examined CloudWatch Logs to investigate repeated SSH authentication failures, isolated the affected instance, and replaced inbound SSH access with Systems Manager Session Manager. I then configured an EventBridge rule to invoke a Lambda-based response workflow and tested it against a second application server.
The most important takeaway is that effective incident response requires both a clear manual procedure and a reliable way to execute repeatable actions. AWS services such as EC2, EBS, S3, CloudWatch Logs, IAM, Systems Manager, EventBridge, and Lambda can work together to support that process.
For cloud security engineers, practicing this complete workflow helps build a better understanding of how investigation, evidence preservation, containment, and automation fit together in a real incident response strategy.
Check the complete documentation here
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article