Building a NOC Monitoring Lab on AWS with Zabbix
Learn how I built a small Network Operations Center (NOC) lab on AWS using Amazon EC2, Zabbix 7.4, PostgreSQL, and Zabbix Agent 2 to monitor Linux infrastructure and simulate real-world incidents.
Part 1: Infrastructure and Monitoring Setup
Why build a NOC monitoring lab?
In an IT environment, service availability is critical.
A server can run normally for hours and suddenly become unreachable, consume excessive CPU, or stop responding to requests. A Network Operations Center (NOC) is responsible for monitoring infrastructure, detecting anomalies, investigating incidents, and helping restore services.
For this project, I wanted to reproduce this operational workflow in a small AWS environment:
Detect → Investigate → Diagnose → Remediate → Verify
The objective was not simply to install a monitoring solution and display a few dashboards.
I wanted to build a small but realistic environment where I could practice:
- AWS infrastructure deployment
- Linux administration
- Network troubleshooting
- Monitoring and observability
- Security Groups
- Incident detection
- Troubleshooting and remediation
For the lab, I used Amazon EC2 as the infrastructure layer and Zabbix 7.4 as the monitoring platform.
1. AWS architecture
I created a small VPC containing two EC2 instances deployed in different subnets.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
AWS VPC
│
┌─────────────┴─────────────┐
│ │
NOC EC2 Monitored EC2
10.0.1.188 10.0.3.138
│ │
┌──────┴──────┐ Zabbix Agent 2
│ │ │
Zabbix Server PostgreSQL │
│ │
Nginx ◄──────── TCP 10050 ──────────┘
│
Zabbix Web UI
The architecture is intentionally simple.
The first EC2 instance acts as the NOC server and hosts:
- Zabbix Server
- PostgreSQL
- Nginx
- Zabbix Web Interface
The second EC2 instance represents a Linux server under monitoring and runs Zabbix Agent 2.
This separation makes the lab more realistic because the monitoring server has to communicate with a remote machine through the AWS network.
2. The NOC EC2 instance
The NOC instance is the central component of the environment.
Its role is to receive and process monitoring data from the monitored infrastructure.
1
2
3
4
5
6
7
NOC EC2
10.0.1.188
│
├── Zabbix Server
├── PostgreSQL
├── Nginx
└── Zabbix Web UI
Zabbix Server acts as the monitoring engine, PostgreSQL stores monitoring data and configuration, and Nginx serves the web interface.
The stack can therefore be represented as:
1
2
3
4
5
6
7
8
9
NOC EC2
│
┌────────────┼────────────┐
│ │ │
Zabbix Server PostgreSQL Nginx
│ │
│ PHP-FPM
│ │
└──────────────► Zabbix Web UI
This architecture helped me understand that a monitoring platform is not simply a single application. Several components have to work together to provide the complete monitoring environment.
3. The monitored EC2 instance
The second EC2 instance represents a Linux server under monitoring.
It runs:
1
2
3
4
Monitored EC2
10.0.3.138
│
└── Zabbix Agent 2
Zabbix Agent 2 collects information from the operating system and makes it available to Zabbix Server.
The main metrics I wanted to monitor included:
- CPU
- Memory
- Disk
- Processes
- System availability
- Network information
The agent configuration is located in:
1
/etc/zabbix/zabbix_agent2.conf
The important parameters in the lab are:
1
2
3
Server=10.0.1.188
ServerActive=10.0.1.188
Hostname=NOC-MONITORED-01
These parameters establish the relationship between the monitored server and the Zabbix server.
4. Network communication
The main communication channel between Zabbix Server and Zabbix Agent 2 uses:
1
TCP 10050
The communication path is therefore:
1
2
3
4
5
6
7
Zabbix Server
10.0.1.188
│
│ TCP/10050
▼
Zabbix Agent 2
10.0.3.138
Before configuring monitoring, I wanted to make sure that the network path was actually working.
From the NOC instance, I tested the connection with:
1
nc -vz 10.0.3.138 10050
A successful connection confirmed that the agent was listening and that the network path allowed the traffic.
This is an important troubleshooting principle:
A service can be running correctly while communication with it is still blocked somewhere in the network.
The lab therefore combines two perspectives:
1
2
3
4
5
Application / Service
+
Network / Security
↓
Complete monitoring path
5. AWS Security Groups
Security Groups were another important part of the lab.
For SSH access, I restricted the source to my public IP instead of allowing SSH access from anywhere.
For monitoring, the monitored instance needed to allow TCP traffic on port
10050 from the Zabbix server.Conceptually:
1
2
3
4
5
6
7
8
9
Internet
│
└── SSH ──► NOC EC2
Source: My IP
NOC EC2
│
└── TCP 10050 ──► Monitored EC2
Source: Zabbix Server
This configuration gave me a practical opportunity to understand the relationship between AWS networking, Security Groups, Linux services, and monitoring software.
It also became useful later during incident troubleshooting.
6. Installing the monitoring stack
On the NOC instance, I installed the main Zabbix components together with PostgreSQL and Nginx.
The resulting stack was:
1
2
3
4
5
6
7
8
9
NOC EC2
│
┌────────────┼────────────┐
│ │ │
Zabbix Server PostgreSQL Nginx
│ │
│ PHP-FPM
│ │
└──────────────► Zabbix Web UI
PostgreSQL stores information such as hosts, items, triggers, events, and monitoring history.
Zabbix Server processes the monitoring data, while the web frontend provides the interface used to operate the platform.
I verified the main services using Linux commands such as:
1
2
3
sudo systemctl status postgresql
sudo systemctl status zabbix-server
sudo systemctl status nginx
I also checked listening ports with:
1
sudo ss -lntp
These commands became particularly useful later when investigating incidents.
7. Installing Zabbix Agent 2
On the monitored EC2 instance, I installed and enabled Zabbix Agent 2:
1
2
sudo apt install zabbix-agent2 -y
sudo systemctl enable --now zabbix-agent2
I then verified that the service was running:
1
sudo systemctl status zabbix-agent2 --no-pager
and that it was listening on port
10050:1
sudo ss -lntp | grep 10050
At this point, the basic communication path was ready:
1
2
3
4
5
6
7
8
9
10
11
AWS VPC
│
├── NOC EC2
│ │
│ └── Zabbix Server
│
│ TCP 10050
│ │
└── Monitored EC2
│
└── Zabbix Agent 2
8. Adding the monitored host to Zabbix
With the infrastructure ready, I added the monitored EC2 instance to Zabbix as a host.
The configuration was:
1
2
3
4
Host name : NOC-MONITORED-01
Interface : Agent
IP address: 10.0.3.138
Port : 10050
I then associated the host with a Linux monitoring template so that Zabbix could automatically begin collecting system metrics.
Once the communication between the server and agent was working, the host became available in the Zabbix interface.
This was the point where the infrastructure became a real monitoring environment rather than simply two connected EC2 instances.
9. Preparing the first monitoring scenario
The first scenario was designed around CPU utilization.
I created a Zabbix trigger that would generate a problem when CPU utilization exceeded 80%.
The purpose was to transform a simple metric into an actionable event.
The logic becomes:
1
2
3
4
5
6
7
8
9
CPU metric
↓
CPU > 80%
↓
Trigger
↓
Problem
↓
NOC investigation
At this point, the infrastructure and monitoring rules were ready.
The next step was to deliberately create an incident and observe how Zabbix reacted.
What comes next?
The infrastructure is now ready.
I have:
- deployed the AWS environment;
- created the NOC and monitored EC2 instances;
- configured the network communication;
- secured access with Security Groups;
- installed Zabbix Server;
- configured PostgreSQL and Nginx;
- installed Zabbix Agent 2;
- connected the monitored host to Zabbix;
- created the first CPU monitoring trigger.
But the most interesting part of the lab starts now.
In Part 2, I will use this environment to simulate real incidents and follow the complete NOC troubleshooting process.
The first incident will simulate abnormally high CPU usage, from the creation of the problem to its detection and recovery.
The second incident will simulate Zabbix Agent 2 becoming unavailable, requiring network, service, and log investigation before restoring the monitoring service.
Part 2: Incident Detection and Troubleshooting
In Part 1, I built the monitoring infrastructure on AWS using two EC2 instances, Zabbix Server, PostgreSQL, Nginx, and Zabbix Agent 2.
The infrastructure was ready. The next step was to test something more important than monitoring itself:
What happens when something goes wrong?
A monitoring system becomes truly useful when it can detect an abnormal situation and give an engineer enough information to investigate and restore the service.
For this lab, I simulated two incidents:
- High CPU utilization
- Zabbix Agent 2 becoming unavailable
The objective was to reproduce a simplified NOC workflow:
1
2
3
4
5
6
7
8
9
10
11
MONITOR
↓
DETECT
↓
INVESTIGATE
↓
DIAGNOSE
↓
REMEDIATE
↓
VERIFY
1. Incident #1 — High CPU utilization
The first incident was designed to simulate a server consuming an abnormal amount of CPU.
The trigger configured in Part 1 was:
1
CPU utilization > 80%
This means that Zabbix should generate a problem when the monitored server exceeds the configured threshold.
1.1 Simulating abnormal CPU usage
To test the trigger, I deliberately generated CPU load on the monitored EC2 instance using
stress-ng.The goal was to reproduce a situation where a server suddenly consumes a large amount of CPU.
The command used for the simulation was:
1
stress-ng --cpu 0 --timeout 5m
The idea was to create a sustained CPU load long enough for Zabbix to detect the abnormal utilization.
This is important because the incident is not manually created inside Zabbix.
Instead:
1
2
3
4
5
6
7
8
9
Linux server
↓
CPU load increases
↓
Zabbix Agent 2 collects the metric
↓
Zabbix Server evaluates the trigger
↓
Problem is generated
1.3 Zabbix detects the incident
After the CPU utilization exceeded the configured threshold, Zabbix detected the problem.
Image 3 — CPU Alert Detected in Zabbix
This is the moment where the monitoring system becomes operationally useful.
The engineer does not need to continuously watch the server's CPU usage.
Zabbix detects the abnormal condition and raises the problem.
The workflow is now:
1
2
3
4
5
6
7
CPU > 80%
↓
Trigger activated
↓
Problem detected
↓
NOC investigation
1.4 Observing the incident over time
I then allowed the incident to continue for several minutes in order to observe how the monitoring data evolved.
This view is useful because monitoring is not only about detecting the initial event.
It also provides historical context.
An engineer can ask:
- When did the problem start?
- How long has it lasted?
- Is the CPU still elevated?
- Is the situation getting better or worse?
This is one of the differences between simply checking a server manually and having a monitoring system continuously collect data.
1.5 Recovery
After the test period, the generated CPU load stopped.
Zabbix then observed that CPU utilization returned to a normal level.
Image 5 — CPU Incident Recovery
The complete lifecycle of the incident was therefore:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
Normal state
↓
CPU load generated
↓
CPU > 80%
↓
Trigger
↓
Problem
↓
CPU load stops
↓
CPU returns to normal
↓
Recovery detected
This first scenario demonstrates a fundamental monitoring concept:
A monitoring system should detect both the problem and the recovery.
2. Incident #2 — Zabbix Agent becomes unavailable
The second scenario was more representative of a typical infrastructure troubleshooting situation.
Instead of creating a resource problem, I deliberately made the monitoring agent unavailable.
The objective was to reproduce a situation where:
The monitored server is running, but the monitoring system can no longer communicate with the agent.
This is interesting because the engineer has to investigate several layers:
1
2
3
4
5
6
7
8
9
Zabbix
↓
Network
↓
Port
↓
Service
↓
Logs
2.1 Establishing the baseline
Before simulating the incident, I checked the current state of the monitored server.
This image should show the results of the commands you used to verify the system, for example:
1
2
sudo systemctl status zabbix-agent2 --no-pager
sudo ss -lntp | grep 10050
If you also checked the connectivity from the NOC server, you can include:
1
nc -vz 10.0.3.138 10050
This establishes an important reference point.
Before the incident:
1
2
3
4
Agent 2 → Running
Port 10050 → Listening
TCP connectivity → Available
Zabbix → Monitoring normally
2.2 Simulating the failure
I then deliberately stopped Zabbix Agent 2 on the monitored EC2 instance.
The command was:
1
sudo systemctl stop zabbix-agent2
At this point, the agent is no longer available to answer requests from Zabbix Server.
The expected result is:
1
2
3
4
5
6
7
8
Zabbix Server
│
│ TCP 10050
✕
│
Zabbix Agent 2
│
└── stopped
2.3 Zabbix detects the unavailable agent
After a short period, Zabbix detected that the agent was no longer available.
The monitoring platform has now detected the symptom.
But an alert alone does not explain the cause.
The NOC engineer has to investigate.
3. Investigating the incident
This is where the lab becomes more interesting.
Instead of immediately restarting the service, I followed a troubleshooting sequence.
The objective was to determine where the communication path was broken.
1
2
3
4
5
6
7
8
9
10
11
ZBX unavailable
↓
Network test
↓
Port check
↓
Service status
↓
Log analysis
↓
Root cause
3.1 Checking the service
The first step was to check the state of Zabbix Agent 2:
1
sudo systemctl status zabbix-agent2 --no-pager
The expected result after the simulation was:
1
Active: inactive (dead)
This immediately indicated that the agent service was no longer running.
3.2 Checking the port
I then checked whether anything was listening on port
10050:1
sudo ss -lntp | grep 10050
Because Zabbix Agent 2 had been stopped, the port was no longer listening.
This helped connect the symptoms together:
1
2
3
4
5
6
7
Zabbix alert
↓
Agent unavailable
↓
Port 10050 unavailable
↓
Agent service stopped
This screenshot is particularly valuable because it demonstrates the troubleshooting process, not just the final result.
3.3 Checking the logs
The next step was to examine the service logs:
1
sudo journalctl -u zabbix-agent2 -n 30 --no-pager
The logs confirmed that the service had been stopped.
This illustrates an important principle in infrastructure operations:
An alert tells you that something is wrong. Logs help explain what happened.
The troubleshooting process therefore moved from the monitoring platform back to the operating system.
4. Remediation and recovery
Once the cause was identified, the corrective action was straightforward.
I restarted Zabbix Agent 2:
1
sudo systemctl start zabbix-agent2
I then verified the service:
1
sudo systemctl status zabbix-agent2 --no-pager
and verified that the port was listening again:
1
sudo ss -lntp | grep 10050
Finally, I tested the network connection again from the NOC server:
1
nc -vz 10.0.3.138 10050
The communication path was restored.
4.1 Zabbix detects the recovery
After the agent became available again, Zabbix detected the recovery.
The complete incident lifecycle was:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
Normal state
↓
Agent stopped
↓
Zabbix detects unavailability
↓
Service investigation
↓
Port investigation
↓
Log analysis
↓
Agent restarted
↓
Port available
↓
Zabbix detects recovery
5. What this lab taught me
These two scenarios demonstrated two different types of monitoring incidents.
Incident 1 — Resource problem
1
2
3
4
5
6
7
8
9
10
11
CPU usage
↓
Threshold exceeded
↓
Trigger
↓
Alert
↓
CPU returns to normal
↓
Recovery
Incident 2 — Service availability problem
1
2
3
4
5
6
7
8
9
10
11
12
13
Agent stopped
↓
Port unavailable
↓
Zabbix alert
↓
Service investigation
↓
Logs
↓
Agent restarted
↓
Recovery
The most important lesson for me was that monitoring is only the first step.
Zabbix can detect an abnormal condition, but the engineer still needs to understand what is happening underneath.
For an unavailable host, for example, several different causes could produce a similar symptom:
1
2
3
4
5
6
7
8
ZBX unavailable
│
├── Agent stopped
├── Port blocked
├── Security Group
├── Network problem
├── Wrong configuration
└── Server unavailable
The role of the NOC engineer is to progressively eliminate these possibilities.
6. From monitoring to NOC operations
This lab allowed me to connect several areas of infrastructure engineering in a single environment:
- AWS EC2
- VPC networking
- Security Groups
- Linux administration
- Zabbix Server
- Zabbix Agent 2
- PostgreSQL
- Nginx
- TCP connectivity
- Linux service management
- Log analysis
- Monitoring and alerting
- Incident troubleshooting
The final workflow can be summarized as:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
MONITORING
│
▼
DETECTION
│
▼
ALERT
│
▼
INVESTIGATION
│
▼
DIAGNOSIS
│
▼
REMEDIATION
│
▼
RECOVERY
│
▼
VERIFICATION
For me, the most valuable part of the project was not simply getting Zabbix to display a green status.
It was deliberately breaking the environment and then using the available tools to understand why it broke and how to restore it.
That is the mindset I wanted to practice with this NOC lab.
Conclusion
This project started as a small AWS monitoring experiment, but it became a practical exercise in infrastructure operations.
By combining AWS, Linux and Zabbix, I was able to reproduce two simplified but realistic incident scenarios and follow them from detection to recovery.
The next evolution of this lab could include:
- disk space exhaustion;
- high memory utilization;
- network connectivity failures;
- automated notifications;
- advanced Zabbix dashboards;
- multiple monitored servers;
- automated remediation;
- integration with an incident-management workflow.
The objective is to continue moving from monitoring infrastructure toward operating and responding to infrastructure incidents.
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article