Building an Infrastructure Monitoring Stack with Prometheus, Grafana, Docker, and Terraform
Explore how I built a production-style infrastructure monitoring stack with real-time metrics, Grafana dashboards, Prometheus alerting, and automated CI validation. This project explores how observability tools and Infrastructure as Code fit into a reliable DevOps workflow.
Building an Infrastructure Monitoring Stack with Prometheus, Grafana, Docker, and Terraform
Modern infrastructure needs more than applications that simply run. Engineers need visibility into system health, resource utilization, service availability, and potential failures before they become larger problems.
To explore these challenges, I built an infrastructure monitoring stack using Prometheus, Grafana, Alertmanager, Node Exporter, Docker Compose, Terraform, and GitHub Actions.
The goal was to create a reproducible monitoring environment with centralized metrics, dashboards, alerting rules, automated health checks, and continuous integration validation.
Project links
- GitHub repository: https://github.com/Cgarg547/AWS-Infrastructure-Monitoring
- Live portfolio: https://aws-infrastructure-monitoring.vercel.app/
- CI workflow: https://github.com/Cgarg547/AWS-Infrastructure-Monitoring/actions/workflows/ci.yml
1. Project Overview
The project brings together several tools commonly used in modern infrastructure monitoring and DevOps workflows.
The stack collects system-level metrics, evaluates monitoring rules, routes alerts, and visualizes infrastructure performance through Grafana dashboards.
The main components are:
- Prometheus: Scrapes and stores time-series metrics and evaluates alerting and recording rules.
- Grafana: Visualizes metrics through monitoring dashboards.
- Node Exporter: Exposes host-level system metrics.
- Alertmanager: Handles alert grouping, routing, and notification policies.
- Docker Compose: Orchestrates the monitoring services in a reproducible local environment.
- Terraform: Defines infrastructure configuration for a potential AWS deployment.
- GitHub Actions: Automates configuration checks and validation in the CI workflow.
Rather than treating monitoring as a collection of separate tools, I wanted to understand how these components work together as one system.
2. Architecture
The architecture separates metric collection, metric evaluation, alert handling, and visualization.
Node Exporter exposes system metrics that Prometheus scrapes. Prometheus stores the collected time-series data and evaluates configured rules. Grafana queries Prometheus to display infrastructure health, while Alertmanager handles alerts generated by Prometheus.
Docker Compose coordinates the services, and GitHub Actions validates important parts of the configuration.
The simplified data flow is:
- Node Exporter exposes system metrics.
- Prometheus scrapes and stores those metrics.
- Recording rules calculate useful derived metrics.
- Alerting rules evaluate configured conditions.
- Alertmanager processes and routes firing alerts.
- Grafana displays metrics and infrastructure status.
This separation makes the stack easier to understand, troubleshoot, and extend.
3. Monitoring Infrastructure Health
A monitoring platform should help answer practical questions about the health of a system.
The Grafana dashboards in this project cover several areas:
CPU utilization
CPU metrics help identify periods of high processing demand and potential resource constraints.
Memory utilization
Memory monitoring helps reveal resource pressure and supports investigation of systems that may be approaching their available memory limits.
Disk utilization
Disk metrics provide visibility into storage capacity and can help identify potential capacity issues before they affect services.
Network traffic
Network metrics help track traffic patterns and provide useful context when investigating connectivity or performance problems.
Service health
The dashboards also include service-status information, including the availability of monitoring components such as Prometheus and Node Exporter.
These visualizations provide a central place to inspect infrastructure metrics rather than checking each service independently.
Dashboard screenshots
- Grafana dashboard overview: https://github.com/Cgarg547/AWS-Infrastructure-Monitoring/blob/main/site/images/grafana-dashboard1.png
- Infrastructure metrics dashboard: https://github.com/Cgarg547/AWS-Infrastructure-Monitoring/blob/main/site/images/grafana-dashboard.png
4. Alerting and Recording Rules
Dashboards help engineers understand what is happening, but they still require someone to inspect them.
Alerting adds another layer by evaluating defined conditions and identifying situations that may require attention.
This project includes Prometheus alert rules and Alertmanager routing for different severity levels, including warning and critical alerts.
The intended workflow is to evaluate configured conditions in Prometheus and pass firing alerts to Alertmanager for processing according to its routing configuration.
Recording rules complement alerting by precomputing frequently used expressions and making derived metrics easier to query.
Together, these features demonstrate how monitoring can evolve from passive visualization into a more structured approach to identifying infrastructure problems.
5. Docker Compose and Reproducibility
Running the monitoring components through Docker Compose makes it easier to bring up the environment consistently.
Instead of configuring each service independently, the project defines the monitoring stack in a coordinated setup.
This is useful for development because it reduces manual setup and makes it easier to reproduce the environment, inspect service logs, and troubleshoot configuration problems.
It also provides a foundation for testing configuration changes before considering a larger deployment.
6. Infrastructure as Code with Terraform
Terraform configuration is included to define infrastructure for a potential AWS deployment.
Infrastructure as Code is valuable because it makes infrastructure definitions version-controlled, reviewable, and repeatable.
For this project, Terraform configuration formatting and validation are included in the CI workflow.
An important implementation detail: the current monitoring stack runs locally using Docker Compose. The Terraform configuration is included and validated, but the project has not been deployed to AWS in this version.
This distinction matters because a validated infrastructure configuration is not the same as a successfully deployed and tested cloud environment.
7. CI Validation with GitHub Actions
Monitoring configurations are part of the system itself. A syntax error or invalid rule can undermine the reliability of the monitoring platform.
To help catch issues earlier, I integrated GitHub Actions into the project.
The CI workflow includes validation tasks for areas such as:
- YAML configuration files
- Prometheus configuration and rules
- Grafana dashboard JSON
- Shell-script syntax
- Terraform formatting and validation
Automating these checks helps identify configuration problems before changes are treated as ready.
It also introduces a useful DevOps principle: infrastructure and monitoring configurations should be tested and reviewed, not simply written and assumed to work.
The workflow is available here:
8. What I Learned
Building this project helped me explore the relationship between observability and infrastructure automation.
Some of the key lessons were:
- Monitoring is most useful when metrics, dashboards, and alerting rules work together.
- Recording rules can simplify frequently used metric calculations.
- Alert routing needs to be configured deliberately so different severity levels can be handled appropriately.
- Docker Compose provides a practical way to reproduce a multi-service development environment.
- CI validation can catch configuration issues before deployment.
- Terraform validation is an important step, but it does not prove that infrastructure will deploy successfully in AWS.
- A clear separation between implemented functionality and planned improvements makes technical projects easier to evaluate and extend.
The project also reinforced the importance of making monitoring configurations part of the software development workflow.
9. Future Improvements
There are several directions in which I would like to extend this project.
Deploying to AWS
Deploy the monitoring stack to AWS infrastructure using Terraform, then validate the deployment with real cloud resources.
Monitoring cloud resources
Explore integrating AWS CloudWatch metrics and alarms to complement host-level monitoring.
Alert notifications
Configure and test an actual notification destination so that alerts can reach the appropriate recipient.
Security hardening
Improve authentication, secrets management, network access controls, and secure exposure of monitoring endpoints before any public cloud deployment.
Operational testing
Introduce more systematic tests for alert behavior, service failures, recovery scenarios, and monitoring availability.
These are future improvements, not capabilities I am claiming are already implemented.
Conclusion
This project combines infrastructure monitoring, visualization, alerting, container orchestration, Infrastructure as Code, and CI validation into one practical engineering exercise.
It provides a foundation for exploring how DevOps teams can make infrastructure more observable and configuration changes more repeatable.
I am continuing to improve the project and explore how the stack could be extended into a deployed AWS monitoring environment.
I'd love to hear from the community: What would you prioritize next for a project like this—AWS CloudWatch integration, deployment using Terraform, more advanced alerting, or security hardening?
If you are working on observability or infrastructure automation, I'd also be interested in learning what tools and practices have worked well for you.
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article