
Building a QA Monitoring Stack with Prometheus, Grafana, and Alert manager
A hands-on look at building and troubleshooting a QA monitoring stack using Node Exporter, PostgreSQL Exporter, Prometheus, Grafana, Loki, and Alert manager, including metrics collection, dashboards, alerting, and troubleshooting.
Introduction
Recently, I got hands-on experience working with a QA infrastructure monitoring setup using open-source observability tools.
The goal was to collect infrastructure and PostgreSQL metrics, visualize them in Grafana, collect logs, and configure alerts for important infrastructure and database conditions.
The monitoring stack included:
- Node Exporter
- PostgreSQL Exporter
- Prometheus
- Grafana
- Loki
- Alert manager
- Docker and Docker Compose
Monitoring Architecture
The overall monitoring flow can be represented as:
Linux Server → Node Exporter → Prometheus → Grafana
PostgreSQL → PostgreSQL Exporter → Prometheus → Grafana
Applications/Services → Logs → Loki → Grafana
Prometheus → Alert Rules → Alertmanager → Notifications
This helped me understand how metrics, logs, dashboards, and alerts work together as part of an observability setup.
1. Node Exporter
Node Exporter is used to expose Linux system-level metrics for Prometheus.
I worked with Node Exporter to monitor:
- CPU utilization
- Memory usage
- Disk usage
- Filesystem capacity
- System load
- Network-related metrics
Node Exporter exposes metrics that Prometheus can scrape.
The default Node Exporter port is:
9100 After configuring the service, I verified that the exporter was running and that metrics were available.
2. PostgreSQL Exporter
For PostgreSQL monitoring, I worked with PostgreSQL Exporter.
The main steps included:
- Creating a dedicated PostgreSQL monitoring user
- Configuring the required permissions
- Configuring PostgreSQL Exporter
- Starting the exporter service
- Connecting it to Prometheus
- Validating the exported metrics
The PostgreSQL Exporter endpoint commonly uses:
9187 One useful metric for validating PostgreSQL exporter connectivity is:
pg_up A value of:
pg_up 1 indicates that the exporter can successfully communicate with PostgreSQL.
3. Prometheus
Prometheus acts as the central metrics collection system.
I worked with the Prometheus configuration file:
prometheus.yml The configuration defines the targets that Prometheus should scrape.
For example, the monitoring flow can include:
- Node Exporter
- PostgreSQL Exporter
- Other application exporters
I also learned how important configuration validation is before restarting Prometheus.
Using
promtool , Prometheus configurations and alert rules can be validated before deployment.This helped reduce configuration-related failures.
4. Grafana
Grafana was used to visualize the metrics collected by Prometheus.
I worked with dashboards to monitor infrastructure and database health.
Some of the important metrics that can be visualized include:
- CPU utilization
- Memory utilization
- Disk utilization
- PostgreSQL health
- Database connections
- Exporter availability
- Other application and infrastructure metrics
Grafana makes it much easier to understand monitoring data through dashboards instead of checking raw Prometheus metrics.
5. Loki
Loki was used for log aggregation.
This added another important part to the monitoring setup because metrics alone don't always explain why something failed.
The combination of:
Metrics + Logs + Dashboards + Alerts
provides a much better troubleshooting experience.
6. Alertmanager
Prometheus alert rules can identify conditions that require attention.
Alertmanager handles the alert-management side of the monitoring system.
I worked with alerts related to conditions such as:
- Exporter unavailable
- PostgreSQL unavailable
- High database connections
- Low disk space
- Infrastructure-related issues
I also tested the alerting flow to make sure alerts could move from Prometheus through Alertmanager to the configured notification channel.
7. Docker and Docker Compose
I also worked with Docker Compose for managing monitoring components.
The environment included services such as:
- Prometheus
- Grafana
- Loki
- Alertmanager
One important lesson was understanding Docker volume mounts.
Configuration files on the host must be correctly mounted inside containers. Otherwise, services may fail because configuration files are missing or inaccessible.
Before making changes to an existing Docker Compose configuration, I also learned the importance of creating a backup.
Troubleshooting Experience
One of the most valuable parts of this work was troubleshooting.
I encountered issues involving:
-
pg_upshowing an unhealthy state
- PostgreSQL exporter connection configuration
- Missing Prometheus alert-rule files
- Container configuration and volume mounts
- File permissions
- Exporter connectivity
- Firewall and network configuration
Instead of immediately changing multiple configurations, I learned to troubleshoot systematically.
My general troubleshooting approach became:
Check Service → Check Configuration → Check Logs → Check Network → Check Permissions → Validate Again
This approach makes it easier to identify the actual root cause.
Key Learnings
This experience helped me understand that monitoring is more than creating a Grafana dashboard.
A complete monitoring solution involves:
Collect → Store → Visualize → Alert → Investigate
I gained practical experience with:
- Linux monitoring
- PostgreSQL monitoring
- Prometheus scraping
- Grafana dashboards
- Log aggregation with Loki
- Alertmanager
- Docker Compose
- Systemd services
- Firewall/network troubleshooting
- Configuration validation
- End-to-end alert testing
Final Thoughts
Working on this QA monitoring setup gave me valuable hands-on experience with DevOps observability.
The biggest lesson for me was that monitoring should help answer three questions:
What is happening?
Why is it happening?
How quickly can we respond?
I'm continuing to build my knowledge in AWS, DevOps, Terraform, Docker, Kubernetes, Prometheus, Grafana, and observability.
I look forward to learning more, building more, and sharing my experiences with the AWS community.
#AWS #AWSDevOps #DevOps #Prometheus #Grafana #Alertmanager #PostgreSQL #Docker #Loki #Monitoring #Observability #Cloud
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article