
AWS DevOps Sentinel
AWS DevOps Sentinel is an AI-powered incident management system that helps DevOps teams detect, investigate, and resolve AWS incidents. It uses AI to analyze logs and metrics, identify root causes, suggest remediation actions, and includes human approval before executing fixes.
AWS DevOps Sentinel – AI Powered Incident Management
Introduction
In our project we tried to build something which is related to DevOps and AI together. The project name is AWS DevOps Sentinel. The main idea of this project is to help developers or DevOps team to find the problems happening in there AWS application and also help them to understand what is the actual reason for the problem.
Normally when a production application goes down, the developer have to check many things like CloudWatch logs, CPU usage, container status, health checks and other metrics. This can take lot of time specially when the problem is urgent. So we thought why not make a system where AI can investigate the incident and suggest a solution.
Our project is mainly a incident management and automation dashboard. It also contains a simulation mode so that we can demonstrate the full working of the system without actually breaking a real AWS service. The visual guide describes the complete flow from incident creation to investigation, approval, execution and finally verification.
What is AWS DevOps Sentinel?
AWS DevOps Sentinel is basically a dashboard for monitoring and handling incidents.
The application shows important information in one place like:
- Active incidents
- Critical incidents
- Incidents waiting for approval
- Resolved incidents
- Incident timeline
- AI analysis
- Root cause
- Remediation plan
There is also a Simulation Mode, which is useful for our demo. In this mode we can create sample incidents such as ECS service unhealthy tasks or EC2 high CPU usage.
The dashboard is designed in a dark theme. The background is dark slate and different colours are used for critical, warning, success and normal states. We tried to keep the UI simple but also make it look like a real DevOps monitoring dashboard.
Why we created this project
One of the main reason for creating this project was to understand how AI can be used in DevOps.
When an incident happens, the team has to first detect the problem, then collect logs and metrics, find the root cause and then decide what action should be taken. If all this is done manually it can take more time specially if the problem is happening in production.
Our idea was to make an AI agent which can help in this process.
The flow of our system is:
START → Simulate → Investigate → Analyze → Approve → Execute → Verify → RESOLVED
The complete demo is expected to take around 6 to 8 minutes.
How the project works
First we open the AWS DevOps Sentinel dashboard. Initially there will be no incidents. The dashboard shows zero active incidents, zero critical incidents, zero awaiting approval and zero resolved incidents.
Then we can use the Demo Simulator.
For example, we can select:
ECS Service Unhealthy Tasks
This is a critical incident where multiple ECS tasks are failing and health checks are also having problems.
After clicking the simulate incident button, an incident is created.
Now the dashboard changes and shows one active incident and one critical incident. The incident card contains information like the service name, time and a small description of the problem.
AI Investigation
After the incident is created, we can open the incident details.
There is a button called Investigate with AI.
When we click this, the AI investigation starts. During this stage the system gives live updates in the event timeline.
For example, first the agent is started, then metrics are fetched from CloudWatch, after that logs are collected and finally the AI completes the analysis.
This is one of the interesting part of the project because we can actually show the steps of an AI agent investigating an incident instead of just showing a final answer. It makes the project more understandable during the demo also.
Finding the Root Cause
After the investigation is completed, the AI gives a root cause analysis.
In our demo incident, the root cause is out of memory errors causing ECS task crashes.
The explanation says that the ECS service is getting repeated OOMKilled container exits. The logs show Java heap space exhaustion and then the containers are terminated. Because of this multiple tasks fail health checks and are replaced again and again, creating a crash loop.
The AI also identifies some contributing factors:
- Container memory limit is too low, around 512MB.
- High traffic is creating more memory pressure.
- There are no proper memory monitoring alerts configured.
The analysis also shows a confidence score, which in the demo is 85%. Ofcourse confidence score does not mean that the AI is always correct, it is just showing how confident the system is with the analysis.
Remediation Plan
Finding the problem is not enough. The system also needs to tell what can be done next.
So AWS DevOps Sentinel generates a remediation plan.
In this demo the recommended action is:
Force restart ECS service to clear crash loop
The action is marked as requiring approval and the risk is shown as medium. It also tells that there can be a short downtime during the restart.
This approval part is important because we don't want AI to blindly execute every action in a real production environment. If the AI makes wrong decision and directly execute something, it can create even more problems.
Human Approval
Before executing the remediation, the system asks for human approval.
The dashboard shows an approval card where the user can see the incident, planned action, approver name and optional comments.
There are two buttons:
Approve & Execute
and
Reject
This creates a human-in-the-loop workflow. So AI can investigate and suggest the fix, but a human can still decide whether the action should actually be executed.
I think this is one of the important part of our project because automation is useful but human decision is also important for risky production changes.
Execution and Verification
After approval, the remediation action is executed.
In our demo, the ECS service is force-redeployed. After that the system checks the metrics again to verify whether the problem is actually solved.
The timeline shows different events such as approval, remediation started, remediation completed and verification. Finally the incident becomes resolved and the metrics return to normal levels.
This verification is important because just executing a command doesn't mean the incident is solved. We need to check the system again and make sure that the service is actually working properly.
Final Result
At the end of the demo, the dashboard shows zero active incidents and zero critical incidents. The resolved count becomes one.
The incident card also shows that the AI analysis had high confidence and the root cause was related to out of memory errors.
Finally the dashboard displays a message saying that the demo is completed and the incident was resolved with AI investigation and human-approved remediation.
So from start to end we can see the complete incident management process in one dashboard.
What I learned from this project
While building this project I learned that DevOps is not only about deploying an application. Monitoring and handling problems is also a very important part.
I also understood how AI agents can be used for more than just chatting. An AI agent can collect information, analyse logs and metrics, find possible root causes and prepare an action plan.
Another important thing I learned is that automation should not always mean fully automatic. For risky operations, having human approval is better.
The project also helped us understand how an incident can be represented using different states like detected, investigating, resolved and waiting for approval.
We also faced some small problems while building the project and connecting different parts together, but those problems helped us understand the workflow better.
Future Improvements
There are many things which can be improved in this project.
For example, the current demo contains simulation scenarios. In future we can connect it with actual AWS CloudWatch alarms and real ECS or EC2 services.
We can also improve the AI analysis by giving it more logs, traces and metrics. This may help the AI to find better root causes.
Another improvement can be adding more remediation actions instead of only restarting an ECS service.
We can also add notifications so that the DevOps team gets an alert when a critical incident happens.
In future we can also add support for more AWS services like Lambda, RDS, EKS and other services. This will make the project more useful for different types of applications.
Conclusion
AWS DevOps Sentinel was built as a small project to demonstrate how AI and DevOps can work together.
The main thing we wanted to show was not just a dashboard but a complete incident workflow.
An incident happens, the system detects it, AI investigates it, the root cause is identified, a fix is suggested, human approval is taken, the fix is executed and finally the system verifies the result.
For me, the most interesting part was seeing how the different steps connect together. It made the concept of AI based incident management much easier to understand.
There is still many things to improve in this project, but as a demonstration and learning project, AWS DevOps Sentinel gives a good idea of how AI agents can help DevOps teams in the future.
Overall this project was a good learning experience for us and also helped us understand that AI can be used not only for generating text but also for solving real technical problems and helping developers during incidents.
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article