
Triage: Building a Self-Adaptive MLOps Controller for Production ML
How we built an explainable MLOps controller that continuously monitors production ML systems, reasons across competing objectives, and takes autonomous corrective actions — without compromising fairness.
Triage: Building a Self-Adaptive MLOps Controller for Production ML
Built during FIRST COMMIT, a hackathon organized by WeMakeDevs and sponsored by AWS.
Machine learning models rarely fail in production in just one way.
Accuracy can drop while latency increases. Data can drift while predictions still look acceptable. Infrastructure costs can rise unexpectedly. And a model can maintain strong overall performance while performing poorly for a particular station or group.
Traditional MLOps pipelines often respond to these problems with predefined rules:
Detect degradation → retrain → deploy.
We wanted to build something more adaptive.
That idea became Triage — a self-adaptive MLOps controller that continuously monitors production ML systems and decides what intervention makes the most sense for the current situation.
From Monitoring to Decision-Making
Triage monitors six metric families:
- Accuracy
- Data drift
- Latency
- Energy consumption
- Cost
- Equity
Instead of reacting to one metric at a time, it evaluates the overall system state and considers multiple possible actions.
These include:
- Continuing with the current model
- Adjusting inference batching
- Reusing an archived model
- Rolling back
- Triggering retraining
- Promoting a candidate model
- Changing serving configuration
- Entering a conservative operating mode
- Escalating for equity review
This allows Triage to ask a more useful question:
Given the current production conditions, what should we do next?
The Important Part: Tradeoffs
Suppose latency suddenly increases.
A traditional pipeline might immediately deploy another model.
But the problem may not be the model itself. Increasing inference batching could solve the latency problem while avoiding an expensive retraining cycle.
Similarly, a newly trained model might have higher accuracy but also higher cost and energy consumption. An older, validated model might provide a better overall operational tradeoff.
Triage evaluates these alternatives before taking action.
Every decision produces a candidate table showing:
Candidate | Performance | Cost | Energy | Equity | Decision |
|---|---|---|---|---|---|
Current model | Degraded | Low | Low | Pass | Rejected |
Archived model | Improved | Medium | Medium | Pass | Selected |
Retraining | High | High | High | Pass | Rejected |
New candidate | High | Medium | Low | Fail | Rejected |
This makes the system's reasoning visible instead of hiding it behind a single automated decision.
Safety Comes First
One of the core design decisions in Triage is that equity is not just another weighted metric.
We enforce a hard constraint:
A model cannot be promoted if per-station equity requirements are violated.
Even if a candidate has better accuracy, lower latency, and lower cost, it is rejected if it violates the equity constraint.
This gives the controller two distinct layers:
Production Metrics ↓Candidate Generation ↓Safety Constraints ↓Tradeoff Evaluation ↓Action Selection ↓Production
In other words:
Optimize wherever possible, but never optimize by violating a safety boundary.
Explainable Autonomy
We designed Triage around the idea that autonomous MLOps should be explainable.
For every control cycle, the system records:
- What the production state looked like
- Which actions were considered
- Which candidates were rejected
- Why they were rejected
- Which action was selected
- Why it was selected
So instead of simply saying:
"Retraining triggered."
Triage can explain:
"Retraining was considered, but an archived model provided comparable accuracy with lower projected latency, cost, and energy consumption while satisfying all equity constraints."
That decision history also creates an audit trail for debugging and future analysis.
What We Built at FIRST COMMIT
We built Triage during FIRST COMMIT, organized by WeMakeDevs and sponsored by AWS.
The hackathon gave us the opportunity to take an idea around autonomous MLOps and turn it into a working system that combines:
Machine Learning + MLOps + Multi-objective Decision Making + Explainability + Responsible AI
The biggest lesson for us was that production ML is not simply about choosing the best model.
It is about continuously making the right operational decision for the current situation — while knowing when the system must stop and ask for human review.
That is what we set out to build with Triage.
Observe. Evaluate. Constrain. Decide. Explain.
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article