AWS Builder Center
Triage: Building a Self-Adaptive MLOps Controller for Production ML

Triage: Building a Self-Adaptive MLOps Controller for Production ML

How we built an explainable MLOps controller that continuously monitors production ML systems, reasons across competing objectives, and takes autonomous corrective actions — without compromising fairness.

Triage: Building a Self-Adaptive MLOps Controller for Production ML

Built during FIRST COMMIT, a hackathon organized by WeMakeDevs and sponsored by AWS.
Machine learning models rarely fail in production in just one way.
Accuracy can drop while latency increases. Data can drift while predictions still look acceptable. Infrastructure costs can rise unexpectedly. And a model can maintain strong overall performance while performing poorly for a particular station or group.
Traditional MLOps pipelines often respond to these problems with predefined rules:
Detect degradation → retrain → deploy.
We wanted to build something more adaptive.
That idea became Triage — a self-adaptive MLOps controller that continuously monitors production ML systems and decides what intervention makes the most sense for the current situation.



From Monitoring to Decision-Making

Triage monitors six metric families:
  • Accuracy
  • Data drift
  • Latency
  • Energy consumption
  • Cost
  • Equity
Instead of reacting to one metric at a time, it evaluates the overall system state and considers multiple possible actions.
These include:
  • Continuing with the current model
  • Adjusting inference batching
  • Reusing an archived model
  • Rolling back
  • Triggering retraining
  • Promoting a candidate model
  • Changing serving configuration
  • Entering a conservative operating mode
  • Escalating for equity review
This allows Triage to ask a more useful question:
Given the current production conditions, what should we do next?



The Important Part: Tradeoffs

Suppose latency suddenly increases.
A traditional pipeline might immediately deploy another model.
But the problem may not be the model itself. Increasing inference batching could solve the latency problem while avoiding an expensive retraining cycle.
Similarly, a newly trained model might have higher accuracy but also higher cost and energy consumption. An older, validated model might provide a better overall operational tradeoff.
Triage evaluates these alternatives before taking action.
Every decision produces a candidate table showing:
Candidate
Performance
Cost
Energy
Equity
Decision
Current model
Degraded
Low
Low
Pass
Rejected
Archived model
Improved
Medium
Medium
Pass
Selected
Retraining
High
High
High
Pass
Rejected
New candidate
High
Medium
Low
Fail
Rejected
This makes the system's reasoning visible instead of hiding it behind a single automated decision.



Safety Comes First

One of the core design decisions in Triage is that equity is not just another weighted metric.
We enforce a hard constraint:
A model cannot be promoted if per-station equity requirements are violated.
Even if a candidate has better accuracy, lower latency, and lower cost, it is rejected if it violates the equity constraint.
This gives the controller two distinct layers:
Production Metrics
↓
Candidate Generation
↓
Safety Constraints
↓
Tradeoff Evaluation
↓
Action Selection
↓
Production

In other words:
Optimize wherever possible, but never optimize by violating a safety boundary.



Explainable Autonomy

We designed Triage around the idea that autonomous MLOps should be explainable.
For every control cycle, the system records:
  • What the production state looked like
  • Which actions were considered
  • Which candidates were rejected
  • Why they were rejected
  • Which action was selected
  • Why it was selected
So instead of simply saying:
"Retraining triggered."
Triage can explain:
"Retraining was considered, but an archived model provided comparable accuracy with lower projected latency, cost, and energy consumption while satisfying all equity constraints."
That decision history also creates an audit trail for debugging and future analysis.



What We Built at FIRST COMMIT

We built Triage during FIRST COMMIT, organized by WeMakeDevs and sponsored by AWS.
The hackathon gave us the opportunity to take an idea around autonomous MLOps and turn it into a working system that combines:
Machine Learning + MLOps + Multi-objective Decision Making + Explainability + Responsible AI
The biggest lesson for us was that production ML is not simply about choosing the best model.
It is about continuously making the right operational decision for the current situation — while knowing when the system must stop and ask for human review.
That is what we set out to build with Triage.
Observe. Evaluate. Constrain. Decide. Explain.
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article