
Engineering the Absence of Heroics: A Pre-Action Gate for Agentic Infrastructure
There's a specific kind of praise every SRE has heard: "great save on that outage last night." It usually comes after someone stayed up until 3 AM piecing together what went wrong from five dashboards and a Slack thread that moved faster than anyone could read it.
There's a specific kind of praise every SRE has heard: "great save on that outage last night." It usually comes after someone stayed up until 3 AM piecing together what went wrong from five dashboards and a Slack thread that moved faster than anyone could read it.
I used to take that praise. Eleven years into banking and infrastructure work, I've stopped. Not because the effort wasn't real, but because I started noticing a pattern: the incidents that generate the best war stories are almost always evidence that something upstream should have stopped the action before it ever reached production. The hero didn't prevent the fire. The hero showed up after the fire started and happened to be good with a hose.
The systems I trust are the boring ones short runbooks, rare failure modes, nothing dramatic to report in the retro. Nobody writes a celebratory post about the quarter where nothing broke. But that quarter is the actual job.
This post is about one piece of infrastructure that tries to make "nothing broke" the default outcome rather than a lucky one: a PreAction SRE Gate, published as part of my opensource
agentsre library. It's not a monitoring tool. It's a guardrail that sits in front of statechanging actions deploys, config pushes, autoscaling events, and increasingly, tool calls made by autonomous agents and decides whether the current reliability posture can actually absorb the risk of that action before it's allowed to run.Why preaction, not postincident
Almost all SRE tooling is reactive by design: alerting fires after a threshold is crossed, autoscalers react after load climbs, rollback automation triggers after error rates spike. That's necessary, but it treats the action itself as a black box something that either succeeds or fails, with risk assessed only after the fact.
The gap gets more dangerous as more actions in a system stop being humaninitiated. When a deploy is triggered by a person, there's an implicit risk evaluation happening in their head: is now a good time to ship this, given what I know about the system? When the same action is triggered by an autonomous agent executing a multistep plan, that implicit check disappears unless it's engineered back in explicitly. That's the problem this gate is built to solve.
The design
The gate evaluates any
ActionRequest against a small set of composable evaluators, each returning a bounded risk score between 0 and 1 with a humanreadable reason. The gate aggregates those into ALLOW , HOLD , or BLOCK .@dataclassclass ActionRequest: name: str blast_radius: str # "singlepod", "az", "region", "global" reversible: bool initiated_by: str # "human" or "agent:<agent_id>" metadata: dict = field(default_factory=dict)Four evaluators ship in the initial release:
ErrorBudgetEvaluator blocks or holds actions when the target service's error budget is already thin. It's injected with a budget_fetcher callable so it stays agnostic to whether you're pulling from Prometheus, Datadog, or CloudWatch the gate doesn't care about your metrics backend, only about the remaining budget number it returns.BlastRadiusEvaluator scales risk by the declared blast radius of the action and penalizes irreversible actions further:RADIUS_WEIGHT = {"singlepod": 0.1, "az": 0.35, "region": 0.65, "global": 0.9}ConcurrentChangeEvaluator penalizes actions that overlap with other inflight risky changes against the same or dependent services the "everyone assumed someone else was watching" failure mode.AgentOriginEvaluator is the piece I think matters most going forward. It adds a conservative risk premium to actions initiated by autonomous agents, unless that specific agent has an established trust score from prior behavior:def __call__(self, action: ActionRequest) > EvaluatorResult: if not action.initiated_by.startswith("agent:"): return EvaluatorResult("agent_origin", 0.0, "humaninitiated") agent_id = action.initiated_by.split(":", 1)[1] trust = self.agent_trust_scores.get(agent_id) if trust is None: return EvaluatorResult("agent_origin", self.default_agent_premium, f"agent {agent_id} has no established trust score") risk = round(max(0.0, self.default_agent_premium * (1 trust)), 2) return EvaluatorResult("agent_origin", risk, f"agent {agent_id} trust_score={trust:.2f}")An agent with no track record gets treated with more caution than a human operator by default. Trust has to be earned through observed behavior, not granted because the caller happens to be automated.
Aggregation: why maxweighted, not averaged
The aggregation logic deliberately avoids simple averaging:
max_risk = max((r.risk_score for r in results), default=0.0)avg_risk = sum(r.risk_score for r in results) / len(results) if results else 0.0aggregate = round(0.7 * max_risk + 0.3 * avg_risk, 2)Pure averaging lets a single catastrophic risk factor hide behind several lowrisk ones a global, irreversible action initiated by an untrusted agent against a service with a healthy error budget might still average out to a comfortable score if you're not careful about weighting. Weighting 70% toward the maximum individual risk score means one evaluator screaming "no" is enough to move the decision toward HOLD or BLOCK, regardless of how calm the others are.
Running it
gate = PreActionSREGate(evaluators=[ ErrorBudgetEvaluator(fake_budget_fetcher), BlastRadiusEvaluator(), ConcurrentChangeEvaluator(fake_active_changes), AgentOriginEvaluator(agent_trust_scores={"deploybot": 0.9}),])
action = ActionRequest( name="rolloutpaymentsapiv42", blast_radius="region", reversible=True, initiated_by="agent:deploybot", metadata={"service": "paymentsapi"},)
result = gate.evaluate(action)print(result.summary())[HOLD] aggregate_risk=0.57 error_budget: 0.56 :: paymentsapi error budget at 12.0%, below hold threshold blast_radius: 0.65 :: blast_radius=region, reversible=True concurrent_change: 0.30 :: 2 concurrent change(s) touching paymentsapi agent_origin: 0.02 :: agent deploybot trust_score=0.90A regional rollout against a service with 12% error budget left, layered on two other concurrent changes, gets flagged for human review even though it's coming from a welltrusted agent and is technically reversible. That's the entire point: the gate isn't asking "can this action succeed," it's asking "can the system currently afford for this action to fail."
Where this fits in a larger governance posture
This gate is one primitive in a broader set I've been building under
agentsre an opensource library of SRE governance patterns aimed specifically at environments where agentic systems are taking more autonomous action against production infrastructure. The gate composes cleanly with upstream policy engines (OPA, Cedar, or a homegrown rules layer) since evaluators are just callables you can wire in organizationspecific risk logic without touching the aggregation core.The code above is trimmed for readability; the full module, along with the rest of the SLI and governance primitives in the library, is on GitHub: github.com/Ajay150313/agentsre. I'd genuinely like to hear how other teams are thinking about preaction risk evaluation for agentinitiated changes it's a problem that's going to get a lot more urgent over the next few years, not less.
I write about SRE, AIOps, and reliability engineering for agentic infrastructure. Previous pieces have covered SLO honesty gaps and multiwindow burnrate alerting. Feedback and PRs on the repo are welcome.
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article