Skill Trials: Optimize Agents One Hypothesis at a Time
Apply an improvement idea across all your skills at once, score each before and after, and promote the winners — instead of optimizing one at a time.
Series: GenAI (3 articles)
- 1Skill Trials: Optimize Agents One Hypothesis at a Time This article
Skill Trials: Optimizing an AI Agent Library One Hypothesis at a Time
Author: Dominick DeBellis
Published: 2026-07-10
Published: 2026-07-10
The problem: good ideas don't spread
If you maintain more than a handful of AI agent skills — structured playbooks that tell an LLM how to investigate, triage, and fix operational problems — you've probably noticed the same thing we did: improving them one at a time doesn't scale.
Our team operates a platform that manages a library of agent skills for operational infrastructure. Each skill is a markdown document describing step-by-step procedures an LLM follows when it receives a work item.
We had an autonomous optimizer that picks one skill, tries an improvement, scores the result, and keeps or discards. It works — but it's slow. If "add a verification step after every remediation" is genuinely a good idea, it probably benefits dozens of skills. The optimizer would need to independently rediscover that insight for each one.
We needed a way to ask: "Does this one idea improve the whole library?" — and act on the answer in a single pass.
The insight: flip the axis
Traditional prompt optimization tools iterate per-skill: pick a skill, try many ideas, keep the best. We inverted that.
A trial takes a single improvement hypothesis and applies it to every eligible skill simultaneously. Each skill is scored before and after. The results tell you whether the idea generalizes:
1
2
3
4
5
Traditional (depth): Trials (breadth):
Skill A → try ideas 1,2,3... Hypothesis H applied to:
Skill B → try ideas 1,2,3... A, B, C, D, ... (all skills)
Skill C → try ideas 1,2,3... → "H helps 31, hurts 4, neutral 45"
(forever) → stage the 31 winners at onceThe hypothesis — not the skill — is the unit of work. A hypothesis that improves 31 of 80 skills is strong evidence it's a real pattern. One that improves 1 of 80 is probably skill-specific noise.
What a hypothesis looks like
A hypothesis is a natural-language instruction, not a code patch or config flag. The optimizer agent reads it and produces a different concrete edit for each skill's structure. Examples from our live history:
- "Add a verification step after remediation that confirms the fix actually took effect."
- "Move the escalation criteria above the remediation steps so the agent decides whether to act before acting."
- "Re-fetch mutable state immediately before modification instead of reusing data captured during an earlier investigation phase."
The hypothesis is the invariant. The realization adapts to each skill.
Where hypotheses come from
We don't invent hypotheses — we mine them from what humans already did.
A background process reads the git history of human skill edits. When someone improves a skill by adding a verification step, the miner extracts the generalized idea: "add verification after remediation." That hypothesis then goes into the trial pipeline to test whether it helps every other skill too.
This is the key insight: what a human already proved on one skill, generalized to all.
Additional sources include feedback from production (the agent notes what it couldn't do) and recycled optimizer insights (the per-skill optimizer's winners, reverse-engineered into hypotheses).
How a trial runs
One trial runs per hour. The algorithm:
- Select a hypothesis from the priority cascade (mined patterns first, then optimizer insights, then feedback-loop priorities, then defaults).
- Fan out — for each eligible skill, prompt the optimizer agent to produce a surgical edit applying the hypothesis to that skill's playbook.
- Score each candidate — a deterministic structural scorer measures whether the edit added investigation steps, warnings, stop-gates, escalation criteria, or code blocks. A delta above the threshold means "improved."
- Aggregate — count improved/degraded/unchanged. The win-rate is itself the credibility signal.
- Promote — the top-N improved skills are staged in a review queue. Nothing reaches production without a human-approved code review.
Two-tier scoring
Running every candidate against real tickets with an LLM judge would be expensive at 80 skills per trial. So we use a cheap gate and an expensive gate:
Structural scoring (during fan-out — all 80 skills, near-zero cost): Regex-based metrics count whether the edit added investigation steps, warnings, code blocks, and stop-gates — or whether it bloated or stripped structure. This is the inner loop gate.
Execution scoring (before promotion — top 5 only, high cost): An LLM runs the candidate skill against test fixtures and judges output quality for completeness, correctness, routing, efficiency, and tool coverage. This is the outer loop gate.
Most bad candidates die cheaply at the structural tier. Only the strongest survivors face the expensive judge.
Results
After several months of continuous operation:
- 1,000+ trials completed
- ~40% average improve rate per hypothesis (of the non-null ones)
- Cross-skill propagation factor: a winning hypothesis typically improves 15–30 skills at once
- All promotions require a human code review — the system stages; humans approve
The highest-performing single trial improved over 60% of eligible skills with one idea: "Add diagnostic steps and remediation paths for failure modes."
How it fits together
Trials is one part of a larger pipeline where every output feeds the next stage:
- Mine — extract improvement patterns from wikis, tickets, and human code reviews
- Build — scaffold new skills from coverage gaps in the ticket corpus
- Optimize (depth) — per-skill optimizer picks one skill, tries many ideas, every few minutes
- Optimize (breadth) — trials applies one idea across all skills, every hour
- Validate — scenario runner and metamorphic testing confirm quality
- Promote — human code review gate before anything reaches production
- Measure — regression tracking, version-diff, and token waste analysis close the loop
Trials' unique contribution is the breadth axis: it's the only part that asks whether an idea generalizes across the library.
What's novel
We did a prior-art scrub across public tools and academic work. Every comparable system — DSPy, SkillOpt, TextGrad, and similar prompt optimization frameworks — optimizes per-skill or per-agent. They iterate depth on one thing.
The trial flips the axis: one hypothesis validated across a population. The closest structural analog we found was a one-shot research study that tested skills-vs-no-skills across dozens of tasks — but that was a study, not a reusable engine in a continuous loop.
The secondary novelty is hypothesis sourcing: mining generalizable ideas from real human edits. The combination — mine from humans, test across all skills, promote winners — has no precedent we could find.
Limitations and what's next
Current limitations:
- Structural scoring doesn't guarantee execution quality (a skill can "look better" without performing better)
- The hypothesis blacklist is simple (2 failures = never retry) — some ideas deserve retry after skills change
- No statistical significance test — threshold + win-rate is the proxy
Target state:
- Execution-confirmation gate: before auto-approval, a candidate must also pass LLM-as-judge testing
- Tiered win-rate: universal (>50%), broad (20–50%), narrow (<20%) — each gets different promotion priority
- Metamorphic guard: a "universal" hypothesis must survive paraphrase invariance testing (if the improvement evaporates when you rephrase the test tickets, it was overfit to wording)
Try it yourself
If you maintain a library of AI agent skills (structured playbooks, prompt libraries, or any multi-skill system), the trial pattern is applicable regardless of framework:
- Keep a log of human edits to your skills (git history works)
- Reverse-engineer generalizable instructions from those edits
- For each instruction, apply it to every skill, score before/after
- The win-rate tells you whether it's a real pattern or noise
- Promote the winners; blacklist the losers
The infrastructure is a loop, a scorer, and a file system. The insight is choosing hypothesis as the unit of iteration instead of skill.
Questions or want to discuss? Reach out to the author.
Series: GenAI (3 articles)
- 1Skill Trials: Optimize Agents One Hypothesis at a Time This article
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article