Auto research on SageMaker, and how an eval cut its cost by 61%
How to build an auto research tool customized for SageMaker, and optimize it for cost
"Auto research" is one of the more interesting ideas to come out of the agentic coding world. You give a coding agent a goal, like building a classification model with an F1 score of 0.7 or better, along with the tools to run experiments and a rule for when to stop. Then you let it loop: try a hypothesis, measure, adjust, and try again. Andrej Karpathy's autoresearch repository is a well-known example, and there are now several general-purpose harnesses for the pattern.
Adapting the loop to SageMaker
I wanted to know how hard it would be to point that loop at a real ML platform. In my case that meant data in Amazon Redshift, training jobs on Amazon SageMaker, experiment tracking in SageMaker's managed MLflow, and the winning model registered in the SageMaker model registry.
The short answer is that it's surprisingly straightforward. I built it as a Claude Code plugin with four pieces:
- a
/sm-researchcommand that interviews you for the goal, the allowed data, and the budget - a skill that tells the agent how to run an experiment
- a small workflow script that fans out experiments each round and decides when to stop
- a handful of Python helpers that do the SageMaker-specific work, like unloading a data slice from Redshift or launching a training job
The agent does the creative part: which tables to join, how to aggregate, which algorithm to try. The helpers do the parts that shouldn't be improvised. They make sure the agent only reads the tables and columns you've allowed, always uses the right AWS profile, asks before creating anything billable, and never deletes anything.
To test it I built a synthetic churn dataset spread across four Redshift tables. No single table gets you above an F1 of about 0.62, but the joined and aggregated data gets you to about 0.73. With the goal at 0.70, the agent has to discover the join to succeed. It did.
Using an eval to make it cheaper
The more interesting part came next. Each experiment is an agent session of twenty-odd tool calls, and a loop runs many of them, so cost adds up quickly. Rather than guessing at optimizations, I built an evaluation for the single "run one experiment" step and then hill-climbed against it, using the
build-eval and hillclimb flows in Claude Code's claude-api skill.The eval has 24 cases, runs each one for real against the test bed, and grades what's actually left behind in AWS. Did the training job finish? Does the MLflow run exist? Does the reported metric match what MLflow recorded? Did any SQL touch data outside the allowlist? I even planted a decoy table with a near-perfect label leak to see if the agent would be tempted. Building the eval turned up bugs before I'd optimized anything. My favorite was the agent discovering the plugin's own workflow and launching the entire research loop from inside a single experiment.
The first useful finding was where the money went. About 80% of the cost was the agent re-reading its own growing context on each turn, so the cost tracked the number of tool calls, not the length of the prompt. Swapping models didn't help:
| configuration | cost vs. baseline | what happened |
|---|---|---|
| Claude Opus 5.5, low effort | 1.5× | fewer turns, but higher token prices |
| Claude Sonnet 5.5, low effort | 0.97× | effort barely changed anything |
| Claude Haiku 5.5, low or medium effort | 0.05–0.07× | too many experiments left unfinished |
What did help was removing the detours the agent was taking. It was hitting avoidable API errors, reading 20 KB log files to find one line, and grepping the plugin's source code to figure out the data layout, because my documentation said one thing and the code did another. Three changes brought the number of tool calls down from about 23 to about 6:
- a one-call helper for the package, launch, and logging steps
- training templates that handle Redshift's decimal columns safely
- a fixed data contract
Here's how the final version compared to where I started, on the held-out cases:
| baseline | final | |
|---|---|---|
| cost per experiment | $0.513 | $0.198 |
| F1 score | 0.684 | 0.683 |
| valid experiments | 96% | 100% |
| allowlist violations | — | 0 |
That's about a 61% cost reduction with no measurable change in model quality. (Costs are priced at Anthropic's list rates from the recorded token usage; I ran Claude Code through Amazon Bedrock, so the absolute numbers will differ for you, but the ratio should hold.)
The allowlist row is there for a reason. The first version of the winning configuration passed on cost and quality, but one attempt out of 36 used a
SELECT * inside a subquery on a table where only some columns were allowed. It turned out the plugin had only ever checked tables, not columns. Because I had registered "zero violations" as a hard requirement before starting, that caught a real data-access bug that a lower bill would otherwise have papered over. I added a structural column check, re-ran the confirmation, and it came back clean.Where to go from here
At the moment this is a prototype tested on synthetic data, and I wouldn't point it at sensitive data without more work. The obvious next step is a proper red-teaming eval. I'd want to know:
- what happens when a value or comment inside a Redshift table contains instructions aimed at the agent
- whether the agent can be talked into reading outside its allowlist through views or system tables
- whether it holds to its guardrails on credentials, deletions, and spending when pushed
I'd also replace my SQL screening with real least-privilege controls, like a Redshift user with column-level grants and a tightly scoped IAM role. Then a clever query fails in the database rather than depending on a regular expression.
But I encourage you to try the pattern yourself. Adapting auto research to your own platform takes less effort than you might think, and a small, honest eval goes a long way toward making it cheaper and safer. You can review the code in the GitHub repository .
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article