Score Any GitHub Repo 0-100 Before You Clone It: An AWS AI Agent

You have done this. You see a repo in a reel or a tweet, someone says "just use this one for agents," and you clone the URL without a second thought. I did it too, until I read that in 2026 attackers planted more than 17,000 fake GitHub repos posing as AI agent and MCP projects, with install steps that steal your credentials. The convincing README is the bait.

That reframed the tool I was building. RepoPilot is a working agent you point at any public GitHub repo. Before it helps you run the repo, it tells you whether you should trust it. Then it gives a brand-new contributor one accurate path from clone to running, and when a command fails it explains the error like a patient teammate instead of a wall of red text.

I built it this weekend for the AWS Builder Center "Build an Agent" challenge. Here is what it does, how I built it, the one thing I obsessed over, and proof that it runs.

It is for the person about to try an unfamiliar repo: a new teammate, an open-source contributor, or you at 11pm acting on a tweet.

The first thing it does is score the repo from 0 to 100, like a credit score. It weighs five dimensions: safety, trust, maintenance, maturity, and documentation, each computed from real signals in the repo's metadata and files. A repo created three weeks ago with forty thousand stars is a known fake-star pattern, so its trust dimension tanks and RepoPilot says exactly why. Then the safety dimension scans the actual setup steps for the things that hide malware: a curl ... | bash that runs code you cannot see, an npm postinstall hook, a base64 blob, a call to a raw IP address. Any one of those is a critical issue, and a critical issue caps the whole grade, because "it runs fine" means nothing if the installer is designed to steal your keys.

This is not just theory about shady repos. Point RepoPilot at ollama/ollama, a tool millions of people install, and it drops to a D with a critical flag, because the official install really is curl -fsSL https://ollama.com/install.sh | sh. RepoPilot does not call it malware. It says, in plain language, "this pipes a remote script into your shell, read it before you run it." That is the honest nuance I wanted: even trusted repos have setup steps worth a second look.

Only after the score does it help you run the thing. It reads the repo's real files, not just the README, and gives you one ordered path: clone, install, configure, run. It flags commands the README mentions that do not exist. It tells you the tool versions you need up front. And when your first command fails, it turns the error into a cause and a single next step.

One more mode: type a username instead of a repo and RepoPilot profiles the developer. Those reels say "follow this dev" as often as "clone this repo," so it shows you the person's footprint, main languages, how long they have been active, and their top repositories by stars, with a short credibility read. Vet the account, not just the code.

The reasoning model is Amazon Nova Pro on Amazon Bedrock. I confirmed invoke access with one Bedrock Converse call before writing any agent code, which came back in about half a second and saved me from building on an assumption. The agent is deployed on AWS Lambda, fronted by an HTTP API Gateway so the web UI can call it from the browser. Locally the CLI uses the Strands Agents SDK; in Lambda I use a small, self-contained Bedrock loop so the package carries no heavy dependencies.

The agent has seven tools, and the newest two are the ones I am proudest of: a trust-assessment tool and a dangerous-setup scanner, which feed a scoring engine that turns their findings into the weighted 0-100 number. I kept all seven tools and the scorer as pure Python with no side effects, so I could unit-test them without AWS. Twenty-five tests, all green, including two hundred randomised repos to prove the weighted score never breaks its own math and that a critical flag always caps the grade. The model never touches correctness: it only narrates the facts the tools produce, in the right order, score and safety first.

One decision mattered more than it should have. I started by vendoring the requests library into the Lambda zip and immediately hit a wall, because the old urllib3 it pulled in tried to import something Python 3.12 had removed. Rather than fight versions, I rewrote the GitHub fetcher on the standard library. The package dropped from 400KB to 13KB and the problem vanished. I also had to tune the danger scanner: my first version flagged Flask because its docs mention 127.0.0.1, so I taught it to ignore localhost and private ranges while still catching real public-IP installers. Fewer false alarms, more trust.

The challenge asks for an agent people enjoy using, and gives "a helpful error message" as an example. So I spent my care on the first failure, the moment a new contributor usually gives up.

When your first install explodes, RepoPilot does not echo the stack trace. It says something like: "A dependency the code imports has not been installed. Run the install step again in the right environment, and activate your venv first. You are close. This is a common first-run snag, not something you broke."

I also gave the web UI two modes so the delight survives contact with reality. Demo mode is pre-configured and instant, so anyone can click around with zero setup and nothing ever hangs. Live mode calls the real deployed agent on AWS, and if a live call ever stalls it falls back to the demo data rather than showing a spinner forever. The agent is never allowed to dead-end on you.

How do I know it works? The metric onboarding teams track is time to a first successful run. RepoPilot's job is to collapse that, and to turn both the "is this safe" moment and the "why did it break" moment into moments where you feel helped, not stuck. Against Flask, FastAPI, Express, langchain, and ollama the agent read the correct stack, the required version, and the true trust signals every time.

Against the Flask repo, the deployed Lambda fetched 236 files, scored it 98 out of 100 (grade A, Excellent) with safety, trust, maintenance, and maturity all at 100 and documentation at 85, detected the Python stack, read python >=3.11 straight from the real pyproject.toml, and Nova Pro returned a clean guide that leads with the score and then the ordered steps. Point it at langchain-ai/langchain and it scores a clean 100. Point it at ollama/ollama and it correctly drops to a D over the real curl | sh install. Type @torvalds and it profiles Linus Torvalds, 263,000 stars across his repos. The code, the architecture diagram, the demo UI, and the twenty-five passing tests are all in the public repo, and the whole thing is live on GitHub Pages.

Check before you clone. Then clone to running, without the guesswork. That is the whole idea.

Video walkthrough: https://youtu.be/CZXNQrqXG7I 
Repo and full source: https://github.com/simplynadaf/repopilot-your-first-run-agent 
Live demo: https://simplynadaf.github.io/repopilot-your-first-run-agent/