Tracking AI Success Rate On An AWS Infrastructure Project

This is part of my series of blog posts on creating an AWS Bootstrap Script to set up secure AI agent infrastructure. 

In the last post I wrote about Code Structure For an AI Generated Infrastructure Deployment Project 

I’ve thought of a lot of different metrics we can use for tracking AI success for the moment I’m going to do something really simple. Here’s what I really care about when it comes to AI Projects - currently 7 things at the time of this writing.

Although I’ve read other reports with complex scientific measurements, what really matters to me first and foremost is if I can use the software I’m trying to build or not and if I can afford it. I also want to know if it is free from defects that will cause problems and if it is secure. If a fancy method is used to evaluate the AI results and it demonstrates something but no one can understand it, how useful is it? I want pragmatic, real world metrics that show me the ROI on my AI investments.

With this tracking I aim for continuous improvement to figure out what works and what doesn’t to get my software to this state of nirvana and how to optimize my token usage. One way to reduce token usage is to not sign up for a $200 monthly plan I can’t use at all. I had that problem in the month of May. I need to make sure that doesn’t happen again. I know someone who said he signed up for a Claude Code plan and then immediately cancelled it. Now I understand why. Unfortunately cancelling a Kiro plan doesn’t work the same way so I need to be careful with that.

I could get into a lot more details like which models I’m using and maybe I’ll do that later. I actually have some ideas on evaluating models but for now I need to keep it simple and focus on producing some useful software so I can get it to a revenue producing state. An MVP (minimum viable product), if you will.

The reason I’m doing this now is two-fold.

First, I’m spending a lot of AI tokens (which is all relative to the business and use case - I’m not getting paid to create this software and in and of itself, it is not making me any money). I want to reduce my spend for this and future projects. Even for revenue generating projects, reducing AI spend will make those projects more profitable.

Secondly, in the past day I found that I was spinning my wheels, testing things over and over again. I tested something and forgot if it worked or not by the next time I ran the project because there are so many moving parts.

Also, something I thought was almost done is now not due to a revision to help pinpoint a bug and prevent future mistakes. However, that caused a huge refactor which broke some things. I want to track what is done and still needs to be fixed. As things change and something breaks, I want to track that I need to re-fix it.

In order to track what is working or not I created a repo with some metrics and a list of features and I’m tracking which ones are done, not done, or broken. I also break my work down into objectives so I’ve broken down an objective into a list of features.

Read the full article here: