AWS Builder Center

Reverse Engineering AWS with Generative AI: From Code Assumptions to Runtime Evidence

A practical troubleshooting technique based on three inputs: the problem, the desired state, and the actual state.

AWS Technical Evangelist
When we use Generative AI inside a development environment, there is an interesting asymmetry.
The model can usually see a lot of code. Depending on the tooling, it may understand the repository structure, Infrastructure as Code definitions, configuration files, deployment scripts, documentation, and even the relationships between different modules.
What it normally does not know nearly as well is what is actually running. And in cloud environments, those two things are not necessarily the same.
I ran into this problem while working with an existing serverless application on AWS. In theory, the application was defined using Infrastructure as Code, mainly AWS SAM and CloudFormation.
In practice, the situation was considerably more complicated.
That experience led me to a simple troubleshooting technique that I still find useful: instead of asking the AI to infer the runtime environment from the repository, I explicitly give it three different views of the problem:
Problem + Desired State + Actual State
The difference sounds small, but it changes the quality of the diagnostic context significantly.

When the repository is not the system

The application I was working with was not exactly a clean greenfield deployment. There were SAM and CloudFormation definitions, but also scripts generating or manipulating required artifacts. Some pieces of configuration or code were duplicated or generated as part of different processes. Deployments were partially manual. CI/CD existed, but some steps were still performed outside the pipeline. Some resources or configuration were outside the Infrastructure-as-Code lifecycle altogether.
And, unsurprisingly, there was infrastructure drift.
Individually, none of these characteristics is particularly unusual in an application that has evolved over time. The difficulty comes from their combination.
Looking at the repository, I could understand what the application was supposed to look like.
Looking at the deployment mechanisms, I could understand what different processes were trying to create.
But neither necessarily represented what was actually running in AWS.
In practice, there were at least three versions of the architecture:
  • what the code suggested existed,
  • what the deployment process intended to create,
  • and what actually existed in AWS.
That distinction became important during troubleshooting, but even more so when trying to understand the application well enough to refactor it.
Eventually, that work contributed to converting the solution into a cleaner Python CDK implementation. The technique described here did not perform that migration, of course. What it did was help me build a more reliable understanding of the platform I was migrating.

Problem, desired state and actual state

A typical AI-assisted troubleshooting conversation often starts with something like this:
This Lambda cannot reach this service. Here is the code and the IaC template…What is wrong?
The model then reasons from the information available to it. That can work very well when the deployed infrastructure accurately matches the repository.
When it does not, the model has an unavoidable gap. It starts making assumptions about runtime state.
I prefer to separate the diagnostic information into three elements.

The problem

What is happening?
For example:
  • a Lambda function cannot reach a private endpoint,
  • traffic crosses a Transit Gateway in one direction but not the other,
  • an API Gateway integration returns an unexpected error,
  • or a deployment behaves differently from what the template suggests.
This is the observable symptom and the expected behaviour.

The desired state

What should the system look like?
This can come from Terraform, CloudFormation, CDK, SAM, architecture diagrams, documentation or simply the design being discussed.
The desired state provides the intention.

The actual state

What does AWS report right now? This is the part I deliberately retrieve through AWS APIs.
Instead of expecting the model to discover everything itself, I ask it to generate a small shell script containing the minimum set of AWS CLI commands required to inspect the components involved in the problem.
I review the script, execute it myself and return the JSON output to the AI.
The diagnostic context then changes from:
Problem + Code + Assumptions
to:
Problem + Desired State + Observed Evidence
That is the useful part of the technique.

A disposable inspection script

The script is not a troubleshooting framework.
It is deliberately disposable.
If I am investigating a routing problem, it might inspect route tables, Transit Gateway attachments and relevant network interfaces.
For another problem, it could inspect Lambda configuration, API Gateway integrations, IAM policies or CloudFormation resources.
The important property is that I already have an approximate idea of the scope.
I am not asking:
Explore this AWS account and find the problem.
I am asking something closer to:
The issue appears to involve these components. Generate the minimum read-only AWS CLI commands required to inspect their current configuration.
The result could be seven commands, ten commands or fifteen commands. The number itself is irrelevant.
What matters is that the inspection scope reflects the problem I am trying to solve.
A simplified workflow looks like this:
Issue → Relevant components → Inspection script → Human execution → JSON evidence → AI analysis → Engineer validation
The model helps decide which evidence would be useful. AWS provides the evidence. The engineer remains responsible for deciding whether the resulting hypothesis is correct.

A small networking example

Imagine a simple routing issue.
Traffic from one network reaches another network through an AWS Transit Gateway, but the response never returns.
The architecture and IaC suggest that routing is symmetric, but the observed behaviour says otherwise.
Instead of continuing to inspect templates, I could ask the AI to generate a small inspection script around the relevant subnets, route tables and Transit Gateway attachment.
A simplified version could look like this:
#!/usr/bin/env bash

aws ec2 describe-route-tables \
--filters "Name=association.subnet-id,Values=$APP_SUBNET_ID" \
> app-route-table.json

aws ec2 describe-route-tables \
--filters "Name=association.subnet-id,Values=$REMOTE_SUBNET_ID" \
> remote-route-table.json

aws ec2 describe-transit-gateway-attachments \
--filters "Name=transit-gateway-id,Values=$TGW_ID" \
> tgw-attachments.json
The commands themselves are not especially sophisticated. That is precisely the point.
I do not need the AI to invent a networking diagnostic platform. I need a small snapshot of the AWS resources involved in this particular problem.
I can then provide the resulting JSON together with the intended routing design.
The AI now has enough information to compare, for example:
  • the routes that should exist,
  • the routes that AWS currently reports,
  • and the network behaviour I am observing.
If a return route is missing, the model can identify that inconsistency and propose it as a hypothesis.
I still need to validate the relationship and decide the change.
The value is not that the model somehow “understands AWS better”. The value is that I have given it better evidence.

The troubleshooting triangle

The AI reasons across the problem, intended design and observed runtime state, but the final validation remains an engineering responsibility.
This diagram represents the three independent inputs used during diagnosis.


Declared state is not always observed state

At this point, an obvious question appears:
Why not simply read Terraform state or inspect the CloudFormation stack?Sometimes that is exactly what I would do.
Terraform state and CloudFormation both provide very valuable information about managed infrastructure. They are often part of the diagnostic context. But they answer a slightly different question.
I find it useful to separate three concepts.
  • Declared state is what the configuration says should exist. That might be a Terraform resource, a CloudFormation template, a SAM definition or CDK code.
  • IaC-managed state represents the infrastructure known and managed by that deployment mechanism. Terraform state, CloudFormation stack information and related deployment metadata are useful here.
  • Actual observed state is what the AWS APIs report about the relevant resources at the moment I inspect them.
Ideally, the three converge. Real systems do not always behave so neatly.
A manually modified security group, a legacy resource outside the stack, an old script executed during deployment or a resource created by another mechanism can create differences between these views.
Drift is one reason, but it is not the only one.
The environment I described earlier included several deployment mechanisms and manual operations. In that context, assuming that the Infrastructure-as-Code view represented the complete system was exactly the assumption I needed to avoid.
The objective is therefore not to replace Terraform state or CloudFormation inspection.
It is to add an independent observation of the AWS resources that matter to the current problem.

Why not just give the AI an AWS MCP?

Another reasonable alternative is to connect the AI directly to AWS through an MCP server or equivalent tooling.
That can absolutely work.
In fact, this workflow could itself be packaged behind an MCP: identify the relevant resources, execute a constrained set of read operations and return a diagnostic snapshot.
The distinction I make is less about the technology and more about the exploration model.
A generic AWS-capable MCP can expose many operations and allow the model to navigate the environment dynamically.
That provides obvious advantages:

Generic MCP

broader autonomy + exploration + convenience
The model can decide that it needs another resource, execute another operation and continue investigating.
For open-ended exploration, that can be extremely useful. But flexibility also introduces a larger search space.
Depending on the client and implementation, more exposed tools can mean more tool definitions available to the model, while exploratory tool calls and their responses accumulate additional context.
More importantly, the model gets to decide progressively what information should be inspected.
For a well-bounded problem, I sometimes want the opposite behaviour.
If I already know that approximately ten AWS components are involved, I can define the boundary before the inspection starts.
I ask for the minimum set of commands required to observe those components, execute them and provide the resulting evidence.
That gives me:

Targeted inspection script

smaller context + explicit scope + human-controlled execution + disposable tooling
Neither model is universally better.
  • An MCP is more convenient when exploration is genuinely part of the problem.
  • A targeted snapshot can be preferable when the problem is already sufficiently constrained.
And there is no reason they cannot be combined. A more sophisticated workflow could use MCP tooling while still applying the same principle: deliberately constrain the evidence gathered for a particular diagnostic question.

Context quality matters more than token counting

There is also a context-window dimension to this approach, although I would be careful not to turn it into a simplistic “save tokens” argument.
I have not measured a universal token reduction, and different models, IDEs and tool integrations behave differently.
The more important effect is qualitative.
If the model has access to many tools and performs broad exploration, the diagnostic conversation can accumulate tool definitions, API calls, responses and intermediate hypotheses.
Some of that information will be useful. Some will not.
A targeted inspection limits the amount of unrelated evidence entering the reasoning process.
Instead of providing every route table, Lambda function, IAM role and VPC in an account, I provide the resources associated with the problem being investigated.
The objective is not simply to make the prompt shorter.
It is to improve the signal-to-noise ratio.
For this kind of troubleshooting, context quality is normally more interesting than context quantity.

Human-controlled AWS inspection loop

The model generates the inspection mechanism, but the engineer reviews and executes it before any interaction with AWS takes place.
This diagram shows the technical flow.


Keeping execution under human control

There is another property of this pattern that I consider useful: the AI does not inherently need autonomous AWS credentials.
  1. The model proposes the commands.
  1. I review them.
  1. I execute them.
This creates a very simple boundary between reasoning and access to the environment.
That does not make generated commands automatically safe. They still need to be inspected before execution, just as generated application code or Terraform changes should be reviewed.
For this use case, I also prefer read-only inspection whenever possible.
Most of what I need during the observation phase comes from  describe ,  get  and  list  operations.
If a diagnostic workflow starts modifying infrastructure before the problem has been understood, then observation and remediation are becoming mixed together. I normally prefer to keep those phases separate.
Again, this is not a limitation of AI tooling. It is a design decision about how much authority I want to delegate during troubleshooting.

Reverse engineering reality

What I found useful about this technique was not the shell script itself.
The script is trivial and disposable.
The important shift was recognising that source code is only one representation of a running platform.
When working with a clean system, the difference may be negligible. The repository, deployment mechanism and AWS environment can align almost perfectly.
When working with a legacy or partially automated platform, that assumption becomes much weaker.
In those environments, I increasingly treat troubleshooting as a comparison between three things:
  • the problem I can observe,
  • the state I intended to create,
  • and the state the cloud provider actually reports.
Generative AI is very good at finding relationships, inconsistencies and plausible explanations once those three views are available.
But the evidence still needs to come from somewhere.
Expecting the model to infer runtime state from source code effectively asks it to fill the missing part with assumptions.
Providing a targeted AWS snapshot replaces part of that uncertainty with observation.
For me, that is the more interesting lesson. This is not really a technique for making AI “debug AWS”.
It is closer to a lightweight form of targeted observability for AI-assisted troubleshooting.
The model can compare evidence and propose hypotheses. The engineer still decides whether the diagnosis is valid, whether additional evidence is required and, ultimately, which change should be made.
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article