AWS Builder Center

Builder Center

How I Test LLM-Generated CloudFormation: A Practical Test Suite for Hallucinated Properties and Unsafe Defaults

LLM-generated CloudFormation can look perfect and still fail on deploy or ship unsafe settings. Here is a pytest-based approach that catches invalid properties, risky defaults, and regressions across a set of prompts.

The problem

When an LLM writes CloudFormation, the output usually looks right. The YAML is clean, the resource names make sense, and the comments are helpful. But three things regularly go wrong:
  • Hallucinated properties: the model invents a property that doesn't exist on that resource type.
  • Unsafe defaults: security groups open to 0.0.0.0/0, public buckets, unencrypted storage.
  • Silent regressions: a prompt that worked last week produces a different template after you tweak the system prompt.
You can't catch these by reading the output once. You need tests that run every time the generator changes.

The approach: treat generated templates like code

I run every generated template through a small pytest suite with three kinds of checks.

1. Syntax and property validation with cfn-lint

cfn-lint validates templates against the official AWS resource specification, so it catches invalid property names and wrong types.
1
2
3
4
5
6
7
8
import subprocess, json

def lint(path):
result = subprocess.run(
["cfn-lint", path, "--format", "json"],
capture_output=True, text=True,
)
return json.loads(result.stdout or "[]")
Tip: pin your cfn-lint version in requirements.txt. Different versions use different spec versions, and mismatches produce confusing errors that don't match what CloudFormation actually does.

2. Policy checks for unsafe defaults

Linting won't tell you that opening SSH to the world is a bad idea. I parse the template and check for rules I care about.
CloudFormation uses short tags like !Ref, so the YAML loader needs to accept them:
1
2
3
4
5
6
7
8
9
10
11
12
13
import yaml

class CfnLoader(yaml.SafeLoader):
pass

def _cfn_tag(loader, suffix, node):
if isinstance(node, yaml.ScalarNode):
return loader.construct_scalar(node)
if isinstance(node, yaml.SequenceNode):
return loader.construct_sequence(node)
return loader.construct_mapping(node)

CfnLoader.add_multi_constructor("!", _cfn_tag)
Then a simple check for open ingress:
1
2
3
4
5
6
7
8
9
def open_ingress(template):
bad = []
for name, res in template.get("Resources", {}).items():
if res.get("Type") == "AWS::EC2::SecurityGroup":
rules = res.get("Properties", {}).get("SecurityGroupIngress", [])
for rule in rules:
if rule.get("CidrIp") == "0.0.0.0/0" and rule.get("FromPort") not in (80, 443):
bad.append(name)
return bad
The same pattern works for public S3 buckets, missing encryption, and oversized instance types.

3. A golden prompt suite

I keep a list of prompts with the resource types each one must produce. pytest runs them all:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
import pytest

GOLDEN = [
("A static website on S3 with CloudFront", {"AWS::S3::Bucket", "AWS::CloudFront::Distribution"}),
("A private RDS PostgreSQL database in a VPC", {"AWS::RDS::DBInstance", "AWS::EC2::VPC"}),
]

@pytest.mark.parametrize("prompt,expected_types", GOLDEN)
def test_generated_template(prompt, expected_types, tmp_path):
text = generate_template(prompt) # your LLM call
path = tmp_path / "template.yaml"
path.write_text(text)

assert lint(str(path)) == [] # valid syntax and properties
template = yaml.load(text, Loader=CfnLoader)
assert open_ingress(template) == [] # no unsafe ingress
types = {r["Type"] for r in template["Resources"].values()}
assert expected_types <= types # right resources present
Now when I change the system prompt or swap the model, I run pytest and immediately see what broke.

What I learned

  1. Most failures are small. A wrong property name or a missing required field, not a completely wrong architecture. Linting catches these cheaply.
  2. Layer your checks. Lint for correctness, policy checks for safety, golden prompts for regressions. Each catches a different class of problem.
  3. Tests change how you prompt. Once failures were visible, I could fix the system prompt with evidence instead of guessing.

Going further

  • Add AWS Config conformance packs for post-deployment compliance auditing.
  • Move complex rules into a policy-as-code tool like OPA instead of hand-written Python.
  • Track pass rate per check over time to measure whether prompt changes actually help.

Wrapping up

An LLM that writes infrastructure code needs the same safety net as any other code: tests that run automatically. A small pytest suite takes an afternoon to build and catches problems before they reach your AWS account.
# cloudformation-resources# testing# infrastructure-as-code
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article