AWS Builder Center

DriftLens: Finding and Explaining Hidden Configuration Changes in Cloud Systems

DriftLens is an agentless tool that detects when staging and production configurations silently drift apart, classifies each divergence by severity, and pinpoints exactly who changed it and when using CloudTrail turning "why does prod behave differently?" from a guessing game into a simple lookup.

The problem

Most software companies do not release changes straight to their live website or app. They keep at least two copies of the system:
  • Production: the live system that real customers use.
  • Staging: a private copy used only for testing. New changes go here first, and if they work, they are released to production.
This process rests on one assumption: if something works in staging, it will work in production. That is only true if the two copies are set up identically, and over time they often stop being identical.
Every system has two parts. One is the code, the program itself. The other is configuration, a set of settings that control how the program behaves. Examples are a switch that turns a new feature on or off, the address of a database, or a limit on how many requests are allowed. The same code can behave very differently depending on these settings.
Configuration drift is when the settings in two environments that should match slowly become different. It usually happens like this:
  • Something breaks in production late at night and an engineer needs to fix it quickly.
  • Instead of going through the normal process, they change one setting directly on the live system. This works and the problem goes away.
  • They intend to make the same change in staging the next day, but they forget, or the change is never written down.
From then on, the two environments differ in a way nobody knows about. The problem appears later, when a new release is tested in staging, passes, and is sent to production, where it fails. The code is the same in both places, so the team has no obvious explanation. Someone has to compare hundreds of settings by hand to find the one difference. Even after they find it, they still have to work out who changed it, when, and why.
DriftLens automates this. It does three things:
  • Capture: it regularly reads the settings from each environment and saves a copy.
  • Compare: it checks the saved copies against each other, or against an earlier copy of the same environment, and lists every setting that differs.
  • Attribute: for each difference, it finds out which person or account made the change and when.

Why existing tools miss this

Two widely used tools are meant to keep environments consistent, but neither covers this problem.
  • Docker bundles a program and everything it needs to run into one package, so the same code and libraries run in every environment. It guarantees the program is identical, but says nothing about the settings applied to it while it runs.
  • Terraform describes the infrastructure in a written file, such as how many servers exist and how powerful they are, and can detect when the real infrastructure no longer matches that file. It covers the hardware and layout, not the individual settings inside running services.
Tool
What it secures
What it misses
Docker
Code & Libraries
Runtime Configuration
Terraform
Physical Infrastructure
Live Application Toggles
DriftLens
Runtime Configuration
N/A
The settings DriftLens looks at are called runtime configuration. They are stored in Amazon Web Services (AWS) and read by the application while it runs. This includes:
  • Parameter Store (part of AWS Systems Manager): a central place for storing named values that applications read, such as a timeout length or a service address.
  • AppConfig: a service for managing feature flags. A feature flag is an on/off switch that lets a company release a feature to some users, or hold it back, without changing the code.
  • Lambda environment variables: settings attached to small programs that AWS runs on demand (called Lambda functions).
Anyone with access can change these values in seconds through the AWS website or a command, and no code or infrastructure file changes when they do. Docker and Terraform therefore never see it happen. This gap is the one DriftLens covers.

How the system works

The team built DriftLens so that nothing has to be installed on the systems being watched. It only reads settings from the outside, using the access AWS already provides. This means there is no extra software to set up or maintain on every server, and the tool can only see what its permissions allow.
The system is built as a sequence of stages. Each stage was built by a different team member and uses a specific AWS service.
Stage 1: Coordinating the process. The stages have to run in the right order, and the system should keep going or report clearly if one of them fails. This is handled by AWS Step Functions, a service for defining a workflow as a series of connected steps. Step Functions starts the collection of settings from the environments at the same time rather than one after another, waits for both to finish, and then passes the results to the next stage.
Stage 2: Collecting and saving settings. A small program (a Lambda function) called the collector reads the settings from the services listed above. It saves what it finds as a snapshot, a record of every setting and its value at that moment. Snapshots are stored in Amazon S3, AWS's file storage service.
The storage bucket has versioning turned on. Normally, saving a file with the same name replaces the old one. With versioning, the old copy is kept and the new one is stored alongside it. This is what allows two kinds of comparison:
  • Staging against production today.
  • Production today against production yesterday, or last week, to see exactly when something changed.
Stage 3: Comparing snapshots. The comparison program is written in Python and takes two snapshots. It does not just compare raw text. It first matches each setting in one snapshot with the same setting in the other, then checks whether the values are the same. Any mismatch is reported as a drift. Each drift is saved as a record in Amazon DynamoDB, a database service suited to storing many small records that are looked up quickly. The dashboard later reads these records.
Stage 4: Finding who made the change. Knowing that a setting differs does not tell you who changed it. AWS keeps a log called CloudTrail that records every action taken in an account: who did it, what they did, and when. A manual change to a setting always leaves an entry there, linked to the person's account (their IAM user, which is AWS's term for a user identity and its permissions).
CloudTrail logs become extremely large, and searching them by hand is not practical. DriftLens uses Amazon Athena, a service that lets you search large amounts of stored data using ordinary database-style queries, without setting up a database. Dev B wrote queries that look through the logs for the specific action that matches each drift and return the account responsible and the time it happened.
Stage 5: The dashboard. The results are shown in a web page built with React, a common tool for building web interfaces. The page is hosted on S3, which can also serve simple websites. It gets its data through Amazon API Gateway, a service that gives the backend programs web addresses the page can request data from. The dashboard shows all the drifts in a table, with colors to indicate how serious each one is.

Github repo link :  https://github.com/DaivyaPatel/bharat_builds_bangalore_tfotr 

Drift Explanations given by AI

Seeing that two values differ is useful, but an engineer still has to work out what that setting does and whether the difference is a problem. Someone unfamiliar with that part of the system can lose a lot of time on this.
For drifts marked as critical, DriftLens sends the details of the difference to a large language model, an AI system that generates text. The model is run through a service called Groq and uses Meta's Llama 3. The model returns a short explanation in plain English, which is saved with the drift record and shown on the dashboard.
For example, if a setting controlling a new checkout page is on in staging and off in production, the explanation says exactly that: the new checkout is still being tested and has not been released to customers. The engineer gets the meaning of the difference directly and does not have to look up what the setting does.

Keeping passwords and secrets safe

A tool that reads settings across a whole environment will eventually come across sensitive information such as passwords or access keys. If it stored them, the tool itself would become a security risk. The team therefore designed it to avoid seeing or keeping such values at all.
  • Blocked access to secrets: AWS has a dedicated service for storing passwords and keys, called Secrets Manager. DriftLens is given an explicit rule that forbids reading from it, so it cannot access those values regardless of what the code does.
  • Checking what it reads: some secrets are stored in ordinary settings, so the collector also examines every value it reads. It uses two checks:
  • It looks at the setting's name. A name containing words like "password" or "token" is treated as sensitive.
  • It looks at the value itself. Real passwords and keys are long strings of random characters, which look statistically different from normal text. Values that look random in this way are treated as sensitive. This is called entropy scanning, and it catches secrets whose names give no hint.
  • Removal before saving: anything flagged is removed from memory before it is written anywhere.
  • Storing a fingerprint instead: in place of the real value, the collector saves a hash, a fixed-length scrambled version produced by a one-way calculation. The same input always gives the same hash, but the original cannot be worked out from it. A random extra value called a salt is added to the input first, which makes the hashes harder to guess or look up. If the hashes for staging and production differ, the values must differ, so DriftLens can report a mismatch without knowing what either password is. Both environments must use the same salt for this comparison to work.
Environment
Original Plaintext Setting
What is actually saved in our S3 Snapshot
Staging
{"DB_PASS": "testPass123"}
{"DB_PASS": "hash:8d969eef6ecad3c29a..."}
Production
{"DB_PASS": "prodSuperS3cr3t!"}
{"DB_PASS": "hash:f2c6948c8b418a0a93..."}
  • Keeping the AI key out of the code: DriftLens needs its own key to use the Groq service. This is supplied to the programs through their environment settings when they are deployed, so it never appears in the project's code on GitHub, where others could see it.

Problems the team ran into

  • The AI service they planned to use was blocked. The original plan was to use Amazon Bedrock, AWS's own AI service. The AWS accounts provided for the hackathon were restricted by Service Control Policies, which are rules set at the organization level that limit what any account underneath it can do. These rules override anything set inside an individual account, so the team could not enable Bedrock themselves. They switched to Groq instead.
  • The first full test found nothing. The comparison stage reported zero differences even though the environments were known to differ. The cause was naming. Resources in each environment carried the environment's name as a prefix, such as staging-checkout, and the production version had a different prefix. The comparison program treated them as two unrelated items and never compared them. The fix was to remove the prefix when the collector saves each item, so that both sides use the same base name. The general lesson is that names have to be made consistent before two environments can be compared.
  • A missing permission stopped the pipeline just before the demo. In AWS, every program needs explicit permission to do anything, including saving data to a database. The program that saves drift records had not been given permission to write to DynamoDB, so the whole pipeline failed. Dev A wrote a script that added exactly the permissions needed and nothing more, and applied it without rebuilding anything else.
  • The planned hosting method was being phased out. Dev C planned to host the dashboard using AWS Amplify, but the version they intended to use, Amplify Gen 1, had been moved into maintenance mode, meaning it was no longer being actively developed. The dashboard was hosted as a plain website on S3 instead, which was enough because it only needs to display pages and request data.

What each team member did and learned

  • Dev A (infrastructure and workflow): built the workflow that coordinates all the stages, including running collection in parallel, and fixed permission and deployment problems under time pressure. Their main takeaway was the value of least privilege, which means giving each part of a system only the access it needs.
  • Dev B (backend, comparison, and AI): wrote the comparison program, the log-search queries that identify who made a change, and the connection to the AI service. Their main takeaway was that data has to be cleaned and made consistent before it can be compared meaningfully.
  • Dev C (frontend and connections): built the dashboard and connected it to the backend, and changed the hosting approach when the first option was no longer usable.
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article