How to Prevent Flaky Tests: Root Causes, Detection, and Structural Fixes

The first time a flaky test blocked a Friday deployment, the team spent two hours investigating before someone suggested just rerunning the pipeline. It passed on the second run. The deployment went out. The test failure was filed under "known flakiness" and nobody looked at it again.

Six weeks later the same test failed. This time something was actually broken. The team reran the pipeline. It passed. The deployment went out. The production incident arrived four hours later.

That is what flaky tests actually cost. Not the investigation time when they fail. The trust they erode until real failures get treated like noise.

A flaky test  is non-deterministic - it produces different results across identical runs without any code change. Understanding why it is non-deterministic is the only path to fixing it rather than working around it.

The sources of non-determinism in test suites fall into distinct patterns. Fix the wrong category and the flakiness comes back. Or it does not come back but a different category keeps producing failures that look identical from the outside.

The most common source of flakiness in modern test suites is timing. A test assumes that an asynchronous operation will complete within a specific time window. On a developer's laptop under light load, it does. On a CI runner handling multiple concurrent jobs, the same operation takes longer and the test times out.

The tell-tale signature: the test passes locally and fails in CI intermittently, with no relationship between failures and code changes.

The fix is replacing time-based waiting with condition-based waiting. Instead of waiting 200 milliseconds and then asserting, wait until the condition being asserted is true - checking repeatedly with a timeout rather than assuming a fixed duration is sufficient. This makes the test independent of execution speed and eliminates the entire category of timing-related flakiness.

A more subtle version of this problem involves event ordering. A subtler version shows up when a test fires two requests simultaneously and then asserts on which response came back first. We ran into this on a search service - two concurrent queries for different product categories, assertion that category A results arrived before category B results. Passed ninety times. Failed on run ninety-one because the network stack happened to reorder them under slightly higher load. The fix was asserting that both result sets arrived and contained the right data, not that they arrived in a specific sequence. The sequence was never guaranteed. The test had just been lucky.

A test that passes alone and fails in the suite is almost always a shared state problem. Something upstream left a record in the database, modified a global variable, wrote to a file, or populated a module-level cache that the failing test did not expect to find.

Run the suite in randomized order and the dependency becomes visible immediately. An order-dependent test fails on some orderings and passes on others. The randomization forces the hidden dependency into the open rather than letting it hide behind a consistent execution sequence that happens to satisfy it every time.

The fix is making each test responsible for its own state. Set it up in the test's setup phase. Tear it down after. Nothing leaks to the next test. Where shared infrastructure like a database is unavoidable, each test uses a unique namespace - a unique table prefix, a unique key prefix, a transaction that gets rolled back that prevents tests from reading each other's writes.

The hardest version of this problem involves test frameworks that share module state across test files. A module that gets modified during one test file's execution retains that modification when the next test file loads it, unless the framework explicitly resets the module registry between files. Configuring the test framework to clear module state between files eliminates this category entirely.

Tests that assert against values that change on every run will always be flaky. Timestamps generated at assertion time. Auto-incremented database IDs. UUIDs generated during object creation. Session tokens. Request correlation IDs.

When a test captures one of these values at a point in time and asserts against it on future runs, the assertion fails the moment the value changes - which is on every run.

The fix has two forms. For values that should be excluded from the assertion entirely - a timestamp on a created record, for instance - exclude them explicitly from the assertion scope. For values that should be consistent but are not - a generated ID that should be stable but changes between test runs - investigate whether the ID generation logic is deterministic and whether the test setup is creating fresh state on each run when it should be reusing existing state.

Test suites that run tests in parallel for speed create interference when those tests share external resources. Two tests that both write to the same database table, connect to the same port, or read and modify the same file will produce results that depend on execution timing rather than on the code under test.

In cloud-native environments this problem is more pronounced than in local development. A CI runner executing ten parallel test workers against a shared database instance generates more contention than a developer's laptop running two parallel workers. The test passes locally and flakes in CI because the resource contention under CI's parallelism level triggers race conditions that do not manifest at lower concurrency.

The structural fix is resource isolation per test worker. Each parallel worker gets its own database instance, its own port range, its own filesystem namespace. The isolation adds setup overhead but eliminates the entire class of parallel interference flakiness.

For AWS-based CI pipelines, this maps to specific service choices. Tests that connect to DynamoDB should use table names namespaced by worker ID, not a shared table name. Tests that use SQS should create queues per test run and delete them in teardown rather than sharing a persistent queue across tests - SQS's eventual consistency means messages from one test run can bleed into another's assertion window if queues are shared. Lambda-backed integration tests that exercise cold start behavior should account for the first invocation's elevated latency in their timeout configuration rather than assuming warm Lambda execution time on every run.

This is the flakiness category that does not look like flakiness. The test does not fail intermittently on the same machine. It passes consistently in one environment and fails consistently in another: local development passes, staging fails, or CI passes, production fails.

The root cause is that the test's behavioral assumptions about upstream dependencies do not match the actual behavior of those dependencies in all environments. The mock file that represents an upstream API reflects the response format from six weeks ago. The real upstream service in the staging environment has deployed twice since then and changed its error schema. The test passes against the mock and fails against the real service.

Most teams investigate this as an infrastructure problem. They check environment variables, compare connection strings, verify network access. None of that finds the actual issue because the actual issue is in the test fixtures, not the environment configuration.

The question that surfaces it faster: what is this test running against in each environment? Local development uses mock files. Staging connects to the real upstream service. The mock files describe how the upstream service behaved six weeks ago. The real service has deployed four times since then. That is the discrepancy. Not a missing environment variable.

The structural fix is changing where the behavioral assumptions come from. Mock files written by developers encode assumptions about upstream service behavior at the time of writing. Those assumptions age as upstream services continue deploying. Keploy addresses this at the capture layer - observing real HTTP traffic between services and generating test fixtures from those actual interactions rather than from developer-authored specifications. Fields that vary between requests, like timestamps and generated identifiers, are identified automatically across multiple observations and excluded from assertions without manual annotation. When an upstream service changes behavior, a new observation session captures the updated behavior and refreshes the fixtures. The behavioral assumptions stay current with what the upstream service is actually doing rather than with what it was doing when someone last had time to update a mock file.

Preventing flaky tests requires detecting them before they become accepted as normal. Three pipeline-level practices make flakiness visible rather than hidden.

Repeat runs on a cadence. Run the full test suite three to five times consecutively without any code change once per week. Any test that produces different results across identical runs is flaky. This surfaces flakiness that only manifests occasionally and would take months to accumulate enough CI failure data to identify.

Track rerun rates per test. When CI systems support test-level retry, log which tests triggered retries. A test with a rerun rate above 5% over a rolling thirty-day window is a flaky test that has learned to hide behind retry configuration. A test that retried twice before passing on six of the last thirty CI runs is flaky. The retry configuration is hiding it. Pulling that data into a visible dashboard is the difference between a team that knows they have eight flaky tests and a team that thinks they have zero flaky tests because nothing is blocking the pipeline.

Quarantine rather than retry. Move identified flaky tests to a separate suite that runs but does not gate the main pipeline. This sounds like the same thing as accepting flakiness, but it is not. Quarantine makes the flakiness explicit and tracked. Retry makes it invisible and growing. Quarantined tests show up in a list that gets reviewed in maintenance windows. Retried tests show up nowhere until they fail at the wrong moment.

Every flaky test is pointing at something real. A timing assumption that does not hold under load. A state dependency that should not exist. A behavioral assumption about an upstream service that stopped being accurate after a deployment. An assertion against a non-deterministic value that should have been excluded.

Teams that prevent flakiness investigate every flaky failure to find what it is pointing at. Teams that work around flakiness configure retries until the failures become infrequent enough to ignore. The second approach is faster in the short term. It produces the situation from the opening of this article - where the pipeline that has learned to pass despite failures passes through a real failure because the team has learned to expect the pipeline to pass despite failures.

The detection practices and structural fixes described here address symptoms. The mindset shift that makes those practices sustainable is treating every flaky failure as information rather than as noise, even when the information is inconvenient to act on immediately.