AWS Builder Center

Builder Center

Terragrunt Performance: From 2m27s to Under 30s

Terragrunt Performance: From 2m27s to Under 30s

Two fixes that cut terragrunt plan startup from 2 minutes 27 seconds to 23 seconds in a large multi-account setup with SOPS secrets and many transitive dependencies.

Serhii Kaidalov

Principal DevOps Engineer // SRE
When terragrunt plan on a single module takes over two minutes before showing a single AWS API call, something is wrong. This post covers two distinct performance problems I found and fixed in a large multi-account Terragrunt setup (~15 AWS accounts, 400+ modules).

The Problem

Running terragrunt plan on our EKS cluster module was taking 2 minutes 27 seconds just to start. With debug logging enabled, the culprit was clear:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
$ time terragrunt plan --terragrunt-log-level debug 2>&1 | head -80

09:20:53 INFO Detected 14 modules in the dependency graph
09:20:53 INFO sops decrypt: acquiring lock for .../secrets/db-secret.enc.yaml
09:20:54 INFO sops decrypt: decrypting .../secrets/db-secret.enc.yaml
09:20:55 INFO sops decrypt: acquiring lock for .../secrets/payment-api-secret.enc.yaml
09:20:56 INFO sops decrypt: decrypting .../secrets/payment-api-secret.enc.yaml
...
09:21:35 INFO Running module .../global/iam/policies
09:21:35 INFO terraform init
09:21:48 INFO terraform output
09:21:52 INFO Running module .../eventbridge/infra-changes-bus
09:21:52 INFO terraform init
09:22:05 INFO terraform output
...
09:23:20 INFO Initializing backend for target module...

real 2m27s
Two separate phases caused the slowness. Both are fixable.

Phase 1: SOPS Decryption During Graph Traversal (~40s)

Root Cause

Terragrunt builds a dependency graph by parsing all transitive dependencies before executing anything. During this traversal it evaluates every module's locals block. If any local calls sops_decrypt_file(), SOPS decrypts the file right then, even if that module is only a transitive dependency and won't execute at all.
Our ssm/secrets-manager was a transitive dependency of the EKS cluster. It had 11 SOPS calls in locals:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
# BEFORE: sops_decrypt_file in locals runs during graph traversal
locals {
secret_db = yamldecode(sops_decrypt_file("${get_terragrunt_dir()}/secrets/db-secret.enc.yaml"))
secret_payment = yamldecode(sops_decrypt_file("${get_terragrunt_dir()}/secrets/payment-api-secret.enc.yaml"))
secret_auth = yamldecode(sops_decrypt_file("${get_terragrunt_dir()}/secrets/auth-secret.enc.yaml"))
# ... 8 more sops calls
}

inputs = {
input_secrets = {
"db-secret" = {
secret_string = jsonencode({
username = local.secret_db.username
password = local.secret_db.password
})
}
# ...
}
}
Every time any module that transitively depended on ssm/secrets-manager was planned, all 11 files were decrypted, whether or not the module itself was being executed.

Fix: Move SOPS Calls into inputs

The inputs block is only evaluated when the module itself executes, not during graph traversal:
1
2
3
4
5
6
7
8
9
10
11
12
# AFTER: sops_decrypt_file in inputs only runs when this module executes
inputs = {
input_secrets = {
"db-secret" = {
secret_string = jsonencode({
username = yamldecode(sops_decrypt_file("${get_terragrunt_dir()}/secrets/db-secret.enc.yaml")).username
password = yamldecode(sops_decrypt_file("${get_terragrunt_dir()}/secrets/db-secret.enc.yaml")).password
})
}
# ...
}
}

"But aren't you decrypting the same file twice?"

No. Terragrunt caches sops_decrypt_file() results within a single run. From Terragrunt source config_helpers.go :
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
var sopsCache = cache.NewCache[string](sopsCacheName)

func decryptFile(ctx context.Context, path string) (string, error) {
if val, ok := sopsCache.Get(ctx, path); ok {
return val, nil
}
locks.EnvLock.Lock()
defer locks.EnvLock.Unlock()
if val, ok := sopsCache.Get(ctx, path); ok {
return val, nil
}
decrypted := decrypt(path)
sopsCache.Put(ctx, path, decrypted)
return decrypted, nil
}
The double-check locking pattern ensures that even concurrent goroutines evaluating the same file only decrypt once. The repeated yamldecode(sops_decrypt_file(...)) pattern in inputs is cosmetic verbosity. The first call per path decrypts, every subsequent call in the same run returns the cached result instantly.
[!TIP]
Rule: Never put sops_decrypt_file() in a locals block. Always inline it at the point of use in inputs or generate blocks.
We migrated 81 files across 15 AWS accounts following this rule.

Pattern Variations

Standard case:
1
2
3
4
5
6
7
8
9
10
11
12
# Before:
locals {
secrets = yamldecode(sops_decrypt_file("${get_terragrunt_dir()}/secrets.enc.yaml"))
}
inputs = {
password = local.secrets.password
}

# After:
inputs = {
password = yamldecode(sops_decrypt_file("${get_terragrunt_dir()}/secrets.enc.yaml")).password
}
Chained locals (a derived local referencing the SOPS local):
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
# Before:
locals {
secrets = yamldecode(sops_decrypt_file("${path}/secrets.enc.yaml"))
api_token = local.secrets.logging_api_tokens.prod.devops
notification_slack = local.secrets.notification_endpoints.slack_channels.prod
}
inputs = {
api_token = local.api_token
notification_endpoints = local.notification_slack
}

# After: inline the full chain
inputs = {
api_token = yamldecode(sops_decrypt_file("${path}/secrets.enc.yaml")).logging_api_tokens.prod.devops
notification_endpoints = yamldecode(sops_decrypt_file("${path}/secrets.enc.yaml")).notification_endpoints.slack_channels.prod
}
Generate blocks (SOPS feeding template rendering):
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
# Before:
locals {
secrets = yamldecode(sops_decrypt_file("${get_terragrunt_dir()}/secrets.enc.yaml"))
}
generate "kustomization" {
contents = templatefile("kustomization.yml.tpl", {
username = local.secrets.dashboard_username
password = local.secrets.dashboard_password
})
}

# After:
generate "kustomization" {
contents = templatefile("kustomization.yml.tpl", {
username = yamldecode(sops_decrypt_file("${get_terragrunt_dir()}/secrets.enc.yaml")).dashboard_username
password = yamldecode(sops_decrypt_file("${get_terragrunt_dir()}/secrets.enc.yaml")).dashboard_password
})
}

Phase 2: Dependency Output Fetching (~70s)

Root Cause

For each dependency block, Terragrunt by default:
  1. Runs terraform init in the dependency directory (downloads providers, initializes backend)
  2. Runs terraform output -json to extract outputs
With 8+ transitive dependencies, this meant ~70 seconds of repeated terraform init and S3/network calls before the target module could even start:
1
2
3
4
5
6
09:21:35 INFO Running module .../global/iam/policies
09:21:35 INFO terraform init # provider downloads, backend init (~8s)
09:21:48 INFO terraform output # reads state via backend (~5s)
09:21:52 INFO Running module .../eventbridge/infra-changes-bus
09:21:52 INFO terraform init # again, every dependency
09:22:05 INFO terraform output

Fix: Read Outputs Directly from S3

Terragrunt can read dependency outputs straight from S3 state files, skipping terraform init and terraform output entirely:
1
export TG_DEPENDENCY_FETCH_OUTPUT_FROM_STATE=true
Add to ~/.zshrc or ~/.bashrc to make it permanent.
After setting the flag:
1
2
3
4
09:21:35 INFO Fetching outputs for .../global/iam/policies from S3 state
09:21:35 INFO Reading s3://acme-terraform-state/global/iam/policies/terraform.tfstate
09:21:36 INFO Fetching outputs for .../eventbridge/infra-changes-bus from S3 state
09:21:36 INFO Reading s3://acme-terraform-state/security/us-east-1/eventbridge/.../terraform.tfstate
One S3 GetObject per dependency instead of a full terraform init + terraform output subprocess.
[!WARNING]
It's TG_DEPENDENCY_FETCH_OUTPUT_FROM_STATE, not TG_FETCH_DEPENDENCY_OUTPUT_FROM_STATE. Easy to mix up. Double-check if it doesn't seem to work.

Bonus: S3 Native State Locking

While in root.hcl, we also dropped DynamoDB for state locking. Terragrunt supports S3-native lock files: a .tflock file written alongside the state. Make sure your version of Terraform or OpenTofu supports it as well.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
# Before: requires a DynamoDB table provisioned in every account
remote_state {
backend = "s3"
config = {
encrypt = true
bucket = "acme-terraform-state-${local.account_id}"
key = "${path_relative_to_include()}/terraform.tfstate"
region = local.aws_region
dynamodb_table = "terraform-locks"
profile = local.aws_profile
}
}

# After: S3-native locking, no DynamoDB needed
remote_state {
backend = "s3"
config = {
encrypt = true
bucket = "my-terraform-state-${local.account_id}"
key = "${path_relative_to_include()}/terraform.tfstate"
region = local.aws_region
use_lockfile = true
profile = local.aws_profile
}
}
Benefits: no DynamoDB table per account, simpler bootstrap for new accounts.

Results

1
2
3
4
5
6
7
# Before (any module with 8+ transitive deps):
$ time terragrunt plan
real 2m27s

# After (both fixes applied):
$ time terragrunt plan
real 0m23s
FixTime saved
SOPS locals to inputs~40s
FETCH_OUTPUT_FROM_STATE=true~70s
Total~110s

Summary

Three changes, one dramatic improvement:
  1. Move sops_decrypt_file() from locals to inputs: prevents SOPS from decrypting during dependency graph traversal. ~40s saved per plan on any module with SOPS-using transitive dependencies.
  2. export TG_DEPENDENCY_FETCH_OUTPUT_FROM_STATE=true: reads dependency outputs directly from S3 state, skips terraform init and terraform output subprocesses per dependency. ~70s saved on modules with many dependencies.
  3. use_lockfile = true in remote_state: removes DynamoDB for state locking, simplifies multi-account setup.
Total: 2m27s to 23s on a module with 8 transitive dependencies and 11 SOPS-encrypted secret files in the dependency graph.
# terraform# amazon-s3# devops# IaC# performance
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article