
How I automated cross-account DocumentDB dump/restore with Step Functions, ECS Fargate and S3 Replication
Serverless pipeline for automated Amazon DocumentDB dump/restore between AWS accounts with cross-region replication, Lambda validation, and Step Functions orchestration. Cost: ~$0.10 per execution.
TL;DR: We built a serverless pipeline that dumps all DocumentDB databases from one AWS account (São Paulo), replicates cross-account/cross-region to another account (Ohio), validates file integrity, and restores automatically — all without human intervention. Cost per execution is ~$0.10 and the full flow takes ~1-2 hours.
1. The Problem
Our team maintains an Amazon DocumentDB cluster in one environment (sa-east-1) that needs to be periodically replicated to another environment in a different region (us-east-2, in a separate AWS account). The scenario:
- Environment A (source): DocumentDB cluster with multiple databases in
sa-east-1(São Paulo)
- Environment B (destination): DocumentDB cluster in
us-east-2(Ohio), separate account
- Frequency: On-demand (with the option to schedule via cron)
- Volume: Variable-sized databases, ~5-20 GB total
What we did before
The process was 100% manual:
- Someone connected via bastion host in the source VPC
- Ran
mongodumpmanually for each database
- Compressed and uploaded to S3
- Downloaded in the destination environment
- Ran
mongorestoreon each database
Problems:
- Took ~2 hours of an engineer's manual work
- Risk of human error (forgetting a database, wrong credentials)
- No consistency guarantee across databases
- No visibility into the process (nobody knew if it worked until manually checking)
2. DocumentDB-Specific Challenges
Amazon DocumentDB is "MongoDB compatible" — but it's not MongoDB. This brings important challenges:
Mandatory TLS
DocumentDB requires TLS/SSL on all connections. You need to download the
global-bundle.pem certificate from AWS before any operation:wget https://truststore.pki.rds.amazonaws.com/global/global-bundle.pemAnd use these flags on every connection:
--ssl --sslCAFile /tmp/global-bundle.pem --sslAllowInvalidHostnamesPrivate Network (VPC-only)
DocumentDB is only accessible from within the VPC. This means any automation must run inside the VPC — you can't do it from a local machine or a Lambda outside the network.
Unsupported Indexes
Not all MongoDB index types work on DocumentDB. During
mongorestore , you'll see errors like "Unsupported feature" . Documents are restored normally — only some advanced indexes fail. We need to treat this as a warning, not a fatal error.retryWrites=false
DocumentDB doesn't support
retryWrites . The connection string must include ?retryWrites=false , otherwise the connection fails silently.
3. Solution Architecture

Components
Component | Purpose | Why this choice |
|---|---|---|
ECS Fargate | Runs mongodump/mongorestore | Runs inside the VPC, no server to maintain, pay only when running |
Step Functions | Orchestrates the flow | Automatic retry, timeout, visibility, logs |
S3 + Replication | Stores dumps and replicates cross-account | AWS-native, cross-region, no code needed |
EventBridge | Automatic trigger | Detects marker file in S3, starts restore |
Lambda | Validates replication | Ensures ALL files arrived before restore starts |
Secrets Manager | Credentials | Injects user/pass into the container without exposing in code |
ECR | Docker images | Stores custom dump/restore images |
Design Decisions
- Fargate Spot for dump: Dump tolerates interruption (Step Functions retry handles it). ~70% savings.
- Fargate On-Demand for restore: Restore cannot be interrupted mid-way (would corrupt the database).
- Marker file: A JSON file that indicates "dump complete". Without it, EventBridge could trigger the restore before all databases are copied.
- S3 Prefixes: Dumps are organized in
dumps/documentdb/with naming convention{database}_{timestamp}.tar.gz.
4. Implementation
📦 Full code (Docker, scripts, Terraform, Lambda) available in the repository: documentdb-cross-account-dump-restore
Below are the most relevant snippets for understanding the solution.
4.1 Docker Image — Dump
The image is based on
debian:bookworm-slim with mongodump , mongosh and AWS CLI installed:FROM debian:bookworm-slim
RUN apt-get update && \
apt-get install -y wget gnupg python3 python3-pip ca-certificates curl && \
rm -rf /var/lib/apt/lists/*
RUN pip3 install --no-cache-dir --break-system-packages awscli
RUN wget -qO - https://www.mongodb.org/static/pgp/server-7.0.asc | apt-key add - && \
echo "deb http://repo.mongodb.org/apt/debian bookworm/mongodb-org/7.0 main" | \
tee /etc/apt/sources.list.d/mongodb-org-7.0.list && \
apt-get update && \
apt-get install -y mongodb-org-tools mongodb-mongosh && \
rm -rf /var/lib/apt/lists/*
COPY dump-docdb.sh /usr/local/bin/dump-docdb.sh
RUN chmod +x /usr/local/bin/dump-docdb.sh
CMD ["/usr/local/bin/dump-docdb.sh"]4.2 Core Dump Logic (simplified)
The script lists databases, runs
mongodump on each, compresses, uploads to S3, and creates a marker file that triggers the restore:# Download TLS certificate (mandatory for DocumentDB)
wget -q https://truststore.pki.rds.amazonaws.com/global/global-bundle.pem -O /tmp/global-bundle.pem
TLS_OPTIONS="--ssl --sslCAFile /tmp/global-bundle.pem --sslAllowInvalidHostnames"
# List databases (excluding admin and local)
DATABASES=$(mongosh "mongodb://$DOCDB_USER:$DOCDB_PASS@$DOCDB_HOST:$DOCDB_PORT/?retryWrites=false" \
$TLS_OPTIONS --quiet \
--eval "db.adminCommand('listDatabases').databases.map(d => d.name).join('\n')" \
| grep -v "^admin$" | grep -v "^local$")
# Dump → compress → S3
for DB in $DATABASES; do
mongodump --host="$DOCDB_HOST:$DOCDB_PORT" --username="$DOCDB_USER" \
--password="$DOCDB_PASS" --db="$DB" --out="/tmp/${DB}_${TIMESTAMP}" $TLS_OPTIONS
tar -czf "/tmp/${DB}_${TIMESTAMP}.tar.gz" -C /tmp "${DB}_${TIMESTAMP}"
aws s3 cp "/tmp/${DB}_${TIMESTAMP}.tar.gz" "s3://$S3_BUCKET/${S3_PREFIX}${DB}_${TIMESTAMP}.tar.gz"
done
# Marker file — signals dump is complete
echo "{\"timestamp\":\"$TIMESTAMP\",\"databases\":$COUNT,\"type\":\"documentdb\"}" \
| aws s3 cp - "s3://$S3_BUCKET/${S3_PREFIX}dump_complete_${TIMESTAMP}.marker"4.3 Core Restore Logic (simplified)
The restore drops the database on the destination and runs
mongorestore :# Drop and restore each database
mongosh "mongodb://..." $TLS_OPTIONS --quiet --eval "db.getSiblingDB('$DB_NAME').dropDatabase()"
mongorestore --host="$DOCDB_HOST:$DOCDB_PORT" --username="$DOCDB_USER" \
--password="$DOCDB_PASS" --db="$DB_NAME" $TLS_OPTIONS "/tmp/restore/${DB_NAME}/"4.4 Step Functions — Orchestration
Step Functions uses
ecs:runTask.sync to execute the Fargate task and wait for completion, with automatic retry on failure:resource "aws_sfn_state_machine" "dump_docdb" {
name = "myapp-docdb-dump-ambiente"
definition = jsonencode({
StartAt = "DumpDocumentDB"
States = {
DumpDocumentDB = {
Type = "Task"
Resource = "arn:aws:states:::ecs:runTask.sync"
Parameters = {
Cluster = aws_ecs_cluster.dump.arn
TaskDefinition = aws_ecs_task_definition.dump_docdb.arn
CapacityProviderStrategy = [{ CapacityProvider = "FARGATE_SPOT", Weight = 1 }]
NetworkConfiguration = { AwsvpcConfiguration = { Subnets = var.private_subnet_ids, SecurityGroups = [aws_security_group.ecs_dump.id], AssignPublicIp = "DISABLED" } }
}
TimeoutSeconds = 7200
Retry = [{ ErrorEquals = ["States.TaskFailed"], MaxAttempts = 2 }]
Next = "DumpSuccess"
}
DumpSuccess = { Type = "Succeed" }
}
})
}4.5 EventBridge — Automatic Trigger
EventBridge detects the marker file in the destination bucket and triggers the restore Step Function:
resource "aws_cloudwatch_event_rule" "s3_marker_docdb" {
name = "myapp-docdb-marker-created-ambiente"
event_pattern = jsonencode({
source = ["aws.s3"]
detail-type = ["Object Created"]
detail = {
bucket = { name = [aws_s3_bucket.dump_destino.bucket] }
object = { key = [{ wildcard = "dumps/documentdb/*.marker" }] }
}
})
}🔗 Full code in the repository: github.com/felipethebrito/documentdb-cross-account-dump-restore — Dump/restore scripts, Dockerfiles, Terraform (IAM, ECS, Step Functions, EventBridge, Lambda, S3, Secrets Manager) — ready to adapt to your environment.
4.6 IAM Roles
The solution requires several IAM roles following least-privilege principle. Here's a summary of each role and its permissions:
Role | Service | Permissions | Purpose |
|---|---|---|---|
ECS Task Execution | ECS | AmazonECSTaskExecutionRolePolicy + secretsmanager:GetSecretValue | Pull Docker image from ECR and inject secrets into the container |
ECS Task | ECS | s3:PutObject , s3:GetObject , s3:ListBucket + secretsmanager:GetSecretValue | Upload/download dumps to/from S3, read credentials at runtime |
Step Functions | States | ecs:RunTask , ecs:StopTask , ecs:DescribeTasks , iam:PassRole , lambda:InvokeFunction | Orchestrate ECS tasks and invoke validation Lambda |
S3 Replication | S3 | s3:GetReplicationConfiguration , s3:ReplicateObject , s3:ReplicateDelete | Cross-account/cross-region replication from source to destination bucket |
EventBridge | Events | states:StartExecution | Trigger the restore Step Function when marker file arrives |
Lambda Validator | Lambda | s3:GetObject , s3:ListBucket + AWSLambdaBasicExecutionRole | Read marker file and list dump files to validate replication completeness |
Key point: The S3 Replication role in the source account needs permissions on the destination bucket (cross-account). The destination bucket policy must explicitly allow the source account's replication role to write objects.
Full Terraform code for all IAM roles is available inorigem/iam-docdb.tfanddestino/iam-docdb.tfin the repository.
5. Orchestration and Scheduling
Manual Execution (on-demand)
aws stepfunctions start-execution \
--state-machine-arn arn:aws:states:sa-east-1:111111111111:stateMachine:myapp-docdb-dump-ambiente \
--region sa-east-1 --profile ambienteAutomatic Execution (scheduled)
To schedule via EventBridge (e.g., every Sunday at 3am):
resource "aws_cloudwatch_event_rule" "weekly_dump" {
name = "myapp-docdb-dump-weekly"
schedule_expression = "cron(0 6 ? * SUN *)" # 3am BRT = 6am UTC
}Monitoring
# Real-time dump logs
aws logs tail /ecs/myapp-docdb-dump-ambiente --follow --region sa-east-1
# Restore logs
aws logs tail /ecs/myapp-docdb-restore-ambiente --follow --region us-east-2Step Functions also provides visual monitoring in the AWS Console — each step shows success/failure with timestamps.
6. Results
Metric | Before (manual) | After (automation) |
|---|---|---|
Engineer time | ~2 hours | 0 (1 command or automatic) |
Total time (dump→restore) | ~2-3 hours | ~1-2 hours |
Success rate | ~90% (human errors) | 100% (with automatic retry) |
Cost per execution | ~$50 (engineer's hour) | ~$0.10 (Fargate + S3) |
Visibility | None | CloudWatch + Step Functions |
Databases supported | Manual per DB | All automatically |
7. Lessons Learned / Gotchas
What went wrong the first time
-
retryWrites=false— The connection failed silently without this flag. It took 1 hour to discover that DocumentDB doesn't support retry writes.
- Unsupported indexes —
mongorestorereturned exit code != 0 even when documents were successfully restored (only indexes failed). We needed to treat this as a warning.
-
dropDatabase()timeout —dropDatabase()on large databases took over 30s. We addedtimeout 30to prevent the script from hanging.
- Expired TLS certificate — We used a fixed certificate in the Docker image. We switched to downloading
global-bundle.pemat runtime (always up to date).
- S3 Eventual Consistency — The marker file arrived before the
.tar.gzfiles in the destination bucket (replication). The validation Lambda needed retry with sleep to handle this.
- 100GB ephemeral storage — Large databases need disk space in the container. Fargate's default is 20GB — we needed to explicitly set
ephemeral_storage { size_in_gib = 100 }.
DocumentDB Gotchas
- It's not MongoDB — Tools that work with MongoDB may not work with DocumentDB
- No
mongos— No native sharding
- Incomplete
changeStreams— Limitations in CDC
- Wire protocol version — Some MongoDB 5+ features don't exist
- Direct connection required — No SRV records (
mongodb+srv://doesn't work)
8. Conclusion and Next Steps
Value delivered
- Zero human intervention — Dump + replication + validation + restore happen automatically
- Cross-account/cross-region — Source in São Paulo, destination in Ohio, separate accounts
- Infrastructure as Code — Everything in Terraform, reproducible and versioned
- Negligible cost — ~$0.10 per execution vs ~$50 of engineer time
- Reusable — The same pattern can be adapted for other databases (RDS, Aurora) with minimal changes
Planned improvements
- [ ] Automatic weekly scheduling via EventBridge cron
- [ ] Selective restore (choose specific databases)
- [ ] Sensitive data masking in the destination environment (see section 9)
- [ ] Slack/SNS notification when dump/restore completes
- [ ] Parallelism in restore (multiple simultaneous ECS tasks)
Shared resources
The infrastructure was designed to share resources where possible (ECS clusters, S3 buckets, VPC), while maintaining independence in DocumentDB-specific components (task definitions, step functions, lambdas). This reduces cost and complexity.
9. Data Masking in the Destination Environment
A natural evolution of this automation is ensuring that sensitive data from the source environment (PII, emails, national IDs, phone numbers) is not exposed in the destination environment. We added a post-restore step that executes masking queries directly on the destination DocumentDB.
Approach
After
mongorestore completes successfully, an additional script connects to the destination DocumentDB and executes updates on collections with sensitive data:// Sensitive data masking - post-restore
db = db.getSiblingDB('my_database');
// Mask emails and phone numbers
db.users.updateMany({}, [{
$set: {
email: { $concat: ["user_", { $toString: "$_id" }, "@masked.local"] },
phone: "11999999999",
document: "00000000000"
}
}]);
// Mask financial data
db.transactions.updateMany({}, [{
$set: {
account_number: "000000000",
card_number: "****-****-****-0000"
}
}]);
// Drop audit collections (not needed in destination)
db.audit_logs.drop();
db.access_logs.drop();Integration in the Flow
Masking is added as a final step in the restore script. After each database is successfully restored, the masking script executes via
mongosh :# After mongorestore
echo "Masking sensitive data..."
mongosh "mongodb://$DOCDB_USER:$DOCDB_PASS@$DOCDB_HOST:$DOCDB_PORT/$DB_NAME?retryWrites=false" \
$TLS_OPTIONS --quiet --file /usr/local/bin/mask-data.jsOr as a separate step in Step Functions:
ValidateReplication → WaitBeforeRestore → RestoreDocumentDB → MaskSensitiveData → SuccessConsiderations
- Performance: Bulk updates on large collections can be slow. Use
bulkWriteor process in batches for collections with millions of documents.
- Idempotency: If masking runs twice, it should not cause issues — updates overwrite with the same masked values.
- Validation: After masking, run a check query to ensure no real data remains (e.g., search for real email/ID patterns).
- Selectivity: Not all collections need masking — only those containing PII. Maintain a list of collections and fields to mask.
Repository
The complete structure includes:
├── origem/ # Terraform — source account
│ ├── ecs-docdb.tf
│ ├── stepfunctions-docdb.tf
│ ├── iam-docdb.tf
│ └── secrets-docdb.tf
├── destino/ # Terraform — destination account
│ ├── ecs-docdb.tf
│ ├── stepfunctions-docdb.tf
│ ├── iam-docdb.tf
│ ├── eventbridge-docdb.tf
│ ├── lambda-validator-docdb.tf
│ └── lambda/index_docdb.py
└── docker/
├── dump-docdb/
│ ├── Dockerfile
│ └── dump-docdb.sh
└── restore-docdb/
├── Dockerfile
├── restore-all-docdb.sh
└── restore-docdb.sh
If you've implemented something similar or have suggestions for improvement, I'd love to exchange ideas in the comments!
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article