Scaling Unified Observability for ECS

Run CloudWatch Agent in replica mode and OpenSearch Ingestion service to scale tracing and log analytics across ECS microservices—no code changes.

Pawan

Published Jun 23, 2026

Last modified Jun 23, 2026

The problem

Plenty of teams have already standardized on Amazon OpenSearch Service as their logging destination. It's where their dashboards, saved searches, and retention policies live. The question for these teams isn't whether to move logs somewhere else—it's how to add distributed tracing and run the log pipeline efficiently around that existing choice.

Two costs tend to creep in. First, the usual way to feed CloudWatch Application Signals is a CloudWatch Agent sidecar in every task. At scale that's one agent container per task, each reserving CPU and memory purely for telemetry. Second, self-managed ingestion—a Logstash fleet parsing and shipping logs into OpenSearch—is another set of instances to size, patch, and scale.

This post shows how to remove both. Deploy the CloudWatch Agent once in replica mode so a single shared agent serves the whole cluster, and replace Logstash with Amazon OpenSearch Ingestion (OSIS), a managed pipeline that parses, buffers, and routes logs and scales on its own.

One thing to set expectations on up front: traces go to Application Signals and logs go to OpenSearch, and the two stay in separate systems. This design does not provide trace-to-log correlation—Application Signals links traces to logs only when logs live in CloudWatch Logs. If your standard is OpenSearch for logs, that separation is the trade you accept, and in return you keep your OpenSearch analytics while gaining the two efficiency wins above.

Architecture overview

The solution runs two parallel pipelines.

Trace pipeline (real-time monitoring):

1
2
3
4
Application container (auto-instrumented)
  -> CloudWatch Agent replica (via Service Connect DNS)
    -> CloudWatch Application Signals
      -> Service maps, traces, metrics

Log pipeline (advanced analytics):

1
2
3
4
5
Application container (JSON logs to shared volume)
  -> Fluent Bit sidecar (reads from volume)
    -> OpenSearch Ingestion pipeline (parse & route)
      -> Amazon OpenSearch Service (service-specific indexes)
        -> OpenSearch Dashboards

The first key idea is CloudWatch Agent replica mode. Instead of a sidecar per task, you deploy one CloudWatch Agent as its own ECS service with Service Connect enabled. Application tasks discover it over DNS (for example cwagent-4316-http:4316) and send traces to it. One agent serves the whole cluster.

The second is OpenSearch Ingestion (OSIS) in place of a self-managed Logstash fleet. OSIS is a fully managed pipeline that parses incoming logs, buffers them durably, routes them to the right index, and auto-scales between the unit bounds you set—no instances to run.

Service	Role
CloudWatch Application Signals	Distributed tracing, service maps, APM metrics
Amazon OpenSearch Service	Log storage, full-text search, aggregations
Amazon OpenSearch Ingestion (OSIS)	Centralized parsing, buffering, dynamic index routing
ECS Service Connect	DNS-based discovery for the shared agent (works on Fargate)
ADOT auto-instrumentation	Injects the OpenTelemetry SDK with no code changes
AWS for Fluent Bit	Lightweight sidecar that forwards logs to OSIS with SigV4 auth

Prerequisites

Before you start, make sure you have:

The AWS CLI v2 installed and configured with credentials.
An existing ECS cluster on Fargate (platform version 1.4.0 or later, which Service Connect requires).
A VPC with at least two private subnets across two Availability Zones.
Docker and an Amazon ECR repository, if you plan to build the sample application image.
Permissions to create resources in ECS, OpenSearch, OSIS, Cloud Map, CloudWatch, and IAM.

Throughout the guide, replace these placeholders with your own values: ACCOUNT (your AWS account ID), REGION (this guide uses ap-south-1), YOUR_CLUSTER_NAME, and YOUR_APP_IMAGE. Capture the endpoints and ARNs the commands return into shell variables as shown—later steps depend on them.

Step 1: Create the IAM roles and trust policies

You need four roles. Each role needs a trust policy (which service is allowed to assume it) and a permissions policy (what it can do). The CloudWatch Agent, Fluent Bit, and ECS task roles are assumed by ECS tasks; the OSIS role is assumed by the ingestion service.

First create the two trust policy documents.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
# Trust policy for roles assumed by ECS tasks
cat > ecs-tasks-trust.json << 'EOF'
{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Principal": { "Service": "ecs-tasks.amazonaws.com" },
    "Action": "sts:AssumeRole"
  }]
}
EOF

# Trust policy for the role assumed by OpenSearch Ingestion
cat > osis-trust.json << 'EOF'
{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Principal": { "Service": "osis-pipelines.amazonaws.com" },
    "Action": "sts:AssumeRole"
  }]
}
EOF

1.1 CloudWatch Agent role — lets the agent publish traces and metrics to Application Signals.

1
2
3
4
5
6
7
aws iam create-role \
  --role-name CloudWatchAgentServerRole \
  --assume-role-policy-document file://ecs-tasks-trust.json

aws iam attach-role-policy \
  --role-name CloudWatchAgentServerRole \
  --policy-arn arn:aws:iam::aws:policy/CloudWatchAgentServerPolicy

1.2 OSIS pipeline role — lets the ingestion pipeline write to your OpenSearch domain.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
aws iam create-role \
  --role-name OSISPipelineRole \
  --assume-role-policy-document file://osis-trust.json

cat > osis-permissions.json << 'EOF'
{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Action": ["es:DescribeDomain", "es:ESHttp*"],
    "Resource": "arn:aws:es:REGION:ACCOUNT:domain/observability-logs/*"
  }]
}
EOF

aws iam put-role-policy \
  --role-name OSISPipelineRole \
  --policy-name OpenSearchAccess \
  --policy-document file://osis-permissions.json

1.3 Fluent Bit ingestion role — lets the log sidecar push records into OSIS.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
aws iam create-role \
  --role-name FluentBitIngestionRole \
  --assume-role-policy-document file://ecs-tasks-trust.json

cat > fluentbit-permissions.json << 'EOF'
{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Action": "osis:Ingest",
    "Resource": "arn:aws:osis:REGION:ACCOUNT:pipeline/*"
  }]
}
EOF

aws iam put-role-policy \
  --role-name FluentBitIngestionRole \
  --policy-name OSISIngest \
  --policy-document file://fluentbit-permissions.json

1.4 ECS task role — the application task assumes this, and it in turn can assume the Fluent Bit role for SigV4 signing.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
aws iam create-role \
  --role-name ECSTaskRole \
  --assume-role-policy-document file://ecs-tasks-trust.json

cat > ecs-task-permissions.json << 'EOF'
{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Action": "sts:AssumeRole",
    "Resource": "arn:aws:iam::ACCOUNT:role/FluentBitIngestionRole"
  }]
}
EOF

aws iam put-role-policy \
  --role-name ECSTaskRole \
  --policy-name AssumeFluentBitRole \
  --policy-document file://ecs-task-permissions.json

This guide also uses the standard ecsTaskExecutionRole for pulling images and writing to CloudWatch Logs. If you don't already have it, create it with the ecs-tasks-trust.json trust policy and attach the AWS managed AmazonECSTaskExecutionRolePolicy.

Step 2: Create a Cloud Map namespace

Service Connect uses Cloud Map for discovery. Create the namespace, wait for the operation to finish, then capture its ARN.

1
2
3
4
5
6
7
8
9
10
11
12
13
VPC_ID=$(aws ec2 describe-vpcs \
  --filters "Name=isDefault,Values=true" \
  --query 'Vpcs[0].VpcId' --output text)

aws servicediscovery create-private-dns-namespace \
  --name cloudwatch-application-signal \
  --vpc $VPC_ID \
  --description "Service Connect namespace for CloudWatch Agent"

# After the operation reports SUCCESS, capture the namespace ARN
NAMESPACE_ARN=$(aws servicediscovery list-namespaces \
  --query 'Namespaces[?Name==`cloudwatch-application-signal`].Arn' \
  --output text)

Step 3: Deploy the OpenSearch domain

Create an access policy that grants the OSIS role access to the domain, then create the domain. Generate the master password and store it in AWS Secrets Manager rather than hardcoding it.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
cat > opensearch-access-policy.json << 'EOF'
{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Principal": { "AWS": "arn:aws:iam::ACCOUNT:role/OSISPipelineRole" },
    "Action": "es:*",
    "Resource": "arn:aws:es:REGION:ACCOUNT:domain/observability-logs/*"
  }]
}
EOF

# Store the master credentials in Secrets Manager
aws secretsmanager create-secret \
  --name observability-opensearch-admin \
  --secret-string '{"username":"admin","password":"<choose-a-strong-password>"}'

aws opensearch create-domain \
  --domain-name observability-logs \
  --engine-version "OpenSearch_2.11" \
  --cluster-config InstanceType=r6g.large.search,InstanceCount=2,ZoneAwarenessEnabled=true,ZoneAwarenessConfig={AvailabilityZoneCount=2} \
  --ebs-options EBSEnabled=true,VolumeType=gp3,VolumeSize=100,Iops=3000 \
  --node-to-node-encryption-options Enabled=true \
  --encryption-at-rest-options Enabled=true \
  --domain-endpoint-options EnforceHTTPS=true,TLSSecurityPolicy=Policy-Min-TLS-1-2-2019-07 \
  --advanced-security-options Enabled=true,InternalUserDatabaseEnabled=true,MasterUserOptions={MasterUserName=admin,MasterUserPassword=<same-password>} \
  --access-policies file://opensearch-access-policy.json

Domain creation takes 15-20 minutes. When DomainStatus.Processing returns false, capture the endpoint.

1
2
3
OPENSEARCH_ENDPOINT=$(aws opensearch describe-domain \
  --domain-name observability-logs \
  --query 'DomainStatus.Endpoint' --output text)

Step 4: Create the OpenSearch Ingestion pipeline

OSIS parses incoming JSON and routes each record to a service-specific index. Centralizing this logic means you update parsing once instead of editing every Fluent Bit config. Save the configuration, substituting your real endpoint, account, and region.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
version: "2"
log-ingestion-pipeline:
  source:
    http:
      path: "/fluentbit/logs"
      max_request_length: "10mb"
  processor:
    - parse_json:
        source: "log"
        tags_on_failure: ["_jsonparsefailure"]
    - date:
        match:
          - key: "timestamp"
            patterns: ["yyyy-MM-dd'T'HH:mm:ss.SSSSSS'Z'"]
        destination: "@timestamp"
    - delete_entries:
        with_keys: ["log", "date"]
  sink:
    - opensearch:
        hosts:
          - "https://OPENSEARCH_ENDPOINT"
        aws:
          region: "ap-south-1"
          sts_role_arn: "arn:aws:iam::ACCOUNT:role/OSISPipelineRole"
        index: "logs-${service_name}-%{yyyy.MM.dd}"

Deploy the pipeline, then capture its ingest endpoint once the status is ACTIVE.

1
2
3
4
5
6
7
8
aws osis create-pipeline \
  --pipeline-name log-ingestion-pipeline \
  --min-units 2 --max-units 4 \
  --pipeline-configuration-body "file://osis-pipeline.yaml"

OSIS_ENDPOINT=$(aws osis get-pipeline \
  --pipeline-name log-ingestion-pipeline \
  --query 'Pipeline.IngestEndpointUrls[0]' --output text)

Step 5: Deploy the CloudWatch Agent in replica mode

Register the agent task definition. The CW_CONFIG_CONTENT variable turns on Application Signals collection, and the two port mappings expose the OTLP (4316) and sampling (2000) endpoints that applications will reach over Service Connect.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
cat > cwagent-task-definition.json << 'EOF'
{
  "family": "cloudwatch-agent-replica",
  "networkMode": "awsvpc",
  "requiresCompatibilities": ["FARGATE"],
  "cpu": "256",
  "memory": "512",
  "taskRoleArn": "arn:aws:iam::ACCOUNT:role/CloudWatchAgentServerRole",
  "executionRoleArn": "arn:aws:iam::ACCOUNT:role/ecsTaskExecutionRole",
  "containerDefinitions": [{
    "name": "cloudwatch-agent",
    "image": "public.ecr.aws/cloudwatch-agent/cloudwatch-agent:latest",
    "essential": true,
    "environment": [{
      "name": "CW_CONFIG_CONTENT",
      "value": "{\"traces\":{\"traces_collected\":{\"application_signals\":{}}},\"logs\":{\"metrics_collected\":{\"application_signals\":{}}}}"
    }],
    "portMappings": [
      { "name": "cwagent-4316", "containerPort": 4316, "protocol": "tcp", "appProtocol": "http" },
      { "name": "cwagent-2000", "containerPort": 2000, "protocol": "tcp", "appProtocol": "http" }
    ],
    "logConfiguration": {
      "logDriver": "awslogs",
      "options": {
        "awslogs-group": "/ecs/cloudwatch-agent-replica",
        "awslogs-create-group": "true",
        "awslogs-region": "ap-south-1",
        "awslogs-stream-prefix": "cwagent"
      }
    }
  }]
}
EOF

aws ecs register-task-definition --cli-input-json file://cwagent-task-definition.json

Define the Service Connect configuration that maps the ports to DNS aliases.

1
2
3
4
5
6
7
8
9
10
11
12
cat > service-connect-config.json << EOF
{
  "enabled": true,
  "namespace": "$NAMESPACE_ARN",
  "services": [
    { "portName": "cwagent-4316", "discoveryName": "cwagent-4316-http",
      "clientAliases": [{ "port": 4316, "dnsName": "cwagent-4316-http" }] },
    { "portName": "cwagent-2000", "discoveryName": "cwagent-2000-http",
      "clientAliases": [{ "port": 2000, "dnsName": "cwagent-2000-http" }] }
  ]
}
EOF

Create a security group that allows the application ports from your VPC CIDR only, then create the service.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
SUBNET_IDS=$(aws ec2 describe-subnets \
  --filters "Name=vpc-id,Values=$VPC_ID" \
  --query 'Subnets[0:2].SubnetId' --output text | tr '\t' ',')

SG_ID=$(aws ec2 create-security-group \
  --group-name cwagent-replica-sg \
  --description "CloudWatch Agent Replica" \
  --vpc-id $VPC_ID --query 'GroupId' --output text)

for port in 4316 2000; do
  aws ec2 authorize-security-group-ingress \
    --group-id $SG_ID --protocol tcp --port $port --cidr 10.0.0.0/16
done

aws ecs create-service \
  --cluster YOUR_CLUSTER_NAME \
  --service-name cloudwatch-agent-replica \
  --task-definition cloudwatch-agent-replica \
  --desired-count 1 --launch-type FARGATE \
  --network-configuration "awsvpcConfiguration={subnets=[$SUBNET_IDS],securityGroups=[$SG_ID],assignPublicIp=DISABLED}" \
  --service-connect-configuration file://service-connect-config.json

Step 6: Configure Fluent Bit and upload it to S3

The Fluent Bit sidecar reads the application's JSON logs and forwards them to OSIS with SigV4 authentication. Store its config in S3 so tasks can load it at startup. Replace OSIS_ENDPOINT with the host portion of the endpoint from Step 4 (no https://).

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
cat > fluent-bit-osis.conf << 'EOF'
[SERVICE]
    Log_Level info

[INPUT]
    Name              tail
    Path              /app/logs/*.log
    Refresh_Interval  5
    Read_from_Head    true
    Skip_Long_Lines   On

[FILTER]
    Name   modify
    Match  *
    Add    service_name demo-service

[OUTPUT]
    Name          http
    Match         *
    Host          OSIS_ENDPOINT
    Port          443
    URI           /fluentbit/logs
    Format        json
    tls           On
    AWS_Auth      On
    AWS_Region    ap-south-1
    AWS_Role_ARN  arn:aws:iam::ACCOUNT:role/FluentBitIngestionRole
    AWS_Service   osis
    Header        Content-Type application/json
EOF

BUCKET="my-observability-configs-$(aws sts get-caller-identity --query Account --output text)"
aws s3 mb s3://$BUCKET
aws s3 cp fluent-bit-osis.conf s3://$BUCKET/fluentbit/fluent-bit-osis.conf

Step 7: Deploy the instrumented application

This task definition has three containers: an init container that copies the OpenTelemetry SDK into a shared volume, the app container that runs your code with auto-instrumentation, and a log_router (Fluent Bit) container that ships logs to OSIS. Two volumes are shared: one for the SDK, one for log files.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
cat > app-task-definition.json << 'EOF'
{
  "family": "demo-app-observability",
  "networkMode": "awsvpc",
  "requiresCompatibilities": ["FARGATE"],
  "cpu": "1024",
  "memory": "2048",
  "taskRoleArn": "arn:aws:iam::ACCOUNT:role/ECSTaskRole",
  "executionRoleArn": "arn:aws:iam::ACCOUNT:role/ecsTaskExecutionRole",
  "containerDefinitions": [
    {
      "name": "init",
      "image": "public.ecr.aws/aws-observability/adot-autoinstrumentation-python:v0.12.0",
      "essential": false,
      "command": ["cp", "-a", "/autoinstrumentation/.", "/otel-auto-instrumentation-python"],
      "mountPoints": [{
        "sourceVolume": "opentelemetry-auto-instrumentation-python",
        "containerPath": "/otel-auto-instrumentation-python"
      }]
    },
    {
      "name": "app",
      "image": "YOUR_APP_IMAGE:latest",
      "essential": true,
      "environment": [
        {"name": "SERVICE_NAME", "value": "demo-service"},
        {"name": "OTEL_RESOURCE_ATTRIBUTES", "value": "service.name=demo-service"},
        {"name": "PYTHONPATH", "value": "/otel-auto-instrumentation-python/opentelemetry/instrumentation/auto_instrumentation:/otel-auto-instrumentation-python"},
        {"name": "OTEL_TRACES_EXPORTER", "value": "otlp"},
        {"name": "OTEL_EXPORTER_OTLP_PROTOCOL", "value": "http/protobuf"},
        {"name": "OTEL_PYTHON_DISTRO", "value": "aws_distro"},
        {"name": "OTEL_PYTHON_CONFIGURATOR", "value": "aws_configurator"},
        {"name": "OTEL_EXPORTER_OTLP_TRACES_ENDPOINT", "value": "http://cwagent-4316-http:4316/v1/traces"},
        {"name": "OTEL_AWS_APPLICATION_SIGNALS_ENABLED", "value": "true"},
        {"name": "OTEL_TRACES_SAMPLER_ARG", "value": "endpoint=http://cwagent-2000-http:2000"}
      ],
      "mountPoints": [
        { "sourceVolume": "opentelemetry-auto-instrumentation-python", "containerPath": "/otel-auto-instrumentation-python" },
        { "sourceVolume": "app-logs", "containerPath": "/app/logs" }
      ],
      "dependsOn": [{ "containerName": "init", "condition": "SUCCESS" }],
      "logConfiguration": { "logDriver": "awsfirelens" }
    },
    {
      "name": "log_router",
      "image": "public.ecr.aws/aws-observability/aws-for-fluent-bit:stable",
      "essential": false,
      "mountPoints": [{ "sourceVolume": "app-logs", "containerPath": "/app/logs", "readOnly": true }],
      "firelensConfiguration": {
        "type": "fluentbit",
        "options": {
          "config-file-type": "s3",
          "config-file-value": "arn:aws:s3:::my-observability-configs-ACCOUNT/fluentbit/fluent-bit-osis.conf",
          "enable-ecs-log-metadata": "false"
        }
      },
      "logConfiguration": {
        "logDriver": "awslogs",
        "options": {
          "awslogs-group": "/ecs/fluent-bit",
          "awslogs-create-group": "true",
          "awslogs-region": "ap-south-1",
          "awslogs-stream-prefix": "fluent-bit"
        }
      }
    }
  ],
  "volumes": [
    { "name": "opentelemetry-auto-instrumentation-python" },
    { "name": "app-logs" }
  ]
}
EOF

aws ecs register-task-definition --cli-input-json file://app-task-definition.json

A few things make this work, all without touching application code:

The init container runs once and copies the SDK to the shared opentelemetry-auto-instrumentation-python volume; dependsOn ensures the app starts only after it succeeds.
PYTHONPATH points the interpreter at that SDK, and OTEL_EXPORTER_OTLP_TRACES_ENDPOINT sends traces to the shared agent over Service Connect DNS.
OTEL_AWS_APPLICATION_SIGNALS_ENABLED turns on the Application Signals metrics and service map.

Create the application service with Service Connect enabled so it can resolve the agent's DNS name.

1
2
3
4
5
6
7
aws ecs create-service \
  --cluster YOUR_CLUSTER_NAME \
  --service-name demo-app-service \
  --task-definition demo-app-observability \
  --desired-count 2 --launch-type FARGATE \
  --network-configuration "awsvpcConfiguration={subnets=[$SUBNET_IDS],securityGroups=[$SG_ID],assignPublicIp=ENABLED}" \
  --service-connect-configuration "{\"enabled\":true,\"namespace\":\"$NAMESPACE_ARN\"}"

For the app itself, write structured JSON logs to /app/logs/. Structured fields (service, level, timestamp) are what let OpenSearch index and aggregate the logs cleanly:

1
2
3
4
5
6
7
8
9
class JSONFormatter(logging.Formatter):
    def format(self, record):
        log_data = {
            "timestamp": datetime.utcnow().isoformat() + "Z",
            "level": record.levelname,
            "service": os.getenv('SERVICE_NAME', 'unknown'),
            "message": record.getMessage(),
        }
        return json.dumps(log_data)

Step 8: Validate both pipelines

Generate some traffic against your application, then confirm each pipeline independently.

On the trace side, open the CloudWatch Application Signals console. Within a few minutes your service appears on the service map with request volume, latency, and error metrics—proof that traces are flowing through the shared replica-mode agent.

On the log side, pull the OpenSearch credentials from Secrets Manager and check that per-service indexes are being created:

1
2
3
4
5
PASSWORD=$(aws secretsmanager get-secret-value \
  --secret-id observability-opensearch-admin \
  --query SecretString --output text | jq -r .password)

curl -u admin:$PASSWORD "https://$OPENSEARCH_ENDPOINT/_cat/indices/logs-*?v"

You should see an index like logs-demo-service-2026.06.23 with a growing document count. Query a few recent documents to confirm parsing worked:

1
2
3
4
curl -u admin:$PASSWORD \
  -X GET "https://$OPENSEARCH_ENDPOINT/logs-*/_search?pretty" \
  -H 'Content-Type: application/json' \
  -d '{ "query": { "match_all": {} }, "size": 5, "sort": [{ "@timestamp": "desc" }] }'

If logs don't show up, check the Fluent Bit container logs (/ecs/fluent-bit) and the OSIS pipeline logs first—the most common causes are an unreachable OSIS endpoint in the Fluent Bit config or the task role missing permission to assume FluentBitIngestionRole.

Managing log retention

Use an OpenSearch Index State Management (ISM) policy to roll indexes through hot and warm states and delete them automatically once they age out. Apply it through an index template that matches logs-* so every new service index inherits the lifecycle without manual setup.

Wrapping up

If OpenSearch is already your logging destination, this architecture gives you tracing and an efficient log pipeline without disturbing that choice. CloudWatch Agent replica mode means one shared agent serves the whole cluster instead of a sidecar in every task, and ADOT auto-instrumentation adds Application Signals tracing with no code changes. OSIS replaces a self-managed Logstash fleet with a managed pipeline that parses, buffers, routes, and auto-scales on its own—so you go from a handful of services to a hundred without standing up more ingestion infrastructure.

Be clear with your team about the boundary: traces live in Application Signals and logs live in OpenSearch, and this design does not correlate the two. You keep OpenSearch's search and analytics; you give up native in-console trace-to-log linking, which would require sending logs to CloudWatch Logs instead.

Start with one or two non-critical services to validate the pattern in your environment, then expand. Build reusable task definition templates so each new service drops into the platform with the same shared agent and managed ingestion already wired in.

Resources

# monitoring-observability # observability # amazon-ecs # aws-cloudwatch # amazon-opensearch

Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.

Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article

Builder Center

Scaling Unified Observability for ECS

Pawan

The problem

Architecture overview

Prerequisites

Step 1: Create the IAM roles and trust policies

Step 2: Create a Cloud Map namespace

Step 3: Deploy the OpenSearch domain

Step 4: Create the OpenSearch Ingestion pipeline

Step 5: Deploy the CloudWatch Agent in replica mode

Step 6: Configure Fluent Bit and upload it to S3

Step 7: Deploy the instrumented application

Step 8: Validate both pipelines

Managing log retention

Wrapping up

Resources