AWS Builder Center

Building a Centralized Observability Dashboard with OpenTelemetry: Unifying Logs, Metrics, and Traces Across Microservices

A practical, Python-first guide for DevOps and platform teams building unified visibility into distributed systems.

Google cloud certified professional machine learning engineer

Why This Topic, and Why Now

Search any observability community today and you'll find the same pattern repeating: teams have dashboards for metrics, a separate tool for logs, and yet another for traces — three browser tabs open just to answer one question: "why is this request slow?"
OpenTelemetry (OTel) was built to close that gap. It gives engineering teams a single, vendor-neutral standard for collecting metrics, logs, and traces, and — critically — for correlating them. The real value isn't in adopting OTel instrumentation alone; it's in what you do with that unified telemetry stream once it reaches a centralized dashboard.
This article walks through the architecture, the Python instrumentation, and the operational payoff of building that centralized pipeline — the kind of system that turns a 45-minute incident investigation into a 5-minute one.



The Problem: Fragmented Signals, Fragmented Teams

Before centralizing anything, it's worth naming the actual pain:
  • Logs tell you what happened, but rarely where in a distributed call chain.
  • Metrics tell you something is wrong (latency spike, error rate climbing), but not why.
  • Traces tell you the path a request took, but are often siloed from the logs generated along that path.
When these three signals live in three different systems with three different query languages and no shared identifiers, engineers spend more time correlating tools than solving problems. This is the single biggest driver of alert fatigue and slow Mean Time to Resolution (MTTR) in microservice environments.



The Architecture: One Pipeline, Three Signal Types

A centralized OpenTelemetry pipeline typically follows this shape:
[ Services / Apps ]
│
(OTel SDK — auto + manual instrumentation)
│
▼
[ OTel Collector (Agent) ] ──► enriches with resource attributes
│
▼
[ OTel Collector (Gateway) ] ──► batches, filters, routes
│
┌────┼────┐
▼ ▼ ▼
Traces Metrics Logs
│ │ │
▼ ▼ ▼
[ Centralized Backend: Grafana / CloudWatch / Tempo / Loki / Prometheus ]
│
▼
[ Unified Dashboard — correlated by trace_id ]
The key architectural decision is the agent-to-gateway pattern: lightweight OTel agents run alongside each service (as a sidecar or DaemonSet), forwarding enriched telemetry to a centralized gateway collector, which handles batching, sampling, authentication, and export to your backend of choice. This keeps instrumentation cheap at the edge and centralizes the expensive operational logic.
The linchpin that makes correlation possible is the trace ID. If every log line and every metric exemplar carries the same trace ID as the span it belongs to, your dashboard can jump from "error rate spike" → "the exact trace" → "the exact log line" in one click.



Instrumenting a Python Service

Here's a minimal but production-shaped example of instrumenting a Flask service with OpenTelemetry, exporting traces, metrics, and correlated logs to a collector.
python
# requirements: opentelemetry-distro, opentelemetry-exporter-otlp,
# opentelemetry-instrumentation-flask, opentelemetry-instrumentation-logging

from flask import Flask
from opentelemetry import trace, metrics
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.metrics import MeterProvider
from opentelemetry.sdk.metrics.export import PeriodicExportingMetricReader
from opentelemetry.exporter.otlp.proto.grpc.metric_exporter import OTLPMetricExporter
from opentelemetry.instrumentation.flask import FlaskInstrumentor
from opentelemetry.instrumentation.logging import LoggingInstrumentor
import logging

# --- Tracing setup ---
trace.set_tracer_provider(TracerProvider())
tracer_provider = trace.get_tracer_provider()
tracer_provider.add_span_processor(
BatchSpanProcessor(OTLPSpanExporter(endpoint="otel-collector:4317", insecure=True))
)
tracer = trace.get_tracer(__name__)

# --- Metrics setup ---
metric_reader = PeriodicExportingMetricReader(
OTLPMetricExporter(endpoint="otel-collector:4317", insecure=True)
)
metrics.set_meter_provider(MeterProvider(metric_readers=[metric_reader]))
meter = metrics.get_meter(__name__)
request_counter = meter.create_counter(
"http_requests_total", description="Total HTTP requests processed"
)

# --- Correlated logging: injects trace_id/span_id into every log record ---
LoggingInstrumentor().instrument(set_logging_format=True)
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)

app = Flask(__name__)
FlaskInstrumentor().instrument_app(app) # auto-creates spans per request

@app.route("/checkout")
def checkout():
request_counter.add(1, {"route": "/checkout"})
with tracer.start_as_current_span("validate-cart") as span:
span.set_attribute("cart.items", 3)
logger.info("Validating cart before checkout") # trace_id auto-attached
# business logic here
return {"status": "ok"}

if __name__ == "__main__":
app.run(port=5000)
Two details matter here more than the boilerplate:
  1.  LoggingInstrumentor().instrument()  is what actually solves the fragmentation problem — it injects the active  trace_id  and  span_id  into every log record automatically, so your log backend and trace backend can be joined on a shared key.
  1. Custom span attributes (like  cart.items ) turn generic traces into debuggable context — this is where "observability" becomes more than "monitoring with extra steps."



The Collector Configuration

The OTel Collector is the traffic controller of this whole system. A gateway configuration that fans telemetry out to a centralized backend looks like this:
yaml
receivers:
otlp:
protocols:
grpc:
http:

processors:
batch:
resource:
attributes:
- key: deployment.environment
value: production
action: upsert

exporters:
otlphttp/tempo:
endpoint: http://tempo:4318
prometheusremotewrite:
endpoint: http://prometheus:9090/api/v1/write
loki:
endpoint: http://loki:3100/loki/api/v1/push

service:
pipelines:
traces:
receivers: [otlp]
processors: [batch, resource]
exporters: [otlphttp/tempo]
metrics:
receivers: [otlp]
processors: [batch, resource]
exporters: [prometheusremotewrite]
logs:
receivers: [otlp]
processors: [batch, resource]
exporters: [loki]
Running this on AWS, the same pattern applies with native integrations: OTLP metrics can be exported directly to Amazon CloudWatch, traces to AWS X-Ray or Grafana Tempo via Amazon Managed Grafana, and long-term metric storage handled by Amazon Managed Service for Prometheus. The AWS Distro for OpenTelemetry (ADOT) provides a secure, supported build of the same collector, which matters for teams that need AWS Support coverage on their observability infrastructure itself.



The Payoff: One Dashboard, One Query

Once logs, metrics, and traces share a  trace_id , a single Grafana (or Managed Grafana) dashboard can support a workflow like this:
  1. An SLO panel shows the checkout service's p99 latency crossing threshold.
  1. Clicking the spike jumps to the exact traces active in that time window.
  1. Clicking a slow trace surfaces the exact span where time was lost.
  1. Clicking that span pulls the exact log lines emitted during it — no separate log search, no manual timestamp correlation.
This is the practical difference between having dashboards and having observability. The former shows you data; the latter shortens the distance between symptom and root cause.



Common Pitfalls to Avoid

  • Instrumenting everything, sampling nothing. High-cardinality, 100%-sampled tracing in production gets expensive fast. Use tail-based or probabilistic sampling at the collector level once volume grows.
  • Skipping resource attributes. Without consistent  service.name ,  deployment.environment , and  service.version  tags, correlation across services breaks down exactly when you need it most — during an incident.
  • Treating the collector as a black box. The gateway collector is itself a production service. Monitor its own health metrics (queue length, export failures, memory) — a silently failing collector means silently missing telemetry.
  • Skipping the logging bridge. Teams often instrument traces and metrics but forget  LoggingInstrumentor , leaving logs as the one signal still disconnected from the rest.



Closing Thought

Centralized OpenTelemetry dashboards aren't a nice-to-have observability upgrade — they're what turns three disconnected tools into one coherent debugging workflow. The technical lift is real, but it's front-loaded: once trace IDs propagate cleanly through your logs and metrics, every future incident gets faster to resolve, not just the next one.
For teams building this out on AWS, the ecosystem (ADOT, CloudWatch native OTLP support, Managed Grafana, Managed Prometheus) has matured to the point where you can build this pipeline without stitching together unsupported open-source components — while still keeping the vendor-neutral instrumentation layer that makes OpenTelemetry worth adopting in the first place.
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article