LLMOps Evaluation: How I Got AgentCore Managed Evaluation Working on Container Runtimes

# requirements.txt
aws-opentelemetry-distro>=0.10.0

# Dockerfile CMD
CMD ["opentelemetry-instrument", "python", "main.py"]

{"Action": ["bedrock-agentcore:PutTelemetry", "bedrock-agentcore:PutSpans", "xray:PutTraceSegments"], "Resource": "*"}

AGENT_OBSERVABILITY_ENABLED=true
OTEL_TRACES_EXPORTER=otlp
OTEL_LOGS_EXPORTER=otlp
OTEL_PYTHON_DISTRO=aws_distro
OTEL_PYTHON_CONFIGURATOR=aws_configurator
OTEL_PROPAGATORS=baggage,xray,tracecontext
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
OTEL_RESOURCE_ATTRIBUTES=service.name=llmops_agent

{"Action": ["bedrock:InvokeModel"], "Resource": ["arn:aws:bedrock:us-east-1::foundation-model/*"]}

from opentelemetry import baggage, context

# In your HTTP handler, before invoking the agent:
ctx = baggage.set_baggage("session.id", session_id)
token = context.attach(ctx)
try:
    response = agent(prompt)
finally:
    context.detach(token)

✅ requirements.txt: aws-opentelemetry-distro>=0.10.0
✅ Dockerfile CMD: opentelemetry-instrument python main.py
✅ NO OTEL_PYTHON_LOGGING_AUTO_INSTRUMENTATION_ENABLED in Dockerfile
✅ Runtime env vars: AGENT_OBSERVABILITY_ENABLED=true + 8 OTEL vars
✅ IAM: bedrock-agentcore:PutTelemetry permission
✅ IAM: bedrock:InvokeModel permission (for custom evaluators)
✅ Code: baggage.set_baggage("session.id", session_id)
✅ Invoke with tool-triggering prompts for ToolUsage evaluator