Connect AWS DevOps Agent to Prometheus and OpenSearch Using Grafana

Learn how to connect AWS DevOps Agent to self-managed Prometheus and Amazon OpenSearch Service using Grafana as a single, private telemetry gateway — so the agent correlates metrics and logs in one investigation, with no public endpoints.

AWS-Hareesh

Published Jun 17, 2026

Last modified Jun 17, 2026

Reading time: ~12 minutes

In this post we connect AWS DevOps Agent to a self-managed Prometheus and an Amazon OpenSearch Service domain — using self-managed Grafana as a single telemetry gateway — so the agent pulls both metrics and logs during an investigation. Everything stays private: no public endpoints, all traffic over Amazon VPC Lattice .

Cost: ~$12–14/day in us-east-1 (EKS, two t3.medium nodes, a single-node
t3.small.search OpenSearch domain, an internal ALB, a NAT gateway, and the VPC
Lattice resource gateway). Clean up when done.

Two ways to connect Grafana

DevOps Agent connects to Grafana either through the built-in Grafana capability
(AWS hosts the Grafana MCP server for you)
or a self-hosted MCP server (you run grafana/mcp-grafana). At the time of
writing, the built-in capability queries Prometheus reliably but rejects Amazon
OpenSearch datasources (type grafana-opensearch-datasource) with a datasource-type
error. The self-hosted server queries both, so we use it here — and show how to switch
to the built-in capability once its OpenSearch support lands. Both share the same EKS,
Grafana, ALB, and VPC Lattice setup.

Architecture

DevOps Agent reaches an internal ALB over VPC Lattice. The ALB host-routes to two
backends: Grafana (OSS) and grafana/mcp-grafana, which exposes Grafana's
datasources as MCP tools. Behind Grafana sit Prometheus (kube-prometheus-stack)
and Amazon OpenSearch Service (logs shipped by Fluent Bit via IRSA/SigV4). A
sample 3-tier app (frontend → orders-service → payment-service) plus a CPU load
generator gives us something to investigate.

The win: DevOps Agent talks to one endpoint and transparently reaches every datasource
behind it — no per-backend wiring, and your team keeps the dashboards they already use.

Prerequisites

An AWS account with permissions for EKS, OpenSearch, VPC Lattice, ELB, ACM, IAM
An AWS DevOps Agent Agent Space
aws CLI v2.34+, kubectl, helm v3, and eksctl or Terraform
An EKS cluster (1.30+) with the OIDC provider enabled for IRSA

VPC Lattice doesn't support a few AZ IDs (e.g. use1-az3). Pick subnets in supported
AZs: aws ec2 describe-subnets --subnet-ids <ids> --query 'Subnets[*].[SubnetId,AvailabilityZoneId]'

Tag everything Project: devops-agent-demo for easy cleanup.

Step 1 — Prometheus

Install open source Prometheus, disabling the chart's bundled Grafana (we run our own):

1
2
3
4
5
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm upgrade --install prometheus prometheus-community/kube-prometheus-stack \
  --namespace observability --create-namespace \
  --set grafana.enabled=false \
  --set prometheus.prometheusSpec.serviceMonitorSelectorNilUsesHelmValues=false

Step 2 — Amazon OpenSearch Service

A single-node, in-VPC domain with IAM fine-grained access control:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
resource "aws_opensearch_domain" "this" {
  domain_name    = "devops-agent-demo"
  engine_version = "OpenSearch_2.15"
  cluster_config { instance_type = "t3.small.search", instance_count = 1 }
  ebs_options    { ebs_enabled = true, volume_size = 10, volume_type = "gp3" }
  vpc_options       { subnet_ids = [var.private_subnet_ids[0]], security_group_ids = [aws_security_group.opensearch.id] }
  encrypt_at_rest         { enabled = true }
  node_to_node_encryption { enabled = true }
  domain_endpoint_options { enforce_https = true, tls_security_policy = "Policy-Min-TLS-1-2-2019-07" }
  advanced_security_options {
    enabled                        = true
    internal_user_database_enabled = false
    master_user_options { master_user_arn = aws_iam_role.opensearch_master.arn }
  }
  tags = { Project = "devops-agent-demo" }
}

The master user is an IAM role; map the Fluent Bit and Grafana roles to OpenSearch
backend roles in the next steps.

Step 3 — Ship logs with Fluent Bit (IRSA + SigV4)

Fluent Bit writes to OpenSearch using SigV4 via an IRSA role (es:ESHttpPost,
es:ESHttpPut on the domain). Output config:

1
2
3
4
5
6
7
8
9
10
11
[OUTPUT]
    Name                opensearch
    Match               kube.*
    Host                <OPENSEARCH_VPC_ENDPOINT>
    Port                443
    TLS                 On
    AWS_Auth            On
    AWS_Region          us-east-1
    Logstash_Format     On
    Logstash_Prefix     app-logs
    Suppress_Type_Name  On

1
2
helm repo add fluent https://fluent.github.io/helm-charts
helm upgrade --install fluent-bit fluent/fluent-bit -n observability -f fluent-bit.values.yaml

With fine-grained access control, map the Fluent Bit role ARN to a writable OpenSearch
role (run the role-mapping API call from an in-cluster pod, since the domain is
VPC-only). Confirm logs land in app-logs-YYYY.MM.DD.

Step 4 — Self-managed Grafana with both datasources

Install OSS Grafana with an IRSA service account, the OpenSearch plugin, both
datasources, and an internal ALB. Key values:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
serviceAccount:
  annotations:
    eks.amazonaws.com/role-arn: "arn:aws:iam::<ACCOUNT_ID>:role/devops-agent-demo-grafana"
plugins:
  - grafana-opensearch-datasource
grafana.ini:
  auth: { sigv4_auth_enabled: true }          # (1) required for OpenSearch signing
  aws:  { allowed_auth_providers: "default,workspace-iam-role,ec2_iam_role" }
ingress:
  enabled: true
  ingressClassName: alb
  annotations:
    alb.ingress.kubernetes.io/scheme: internal
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS":443}]'
    alb.ingress.kubernetes.io/certificate-arn: "<ACM_CERT_ARN>"
    alb.ingress.kubernetes.io/group.name: doa-demo
  hosts: ["grafana.internal.devops-agent-demo"]
datasources:
  datasources.yaml:
    apiVersion: 1
    datasources:
      - name: Prometheus
        type: prometheus
        url: http://prometheus-kube-prometheus-prometheus.observability.svc.cluster.local:9090
      - name: OpenSearch
        type: grafana-opensearch-datasource
        url: "https://<OPENSEARCH_VPC_ENDPOINT>"
        jsonData:
          database: "app-logs"                # (2) point at an index/alias that exists
          flavor: opensearch
          version: "2.15.0"
          timeField: "@timestamp"
          sigV4Auth: true
          sigV4AuthType: default              # uses the pod's IRSA role
          sigV4Region: us-east-1

Two gotchas. (1) SigV4 must be on in both the datasource (sigV4Auth: true)
and core Grafana (grafana.ini → auth.sigv4_auth_enabled); set the latter via
grafana.ini, not env vars. (2) The OpenSearch datasource "Save & test" can report
"Index not found" even when queries work — validate with a real query against an
index/alias that exists rather than trusting the health check.

Step 5 — Deploy `grafana/mcp-grafana`

This gives DevOps Agent its query tools. Run it in streamable-HTTP mode against
in-cluster Grafana, with the Prometheus and Elasticsearch/OpenSearch tools enabled
(use a Grafana service-account token, Viewer role):

1
2
3
4
5
6
7
8
9
10
containers:
  - name: mcp-grafana
    image: grafana/mcp-grafana:latest
    args: ["-t","streamable-http","--address","0.0.0.0:8000",
           "--enabled-tools","search,datasource,prometheus,elasticsearch,alerting",
           "--disable-write"]
    env:
      - { name: GRAFANA_URL, value: "http://grafana.monitoring.svc.cluster.local" }
      - name: GRAFANA_SERVICE_ACCOUNT_TOKEN
        valueFrom: { secretKeyRef: { name: mcp-grafana-token, key: token } }

Its query_elasticsearch tool (enabled by the elasticsearch category) proxies
through Grafana's datasource API, so it handles grafana-opensearch-datasource
datasources via the same SigV4/IRSA path. Expose it through the same ALB on a
second hostname (mcp-grafana.internal.devops-agent-demo) using the shared
group.name: doa-demo.

Security: This demo's MCP endpoint doesn't authenticate inbound requests — the
private connection (no public access) is the boundary, and the Grafana token lives
in the pod. In production, front the MCP endpoint with authentication and scope the
service account to the minimum datasources.

Step 6 — Private connection over VPC Lattice

Both Grafana and the MCP server sit behind one internal ALB (shared group, two host
rules). DevOps Agent reaches the ALB over VPC Lattice:

1
2
3
4
5
6
7
aws devops-agent create-private-connection \
  --name doa-grafana-conn \
  --mode '{"serviceManaged":{"hostAddress":"<INTERNAL_ALB_DNS>","vpcId":"vpc-...",
    "subnetIds":["subnet-a","subnet-b"],"portRanges":["443"],"ipAddressType":"IPV4",
    "certificate":"<PEM_PUBLIC_KEY>"}}' \
  --tags Project=devops-agent-demo --region us-east-1
# wait for status ACTIVE (a few minutes)

Routing: The host address (ALB DNS) is what Lattice resolves for routing.
The endpoint URL you give at registration is the HTTP Host/TLS SNI — not DNS.
So one connection can back multiple integrations, each with its own hostname, and the
ALB host-routes them. That's how we serve both Grafana and the MCP server through one
connection. (Newer CLI/console versions nest the gateway config under
resourceGatewayConfig.create.)

Ensure the ALB security group allows inbound 443 from the VPC CIDR and has a host rule
for each endpoint hostname.

Step 7 — Register the MCP server (use the console)

In the DevOps Agent console →
Capability Providers → MCP Server → Register:

Endpoint URL: https://mcp-grafana.internal.devops-agent-demo/mcp
Private connection: doa-grafana-conn
Auth: the console requires OAuth, API key, or SigV4. Our endpoint ignores inbound
auth, so use API key with a placeholder (header x-api-key, value unused-demo-key).

Associate it with your Agent Space and enable list_datasources, query_prometheus,
and query_elasticsearch. Confirm in a fresh chat:

1
List the tools available from the grafana-mcp MCP server.

Step 8 — Validate end to end

Ask the agent to query both datasources:

Prometheus — query_prometheus for count(kube_pod_info):

1
{"data": [{"metric": {}, "value": [..., "33"]}]}

OpenSearch — query_elasticsearch on index app-logs:

1
2
3
{"_index":"app-logs-...","_source":{
  "kubernetes":{"namespace_name":"demo-apps","container_name":"payment-service"},
  "log":"INFO payment-service Payment processed: order=41dca928 amount=$496"}}

The agent now introspects metrics and logs through a single Grafana gateway —
during an incident it correlates a CPU spike (Prometheus) with the application's
behavior (OpenSearch) in one investigation.

Optional — auto-trigger from a Grafana alert. Create an alert on pod CPU > 80% and
a webhook contact point posting to your Agent Space webhook. DevOps Agent expects a
specific incident schema, so reshape Grafana's payload with a notification template:

1
2
3
4
5
6
7
8
{{ define "devops-agent-payload" }}
{"eventType":"incident",
 "incidentId":"{{ (index .Alerts 0).Labels.alertname }}-{{ (index .Alerts 0).Fingerprint }}",
 "action":"{{ if eq .Status "resolved" }}resolved{{ else }}created{{ end }}",
 "priority":"HIGH","title":"{{ (index .Alerts 0).Labels.alertname }}",
 "description":"{{ (index .Alerts 0).Annotations.summary }}",
 "service":"{{ (index .Alerts 0).Labels.job }}","timestamp":"{{ (index .Alerts 0).StartsAt }}"}
{{ end }}

Set the contact point's Custom Payload to {{ template "devops-agent-payload" . }}.
(Custom Payload needs Grafana 9.0+; on some 11.x OSS builds it's behind a feature
toggle. Verify the webhook works first with a curl of a correctly-shaped payload.)

The cleaner approach: the built-in capability

Once you no longer want to run your own MCP server, register the built-in Grafana
capability — same private connection plus a Grafana service-account token, nothing to
deploy:

1
2
3
4
5
6
aws devops-agent register-service --service mcpservergrafana \
  --private-connection-name doa-grafana-conn \
  --service-details '{"mcpservergrafana":{"name":"grafana",
    "endpoint":"https://grafana.internal.devops-agent-demo",
    "authorizationConfig":{"bearerToken":{"tokenName":"grafana-sa-token","tokenValue":"<glsa_TOKEN>"}}}}' \
  --region us-east-1

Current limitation: the built-in capability's query_elasticsearch rejects
grafana-opensearch-datasource datasources (datasource <UID> is of type grafana-opensearch-datasource, not elasticsearch). Until that's resolved, use the
self-hosted path above for OpenSearch logs; Prometheus works on both. Switch to the
built-in capability once OpenSearch support lands — the rest of your setup is
identical. See Connecting Grafana .

Cleanup

1
2
3
4
5
6
# Remove the capability (console: Capabilities → grafana-mcp → Remove), then:
aws devops-agent delete-private-connection --name doa-grafana-conn --region us-east-1
helm -n monitoring uninstall grafana
helm -n observability uninstall fluent-bit prometheus
kubectl delete ns monitoring observability demo-apps
terraform destroy -auto-approve

Takeaways

Grafana as a gateway keeps the integration simple: one endpoint, every datasource behind it.
SigV4 for OpenSearch needs enabling on both the datasource and core Grafana, with IRSA — no static keys.
VPC Lattice decouples DNS routing (host address) from HTTP Host/SNI (endpoint URL), so one ALB serves multiple integrations.
Register capabilities via the console for reliable MCP tool binding.
For OpenSearch today, self-host grafana/mcp-grafana; the built-in capability's OpenSearch support is still rolling out.

Learn more: Connecting Grafana ·
Connecting to privately hosted tools ·
Securely connect DevOps Agent to private services ·
grafana/mcp-grafana

# devops # agentic-ai # agents

Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.

Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article

Builder Center

Connect AWS DevOps Agent to Prometheus and OpenSearch Using Grafana

AWS-Hareesh

Two ways to connect Grafana

Architecture

Prerequisites

Step 1 — Prometheus

Step 2 — Amazon OpenSearch Service

Step 3 — Ship logs with Fluent Bit (IRSA + SigV4)

Step 4 — Self-managed Grafana with both datasources

Step 5 — Deploy grafana/mcp-grafana

Step 6 — Private connection over VPC Lattice

Step 7 — Register the MCP server (use the console)

Step 8 — Validate end to end

The cleaner approach: the built-in capability

Cleanup

Takeaways

Step 5 — Deploy `grafana/mcp-grafana`