From kubectl to agentic ops: A practical guide to operating Kubernetes with k8sgpt and HolmesGPT
Learn how to move from manual kubectl troubleshooting to LLM-assisted operations on Amazon EKS. This article shows how k8sgpt and HolmesGPT work together, one detecting misconfigurations deterministically, the other investigating root causes through an agentic loop.
Your team runs a microservices stack on Kubernetes. One night, the payment-service stops receiving traffic. Your phone rings at 2am. You SSH into the bastion, run kubectl describe, scroll through events, open Grafana in another tab, query Loki for the right time window, check the latest deployment in ArgoCD, and start correlating. Twenty minutes in, you discover it's not one issue but three compounding: a typo in the Ingress, a wrong label on the Service selector, and a bad image tag on the Pods. No single resource told you that, you had to walk the path yourself.
This is how a large share of teams still operate Kubernetes today. The stack itself hasn't changed much in five years. What is changing fast is the brain layer above it.
Two Cloud Native Computing Foundation ( CNCF ) Sandbox projects illustrate that shift: k8sgpt (accepted in December 2023), a cluster scanner that detects misconfigurations, and HolmesGPT (accepted in October 2025), an agentic investigator that performs root-cause analysis. While often mentioned in the same breath, they represent two distinct paradigms of how an LLM can sit on top of a Kubernetes cluster. Understanding the difference is the entry point to making sane architectural choices in this space.
This article walks through both projects, their patterns, how to deploy them on Amazon EKS with Amazon Bedrock as inference backend, and how to embed them into a pipeline where detection feeds investigation automatically. At the end, you should have a mental model you can use to evaluate this category of tooling.
Note: The patterns discussed in this article incur AWS charges, including Amazon EKS cluster hours, Amazon Bedrock model invocations, and data transfer if you cross VPC or region boundaries. See AWS Pricing for current rates.
Prerequisites
This article assumes working knowledge of Kubernetes (objects, controllers, RBAC), familiarity with Amazon EKS ( managed node groups , EKS Pod Identity ), and basic Amazon Bedrock awareness (model invocation, model access requests). Familiarity with Amazon CloudWatch Logs helps in the pipeline sections.
The two paradigms
Before deploying anything, internalize this distinction. Everything else flows from it.
k8sgpt: scanner with LLM-as-translator
k8sgpt is a cluster scanner with an LLM bolted on at the explanation step. Run k8sgpt analyze, and a set of deterministic Go-based analyzers walks your cluster’s resources looking for known anti-patterns: pods stuck in CrashLoopBackOff, services with no endpoints, ingresses pointing nowhere, PVCs in Pending, HPAs targeting non-existent deployments, NetworkPolicies that orphan pods. Detection is deterministic. The LLM only intervenes at the tail end to translate raw findings into human-readable explanations and suggested fixes.
Mental model: it's a kubectl describe you can ask "and what does this mean?".
It runs in two modes:
- CLI: a one-shot scan from your terminal, useful for spot-checks and onboarding to unknown clusters.
- Operator: a Kubernetes operator that watches the cluster continuously and materializes findings as Result custom resources. It exposes a /metrics endpoint scrapable by Prometheus, and findings can be exported to CloudWatch Logs for integration with other tooling.
HolmesGPT: agentic investigator
HolmesGPT takes a fundamentally different approach. The LLM sits inside the loop, not at the end of it. Given a question or an alert, HolmesGPT runs a ReAct loop: it reasons about the problem, selects and calls a tool, observes the result, and iterates until it converges on a root-cause hypothesis. Tools are pluggable through "toolsets": Kubernetes API, AWS CloudWatch, bash, connectivity checks, and custom toolsets you define yourself.
Mental model: a junior SRE with read access to your stack and a methodology for correlating resources, logs, events, and cluster state.
It also runs in two modes:
- CLI: investigate a specific alert from your terminal.
- Deployment (via the official HolmesGPT Helm chart ): a long-running HTTP service that exposes an investigation API, queryable from the CLI, a CronJob, or any custom integration.
The single line that captures the difference
k8sgpt detects first, then asks the LLM to explain. HolmesGPT lets the LLM drive the investigation from start to finish and reports back with supporting data for the suggested fix.
That difference drives every other trade-off in this article.
Where each fits in the operations workflow
Phase | Tool | Why |
|---|---|---|
Phase | Tool | Why |
Cluster onboarding / inventory | k8sgpt | Walks the resources, surfaces hygiene issues you didn't write a query for |
Continuous hygiene scanning | k8sgpt-operator | Findings materialized as CRDs, exportable to CloudWatch Logs |
Pre-prod review before merge | k8sgpt CLI in CI | Deterministic enough to gate a PR |
Investigation of detected issues | HolmesGPT | Reads findings from CloudWatch, correlates with live cluster state |
Open-ended troubleshooting | HolmesGPT | Multi-layer investigation following network/resource paths |
Question about a specific pod | k8sgpt | One-shot, fast, cheap |
These are not mutually exclusive. The stable production pattern is k8sgpt detecting continuously, HolmesGPT investigating on demand. Detection runs in the background and exports to CloudWatch; investigation triggers when something needs deeper analysis.
Architecture on Amazon EKS
The two projects share the same deployment surface, a Kubernetes cluster — but differ in their operational profile.
Common building blocks
Both need:
- A model backend. Both projects are model-agnostic. In this article, we use Amazon Bedrock as the inference backend — it provides managed access to foundation models with no infrastructure to operate. Use Amazon Nova Pro for k8sgpt (explanation tasks require moderate reasoning). Use Claude Sonnet via Bedrock for HolmesGPT (the agentic investigation loop requires strong reasoning and tool-use capabilities).
- An IAM identity for the pod. Use Amazon EKS Pod Identity to bind an IAM role to a Kubernetes ServiceAccount. Ensure the eks-pod-identity-agent addon is installed on the cluster — without it, Pod Identity associations exist but credentials are never injected into pods, and the pod silently falls back to the node instance role.
- Network egress to Bedrock. If your nodes run in private subnets without NAT, create a VPC interface endpoint for bedrock-runtime (and bedrock for management calls). Without it, the pods can't reach the model.
- Read RBAC on the cluster. k8sgpt is comfortable with the built-in view ClusterRole. HolmesGPT needs access to the resources it investigates — including k8sgpt's Result CRDs if you connect the two tools.
k8sgpt CLI on EKS — one-shot analysis
The CLI uses the AWS credentials from your environment (environment variables or profile). No additional configuration needed — if you can run aws sts get-caller-identity, k8sgpt can reach Bedrock.
Install k8sgpt and configure the Bedrock backend:
$ brew install k8sgpt$ k8sgpt auth add --backend amazonbedrock \ --model ${MODEL_ID}\ --providerRegion ${AWS_REGION}Run an analysis:
$ k8sgpt analyze --explain --backend amazonbedrockYou can filter by resource type or namespace:
$ k8sgpt analyze --explain --backend amazonbedrock \ --filter Pod,Service,Ingress,PersistentVolumeClaim$ k8sgpt analyze --explain --backend amazonbedrock --namespace defaultExample output for a PVC referencing a non-existent StorageClass:
0: PersistentVolumeClaim default/stuck-pvc()- Error: storageclass.storage.k8s.io "non-existent-class" not foundError: The specified StorageClass "non-existent-class" could not be found in the Kubernetes cluster.Solution: Verify StorageClass names with `kubectl get sc`, update your resource YAML to use an existing class, or create the missing class with `kubectl apply -f storageclass.yaml`.One finding, one explanation, one actionable fix — that is the k8sgpt pattern at its simplest.
k8sgpt-operator on EKS — continuous scanning
The operator requires three things: the Helm chart, an IAM role with Pod Identity, and a K8sGPT custom resource.
Install the EKS Pod Identity Agent addon . Without it, Pod Identity associations exist but credentials are never injected into pods — the pod silently falls back to the node instance role.
$ aws eks create-addon \ --cluster-name ${CLUSTER_NAME}\ --addon-name eks-pod-identity-agent \ --region ${AWS_REGION}Create an IAM role for the operator.
Trust policy (allows EKS Pod Identity to assume the role):
{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "Service": "pods.eks.amazonaws.com" }, "Action": ["sts:AssumeRole", "sts:TagSession"] } ]}Permissions policy:
{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "bedrock:InvokeModel", "bedrock:InvokeModelWithResponseStream" ], "Resource": "arn:aws:bedrock:${AWS_REGION}:${ACCOUNT_ID}:inference-profile/${MODEL_ID}" } ]}Install the Helm chart and create the Pod Identity association:
$ helm repo add k8sgpt https://charts.k8sgpt.ai/$ helm install k8sgpt-operator k8sgpt/k8sgpt-operator \ -n k8sgpt-operator-system --create-namespace $ aws eks create-pod-identity-association \ --cluster-name ${CLUSTER_NAME}\ --namespace k8sgpt-operator-system \ --service-account k8sgpt-k8sgpt-operator-system \ --role-arn arn:aws:iam::${ACCOUNT_ID}:role/k8sgpt-bedrock \ --region ${AWS_REGION} The ServiceAccount name is k8sgpt-k8sgpt-operator-system — not k8sgpt. Verify with kubectl get sa -n k8sgpt-operator-system after the Helm install.
Create the K8sGPT custom resource:
apiVersion: core.k8sgpt.ai/v1alpha1kind: K8sGPTmetadata: name: k8sgpt-bedrock namespace: k8sgpt-operator-systemspec: ai: enabled: true model: ${MODEL_ID} backend: amazonbedrock region: ${AWS_REGION} noCache: falseNo API keys, no Secrets — Pod Identity handles auth to Bedrock. Within 30–60 seconds, findings appear as Result custom resources:
$ kubectl get results -n k8sgpt-operator-systemNAME KIND BACKENDdefaultbrokenapp Pod amazonbedrock$ kubectl describe result defaultbrokenapp -n k8sgpt-operator-systemSpec: Backend: amazonbedrock Details: Error: The pod named "broken-app" has a container that failed to start or run due to an error. Solution: Check pod logs using `kubectl logs broken-app`, fix issues, delete pod with `kubectl delete pod broken-app`, and let Kubernetes recreate it. Error: Text: the last termination reason is Error container=broken-app Kind: Pod Name: default/broken-appExporting findings to Amazon CloudWatch Logs
Result CRDs are queryable inside the cluster, but most teams want findings visible in their existing observability layer. A lightweight CronJob can export them to Amazon CloudWatch Logs on a schedule. The job reads all Result CRs and pushes them as structured JSON events to a dedicated log group.
The CronJob needs its own IAM role (via Pod Identity) with logs:CreateLogStream and logs:PutLogEvents scoped to the target log group, plus a ClusterRole granting get and list on core.k8sgpt.ai/results.

The operator also exposes a /metrics endpoint (port 8081) scrapable by Prometheus, and findings can be visualized in Grafana dashboards by kind, namespace, and trend over time.
k8sgpt in CI — gating pull requests
The CLI mode is deterministic enough to run in a CI pipeline. The same k8sgpt analyze command, wrapped in a GitHub Actions step or a Jenkins stage, can gate PRs that introduce known anti-patterns into manifests. No additional tooling — it is the same binary, the same analyzers, the same Bedrock backend.
HolmesGPT on EKS — CLI mode
Install the CLI (installation docs):
$ brew tap robusta-dev/homebrew-holmesgpt$ brew install holmesgptHolmesGPT uses your local kubeconfig and AWS credentials. Configure Bedrock as the inference backend by passing the model with the bedrock/ prefix:
$ holmes ask "what pods are failing in namespace default and why?" \ --model="bedrock/${MODEL_ID}"The bedrock/ prefix tells litellm (the underlying LLM abstraction layer) to route the call through the Bedrock Converse API.
HolmesGPT on EKS — deployed mode
For continuous or automated investigations, deploy HolmesGPT as an in-cluster HTTP service via the official Helm chart:
$ helm repo add holmesgpt https://robusta-charts.storage.googleapis.com$ helm install holmesgpt holmesgpt/holmes \ -n holmesgpt --create-namespace -f values.yaml# values.yamladditionalEnvVars: - name: AWS_REGION_NAME value: "${AWS_REGION}" modelList: bedrock-claude: model: bedrock/${MODEL_ID} aws_region_name: ${AWS_REGION} toolsets: kubernetes/core: enabled: true aws/cloudwatch: enabled: trueSame Pod Identity story as k8sgpt for Bedrock access. Create the IAM role, then bind it to the chart's ServiceAccount (holmesgpt-holmes-service-account):
$ aws eks create-pod-identity-association \ --cluster-name ${CLUSTER_NAME}\ --namespace holmesgpt \ --service-account holmesgpt-holmes-service-account \ --role-arn arn:aws:iam::${ACCOUNT_ID}:role/holmesgpt-bedrock \ --region ${AWS_REGION}The IAM role needs bedrock:InvokeModel and bedrock:InvokeModelWithResponseStream on the model used. If you enable the CloudWatch toolset, add read permissions scoped to the target log group:
{ "Effect": "Allow", "Action": [ "logs:DescribeLogGroups", "logs:DescribeLogStreams", "logs:GetLogEvents", "logs:FilterLogEvents", "logs:StartQuery", "logs:GetQueryResults" ], "Resource": "arn:aws:logs:${AWS_REGION}:${ACCOUNT_ID}:log-group:/k8sgpt/findings:*"}Once deployed, the service exposes an HTTP API on port 80. Trigger an investigation with:
$ curl -s http://holmesgpt-holmes.holmesgpt/api/chat -X POST \ -H "Content-Type: application/json" \ -d '{"ask": "why is payment-service not receiving traffic?", "model": "bedrock-claude"}'The model field references the name defined in modelList. The response contains the full investigation report with root causes and suggested fixes.
Comparison
Aspect | k8sgpt | HolmesGPT |
|---|---|---|
Trigger | Manual / scheduled / Operator watch | On demand (API call, CLI, or automation) |
LLM role | Explains pre-detected findings | Drives the investigation loop |
Determinism | High (analyzers are deterministic) | Low (agent decides next step) |
Source breadth | Kubernetes API only | K8s + CloudWatch + bash + custom toolsets |
Tokens per run | ~5–15k | 20k–200k+ depending on depth |
Latency | Seconds | Tens of seconds to minutes |
Output shape | List of findings + suggested fixes | Investigation report with evidence |
Strengths | Hygiene, audit, predictable cost | Cross-resource correlation, RCA |
Weaknesses | Bounded by analyzer coverage | Variable cost, depends on model quality |
CNCF status | Sandbox (Dec 2023) | Sandbox (Oct 2025) |
Use cases
Use case 1 — Pre-prod hygiene with k8sgpt
You inherit a cluster from another team. You don't know what's in there. Run:
$ k8sgpt analyze --explain --backend amazonbedrock --namespace defaultHere is a real output from a cluster with a misconfigured payment-service deployment — an image tag that doesn't exist, a Service pointing at a non-existent selector, an Ingress referencing a typo'd Service name, and a PVC requesting a StorageClass that was never created:
0: Deployment default/payment-service()- Error: Deployment has 3 replicas but 0 are availableError: The payment-service deployment should have 3 running instances, but none are currently running.Solution: Check pod logs for errors, ensure resources are sufficient, and review deployment configuration for mistakes. 1: PersistentVolumeClaim default/payment-data()- Error: storageclass.storage.k8s.io "non-existent-class" not foundError: The specified StorageClass could not be found in the cluster.Solution: Verify StorageClass names with `kubectl get sc`, update your resource YAML to use an existing class. 2: Ingress default/payment-ingress()- Error: Ingress does not specify an Ingress class.- Error: Ingress uses the service default/payment-service-typo which does not exist.Error: Ingress lacks class and references non-existent service.Solution: Correct service name in Ingress spec and specify Ingress class. 3: Service default/payment-service()- Error: Service has no endpoints, expected label app=payment-wrong-labelError: Service lacks endpoints due to incorrect label.Solution: Verify label on pods matches service selector. 4: Pod default/payment-service-5c7bcd5c75-8phvk()- Error: Back-off pulling image "nginx:99.99.99"Error: Kubernetes failed to pull the specified image version.Solution: Verify the correct image tag, update the deployment YAML.Five injected issues, five detected — in a single pass, in seconds. The detection is deterministic (the Go analyzers find the anti-patterns); the LLM adds actionable explanations and suggested fixes. No hallucinations in the remediation steps.
These are not exotic findings. They are the common hygiene issues that accumulate in clusters with multiple teams pushing manifests. k8sgpt surfaces them in one command.
Use case 2 — Multi-layer investigation with HolmesGPT
Same cluster, same payment-service deployment. But this time, instead of asking "what's broken?", we ask HolmesGPT an open-ended question: why is no traffic reaching the service?
$ holmes ask "the payment-service has no traffic reaching it, investigate why" \ --model="bedrock/${MODEL_ID}"HolmesGPT investigates layer by layer — Ingress, Service, Pods — and correlates the findings:
Found three compounding issues preventing any traffic from reaching payment-service: 1. Ingress points to a non-existent service (typo) payment-ingress routes payments.example.com/ to payment-service-typo:80 Fix: patch ingress backend to payment-service 2. Service selector uses wrong label Selects app=payment-wrong-label but pods have app=payment-service → Endpoints is empty Fix: patch selector to app=payment-service 3. All pods are in ImagePullBackOff Image nginx:99.99.99 doesn't exist Fix: update deployment image to a valid tag Layer | Issue | Fix Ingress | Backend service name typo | Patch ingress Service | Selector matches no pods | Patch selector Pods | Image doesn't exist | Update image tag "All three issues must be fixed for traffic to flow end-to-end."Compare this with the k8sgpt output above: k8sgpt listed five issues independently — a flat list of findings with no relationship between them. HolmesGPT followed the network path (Ingress → Service → Pods), identified that the problems compound, and concluded that all three layers must be fixed together for traffic to flow. That correlation is what the ReAct loop buys you: the agent reasons about the system as a whole, not resource by resource.
Use case 3 — k8sgpt findings → CloudWatch → HolmesGPT investigation
The two tools become more powerful when connected. k8sgpt detects issues continuously and exports findings to CloudWatch Logs. HolmesGPT reads those findings and investigates them with full cluster context — correlating the detection data with live state, logs, and metrics.
The pipeline:
k8sgpt-operator (scan every 30s) │ ▼Result CRDs (in-cluster) │ ▼CronJob (every 5 min) → CloudWatch Logs /k8sgpt/findings │ ▼HolmesGPT (custom CloudWatch toolset) → reads findings → investigatesWhen HolmesGPT is asked to check CloudWatch for k8sgpt findings, it queries the log group, retrieves the structured JSON events, and then uses its Kubernetes toolset to investigate each issue in depth. Here is a real output from this pipeline:
Log Streams Found: findings-2026-08-05 | Last Event: 2026-08-05 13:15:01 UTC findings-2026-08-04 | Last Event: 2026-08-04 22:20:00 UTC k8sgpt Detected Issues (from CloudWatch): 1. Pod: broken-app (default) Error: Container keeps crashing — last termination reason is Error. Fix: kubectl logs broken-app -n default 2. PVC: payment-data (default) Error: StorageClass "non-existent-class" does not exist — PVC stuck unbound. Fix: kubectl get sc, then update PVC to use an existing class. 3. Ingress: payment-ingress (default) Error: No Ingress class specified + references service "payment-service-typo" which does not exist. Fix: Add ingressClassName, correct backend service name. 4. Service: payment-service (default) Error: No endpoints — selector expects app=payment-wrong-label, no pods match. Fix: Patch selector to match actual pod labels. 5. Pods: payment-service deployment (3 replicas) Error: ErrImagePull — image nginx:99.99.99 does not exist. Fix: kubectl set image deployment/payment-service <container>=nginx:latest All five issues must be resolved to restore the cluster to a healthy state.The key difference from running each tool in isolation: HolmesGPT doesn't just restate the k8sgpt findings — it investigates them further using the Kubernetes API, correlates across resources, and identifies that the issues compound. k8sgpt provides the detection breadth (what's wrong); HolmesGPT provides the investigation depth (why it matters and how the issues interact).
This pipeline requires:
- The CloudWatch export CronJob (described in "Exporting findings to Amazon CloudWatch Logs" above)
- A custom CloudWatch toolset in the HolmesGPT configuration that exposes cloudwatch_get_log_streams, cloudwatch_get_log_events, and cloudwatch_filter_log_events as callable tools
toolsets: aws/cloudwatch: description: "Query AWS CloudWatch Logs to retrieve k8sgpt findings" prerequisites: - command: "python3 -c 'import boto3'" tools: - name: "cloudwatch_get_log_streams" description: "List log streams in a CloudWatch Logs log group, ordered by most recent event time. Use this to find available log streams before querying events." script: | #!/bin/bash python3 -c " import boto3, json from datetime import datetime client = boto3.client('logs', region_name='{{ region }}') r = client.describe_log_streams( logGroupName='{{ log_group }}', orderBy='LastEventTime', descending=True, limit=10 ) streams = r.get('logStreams', []) result = [] for s in streams: ts = s.get('lastEventTimestamp', 0) result.append({ 'name': s['logStreamName'], 'last_event': datetime.utcfromtimestamp(ts/1000).strftime('%Y-%m-%d %H:%M:%S UTC') if ts else 'N/A' }) print(json.dumps(result, indent=2)) " - name: "cloudwatch_get_log_events" description: "Get log events from a specific CloudWatch Logs log stream. Returns the most recent events from the specified log group and stream. Use this to read k8sgpt findings from the /k8sgpt/findings log group." script: | #!/bin/bash python3 -c " import boto3, json from datetime import datetime client = boto3.client('logs', region_name='{{ region }}') r = client.get_log_events( logGroupName='{{ log_group }}', logStreamName='{{ log_stream }}', startFromHead=False, limit=50 ) events = r.get('events', []) result = [] for e in events: ts = datetime.utcfromtimestamp(e['timestamp']/1000).strftime('%Y-%m-%d %H:%M:%S UTC') try: msg = json.loads(e['message']) except: msg = e['message'] result.append({'timestamp': ts, 'message': msg}) print(json.dumps(result, indent=2)) " - name: "cloudwatch_filter_log_events" description: "Filter and search log events across all streams in a CloudWatch log group. Use filter_pattern to search for specific text. Returns matching events from the last 24 hours." script: | #!/bin/bash python3 -c " import boto3, json, time from datetime import datetime client = boto3.client('logs', region_name='{{ region }}') start = int((time.time() - 86400) * 1000) kwargs = { 'logGroupName': '{{ log_group }}', 'startTime': start, 'limit': 50 } filter_pat = '{{ filter_pattern }}' if filter_pat and filter_pat != '': kwargs['filterPattern'] = filter_pat r = client.filter_log_events(**kwargs) events = r.get('events', []) result = [] for e in events: ts = datetime.utcfromtimestamp(e['timestamp']/1000).strftime('%Y-%m-%d %H:%M:%S UTC') try: msg = json.loads(e['message']) except: msg = e['message'] result.append({'timestamp': ts, 'stream': e.get('logStreamName',''), 'message': msg}) print(json.dumps(result, indent=2)) "
- IAM permissions for the HolmesGPT role to read CloudWatch Logs (logs:DescribeLogStreams, logs:GetLogEvents, logs:FilterLogEvents)
k8sgpt is the hygiene radar; HolmesGPT is the incident responder. Different cadence, different cost profile, complementary outputs.
Security considerations and pitfalls
Treat LLM prompts as data exfiltration risk
When HolmesGPT calls kubectl logs, those logs become part of the prompt sent to Bedrock. Application logs routinely contain JWTs, email addresses, customer IDs. Bedrock does not use this data for training, but it still leaves your application's trust boundary.
Mitigations:
- Strip PII at the log shipper (Fluent Bit, Vector) before logs reach Loki / CloudWatch.
- Apply Amazon Bedrock Guardrails with PII masking policies on the model invocation.
- Restrict toolsets so HolmesGPT only sees logs from non-sensitive namespaces by default.
Scope IAM precisely
The IAM role bound to the agent's ServiceAccount should:
- List exact Bedrock model ARNs in Resource. No bedrock:*.
- Grant bedrock:InvokeModel and the streaming variant only.
- For HolmesGPT, restrict CloudWatch and other AWS data sources to specific log groups, namespaces, or tags.
A wildcard IAM role on an LLM-driven agent is a confused-deputy attack waiting to happen.
Lock down egress
Outside Bedrock and your in-cluster data sources, the agent should not reach the public Internet. Use NetworkPolicies that allow only:
- The Bedrock VPC endpoint
- In-cluster Prometheus, Loki, AlertManager
- The Kubernetes API server
- AWS CloudWatch endpoints if used
Block everything else. An agent with unrestricted egress is one prompt-injection away from exfiltrating cluster state.
Cost guardrails from day one
The real budget killer is not steady-state per-scan token usage; it's an agent that loops on an under-specified problem (incomplete context, missing toolset) and burns thousands of tokens going nowhere. Set the following from day one — not as optional guardrails, as deployment prerequisites:
- Per-investigation step cap. HolmesGPT exposes a max_steps parameter (default 100) that bounds the number of LLM calls in the ReAct loop. Treat this as the primary safety net against runaway agents. Lower the default in early deployments while you observe real investigations.
- AWS Budgets on Bedrock spend per service tag, with email alerts at 50% / 80% / 100% thresholds.
- Provisioned Throughput on Bedrock only if you have measured steady-state usage; on-demand is fine for most teams.
Treat LLM root-cause hypotheses as hypotheses
HolmesGPT's confidence labels are not ground truth. The output is an enriched starting point for the on-call human, not a verdict. Teams that treat the report as a hypothesis ship faster RCAs; teams that treat it as truth ship incorrect post-mortems faster.
Decision tree
Need: act on a Kubernetes operational concern.│├── Is the issue already identified, with a known fix?│ └── Yes → no agent needed. Apply the fix.│├── Is it a one-shot question about a specific pod / resource?│ └── Yes → k8sgpt analyze (CLI).│├── Are you onboarding to an unknown cluster, or want continuous hygiene?│ └── Yes → k8sgpt-operator + CloudWatch export.│├── Is there a live incident with multi-source symptoms?│ └── Yes → HolmesGPT (alert-triggered or CLI).│└── Do you want automated investigation of detected issues? └── Yes → k8sgpt → CloudWatch → HolmesGPT pipeline. Detection feeds investigation; both run continuously.Clean-up
If you experimented with the deployment patterns described above, remove resources to avoid ongoing charges:
- The Helm releases (helm uninstall k8sgpt-operator -n k8sgpt-operator-system, helm uninstall holmesgpt -n holmesgpt).
- The associated namespaces.
- The K8sGPT custom resource and any Result CRs.
- The CloudWatch Logs log group (/k8sgpt/findings) and the findings export CronJob.
- IAM roles and Pod Identity associations created for Bedrock and CloudWatch access.
- VPC interface endpoints for Bedrock if created exclusively for this work.
- AWS Budgets and Amazon CloudWatch alarms if you set them.
Conclusion
The shift from kubectl-driven operations to agent-assisted operations is not about replacing the cluster. It's about replacing the operator's cognitive loop: from "I think, I type, I read, I correlate" to "the agent investigates, I review, I decide." The cluster, the YAML, the controllers, the operators — none of that goes away.
k8sgpt and HolmesGPT illustrate the two shapes this layer takes. A scanner pattern, where deterministic detection is enriched with LLM-generated explanations, gives you predictable cost and high signal on cluster hygiene. An agentic pattern, where the LLM drives a tool-using investigation loop, gives you cross-stack root-cause analysis at a higher and more variable cost.
The real power emerges when you connect the two: k8sgpt detects continuously, exports structured findings to CloudWatch Logs, and HolmesGPT reads those findings and investigates them with full cluster context. Detection feeds investigation. The scanner catches the signals; the agent correlates and explains them.
Neither replaces operational competence. An agent on a poorly-architected cluster is a faster path to a bigger outage. RBAC, network segmentation, observability hygiene, IAM scoping — these foundational practices still belong to the human. The agent is a force multiplier, not a substitute.
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article