
From Noisy Alerts to Safe Incident Response: How We Built Calyx on AWS
Calyx is an AI-native incident investigator for small engineering teams. Learn how we used Amazon ECS, AWS Fargate, CloudWatch, RDS, ElastiCache, S3, IAM, and Terraform to turn production telemetry into evidence-backed investigations while keeping remediation under human control.
Calyx : AI-Native Observability and Incident Investigation Platform
Building and deploying software has become increasingly fast, but understanding production failures is still difficult.
When an application starts returning errors, developers often need to search logs, inspect dashboards, review deployments, compare commits, contact teammates, and decide whether a rollback is safe. Large organizations have dedicated SRE teams for this work. Students, solo developers, startups, and small engineering teams usually do not.
We built Calyx to make incident investigation accessible to these teams.
Calyx is an AI-native observability and incident-investigation platform. It collects production telemetry, detects abnormal behaviour, creates durable incidents, and helps developers answer:
- What failed?
- When did it begin?
- Which services or users may be affected?
- What changed immediately before the failure?
- What evidence supports the suspected cause?
- What is the safest next action?
Calyx is not simply a chatbot connected to logs. It is a durable incident workflow with evidence tracking, tenant isolation, controlled automation, and human approval.
The Calyx Investigation Loop
Calyx follows an automated but safety-conscious workflow:
Application produces telemetry ↓Calyx receives and normalizes events ↓Scheduled detectors identify abnormal behaviour ↓A durable incident is created ↓The investigation agent gathers bounded evidence ↓Developers receive a timeline and root-cause hypothesis ↓A human approves or rejects any proposed action
Developers can investigate incidents through a Slack-style web workspace, Slack, authenticated APIs, or an MCP-compatible coding assistant.
Separating Facts from AI Hypotheses
A major concern with AI-assisted incident response is that generated explanations may sound more certain than the available evidence supports.
Calyx therefore separates investigation output into four categories:
Category | Meaning |
|---|---|
Observed | Directly returned by logs, metrics, or provider APIs. |
Correlated | Related by time, identifiers, deployments, or commits. |
Hypothesis | An AI-generated explanation that is still uncertain. |
Verified | Confirmed through an explicit check or controlled workflow. |
For example:
"The error rate increased three minutes after deploymentabc123."
This is a correlation. It does not automatically prove that deployment
abc123 caused the incident.This evidence model helps developers understand what Calyx knows, what it inferred, and where more investigation is required.
Why We Chose Amazon ECS and AWS Fargate
Calyx contains multiple workloads with different responsibilities:
- A Next.js web application
- A Go collaboration and chat API
- A telemetry intake service
- A Redis stream consumer
- A scheduled anomaly detector
- A hosted MCP server
- An optional Slack service
We wanted to deploy these workloads independently without managing EC2 instances or introducing Kubernetes.
We selected Amazon ECS with AWS Fargate as the application runtime.
Each workload runs with its own:
- Task definition
- Health checks
- Environment variables
- Secrets
- IAM permissions
Public APIs are connected to an Application Load Balancer, while background consumers remain private.
Fargate allowed us to preserve our existing container architecture while removing the operational burden of managing servers.
Our AWS Architecture
The production architecture uses:
- Amazon ECS on AWS Fargate — Application workloads
- Amazon ECR — Container images
- Amazon RDS for PostgreSQL — Authoritative application state
- Amazon ElastiCache for Redis — Event streams and realtime delivery
- Amazon S3 — Private workspace uploads
- Application Load Balancer — HTTPS and host-based routing
- AWS Certificate Manager — TLS certificates
- Amazon Route 53 — DNS
- AWS Secrets Manager — Credentials
- AWS Identity and Access Management (IAM) — Task-specific permissions
- Amazon CloudWatch — Logs, metrics, dashboards, and alarms
- AWS Lambda — Optional CloudWatch Logs forwarding
- Amazon SNS — Operational notifications
Public Endpoints
app.example.com → Next.js web applicationchat.example.com → Go chat APIintake.example.com → Telemetry intake APImcp.example.com/mcp → Hosted MCP server
Only the Application Load Balancer is public.
ECS tasks, PostgreSQL, and Redis remain inside private subnets.
PostgreSQL as the Source of Truth
We deliberately give RDS and ElastiCache different responsibilities.
Amazon RDS for PostgreSQL
RDS stores durable information, including:
- Telemetry events
- Alert contexts
- Incidents and evidence
- Deployment and commit correlations
- Workspace and chat data
- Remediation requests
- Immutable audit events
- Coding-agent jobs
Amazon ElastiCache for Redis
Redis handles ephemeral concerns:
- Ingestion streams
- Distributed rate limits
- Realtime fan-out
- Short-lived coordination
If Redis is temporarily unavailable, incidents, approvals, evidence, and audit history remain stored in PostgreSQL.
Redis can recover as a delivery layer without becoming a second source of truth.
Secure S3 Access with ECS Task Roles
The Go API stores workspace uploads in a private, encrypted, versioned S3 bucket.
We did not want long-lived AWS access keys inside environment files or containers. The API therefore uses the AWS SDK for Go v2 default credential chain:
cfg, err := awsconfig.LoadDefaultConfig(ctx)if err != nil { return nil, fmt.Errorf("load AWS configuration: %w", err)}
client := s3.NewFromConfig(cfg)
On ECS, the SDK automatically receives temporary credentials from the task role.
The task role is limited to the required S3 bucket and object operations.
The browser never receives AWS credentials or unrestricted object URLs.
Every download passes through the authenticated Go API, which checks workspace membership before streaming the object.
Sending CloudWatch Logs to Calyx
Calyx includes a native CloudWatch Logs ingestion path.
An optional AWS Lambda function:
- Receives a CloudWatch Logs subscription event.
- Decodes and decompresses the AWS Logs envelope.
- Retrieves its Calyx source token from Secrets Manager.
- Sends the bounded payload to the intake service.
- Lets Calyx normalize the records into its common event schema.
Preventing Recursive Ingestion
One subtle challenge was avoiding recursive ingestion.
If the intake service's own CloudWatch log group were subscribed to the forwarding Lambda, each forwarded request could generate another intake log and trigger another invocation.
We prevent that by excluding the intake log group from the Terraform subscription-filter set:
for_each = var.cloudwatch_ingestion_enabled ? { for key, group in aws_cloudwatch_log_group.service : key => group if key != "intake"} : {}
This small safeguard prevents an expensive feedback loop.
Immutable Images and Safe Release Gates
We use Amazon ECR to store the Node.js, Go, and Next.js images.
ECS task definitions reference immutable image digests instead of mutable tags such as
latest :123456789012.dkr.ecr.us-east-1.amazonaws.com/calyx-node@sha256:...
We also wanted to ensure that services never started against an outdated database schema.
Our release process therefore:
- Creates the ECR repositories.
- Builds Linux/AMD64 production images.
- Pushes the images to ECR.
- Resolves their immutable SHA-256 digests.
- Provisions services at zero replicas.
- Runs the Node.js database migration as an ECS task.
- Runs the Go database migration as an ECS task.
- Runs the legacy upload backfill.
- Creates and stores the management token.
- Enables the ECS services.
- Waits for service stability.
- Probes each public HTTPS endpoint.
If a migration or backfill fails, deployment stops before application services start.
ECS deployment circuit breakers protect subsequent service updates.
Human Approval Before Remediation
An investigation agent can recommend an action, but a recommendation is not execution authority.
Calyx persists remediation proposals in PostgreSQL with:
- Tenant and incident identity
- Proposed operation
- Reason and dry-run result
- Approval status
- Immutable transition events
Execution requires approval from an authenticated workspace administrator or an allowlisted Slack identity.
The transition from
pending to executing is atomic.If an external operation returns an ambiguous result, Calyx does not automatically repeat it. A human reconciles the action using its durable request ID.
Coding-Agent Safety
Coding agents follow the same safety model.
They can:
- Create a branch
- Draft a pull request
They cannot:
- Merge pull requests
- Deploy changes
This ensures that AI-assisted investigation remains useful without giving agents uncontrolled production authority.
Infrastructure as Code
We defined the AWS environment with Terraform.
Terraform provisions:
- A two-availability-zone VPC
- Public and private subnets
- Routing and security groups
- ECS services and release tasks
- ECR repositories
- RDS PostgreSQL
- ElastiCache Redis
- Private S3 storage
- Application Load Balancer
- ACM certificates
- Route 53 records
- Secrets Manager secrets
- IAM roles and policies
- CloudWatch dashboards and alarms
- Optional Lambda log forwarding
- SNS notifications
This makes the environment reproducible and documents the purpose of every AWS service.
Because Terraform state contains generated infrastructure credentials, we treat it as sensitive and keep it outside source control.
Cost Decisions
For our hackathon environment, we selected cost-oriented defaults:
- One NAT Gateway
- One small RDS instance
- One small Redis node
- One task per application service
- Small Fargate CPU and memory allocations
- Limited CloudWatch log retention
The main continuously billed resources are:
- NAT Gateway
- Application Load Balancer
- RDS
- ElastiCache
- Fargate
- CloudWatch Logs
For a larger deployment, we would evaluate:
- Multi-AZ databases
- Additional NAT Gateways or VPC endpoints
- Autoscaling policies
- Longer backup retention
- Cross-region recovery
We avoided adding AWS services only to make the architecture diagram larger. Every service has a specific runtime, security, storage, or operational responsibility.
What We Learned
Our biggest lesson was that AI incident response is not primarily a prompt-engineering problem.
The difficult parts are:
- Capturing reliable evidence
- Maintaining tenant isolation
- Distinguishing facts from hypotheses
- Preserving durable incident state
- Handling retries safely
- Preventing duplicate external actions
- Protecting credentials
- Making deployment and recovery predictable
AWS provided the infrastructure boundaries needed to solve these problems.
ECS Fargate allowed us to operate multiple workloads without managing servers.
RDS gave us a durable source of truth.
ElastiCache provided low-latency coordination.
IAM task roles eliminated static S3 credentials.
CloudWatch provided both infrastructure visibility and real telemetry that Calyx could investigate.
Conclusion
Calyx transforms production failures into evidence-backed investigations.
Instead of giving small teams another noisy alert, it provides:
- A timeline
- Relevant evidence
- Deployment context
- An explicitly labelled hypothesis
- A safely controlled next action
By combining ECS Fargate, ECR, RDS, ElastiCache, S3, CloudWatch, IAM, Secrets Manager, Route 53, ACM, Lambda, and SNS, we created an AWS-native architecture in which AI helps gather and explain evidence while humans retain control over consequential actions.
Links
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article