Weekend Challenge - Kafka Lag Watcher
The Kafka Lag Watcher: An Agent That Tells You Why Your Consumers Are Behind
I work with Kafka every day. The worst part of my job is not the code. It is the 2 AM message that says "orders are delayed" and the dashboard that shows one number: lag, 250,000 messages. That number tells you nothing about why. So I built an agent that watches consumer lag and explains it in plain words.
What and who:
The Kafka Lag Watcher is for developers and on-call engineers who run Kafka consumers in production. It checks a consumer group on a schedule, reads the lag on every partition, and tells you what is actually happening. It does not just say "lag is high." It says things like: "One partition is stuck while the rest are healthy. That is the classic poison-pill pattern: a single bad record the consumer keeps failing on." Then it lists the fix steps in order.
It recognizes four patterns. A spike, where one partition has a big backlog but the consumer is still healthy, so you just wait or scale up. A poison pill, where one partition is stuck on a bad record. A slow consumer group, where everything drifts behind because the group is under-provisioned. And a rebalance storm, where partitions keep moving between consumers so nobody makes progress. Each pattern gets a different verdict and different actions, because the wrong fix for the right problem is still the wrong fix.
How I built it:
The agent is one Python file with two modes. Demo mode runs on built-in synthetic data and needs zero setup, which is how I tested and filmed it. Real mode connects to a live cluster with kafka-python, reads end offsets and committed offsets per partition, and runs the same analysis.
The analysis step is pure Python. It computes lag per partition, flags partitions over thresholds, compares consume rates, and classifies the pattern with simple rules: a single slow partition with a lag far above the rest means a poison pill; one hot partition at a normal rate means a spike; everything slow means under-provisioning; mixed slow and fast means rebalancing trouble. It also estimates the time to clear a backlog from the current consume rate.
On AWS, the design runs as a scheduled serverless job. EventBridge fires the check every five minutes. A Lambda function reads the offsets, runs the analysis, and calls Amazon Bedrock to write the plain-English explanation and the action list. The result goes to an SNS topic so the on-call engineer gets a message that reads like a senior teammate wrote it, not a dashboard alert. The core logic is the same code in the demo and in Lambda, so I could build and test everything locally first.
The delightful detail:
The one thing that makes it enjoyable to use is the ETA line. When the agent finds a spike, it tells you the backlog will clear in about two minutes at the current rate, and that no code change is needed if that is acceptable. That single sentence kills half of all Kafka pages. Most lag spikes are self-healing, and the agent knows it. It tells you to relax instead of telling you to panic. I know it worked because the first time I ran it on a real-looking backlog, it said "no code change needed," and that is exactly what a good senior engineer would have said.
Proof it works
I ran the agent on three built-in scenarios. Here is the poison-pill one:
The Kafka Lag Watcher is for developers and on-call engineers who run Kafka consumers in production. It checks a consumer group on a schedule, reads the lag on every partition, and tells you what is actually happening. It does not just say "lag is high." It says things like: "One partition is stuck while the rest are healthy. That is the classic poison-pill pattern: a single bad record the consumer keeps failing on." Then it lists the fix steps in order.
It recognizes four patterns. A spike, where one partition has a big backlog but the consumer is still healthy, so you just wait or scale up. A poison pill, where one partition is stuck on a bad record. A slow consumer group, where everything drifts behind because the group is under-provisioned. And a rebalance storm, where partitions keep moving between consumers so nobody makes progress. Each pattern gets a different verdict and different actions, because the wrong fix for the right problem is still the wrong fix.
How I built it:
The agent is one Python file with two modes. Demo mode runs on built-in synthetic data and needs zero setup, which is how I tested and filmed it. Real mode connects to a live cluster with kafka-python, reads end offsets and committed offsets per partition, and runs the same analysis.
The analysis step is pure Python. It computes lag per partition, flags partitions over thresholds, compares consume rates, and classifies the pattern with simple rules: a single slow partition with a lag far above the rest means a poison pill; one hot partition at a normal rate means a spike; everything slow means under-provisioning; mixed slow and fast means rebalancing trouble. It also estimates the time to clear a backlog from the current consume rate.
On AWS, the design runs as a scheduled serverless job. EventBridge fires the check every five minutes. A Lambda function reads the offsets, runs the analysis, and calls Amazon Bedrock to write the plain-English explanation and the action list. The result goes to an SNS topic so the on-call engineer gets a message that reads like a senior teammate wrote it, not a dashboard alert. The core logic is the same code in the demo and in Lambda, so I could build and test everything locally first.
The delightful detail:
The one thing that makes it enjoyable to use is the ETA line. When the agent finds a spike, it tells you the backlog will clear in about two minutes at the current rate, and that no code change is needed if that is acceptable. That single sentence kills half of all Kafka pages. Most lag spikes are self-healing, and the agent knows it. It tells you to relax instead of telling you to panic. I know it worked because the first time I ran it on a real-looking backlog, it said "no code change needed," and that is exactly what a good senior engineer would have said.
Proof it works
I ran the agent on three built-in scenarios. Here is the poison-pill one:
KAFKA LAG WATCHER REPORT
Group: orders-consumer Topic: orders.events
Group: orders-consumer Topic: orders.events
Total lag: 38,300 messages
Worst partition: 4 (38,000 messages behind)
Worst partition: 4 (38,000 messages behind)
DIAGNOSIS
One partition is stuck while the rest are healthy. That is the
classic poison-pill pattern: a single bad record the consumer
keeps failing on.
One partition is stuck while the rest are healthy. That is the
classic poison-pill pattern: a single bad record the consumer
keeps failing on.
WHAT TO DO
- Check the consumer logs for repeated failures on partition 4.
- Find the record at the stuck offset and move it to a dead-letter topic.
- Add a try/except around record processing with a max-retry limit.
- Consider a circuit breaker so one bad record cannot stall the partition.The spike and rebalance scenarios classify correctly too. The code is on GitHub, and the screenshot below (demo-screenshot.png, real terminal output of the agent running) is my proof it works.
#agents #buildercommunity
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article