AIdeas: LuminaLog — Privacy-First Observability
LuminaLog is a serverless, privacy-first observability platform. It features SmartScrub™, a Kinesis-driven engine that replaces PII with realistic synthetic data before storage, preserving both data privacy and debuggability. Combined with real-time WebSocket live tailing and automated Gemini 2.5 Flash error analysis, it gives startups in regulated markets enterprise-grade compliance and insights without the enterprise price tag.
App Category
Commercial Solutions
My Vision
I didn't start LuminaLog because observability is a glamorous problem. I started it because I was
building a product for the Nigerian market and realized that every logging tool I tried was designed
for someone else.
building a product for the Nigerian market and realized that every logging tool I tried was designed
for someone else.
Datadog costs more per month than most African startups can justify at early stage. New Relic's
pricing is opaque. Self-hosting ELK requires a DevOps team you probably can't afford yet. And
none of them, not one treat PII protection as a first-class feature. They bolt it on. Or
they leave it to you entirely.
pricing is opaque. Self-hosting ELK requires a DevOps team you probably can't afford yet. And
none of them, not one treat PII protection as a first-class feature. They bolt it on. Or
they leave it to you entirely.
Meanwhile, healthcare startups in Lagos are logging patient data. Fintechs in Accra are logging
transaction records. Insurance platforms across West Africa are logging everything and most of
it is going into log files completely unprotected.
transaction records. Insurance platforms across West Africa are logging everything and most of
it is going into log files completely unprotected.
So I built LuminaLog. The pitch is simple: you should be able to ship fast, observe everything,
and not accidentally expose your users' data in the process.
and not accidentally expose your users' data in the process.
LuminaLog is a cloud-native observability platform with four capabilities that work together:
- Real-time log ingestion and live tail — Send logs from any application, watch them stream
in real time via WebSocket - SmartScrub™ privacy engine — Automatic PII detection that replaces sensitive values with
realistic synthetic data before logs ever reach storage - AI-powered error analysis — Gemini 2.5 Flash automatically classifies new errors and
generates root-cause summaries as they arrive - Interactive AI Debug — Paste a stack trace, get a structured diagnosis in seconds
The target customer is the engineering team at a Fintech, Healthtech, or SaaS startup that is
serious about their product but hasn't had the time or budget to build a proper observability
layer especially one that won't get them fined under GDPR, NDPR, or HIPAA.
serious about their product but hasn't had the time or budget to build a proper observability
layer especially one that won't get them fined under GDPR, NDPR, or HIPAA.
Why This Matters
Two things happened that convinced me this was urgent.
The first: a founder I know got a legal notice. Not a hack, not a breach in the dramatic sense,
just log files. Their application was logging user emails and IP addresses in plain text to a
third-party logging service. Their legal team flagged it six months into launch. It cost them weeks
of engineering time and a compliance audit they weren't ready for.
just log files. Their application was logging user emails and IP addresses in plain text to a
third-party logging service. Their legal team flagged it six months into launch. It cost them weeks
of engineering time and a compliance audit they weren't ready for.
The second: I priced out setting up a compliant logging infrastructure for a side project of my
own. Just to be GDPR-safe and get proper error tracking, I was looking at a minimum of $80–100/month before I'd written a single user. That's a rent payment in most of the markets I want to serve.
own. Just to be GDPR-safe and get proper error tracking, I was looking at a minimum of $80–100/month before I'd written a single user. That's a rent payment in most of the markets I want to serve.
Here's the tension that LuminaLog is designed to solve: the markets that most need proper
observability are the ones that can least afford the tools built for enterprise teams in San
Francisco. A bootstrapped Healthtech startup in Nairobi shouldn't have to choose between
"observe your system properly" and "protect your patients' data" and "stay within your runway."
They shouldn't be making that trade-off at all.
observability are the ones that can least afford the tools built for enterprise teams in San
Francisco. A bootstrapped Healthtech startup in Nairobi shouldn't have to choose between
"observe your system properly" and "protect your patients' data" and "stay within your runway."
They shouldn't be making that trade-off at all.
How I Built This
I'll be honest about the architecture choices because they were deliberate and sometimes painful.
The Ingestion Layer
Every log batch hits an AWS API Gateway HTTP API backed by a Go Lambda function. Go was
the right call, cold start times are low, memory overhead is minimal, and the concurrency model
is clean. The Lambda validates the API key (checked against DynamoDB), enforces per-org rate
limits, and queues the batch to Amazon SQS.
the right call, cold start times are low, memory overhead is minimal, and the concurrency model
is clean. The Lambda validates the API key (checked against DynamoDB), enforces per-org rate
limits, and queues the batch to Amazon SQS.
A second Lambda , the
PII scrubbing, error fingerprinting, writing enriched records to Amazon Kinesis Firehose,
which delivers everything to S3 partitioned by org, project, and date.
ingestion-processor,picks up from SQS and does the heavy work:PII scrubbing, error fingerprinting, writing enriched records to Amazon Kinesis Firehose,
which delivers everything to S3 partitioned by org, project, and date.
I chose this two-step pattern (API Lambda → SQS → Processor Lambda) specifically so the HTTP
response time stays fast regardless of what the processing pipeline is doing. Ingestion is
acknowledged immediately; the SDK doesn't block on our analysis work.
response time stays fast regardless of what the processing pipeline is doing. Ingestion is
acknowledged immediately; the SDK doesn't block on our analysis work.
Data Flow
SmartScrub™ — The Privacy Engine
This is the part I'm most proud of, and the part that most people get wrong when they first
hear about it.
hear about it.
SmartScrub™ does not just redact. It replaces.
Most PII scrubbers drop the value and put
debugging. You can't tell if an email address was formatting correctly. You can't trace a user's
session. You can't reproduce a bug. The log becomes useless.
[REDACTED] in its place. That breaks your logs fordebugging. You can't tell if an email address was formatting correctly. You can't trace a user's
session. You can't reproduce a bug. The log becomes useless.
SmartScrub™ uses deterministic synthetic replacement. Every PII value is replaced with a
realistic fake value generated from a SHA-256 hash of the original, seeded with the user ID.
This means:
realistic fake value generated from a SHA-256 hash of the original, seeded with the user ID.
This means:
patient.name@hospital.ng→user_3f8a92b1@example.com(valid email, different inbox)192.31.4.210→192.168.174.23(still a real-looking IP, now private range)sk_live_abc123...→sk_test_3f8a92b1...(still a Stripe key format, not a live key)123-45-6789(SSN) →XXX-XX-7f3a(format preserved, value destroyed)
The log stays readable. Your stack traces still make sense. Your error patterns still cluster
correctly. But the actual sensitive value is gone, replaced by something that looks right
without being real.
correctly. But the actual sensitive value is gone, replaced by something that looks right
without being real.
The replacement is also deterministic: if the same email appears 100 times across 100 log
entries, it always maps to the same synthetic value. That means you can still trace a user's
session through the logs , you just can't identify who they are without the mapping key.
entries, it always maps to the same synthetic value. That means you can still trace a user's
session through the logs , you just can't identify who they are without the mapping key.
The scrubber runs as a Kinesis Firehose data transformation Lambda — every record passes
through it before hitting S3. It checks 40+ pattern types depending on tier:
through it before hitting S3. It checks 40+ pattern types depending on tier:
- Indie tier: 30+ patterns — emails, credit cards, SSNs, API keys, JWTs, IBANs, crypto
wallets, medical IDs, Stripe keys, AWS keys, GitHub tokens, Slack webhooks, MongoDB URIs,
Nigerian phone numbers, and more - Bootstrapper and above: Adds name detection (English, Yoruba, Igbo, Arabic, Chinese,
Indian names) and address detection - Scale tier: Edge cases go to an AI resolver (via OpenRouter) that determines whether an
ambiguous value is truly PII before replacing it
AI Error Analysis
When the processor encounters an error-level log with a structured error payload, it checks the
org's tier and invokes the AI analyzer accordingly.
org's tier and invokes the AI analyzer accordingly.
For Compliance and Scale tiers, AI analysis runs synchronously before the alert fires, the
developer gets a root-cause summary in the alert email itself. For Bootstrapper tier, the
AI analyzer Lambda is invoked asynchronously after the alert, so the alert isn't delayed but
the analysis appears in the dashboard shortly after.
developer gets a root-cause summary in the alert email itself. For Bootstrapper tier, the
AI analyzer Lambda is invoked asynchronously after the alert, so the alert isn't delayed but
the analysis appears in the dashboard shortly after.
The deduplication system took a while to get right. Errors cluster, the same bug can fire
thousands of times per minute. I implemented two layers: in-memory dedup using a sync.Map
per Lambda invocation, and cross-invocation dedup using a DynamoDB conditional write that acts
as an atomic claim. Whichever Lambda instance wins the claim does the analysis. Everyone else
skips it.
Live Tail — WebSocket Streaming
This is the feature that makes the demo visceral.
Open the Logs page, enable Live Tail, and watch your logs appear in the browser the moment
they're ingested — not on a refresh, not on a timer. In real time.
they're ingested — not on a refresh, not on a timer. In real time.
It runs on AWS API Gateway WebSocket API. On connect, the browser authenticates with the
user's session. On subscribe, it sends the project ID it wants to watch. On the processor
side, after each batch is written, the broadcaster Lambda is invoked , it scans active
connections in DynamoDB, matches them to the incoming project ID, and pushes matching logs
over WebSocket using
user's session. On subscribe, it sends the project ID it wants to watch. On the processor
side, after each batch is written, the broadcaster Lambda is invoked , it scans active
connections in DynamoDB, matches them to the incoming project ID, and pushes matching logs
over WebSocket using
PostToConnection.Connection lifecycle management had a gotcha: WebSocket connections go stale when a user closes
their laptop. The broadcaster catches GoneException (HTTP 410) responses and automatically
removes the dead connection from DynamoDB.
The AWS Production Stack
| Service | Role in LuminaLog |
|---|---|
| AWS Lambda | Serverless microservices (Go): API endpoints, PII scrubbing, AI analysis, alerting, and WebSocket broadcasting |
| Amazon API Gateway | HTTP front-door for rapid log ingestion and WebSocket API for real-time Live Tail streaming |
| Amazon SQS | Decoupling the fast ingestion API from the intensive PII scrubbing processor |
| Amazon Kinesis Firehose | High-throughput delivery stream for micro-batching and writing scrubbed logs |
| Amazon S3 | Durable, partitioned long-term storage for all log data and Terraform state |
| Amazon DynamoDB | Fast NoSQL state: API keys, WebSocket connections, error fingerprints, limits, and AI claim locks |
| Amazon Comprehend | ML-based natural language processing for auditing ambiguous PII edge cases (Scale tier) |
| AWS Secrets Manager | Secure storage and runtime injection for external credentials (ZeptoMail, AI API keys) |
Everything is Terraform-managed, split into modules: ingestion, storage, database,
websocket-api, authorizer, ai-analyzer, alert-dispatcher, pii-scrubber, billing-alert-handler,
subscription-processor, and more.
websocket-api, authorizer, ai-analyzer, alert-dispatcher, pii-scrubber, billing-alert-handler,
subscription-processor, and more.
Email notifications (alerts, invitations, billing warnings) run through a shared internal Go
package that supports SES and ZeptoMail, including a dual-send migration mode. The abstraction
means swapping providers doesn't touch any service code.
Accelerated by AWS Kiro
Building a distributed, privacy-first observability platform is typically a multi-engineer effort. To accelerate development, I relied heavily on AWS Kiro, Amazon's agentic AI IDE, to build out the infrastructure. Kiro's agentic capabilities were instrumental in rapidly scaffolding the complex Terraform modules, authoring the Go Lambda functions, and debugging the event-driven architecture. Kiro allowed me to translate architectural requirements into production-ready, deployed AWS infrastructure significantly faster than traditional workflows.
Zero-to-Scale Pricing (The $200 Free Tier Advantage)
A core tenet of LuminaLog is that startups shouldn't be priced out of compliance. Despite the comprehensive feature set, this entire stack was explicitly designed to operate comfortably within the standard AWS Free Tier or initial $200 startup credits.
Because the architecture is 100% serverless and scales to zero—relying entirely on API Gateway, Lambda, SQS, Kinesis Firehose, and DynamoDB (On-Demand)—there are zero fixed compute costs. You only pay for what you actually use. There are no idle EC2 instances to maintain, no expensive NAT Gateways, and no standing database clusters demanding hourly fees. A startup can launch with enterprise-grade observability and pay mere pennies until their traffic meaningfully scales.
Demo
Dashboard Overview
Logs View(With Live tail)
AI debugging
PII scrubbing view
Api Keys
Billing page(with plans and pricing)
What I Learned
1. Privacy is architecture, not a feature.
I tried adding PII scrubbing after the fact in an early prototype. It's messy, unreliable, and
gives you false confidence. The right approach is to make scrubbing a mandatory pipeline stage —
data never moves forward without passing through it. Once I restructured the ingestion pipeline
around that principle, everything else got cleaner.
I tried adding PII scrubbing after the fact in an early prototype. It's messy, unreliable, and
gives you false confidence. The right approach is to make scrubbing a mandatory pipeline stage —
data never moves forward without passing through it. Once I restructured the ingestion pipeline
around that principle, everything else got cleaner.
2. Redaction kills debuggability. Replacement preserves it.
My first version did simple
My first version did simple
[REDACTED] replacement. A stack trace with [REDACTED] where the user ID should be is useless. Switching to deterministic synthetic replacement, same input always produces the same fake output, was the unlock. The log stays readable. The PII is gone.3. The deduplication problem is harder than it looks.
I shipped AI analysis without proper deduplication and immediately hit a loop where a single
recurring error generated hundreds of AI calls in minutes. The two-layer approach (in-memory
sync.Map + DynamoDB conditional write) was the fix, but it took two failed attempts to land it.
Distributed systems are humbling.
I shipped AI analysis without proper deduplication and immediately hit a loop where a single
recurring error generated hundreds of AI calls in minutes. The two-layer approach (in-memory
sync.Map + DynamoDB conditional write) was the fix, but it took two failed attempts to land it.
Distributed systems are humbling.
4. Serverless cold starts are real — design around them.
The separation of the HTTP handler Lambda from the processing Lambda was partly driven by cold
start management. The HTTP Lambda stays warm (it gets traffic constantly). The processor Lambda
can cold start occasionally, but the user never feels it because the job was already queued.
The separation of the HTTP handler Lambda from the processing Lambda was partly driven by cold
start management. The HTTP Lambda stays warm (it gets traffic constantly). The processor Lambda
can cold start occasionally, but the user never feels it because the job was already queued.
5. Your pricing tiers are your product roadmap in disguise.
When I defined Indie, Bootstrapper, Compliance, and Scale, I was forced to answer: what does a
compliance-focused customer actually need that a solo developer doesn't? That question shaped
every feature decision. Tiers make you rigorous about value. You can't add a feature without
deciding who it's for.
When I defined Indie, Bootstrapper, Compliance, and Scale, I was forced to answer: what does a
compliance-focused customer actually need that a solo developer doesn't? That question shaped
every feature decision. Tiers make you rigorous about value. You can't add a feature without
deciding who it's for.
6. Building for markets that "aren't tech hubs" is a different discipline.
The product has to work on variable connectivity. It has to be priced in a way that doesn't
assume USD salaries. The support experience has to be asynchronous-first. These aren't
weaknesses ,they're design constraints that make you build something more resilient. The name
detection in SmartScrub™ covers Yoruba, Igbo, Arabic, Chinese, and Indian names specifically
because those are the users I'm building for. That's not an afterthought. It's the point.
The product has to work on variable connectivity. It has to be priced in a way that doesn't
assume USD salaries. The support experience has to be asynchronous-first. These aren't
weaknesses ,they're design constraints that make you build something more resilient. The name
detection in SmartScrub™ covers Yoruba, Igbo, Arabic, Chinese, and Indian names specifically
because those are the users I'm building for. That's not an afterthought. It's the point.
LuminaLog is live at luminalog.cloud . If you're building in a regulated space and you've been
putting off proper observability because the options felt too expensive or too complex, this is
what I built for you.
putting off proper observability because the options felt too expensive or too complex, this is
what I built for you.
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article