AWS Builder Center

Builder Center

AIdeas: Qleam AI - A Self-Learning Baby Cry Intelligence

AIdeas: Qleam AI - A Self-Learning Baby Cry Intelligence

Qleam AI is an AI-powered baby cry analysis and early childhood communication system designed for modern parenting. Using machine learning and self-learning models, it analyzes short audio recordings of baby sounds (0–24 months) to identify patterns and signals, then translates them into clear, structured insights.

Tanvir Ahmed

Lead Engineer (DevOps & Cloud)
Team Name: Qlitch
Members:
Tanvir Ahmed
Rahat Mahmud

App Category

Daily Life Enhancement

My Vision

Imagine a soul trapped in a parallel universe, trying to reach you. Crying, calling, seeking help, sending nothing but signals.
In that world, nothing matters except a few familiar faces they remember and hope will understand. Maybe they are hungry, in pain, uncomfortable, or simply want to be held for a moment so they can finally fall asleep in peace.
But their only language is sound. Within every cry are patterns, signals of emotion and need that are often too subtle for the human ear to fully interpret.
Qleam AI helps uncover these signals by analyzing emotional patterns and acoustic cues, supporting caregivers in recognizing what might otherwise be missed.

Why This Matters

We are building an AI-powered world.
Everything around us is accelerating; innovation, automation, even our dreams of reaching Mars. Yet in this fast-moving reality, one of the most fundamental forms of human communication often remains locked inside a small box. Not because we do not care, but because we struggle to understand it.
Babies constantly send emotional signals through sound, rhythm, tone, and behavior. But these signals are often difficult to interpret, especially during the first intense years of life.
For caregivers, crying can be frequent, ambiguous, and emotionally overwhelming. Nights become cycles of uncertainty; trying to guess whether the baby is hungry, uncomfortable, tired, or in need of comfort.
Qleam AI matters because it brings clarity to these everyday moments.
  • It reduces trial-and-error in immediate care decisions.
  • It offers a consistent interpretation framework instead of random guesswork.
  • It creates a feedback loop where the system improves through real caregiver-confirmed outcomes.
This is Daily Life Enhancement in practice: less confusion, faster response, and greater emotional confidence for families.

How I built this

Architecture Overview

Qleam AI runs on a serverless AWS architecture built for low-latency audio inference and continuous model improvement.
The React frontend records short audio samples (typically 5-30 seconds) and uploads them to Amazon S3 through pre-signed URLs. API Gateway + Lambda handle authentication, session lifecycle, and ownership checks.
AWS Step Functions orchestrates two core stages:
  • feature_extraction: decoding, quality gating, diarization, acoustic computation, and optional embedding inference
  • insight_generator: cry/speech/laugh routing, emotion synthesis, and final parent-facing insight payload
Learning services run as separate control loops:
  • feedback_processor: validates caregiver confirmations/corrections
  • training_check: evaluates retraining readiness from confirmed sample growth
  • model_trainer: load -> train -> validate -> promote model versions

AWS Services Used

Frontend & Auth: React (Amplify UI) + Cognito (JWT) + CloudFront
API Layer: API Gateway → Lambda
Orchestration: Step Functions + EventBridge
Storage: S3 + DynamoDB
AI/Processing: Transcribe + Bedrock + SageMaker
Infrastructure: VPC + NAT Gateway + IAM + ECR
Monitoring & Ops: CloudWatch + X-Ray + SNS + SSM
Qleam Architecture and Data flow


Service Roles Justification

API Gateway – Exposes and secures backend APIs
Lambda – Executes backend logic with serverless scaling
Step Functions – Orchestrates audio and training workflows
S3 – Stores audio files, assets, and model artifacts
DynamoDB – Provides low-latency NoSQL data storage
Cognito – Manages authentication and JWT-based access
CloudFront – Delivers frontend with caching and CDN
VPC + NAT Gateway – Ensures network isolation and outbound access
IAM – Controls service permissions (least privilege)
ECR – Stores and versions container images
CloudWatch – Collects logs, metrics, and triggers alerts
SNS – Sends notification alerts from monitoring
EventBridge – Triggers scheduled and event-driven workflows
SSM Parameter Store – Stores configuration outside application code
Transcribe – Converts speech to text for analysis
Bedrock – Generates AI-based insight output
SageMaker – Performs advanced acoustic inference
X-Ray – Provides tracing for debugging and performance

Optimization Strategy

DynamoDB on-demand billing (PAY_PER_REQUEST)
S3 lifecycle policies (archive + expiration)
CloudFront cost optimization (PriceClass_100)
ECR lifecycle (retain last 10 images)
Log retention (30 days for DEV)
VPC endpoints for S3 and DynamoDB
Advanced inference disabled by default

Cost Estimation

Assumptions :
10,000 requests/month
Average audio length: 20–30 seconds
Transcribe billed per second ($0.0004/sec)
SageMaker inference: 20 sec/request (3GB)
DEV includes full infrastructure baseline
Cost Component0 Traffic (USD/month)10,000 Requests – Basic Mode (USD/month)10,000 Requests – Advanced (USD/month)
VPC (NAT + IPv4)36.5036.5036.50
NAT Data0.000.900.90
S3 (Storage + Requests)0.460.520.52
DynamoDB (Ops + Backup)0.200.220.22
API Gateway0.000.040.04
Lambda0.000.000.00
Step Functions0.001.151.15
CloudWatch4.304.304.30
ECR0.800.800.80
Bedrock0.003.133.13
Transcribe0.0075.0075.00
SageMaker0.00—13.24
Other (Cognito, SNS, etc.)0.000.000.00
TOTAL42.26122.56135.80

System Scope and Runtime Modes

Qleam AI right now is a web application for parent-guided baby sound analysis for children aged 0-24 months.
Core behavior:
  • Parent records audio in browser (audio/webm).
  • Audio is uploaded to S3 via pre-signed URL.
  • AWS Step Functions orchestrates classification and insight generation.
  • Session insight is stored in DynamoDB and shown in frontend.
  • Optional parent feedback can convert a session into confirmed training data.
Runtime AI modes:
  • Basic: rule-based cry emotion path (no HuBERT call).
  • Advanced: HuBERT embedding extraction + classifier path for cry/mixed when endpoint and flag are enabled.
Mode decision source:
  • API /status reports ai_mode = "Advanced" only when:
    • USE_SAGEMAKER_INTENT_ENDPOINT = true
    • SAGEMAKER_HUBERT_ENDPOINT is non-empty
  • Otherwise ai_mode = "Basic".
Lambda responsibilities:
  • api_handler: Auth, child/session CRUD, upload/start endpoints, feedback forwarding, status
  • feature_extraction: Audio decode, quality/reject gates, diarization, sound classification, adult detection, optional HuBERT/classifier
  • insight_generator: Final parent-facing insight assembly and session update
  • feedback_processor: Feedback persistence + training anonymization gates
  • training_check: Scheduled threshold check for retraining
  • model_trainer: Load/train/validate/promote emotion classifier weights
Step Functions
  • Processing pipeline: AudioClassifier -> (FastReject Choice) -> InsightGenerator -> Succeed
  • Training pipeline (enabled in dev Terraform root): TrainingCheck -> Load -> Train -> Validate -> Promote


Session-Time Inference Pipeline

Each recording passes through a structured inference graph.
1. Quality gating first Audio is decoded and tested for duration, silence ratio, clipping, voiced activity, and SNR. If quality is poor, Qleam AI returns a rejection insight instead of a potentially misleading interpretation.
2. Acoustic preprocessing Energy-based diarization isolates likely baby segments. Lightweight denoising and pre-emphasis improve signal quality before feature extraction.
3. Acoustic intelligence path (deterministic) The system computes interpretable features such as:
  • f0_mean, pitch instability, voiced fraction
  • RMS/energy variability
  • spectral centroid/flatness/rolloff
  • rhythmic signals (syllable rate, burst behavior)
    These features drive sound-type routing (cry, speech, laugh, silence, noise, mixed) and rule-based cry analysis.
4. Advanced embedding path (model-based) For cry-relevant audio, HuBERT on SageMaker produces frame embeddings and mean-pools them into a 768-dimensional representation. That vector feeds the emotion classifier.
This path runs only when all are true:
  • USE_SAGEMAKER_INTENT_ENDPOINT = true
  • SAGEMAKER_HUBERT_ENDPOINT configured
  • sound_type in ("cry", "mixed")
Then:
  1. HuBERT embedding is requested from endpoint.
  2. If embedding shape is 768, classifier predicts emotion.
  3. TrainingFeatures candidate is stored (S3 .npy + DDB metadata).
If HuBERT/classifier call fails and advanced path is unavailable, Qleam AI automatically falls back to deterministic acoustic analysis.

Insight Generation

Insights combine probabilistic modeling with explainable acoustic evidence.
  • If classifier output is valid, emotion is selected from model probabilities.
  • If not, rule-based cry scoring compares acoustic features against emotion profiles and normalizes scores.
The insight payload includes:
  • emotion distribution and confidence
  • acoustic evidence profile (pitch, intensity, stability, brightness, variation)
  • alternatives/ambiguity signals
  • caregiver-facing guidance
For visualization, values are normalized into emotion radar and acoustic profile charts. This keeps output interpretable and auditable rather than black-box only.
Failure Semantics:
  • Controlled reject (quality/adult/noise gates) -> reject insight is generated.
  • Hard pipeline failure before insight write -> session can remain in processing state until timeout/retry logic at client side.

Self-Learning Dataset and Daily Growth

Qleam AI uses a feedback-verified learning pipeline instead of training directly on raw session predictions.
Each eligible cry session can create a training candidate containing:
  • embedding vector reference
  • acoustic feature summary
  • predicted label + confidence
  • quality metadata
Candidates become confirmed training data only after gate checks pass:
  • valid caregiver feedback
  • minimum audio quality
  • correct sound-type scope
  • confidence-policy checks for "was correct"
  • deduplication protection
After acceptance, records are anonymized and promoted into the training pool.
Retraining is day-scheduled and threshold-based (minimum pool size + new confirmed sample growth), so model updates are controlled rather than noisy.
Candidate creation:
Candidates are generated only when:
  • session is cry or mixed
  • advanced HuBERT path executes
  • 768-d embedding is produced
Output:
  • embeddings in S3: training-features/{feature_id}/embeddings.npy
  • metadata row in TrainingFeatures
Feedback intake and gates:
API sends cry feedback to feedback_processor.
Applied gates (apply_quality_gates):
  1. Embeddings exist (TrainingFeatures row found by session_id)
  2. Confirmed emotion is valid (not skip/not_sure/unknown)
  3. Quality threshold: SNR >= 10, duration >= 3, sound_type in (cry,mixed)
  4. Confidence gate for confirmations: if was_correct=true, model confidence must be >= 0.4
  5. Dedup: same session cannot be confirmed twice
Gate behavior nuance:
  • was_correct=false corrections bypass gate #4 and can be accepted if other gates pass.
Anonymization:
When accepted
  • Adds confirmed label fields
  • Sets is_confirmed=true
  • Removes session_id from TrainingFeatures
  • Keeps de-identified fields needed for model training and audit
Retraining trigger:
training_check thresholds:
  • total confirmed must be >= 30
  • trigger when either:
    • new confirmed since last training >= 50, or
    • new confirmed >= 10 and at least 7 days since last training
Trainer and promotion:
model_trainer steps:
  1. Load confirmed embeddings + age encodings
  2. Train two-branch model (embedding branch + age branch)
  3. Validate against active model
  4. Promote if improved by margin (+0.02), or bootstrap rule when no active model and accuracy >= 0.30
Promotion writes:
  • S3 model artifact: models/emotion_classifier/v{N}/model_weights.npz
  • ModelVersions record with active=true
  • marks features with included_in_training
Legacy Research Datasets and Modernization
Qleam AI starts from pre-2025 public cry research priors (baseline signal behavior), then modernizes performance with continuously confirmed real-world samples.
In production behavior, the model confidence layer is not static:
  • legacy baseline stabilizes early-stage behavior
  • confirmed daily samples progressively increase influence
  • blending gradually shifts weight toward self-learned data as evidence accumulates
This avoids discarding prior science while preventing the model from staying frozen in controlled-lab assumptions.
Research/data governance note: when additional external datasets are licensed and approved, they can be integrated as calibrated baseline layers. The long-term objective is stronger alignment with Qleam AI's own verified dataset, not replacement-by-claim.


Data Deletion and Privacy

Deletes direct child's personal data:
  • all child sessions
  • feedback rows linked to each session
  • S3 audio objects under {child_id}/{session_id}/
  • child profile row
Not deleted by this operation:
  • de-identified TrainingFeatures already anonymized for shared model training
  • ModelVersions shared model metadata

Demo

👉 ai.qleam.com 
Try me Now

What I Learned

I learn five practical lessons while building Qleam AI:

  • Real-world AI quality depends on data quality gates as much as model design.
  • Explainable outputs build user trust faster than black-box predictions.
  • Feedback-confirmed samples are the highest-value signal for improvement.
  • A hybrid design (deterministic acoustic logic + advanced embeddings) is more reliable in production.
  • Continuous retraining with validation/promotion rules improves performance without destabilizing the system.
  • Serverless architecture helps scale quickly while keeping operations simple.

Technologies I learned and strengthened

  • Building end-to-end serverless workflows with AWS Lambda, API Gateway, and Step Functions.
  • Designing secure media ingestion and artifact pipelines with Amazon S3 + pre-signed URLs.
  • Structuring model and session state for production using Amazon DynamoDB.
  • Applying audio ML inference patterns with SageMaker HuBERT embeddings.
  • Implementing feedback-driven model lifecycle logic: load -> train -> validate -> promote.
  • Building production-ready frontend-to-cloud integration with React + Cognito auth flows.
Thank you, appreciate your time.
"Think, think, practice thinking"
Ritwik Ghatak
# aideas-2025# daily-life-enhancement# APJC# ai-ml# social-impact
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article