Building an AI Recruiter That Actually Calls
Building TimekeeperX AI Hiring Agent with Amazon Nova Sonic for automated, fair, and scalable phone screening
The Problem: Hiring is a Human Bottleneck
Every growing company faces the same challenge: screening candidates doesn't scale. A recruiter handling 50 open positions might need to conduct hundreds of phone screens per month — each one a 10–15 minute call that requires scheduling, preparation, and note-taking. The result is predictable: backlogs grow, candidates wait weeks for a response, and top talent slips away to faster-moving competitors.
But the cost isn't just efficiency — it's fairness. Research consistently shows that human phone screens are susceptible to unconscious bias. The candidate who calls at 4 PM on a Friday gets a different experience than the one who calls at 10 AM on a Tuesday. Accents, speech patterns, and even voice pitch can trigger snap judgments that have nothing to do with qualifications.
We asked ourselves: what if we could build a phone screener that is endlessly patient, always consistent, and evaluates every candidate against the same criteria?
What We Built
TimekeeperX AI Hiring Agent is a production-grade voice AI system that conducts automated phone screening interviews using Amazon Nova Sonic. It's not a chatbot — it's a voice agent that calls candidates on their actual phone, conducts a natural conversation grounded in their resume and the job description, and produces a scored evaluation.
How It Works
The system is integrated into our existing Applicant Tracking System (ATS). When a recruiter moves a candidate to the "Screening" stage:
- The candidate receives an email with a scheduling link
- They pick a time that works for them
- At the scheduled time, the AI calls their phone
- A natural, conversational interview happens — typically 3–7 minutes
- The call recording, transcript, and AI-generated score are delivered back to the recruiter
The entire flow requires one click from the recruiter and zero real-time human involvement.
Why Amazon Nova Sonic?
The key insight is that phone screening is speech-native. You can't replace a phone call with a text survey — communication skills, confidence, articulation, and cultural fit are best assessed through voice. Nova Sonic's bidirectional speech-to-speech streaming made this possible in a way that traditional speech-to-text → LLM → text-to-speech pipelines never could:
- Sub-second latency: Nova processes speech and generates responses in real time, producing natural conversational rhythm
- Interruption handling: When a candidate speaks over the AI, Nova detects the overlap and re-engages naturally — just like a human would
- Tool calling during speech: Nova can invoke functions (like ending the call) without breaking the conversational flow
- Multilingual: Our system seamlessly switches between English and Hindi based on the candidate's language
The Technical Journey: From Sample Code to Production
We started with Amazon's sample SIP server for Nova Sonic — a reference implementation designed for inbound demo calls. Turning it into a production hiring system required solving problems across every layer of the stack.
Challenge 1: SIP in the Cloud
SIP (Session Initiation Protocol) is the backbone of telephony. Unlike HTTP, it uses UDP for signaling and RTP for media streams. This means no load balancers, no NAT gateways, no container networking abstractions. Early deployments had silent audio — RTP packets were being silently dropped by cloud NAT.
Our solution: We deploy the SIP server on a dedicated Oracle Cloud VM with host networking, carefully configured firewall rules for SIP (UDP 5060) and RTP (UDP 10000–10400), and explicit handling of fragmented UDP packets.
Challenge 2: Scaling to 200 Concurrent Calls
A production hiring platform needs to handle many calls simultaneously. We built:
- An RTP port pool with 400 ports supporting 200 concurrent sessions
- Admission control that returns SIP
503when capacity is reached - Session lifecycle guards (setup timeouts, max-duration backstops)
- Call-ID based session tracking to prevent collisions
Challenge 3: Making the AI Sound Like a Recruiter
The difference between a demo and a product is in the details. We invested heavily in prompt engineering to make the AI behave like a skilled recruiter:
- One question at a time (no rapid-fire interrogation)
- Resume-grounded follow-ups ("I see you worked at X — tell me about that project")
- Natural pacing (3–7 minutes, not 30 seconds)
- Graceful call ending (the AI decides when it has enough information and hangs up)
Challenge 4: Scoring Voice, Not Just Words
If you only analyze the call transcript, you miss half the interview. A candidate's confidence, professionalism, and conversational flow are completely lost in text.
To solve this, we built a Hybrid Multimodal Analysis Pipeline:
- Technical Validation (Text): The call transcript is evaluated by the LLM against the job description and the candidate's parsed resume to verify their actual knowledge.
- Subjective Validation (Audio): The actual
.wavrecording is processed by a multimodal model (Google Gemini) that listens to the candidate's speech. It scores their fluency, communication skills, and vocal confidence.
This combined approach means technical knowledge is scored from what the candidate said, while communication skills are scored from how they said it. Together, they create a highly accurate 0-100 composite score.
Community Impact: Why This Matters
1. Democratizing Access to Fair Screening
Small and medium businesses often can't afford dedicated recruiters. They rely on whoever is available — a hiring manager squeezing in calls between meetings, or worse, simply not screening at all and making decisions based on resumes alone. AI-powered screening gives every company access to consistent, thorough candidate evaluation regardless of their HR budget.
2. Reducing Bias in Initial Screening
The AI evaluates every candidate against the same criteria: job requirements and resume alignment. It doesn't have bad days. It doesn't form impressions based on accents or names. While no system is perfectly unbiased, removing the most variable part of the process — a tired human making gut decisions — is a meaningful step toward fairer hiring.
3. Respecting Candidates' Time
With AI screening, candidates get to choose their interview time, receive the call exactly when scheduled, and get a decision faster. No more "we'll get back to you" followed by weeks of silence. The system processes results in minutes, not days.
4. Multilingual Accessibility
Our system automatically adapts to the candidate's language. A candidate who is more comfortable speaking Hindi can do so, and the AI will seamlessly switch — without requiring the company to hire bilingual recruiters. This is particularly impactful in diverse markets like India, where we operate.
5. Enabling Remote and Rural Hiring
Phone screening works everywhere there's a phone signal. Unlike video interviews that require good internet, or in-person screens that require proximity, AI phone screening can reach candidates in rural areas, small towns, and regions with limited connectivity. This expands the talent pool for companies and opens opportunities for candidates who would otherwise be overlooked.
Architecture at a Glance
High-level architecture of the TimekeeperX AI voice screening pipeline, highlighting the integration between the cloud SIP server, Amazon Nova Sonic, and the post-call multimodal scoring engines.
Real-World Application
TimekeeperX is not a hackathon prototype — it's a module within an actively developed HR SaaS platform used by organizations in India. The AI Hiring Agent integrates with:
- Requisition management (approval workflows, budget tracking)
- Job posting (career pages, multi-channel distribution)
- Resume parsing (OCR, text extraction, vector embeddings)
- Interview scheduling (Google Meet, Webex integration)
- Offer management (template generation, e-signatures)
- Onboarding (document verification, task tracking)
The voice screening module slots into this existing pipeline, turning what was a manual bottleneck into an automated, measurable step.
Encouraging Adoption
We believe AI-powered voice screening will see adoption through three vectors:
- Start with high-volume roles. Customer support, retail, and BPO positions require hundreds of screens per month. The ROI is immediate and measurable.
- Transparent AI disclosure. We instruct the AI to identify itself as a hiring agent. Candidates know they're talking to AI, which builds trust and sets appropriate expectations.
- Human-in-the-loop. The AI doesn't make final decisions — it produces scores and recommendations that recruiters review. This keeps humans accountable while eliminating the drudgery of initial screening.
- Open architecture. Our SIP server can connect to any SIP trunk provider, making it adaptable to different telephony setups and regions. The system prompt is fully customizable per job, per company.
What's Next
- Multi-round screening — technical, cultural, and behavioral rounds with different AI personas
- Expanded language support — regional Indian languages, Spanish, Arabic
- Candidate experience scoring — let candidates rate their experience to improve the AI
- Video screening — extend to WebRTC-based video interviews
Conclusion
The hiring industry is ripe for AI transformation, and voice is where it matters most. Amazon Nova Sonic gave us the real-time, speech-to-speech capability needed to build something that doesn't just sound good in a demo — it works in production, at scale, with real candidates receiving real phone calls.
If a single click can replace weeks of phone tag between recruiters and candidates, while making the process fairer and faster for everyone involved, that's a future worth building.
Built with Amazon Nova Sonic, Amazon Bedrock, AWS (ECS, S3, CloudFront, Route 53, ElastiCache, SES, WAF, Textract), Oracle Cloud Infrastructure, Node.js, Next.js, MongoDB, SignalWire, SIP/RTP, BullMQ, Docker, Terraform, and GitHub Actions.
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article