MedScribe gives AI exactly one job, turning a doctor's spoken consultation into a faithful structured record, then lets plain data drive the PDF, the voice note, the reminders and the patient's questions. Two people, three days, thirteen managed services, no servers.
The idea
AI gets one job in MedScribe: listen to unstructured speech and produce a structured record of what was said, without inventing, correcting or deciding anything. Deterministic code verifies that record. The doctor approves it. From then on, every feature in the product is a plain function of the record. No feature calls the model to "figure out" what the prescription means; the prescription already means something, precisely, because it is data.
We drew that line after watching the alternative fail. Ask a language model to turn a consultation into a prescription and it will do it beautifully, right up to the moment it does it wrong. Say "Dolo six-fifty le lena" to a fuzzy matcher with a generous threshold and it hands you a blood-pressure tablet with a similar-sounding name, and nothing in the output tells you it happened. For anything that touches a prescription, the output is either verifiable or it is dangerous.
What the record is
By the time the doctor presses Approve, a consultation has become:
- Symptoms, diagnosis, tests advised, next visit, as the doctor stated them. The diagnosis is inferred only when the symptoms make a common one obvious, and only from a fixed condition list.
- Medicines, each with the name exactly as heard, the verified brand it resolved to, that brand's salts, dose, timing, food relation and duration, plus the resolver's confidence, the clinical verdict against the diagnosis, and whether the doctor edited, confirmed, added or removed it.
- Provenance: which transcript lines it came from, what the doctor changed, and for any brand that was not on the verified list, the catalogue evidence it was admitted on.
Everything below is derived from that object.
How the record is made: AI with its hands tied

Speech. The browser streams the microphone to Sarvam AI's Indian-language speech-to-text; transcript lines arrive over an API Gateway WebSocket. Brand names come back in Latin script, which hinged on keeping the STT prompt short.
Extraction on Amazon Bedrock. Every twelve seconds the full transcript goes to OpenAI's open-weight gpt-oss-120b through Bedrock's OpenAI-compatible endpoint. The model returns the fields above with each medicine name verbatim; any id it emits is stripped. We chose it empirically: five candidates on identical Hinglish transcripts, scored on rule violations (inventing a brand, dropping a medicine, normalising a name). gpt-oss-120b was the only one at zero, at about 1.7 seconds per call. Bedrock's account-level "no data retention" setting satisfies the clinical requirement outright.
Verification in plain code. A phonetic-plus-fuzzy resolver matches each spoken name against a curated list of Indian brands, salts and conditions; a validator checks duplicate salts, fit with the diagnosis and look-alike names. Anything unresolved stays red and blocks approval. When the list has no credible match, the row looks the name up in a live pharmacy catalogue and shows the doctor the spelling, composition, therapy class and a source link; the model's memory is only a fallback. Speech mangling that used to dead-end now resolves: "New Sporin" → Neosporin, "Zenflex UTI" → Zenflox UTI.
Merging without losing anything. Each extraction pass is folded into the prior draft by one pure function. This is the whole safety model in fifteen lines:
def process(transcript_text, prior_state=None, client=None, lines=None) -> dict:
"""`lines` lets a medicine the doctor removed come back when it is spoken
again later; without it, removal is final."""
prior = prior_state or empty_draft()
try:
ex = extract(transcript_text, client=client)
except LLMError as e:
return {**prior, "extraction_error": str(e)} # fail-safe: last good draft
state = dict(prior)
for f in TOP_LEVEL_FIELDS:
if f not in prior.get("locked_fields", []): # doctor edits win
state[f] = ex.get(f)
new_meds = [_resolve_one(m, _resolver_instance()) for m in ex["medicines"]]
state["medicines"] = _merge_medicines(prior["medicines"], new_meds, lines) # union + tombstones
return validate_state(state) # deterministic clinical checksThe model can fail, stutter or change its mind between passes; the draft only ever moves forward, and nothing the doctor touched moves at all.
What the record unlocks, with no further AI
Once approved, the rest is derivation. Each row is a small deterministic function of the same object, and each maps onto a managed service.
Derived from the record | How | Service |
|---|---|---|
Signed prescription PDF | Template over the record; digitally-signed stamp with the approval time | AWS Lambda + WeasyPrint layer, Amazon S3 |
Voice note in the patient's language | Script generated from the dose fields, not by the model; Hindi or English | Amazon Polly (Kajal, bilingual) |
Dose chips and "how to take" | Timing codes expanded per language | AWS Amplify Hosting (patient app) |
"Know your medicine" on WhatsApp | Salts, class and indications from the salt table the record points to | Amazon DynamoDB |
Alternative brands | Same-salt lookup across the brand table | Amazon DynamoDB |
Order links, nearest pharmacy | Brand names from the record; patient's shared location | Meta WhatsApp Cloud API via Lambda |
Dose reminders and TAKEN replies | One schedule per dose from frequency and duration; mutes honoured | Amazon EventBridge Scheduler |
The patient's questions | A records-only assistant that answers from their own prescriptions and refuses to guess | Amazon Bedrock over the record only |
Follow-up booking | Doctor and clinic from the record; date and time parsed from Hinglish text or a voice note into a fixed form | Amazon Bedrock (form-filling), Amazon DynamoDB |
A better verified list | A brand the doctor added is written on approval with salts, composition, aliases, source and evidence, and resolves instantly for every doctor thereafter | Amazon DynamoDB, re-indexed at runtime |
Notice what this buys. The voice note cannot say a medicine the PDF does not show. "Know your medicine" cannot describe a salt the prescription does not contain. The patient's assistant cannot invent an instruction, because it only has the record to read. When one object feeds everything, consistency is not a feature you build; it is a property you get.
Control: one write, one approver, one audit trail
Approval is the only event that sends anything to the patient and the only event that admits a new brand to the verified list, with the doctor's identity and the catalogue evidence recorded beside it. The post-approval pipeline is an AWS Step Functions state machine: FinalExtract → ResolveValidate → gate → RenderPDF → RenderAudio → PersistRx → SendPrescription. Retries are declarative, every step is idempotent, and the execution history is the audit trail.
Image: the Step Functions execution graph
Five services that carried the most weight
- Amazon Bedrock. One
converse_jsoninterface serves extraction, diagnosis inference, appointment parsing and catalogue fallback, so the provider is a flag. Two things to know: reasoning models count hidden reasoning againstmax_tokens(a 1,024 cap silently truncated a nine-medicine consult; 4,096 withreasoning_effortlow fixed it and cut latency by two thirds), and design the caller so the model is optional.
- AWS Step Functions. The pipeline above maps one-to-one onto states; a failure returns the consult to the doctor with the reason. Median Approve-to-WhatsApp time across the hackathon's runs: 23 seconds.
- Amazon DynamoDB. A single table for patients, doctors, daily-token links, consults, drafts, prescriptions, appointments and reminders, plus three reference tables. Conditional writes give exactly-once handling of webhook redeliveries; atomic counters issue tokens and meter sends. No capacity planning at any point.
- AWS SAM. One template for both APIs, six Lambda functions, the state machine, tables, bucket, Cognito pools, schedules and IAM.
sam build --use-containerandsam deploywere the entire release process, about forty times in three days.
- AWS Amplify Hosting. Three React apps built by one spec into a single HTTPS origin (
/,/doctor/,/patient/), redeployed on every push in about ninety seconds. HTTPS is not optional: microphone capture and the WebSocket both need it.
Around them: Lambda, API Gateway (HTTP with Cognito JWT authorisers, plus WebSocket), Cognito, S3, Polly, EventBridge Scheduler, Systems Manager Parameter Store for every secret, and CloudWatch. All in ap-south-1, so patient data stays in India. Everything stayed inside the free tier; the $100 credit budget was never drawn on.
Results
- Fidelity. Only one of five candidate models produced zero rule violations on identical Hinglish transcripts; with catalogue grounding, every misheard brand in our test set resolved to a real product with composition and source.
- Speed. Extraction about 1.7 s; live refresh every 12 s; Approve to PDF-on-WhatsApp median 23 s, fastest 9 s, over 31 pipeline runs.
- Usage during the hackathon. 232 consultations, 32 approved prescriptions, 89 WhatsApp messages sent and 19 received (voice notes included), 3 brands admitted to the verified list on approval.
Things worth knowing before you build one
- Make the record first; make it boring. Decide the schema before the prompt. Every downstream feature got cheaper the moment the record was fixed.
- Keep the spoken name verbatim. Every attempt to clean names before matching reduced accuracy. Match on a transliterated, unit-stripped copy; store what was heard.
- Ground unknowns in a source, not in the model. A catalogue with a composition and a URL beats the model's memory; its first guess for "New Sporin" was a diuretic.
- Derive, don't re-ask. If a feature can be computed from the record, compute it. The model runs after approval in exactly two places, and both are constrained to a schema.
- Test offline, verify live. 257 tests run against moto with no account, and the safe core was green before the account saw a request. They caught almost everything; the live WhatsApp round trip caught the rest.
Team
- Jasmeet Singh: architecture and AWS infrastructure, the WhatsApp integration, accounts and the clinic queue, the three front ends and hosting.
- Anshika Sharma: the extraction layer and draft merge, resolver and validator integration, the approval pipeline (PDF, voice, delivery), reminders, the WhatsApp menu and booking, and the test suite.
Try it
- Youtube: https://youtu.be/kh0xCpTxhdY
#amazon-bedrock #serverless #aws-lambda #aws-step-functions #amazon-dynamodb #amazon-polly #aws-amplify #hackathon #healthcare
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article