Automate Insurance Document Intake with Amazon Bedrock Data Automation
Serverless pipeline that parses insurance emails and uses Amazon Bedrock Data Automation to split, classify, and file PDF attachments into Salesforce.
Insurance carriers and distributors process a constant stream of policy applications that arrive by email. A broker sends an email to an intake mailbox with a single PDF attachment — except that PDF often bundles several documents together: an annuity application, a life insurance application, and supporting documents like beneficiary forms or identification. Someone on the operations team has to open each email, scroll through the pages to figure out where one document ends and the next begins, split the file, classify each piece, create a case in Salesforce, and upload the documents individually.
This article shows a serverless pipeline that does all of that automatically. It parses inbound emails, deduplicates them, uses Amazon Bedrock Data Automation (BDA) to split and classify PDF attachments, and creates a Salesforce Case with each classified document attached. The full source code is available on GitHub sample-insurance-email-intake-bedrock-data-automation , so you can deploy it in your own account and adapt it to your workflow.
The Problem
A typical broker submission looks like this: an email lands in an intake mailbox with a subject like "Application Package - Annuity + Life Insurance" and a single PDF attached. That PDF might contain a 2-page annuity application followed by a 3-page life insurance application, with no clear separator. A human has to recognize the boundaries, split the file, label each part, and create the right records downstream.
The goals for automating this worflow:
- Detect and skip duplicate emails so we never create the same case twice
- Split combined PDFs into individual documents at the correct page boundaries
- Classify each document (annuity application, life insurance application, supporting document)
- Create a CRM case with the classified documents attached, in a single atomic operation
- Keep the whole thing serverless, auditable, and cheap to run
Architecture Overview

The pipeline is event driven and uses a small set of managed services:
The services involved:
- Amazon S3 stores raw emails, extracted attachments, and split documents. An object-created event triggers the pipeline.
- AWS Lambda runs three focused functions rather than one monolith.
- AWS Step Functions orchestrates the asynchronous wait for BDA using a native Wait state, so no Lambda sits idle burning compute.
- Amazon Bedrock Data Automation does the AI-powered document splitting and classification.
- Amazon DynamoDB holds a deduplication registry with a TTL so entries expire automatically.
- AWS Secrets Manager stores Salesforce credentials.
- Salesforce Composite REST API creates the case and uploads documents in one transactional call.
Why Step Functions Instead of One Big Lambda
The first version of this pipeline was a single Lambda that invoked BDA and then polled for completion in a loop with
time.sleep(15) . That works, but it has real drawbacks. The function pays for compute while it sleeps, it risks hitting the 15-minute Lambda timeout on large documents, and there is no visibility into which step failed when something goes wrong.Moving the orchestration to Step Functions solves all three. BDA is asynchronous, so the natural pattern is: start the job, wait, check status, and loop until it completes. Step Functions expresses this with a
Wait state (which costs nothing while waiting), a Check Status task, and a Choice state that either proceeds, fails, or loops back to wait. Each Lambda now runs for seconds instead of minutes, retries are declarative, and the execution graph in the console shows exactly where any failure occurred.The three Lambdas each have a single responsibility:
- Parse Email — triggered by S3, it parses the MIME message, checks DynamoDB for duplicates, extracts attachments to S3, invokes BDA asynchronously, and starts the Step Functions execution.
- Check BDA Status — a lightweight function called repeatedly in the wait loop; it just returns a normalized status.
- Process Results — once BDA finishes, it parses the output, physically splits the PDF by the page indices BDA returns, creates the Salesforce Case, and marks the email processed.
How It Works, Step by Step
When a
.eml file lands in the raw-emails/ prefix of the S3 bucket, S3 fires an event that invokes the Parse Email Lambda. The function reads the raw bytes and parses the MIME structure. Email attachments travel as base64-encoded MIME parts inside the message, so the PDF is embedded directly in the .eml file — there is nothing to upload separately.Before doing any work, the function computes a SHA-256 hash from the email metadata and attachment checksums, then checks DynamoDB. If the hash already exists, the email is a duplicate and processing stops immediately. This is what keeps the system from creating duplicate cases when a broker resends the same package or when S3 delivers an event more than once.
Assuming it is new, the function extracts each attachment and writes it to S3 under
attachments/{email_id}/ . Attachment filenames come from untrusted input, so they are sanitized to strip path-traversal characters before being used in an S3 key. The function then calls InvokeDataAutomationAsync to start the BDA job and hands the invocation ARN to Step Functions.Step Functions takes over the waiting. If there were no PDFs (or BDA is not configured), it routes straight to Process Results with the attachments passed through unsplit. Otherwise it enters the Wait → Check → Choice loop. The Wait state pauses 15 seconds at no cost, the Check Status Lambda queries BDA, and the Choice state decides whether the job is complete, has failed, or is still running. When BDA reports completion, it moves on to Process Results.
Process Results reads the BDA output metadata, which tells it where each logical document begins and ends and how it was classified against the blueprints you configured (for example, "Annuity Policy Application" or "Life Insurance Application"). Using the page indices BDA returns, the function physically splits the original PDF into separate files with
pypdf , uploads each to split-docs/ , and then calls the Salesforce Composite API. That single call creates a Case and attaches every classified document atomically with allOrNone set to true, so either everything succeeds or nothing does. Finally, the email hash is written to DynamoDB to prevent reprocessing.Deployment
Step 1: Deploy Infrastructure
cd infrapython3 -m venv .venvsource .venv/bin/activatepip install -r requirements.txt
cdk bootstrap # First time onlycdk deployNote the outputs:
-
BucketName— your S3 bucket
-
ParseEmailFunctionName— the trigger function
-
StateMachineArn— the Step Functions state machine
-
DedupTableName— DynamoDB table
Step 2: Configure Bedrock Data Automation
2a: Create Blueprints (AWS Console)
- Go to Amazon Bedrock → Data Automation → Custom output setup
- Click "Create Blueprint"
- Upload
samples/annuity_policy_application.pdfas a sample document
- Enter this blueprint prompt:
Extract the following from this annuity policy application:- Full name of the annuity applicant- Date of birth of the applicant- Type of annuity selected (Fixed, Variable, Indexed, Immediate)- Initial premium or funding amount- Name of the primary beneficiary- Policy or application number if present- Whether the applicant signature is present on the form- Click "Generate blueprint"
- Name it:
AnnuityPolicyApplication
- Save the blueprint
Repeat for Life Insurance:
- Click "Create Blueprint" again
- Upload
samples/life_insurance_application.pdf
- Enter this blueprint prompt:
Extract the following from this life insurance application:- Full name of the insured/applicant- Date of birth of the applicant- Type of life insurance (Term, Whole Life, Universal, Variable)- Coverage amount (face value or death benefit amount)- Name of the primary beneficiary- Relationship of beneficiary to insured- Policy or application reference number- Whether the applicant signature is present- Click "Generate blueprint"
- Name it:
LifeInsuranceApplication
- Save the blueprint
2b: Create a Project (AWS Console)
- In Data Automation → click "Create project"
- Name it:
InsuranceEmailDocumentProcessor
- Click "Create project"
- Go to the Custom output tab → Add blueprint → add both blueprints you created
2c: Enable Document Splitting (CLI — not available in console)
Document splitting must be enabled via CLI:
aws bedrock-data-automation update-data-automation-project \ --project-arn "<YOUR_PROJECT_ARN>" \ --override-configuration '{"document": {"splitter": {"state": "ENABLED"}}}'Get your project ARN from the project details page in the console.
2d: Update Lambda with the Project ARN
aws lambda update-function-configuration \ --function-name InvokeBDA \ --environment "Variables={ BUCKET_NAME=$(aws cloudformation describe-stacks --stack-name EmailProcessingStack \ --query \"Stacks[0].Outputs[?OutputKey=='BucketName'].OutputValue\" --output text), BDA_PROJECT_ARN=<YOUR_PROJECT_ARN> }"Step 3: Configure Salesforce Integration
Generate a certificate pair:
openssl req -x509 -nodes -days 3650 \ -newkey rsa:2048 \ -keyout salesforce_private_key.pem \ -out salesforce_certificate.crt \ -subj "/CN=AWS Email Processing/O=MyCompany"Create an External Client App in Salesforce:
- Setup → App Manager → New External Client App
- Enable OAuth → add scopes:
api,refresh_token, offline_access
- Enable JWT Bearer Flow → upload
salesforce_certificate.crt
- Pre-authorize your user profile under Policies tab
Store credentials in Secrets Manager:
aws secretsmanager update-secret \ --secret-id salesforce/integration \ --secret-string "$(python3 -c "import jsonprivate_key = open('salesforce_private_key.pem').read()print(json.dumps({ 'consumer_key': '<YOUR_CONSUMER_KEY>', 'service_user_email': '<YOUR_SF_USERNAME>', 'private_key': private_key}))")"Step 4: Test End-to-End
Upload the sample email to trigger the pipeline:
BUCKET=$(aws cloudformation describe-stacks --stack-name EmailProcessingStack \ --query "Stacks[0].Outputs[?OutputKey=='BucketName'].OutputValue" --output text)
aws s3 cp samples/test_raw_email.eml s3://$BUCKET/raw-emails/test_email.emlVerify in Salesforce:
- Open Cases → you should see a new Case with split, classified documents attached
Adding SES for Production (Optional)
To receive real emails instead of manual S3 uploads:
- Verify your domain in Amazon SES
- Add MX record to DNS pointing to SES
- Create a receipt rule that delivers to
s3://<bucket>/raw-emails/
- Activate the rule set
The sample PDFs and test emails in the repository contain synthetic data generated for testing purposes and do not represent real individuals.
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article