How Step Functions Orchestration Won Me a 7,200€ CHECK24 Scholarship
A deep dive into Step Functions patterns that handle 5 unreliable APIs with nested parallelism, comprehensive error handling, and real-time streaming.
The Challenge
CHECK24's 7th round GenDev scholarship presented a specific integration challenge: build a real-time internet provider comparison platform using 5 given provider APIs with different characteristics:
- WebWunder: SOAP web service requiring XML request/response handling
- ByteMe: CSV API with known duplicate data issues
- VerbynDich: Non-standard text-based API with pagination (no proper spec)
- Servus Speed: Two-step REST workflow (get products → get details for each)
- Ping Perfect: API requiring custom HMAC-SHA256 request signing
The APIs are intentionally unreliable with expected failures and delays. The platform needed to stream results to users in real-time as each provider responds while handling partial failures gracefully.
Architecture Overview
The solution uses serverless AWS components with Step Functions handling the provider orchestration:
AWS serverless architecture demonstrating Step Functions orchestration of 5 parallel provider API calls with comprehensive fault tolerance and real-time result streaming.
Request Flow:
- WebSocket Connection via API Gateway with Lambda authorizer validation and Google Maps address verification
- Request Handler Lambda generates unique IDs, checks ElastiCache for cached results, and queues requests in SQS
- Requestor Handler Lambda polls SQS and initiates the Step Functions workflow with parallel provider calls
- Step Functions HTTP Integration makes direct API calls to all 5 provider endpoints with provider-specific retry logic and timeout configurations
- Results Processing - successful responses go to results queue, failures to DLQ for analysis and potential retry
- Result Handler Lambda processes responses, stores in DynamoDB for session management and shared links, archives to S3, and updates ElastiCache
- Real-time Streaming - progressive results sent back to clients through the persistent WebSocket connection
Key Components:
- ElastiCache: Fast result caching and session management
- Step Functions: Direct HTTP API integration with nested Map states for parallel orchestration
- SQS with DLQ: Reliable message queuing and comprehensive failure handling
- AWS Secrets Manager: Secure provider credential storage
- DynamoDB: Result storage with share link functionality
- S3 + S3 Glacier: Data archiving with cost-effective long-term storage
- CloudWatch: Comprehensive monitoring of queue lengths, API response times, and system health
The Step Functions workflow uses AWS's native HTTP integration to call provider APIs directly, eliminating the need for Lambda wrapper functions while maintaining sophisticated orchestration patterns.
Step Functions Implementation
The core challenge was handling 5 providers with completely different API patterns while maintaining fast user response times. Step Functions with native HTTP integration enabled declarative workflows that call provider APIs directly.
Multi-Level Parallel Structure
The workflow implements nested parallelism: a top-level Parallel state for all 5 providers, with each branch containing Map states for provider-specific parallel processing.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
{
"ParallelProviders": {
"Type": "Parallel",
"Branches": [
{
"StartAt": "CallByteMe",
"States": { /* Simple HTTP call */ }
},
{
"StartAt": "PrepareVerbynDichPages",
"States": {
"VerbynDichParallelFetch": {
"Type": "Map",
"MaxConcurrency": 6,
"ItemsPath": "$.page_numbers",
"Iterator": { /* Fetch each page concurrently */ }
}
}
},
{
"StartAt": "CallServusSpeedAvailableProducts",
"States": {
"ServusSpeedParallelDetails": {
"Type": "Map",
"MaxConcurrency": 5,
"ItemsPath": "$.products_response.ResponseBody.availableProducts",
"Iterator": { /* Get details for each product */ }
}
}
}
]
}
}Result: All providers process simultaneously, reducing total response time from 60+ seconds to ~8 seconds.
Pattern 1: Dynamic Pagination (VerbynDich)
VerbynDich requires paginating through up to 18 pages. Instead of sequential requests, Map states fetch all pages concurrently:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
{
"VerbynDichParallelFetch": {
"Type": "Map",
"MaxConcurrency": 6,
"ItemsPath": "$.page_numbers",
"Iterator": {
"StartAt": "CallVerbynDichSinglePage",
"States": {
"CallVerbynDichSinglePage": {
"Type": "Task",
"Resource": "arn:aws:states:::http:invoke",
"Parameters": {
"ApiEndpoint": "https://verbyndich.../data",
"QueryParameters": {
"page.$": "States.Format('{}', $.page_number)"
}
},
"Retry": [
{
"ErrorEquals": ["States.Http.StatusCode.429"],
"IntervalSeconds": 5,
"MaxAttempts": 2,
"BackoffRate": 3.0
}
],
"Catch": [
{
"ErrorEquals": ["States.ALL"],
"Next": "HandlePageError"
}
]
}
}
}
}
}Individual page failures don't break the entire request. Failed pages are retried in subsequent processing batches (detailed in Error Handling below).
Pattern 2: Multi-Step Parallel Processing (Servus Speed)
Servus Speed requires two API calls: first to get available products, then to fetch details for each product. Map states enable dynamic parallelism based on the first response:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
{
"ServusSpeedParallelDetails": {
"Type": "Map",
"MaxConcurrency": 5,
"ItemsPath": "$.products_response.ResponseBody.availableProducts",
"Iterator": {
"StartAt": "CallProductDetails",
"States": {
"CallProductDetails": {
"Type": "Task",
"Resource": "arn:aws:states:::http:invoke",
"Parameters": {
"ApiEndpoint.$": "States.Format('https://servus-speed.../product-details/{}', $.product_id)"
}
}
}
}
}
}The number of parallel executions scales with the API response. If 15 products are available, 15 parallel detail calls execute within the concurrency limit.
Pattern 3: HMAC Authentication with Parallel Calls (Ping Perfect)
Ping Perfect requires HMAC-SHA256 signatures. The workflow orchestrates signature generation in Lambda, then makes parallel authenticated calls:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
{
"PingPerfectParallelFetch": {
"Type": "Map",
"ItemsPath": "$.call_types",
"Iterator": {
"StartAt": "GeneratePingPerfectSignature",
"States": {
"GeneratePingPerfectSignature": {
"Type": "Task",
"Resource": "arn:aws:states:::lambda:invoke",
"Parameters": {
"FunctionName": "${aws_lambda_function.ping_perfect_signer.function_name}"
},
"Next": "CallPingPerfectAPI"
},
"CallPingPerfectAPI": {
"Type": "Task",
"Resource": "arn:aws:states:::http:invoke",
"Parameters": {
"Headers": {
"X-Signature.$": "$.signature_response.Payload.signature",
"X-Timestamp.$": "$.signature_response.Payload.timestamp"
}
}
}
}
}
}
}EventBridge Connections
Authentication is handled through EventBridge connections, eliminating secrets in the workflow definition:
1
2
3
4
5
{
"Authentication": {
"ConnectionArn": "${aws_cloudwatch_event_connection.servus_speed_connection.arn}"
}
}Terraform manages the connections with appropriate authentication methods:
1
2
3
4
5
6
7
8
9
resource "aws_cloudwatch_event_connection" "servus_speed_connection" {
authorization_type = "BASIC"
auth_parameters {
basic {
username = "user_XXXXX"
password = var.servus_speed_auth
}
}
}Error Handling
Cross-Batch Failure Recovery
The most sophisticated aspect is the cross-batch retry logic within Map states. When individual items fail in one batch, they're automatically retried in subsequent batches rather than immediately failing.
Example with VerbynDich (18 pages, MaxConcurrency: 6):
- Batch 1: Pages 0-5 process concurrently
- If page 2 fails → marked for retry, not final failure
- Batch 2: Pages 6-11 + retry of page 2 from Batch 1
- Batch 3: Pages 12-17 + any remaining retries
This ensures maximum data retrieval while maintaining efficient parallel processing.
Comprehensive Error Boundaries
Each provider branch includes comprehensive error handling that preserves partial results:
1
2
3
4
5
6
7
8
9
{
"Catch": [
{
"ErrorEquals": ["States.ALL"],
"Next": "HandleProviderError",
"ResultPath": "$.error_info"
}
]
}Results are sent to SQS regardless of partial failures, ensuring users receive available data while failed requests are captured in dead letter queues for analysis. This ensures users always receive available data while operational teams get comprehensive failure analytics.
Implementation Details
Concurrency Management:
- VerbynDich: MaxConcurrency 6 (API rate limits)
- Servus Speed: MaxConcurrency 5 (server capacity)
- WebWunder: MaxConcurrency 3 (SOAP endpoint limitations)
Performance:
- Parallel execution reduces total response time by approximately 70%
- Individual provider failures don't block other providers
- Partial results stream to users as they become available
Cost:
- Map states incur no additional cost beyond state transitions
- EventBridge connections eliminate Lambda invocations for authentication
- Parallel execution reduces overall workflow duration
Results
The platform processes requests with multi-level parallelism and fault tolerance. Users receive results as they become available, with failed providers handled gracefully without affecting successful ones.
CHECK24 awarded this implementation their €7,200 GenDev scholarship, recognizing the advanced orchestration patterns that handle complex API integration challenges through declarative workflows rather than imperative coordination code.
The Step Functions approach demonstrates how nested Map states with dynamic scaling can solve real-world integration problems that would otherwise require complex Lambda orchestration logic.
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article