AWS Builder Center

Builder Center

ChDB or DuckDB on AWS Lambda: A Technical Comparison

DuckDB and ChDB both bring powerful analytical capabilities into lightweight environments, but running them inside AWS Lambda comes with its own challenges. This article explores how each engine can be installed and configured with extensions in Lambda, and what these approaches mean for building serverless data pipelines.

Introduction

AWS Lambda is designed around a simple idea: run code without managing servers.
For data engineers, however, things get interesting when the code needs an analytical database engine.
Two interesting options are DuckDB and ChDB. Both can process analytical workloads locally within a Lambda execution environment, but they have different architectures, installation methods, and extension requirements.
The first step in comparing them is getting them to run reliably.

Why Lambda Changes the Setup

Installing a database engine on a normal machine is usually straightforward.
Lambda adds constraints such as:
  • Deployment package size
  • Runtime compatibility
  • CPU architecture
  • Temporary storage
  • Cold-start time
  • Native binary dependencies
  • Extension availability
A solution that works perfectly on a developer's laptop may fail when packaged into Lambda.
So environment setup isn't just a preliminary step. It is part of the technical comparison.

Running DuckDB in Lambda

DuckDB provides Python and other language bindings that can be packaged with a Lambda function.
A simplified Python example looks like:
1
2
3
4
5
6
7
8
9
10
11
import duckdb

def lambda_handler(event, context):
result = duckdb.sql("""
SELECT 42 AS answer
""").fetchall()

return {
"statusCode": 200,
"body": str(result)
}
The challenge is not the SQL.
The challenge is packaging the correct DuckDB runtime and ensuring that its native components are compatible with the Lambda environment.
For more complex workloads, extensions may also be required.
Examples include extensions for working with formats or external data sources.
The deployment therefore needs to account for both:
1
2
3
4
5
6
7
Lambda Function
↓
DuckDB Runtime
↓
Required Extensions
↓
Data Source

Running ChDB in Lambda

ChDB takes a different approach.
It provides an embedded way to run ClickHouse capabilities inside an application rather than operating a traditional ClickHouse server.
A simplified example might look like:
1
2
3
4
5
6
7
8
9
10
11
import chdb

def lambda_handler(event, context):
result = chdb.query("""
SELECT 42
""")

return {
"statusCode": 200,
"body": result
}
Again, the interesting part begins when we move from a local development environment to Lambda.
Native dependencies, package compatibility, architecture, and extension support all become important.

Extensions Change the Comparison

The database engine itself is only one part of the environment.
A realistic data pipeline may need capabilities such as:
1
2
3
4
5
Parquet
JSON
HTTP
S3
External data sources
If an extension or feature is required, we need to consider:
  1. Is it available for the selected engine?
  2. Can it run inside Lambda?
  3. Does it require additional binaries?
  4. Does it increase deployment size?
  5. Does it affect cold-start performance?
  6. Can it access the required AWS resources securely?
This is where two engines that appear similar on paper can behave very differently in a serverless environment.

Lambda Layers and Container Images

There are several ways to package these dependencies.

Deployment Package

The libraries and native dependencies are included directly with the Lambda function.
This is simple for smaller setups but can become difficult as dependencies grow.

Lambda Layers

Common dependencies can be separated into a Lambda Layer.
1
2
3
4
5
6
7
Lambda Function
+
Lambda Layer
↓
DuckDB / ChDB
↓
Extensions
This can make dependency management cleaner, particularly when multiple functions use the same environment.

Container Images

For more complex native dependencies, Lambda container images can provide more control over the runtime environment.
This can be particularly useful when the database engine or extensions require additional system libraries.

What Should We Compare?

Once both environments are working, the comparison becomes much more meaningful.
Important dimensions include:
AreaDuckDBChDB
InstallationPackage/binary dependentPackage/binary dependent
Native dependenciesImportantImportant
ExtensionsDepends on supported extensionsDepends on ClickHouse capabilities
Lambda packagingPackage, Layer, or containerPackage, Layer, or container
Cold startNeeds measurementNeeds measurement
Memory usageNeeds measurementNeeds measurement
Data formatsStrong analytical supportStrong analytical support
Server requiredNoNo
The table is only a starting point. Actual performance should be measured in the same Lambda environment rather than assumed from local benchmarks.

A Practical Testing Environment

For a fair comparison, both engines should be tested under the same conditions.
For example:
1
2
3
4
5
6
Runtime: Python
Architecture: x86_64 or arm64
Memory: Fixed value
Timeout: Fixed value
Dataset: Same files
Lambda configuration: Same
Then measure:
  • Deployment size
  • Initialization time
  • Cold-start time
  • Warm execution time
  • Memory consumption
  • Query execution time
  • Extension loading time
  • Failure rate
This turns the comparison from “Which database sounds better?” into an actual engineering experiment.
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article