AWS Builder Center

Builder Center

Building a Scalable Synthetic Data Pipelines with Amazon S3 Tables, Apache Iceberg and Faker

Are you keen to try out Amazon S3 Tables? Learn how to generate millions of highly realistic synthetic financial transactions using Python's Faker library and AWS Glue, then store them in S3 Tables with Apache Iceberg for testing and development without exposing sensitive production data. This hands-on tutorial demonstrates partition optimization and reproducible data generation at scale.

David McAmis

In today's data-driven world, organizations face a persistent challenge: how do you develop, test, and optimize data systems without access to production data? Privacy regulations make it increasingly difficult to use real customer data in development environments. Even when regulations aren't a concern, production data often contains sensitive information that shouldn't be exposed to every developer or data scientist on your team. Yet, testing with small, unrealistic datasets leads to systems that fail when confronted with real-world scale and complexity.
This is where synthetic data generation becomes invaluable. By creating highly realistic, large-scale datasets that mimic production characteristics without containing any actual customer information, teams can build and test data pipelines with confidence. But not all synthetic data solutions are created equal. Many approaches struggle to generate data at the scale needed to truly stress-test modern data systems, or they produce data that's too simplistic to reveal real-world issues.

Why Faker for Scalable Synthetic Data?

The  Python Faker library  has emerged as a powerful tool for generating synthetic data that strikes the right balance between realism and performance. Unlike simple random data generators that produce meaningless strings of characters, Faker understands semantic data types. It can generate realistic names, addresses, companies, phone numbers, email addresses, and dozens of other data types that look and behave like real data.
While there are other libraries and methods out there, what makes Faker particularly valuable for large-scale data generation is its combination of realism and reproducibility. The library uses localized data providers, meaning it can generate culturally appropriate names and addresses for different regions. A transaction in Australia will have Australian company names and locations, while one in the UK will reflect British conventions. This level of detail is crucial when testing systems that need to handle international data or when creating demo environments that need to look authentic.
Crucially, Faker supports seeding, which allows you to generate the same "random" data every time you run your pipeline. This reproducibility is essential for testing and debugging. When a data pipeline fails, you need to be able to regenerate the exact same input data to investigate the issue. Without deterministic data generation, debugging becomes a frustrating game of trying to reproduce intermittent failures.
When combined with distributed computing frameworks like Apache Spark, Faker can generate billions of rows of realistic data in minutes rather than hours or days. This scalability transforms synthetic data from a development convenience into a production-grade testing tool that can validate systems under realistic load conditions.

Amazon S3 Tables: Modern Data Lake Architecture

 Amazon S3 Tables  represents a significant evolution in cloud data storage, bringing together the scalability and cost-effectiveness of S3 with the transactional guarantees and query performance traditionally associated with data warehouses. Built on  Apache Iceberg , S3 Tables provides a managed service that handles the complexity of maintaining table metadata, managing file layouts, and optimizing query performance.
Traditional data lakes built directly on S3 suffer from several challenges. Without a proper table format, you can't perform atomic updates or deletes, making it difficult to correct errors or comply with data deletion requests. Query performance degrades as data grows because there's no efficient way to skip irrelevant files. Schema evolution becomes a nightmare, often requiring complete data rewrites. And forget about time travel or auditing capabilities.
S3 Tables solves these problems by leveraging Apache Iceberg's sophisticated metadata layer. Every write operation is atomic and isolated, meaning concurrent readers and writers don't interfere with each other. The system maintains detailed statistics about data files, allowing queries to skip files that definitely don't contain relevant data. Schema changes are tracked in metadata, allowing you to add, remove, or modify columns without touching the actual data files. And every change is versioned, so you can query historical states of your data or roll back problematic updates.

Using Synthetic Data to Explore S3 Tables Features

Synthetic data provides an ideal way to explore and validate S3 Tables capabilities without the risk and complexity of using production data. You can generate datasets with specific characteristics to test particular features. Want to see how partition pruning improves query performance? Generate a dataset with clear partition boundaries and run queries with and without partition filters. Curious about time travel capabilities? Create multiple versions of your data and query historical snapshots. Need to test schema evolution? Generate an initial dataset, modify the schema, and add new data with additional columns.
The financial transaction dataset we'll build in this tutorial is particularly well-suited for exploring S3 Tables features. Transactions naturally partition by dimensions like currency and merchant category, making them perfect for demonstrating partition pruning. The time-series nature of transactions allows us to explore time-based queries and retention policies. And the realistic data generated by Faker ensures that our tests reflect actual query patterns and data distributions.
By generating millions or billions of synthetic transactions, you can stress-test query performance, validate that partition strategies actually improve query times, experiment with different table layouts and configurations, and build confidence in your data architecture before deploying to production. This approach dramatically reduces the risk of discovering performance problems or architectural limitations after you've already committed to a particular design.

The Architecture: Bringing It All Together

Our pipeline combines  AWS Glue  for serverless distributed computing, the Faker library for realistic data generation, S3 Tables for managed Iceberg storage, and Apache Spark for parallel processing. This architecture can scale from thousands to billions of rows while maintaining reproducibility and data quality. Let's walk through how each component works together to create a production-ready synthetic data pipeline.
To get started, you will need to have:
  • An AWS Account which you can use
  • AWS Glue configured and an AWS Glue Studio Notebook


Before you start

When you first start a fresh Glue Studio notebook, there is a cell at the top to start the Glue Interactive Session. Make sure to add the following to the configuration to include the faker library:
Python
%additional_python_modules faker

Step 1: Configuring the S3 Tables Catalog

Python
sc = SparkContext.getOrCreate()
glueContext = GlueContext(sc)
spark = glueContext.spark_session
The first thing we need to do is initialize the AWS Glue environment. The  GlueContext  wraps the standard Spark context and provides AWS-specific functionality that we'll need throughout our pipeline. We use the  getOrCreate()  method to ensure we don't accidentally create duplicate contexts if the notebook is re-run, which is a common scenario during development and testing.

Catalog Configuration

Python
spark.conf.set("spark.sql.catalog.s3tables", "org.apache.iceberg.spark.SparkCatalog")
spark.conf.set("spark.sql.catalog.s3tables.catalog-impl", "org.apache.iceberg.rest.RESTCatalog")
spark.conf.set("spark.sql.catalog.s3tables.uri", "https://s3tables.ap-southeast-2.amazonaws.com/iceberg")
spark.conf.set("spark.sql.catalog.s3tables.warehouse", "arn:aws:s3tables:ap-southeast-2:1234567890:bucket/synthetic-data-bucket-1234567890")
spark.conf.set("spark.sql.catalog.s3tables.io-impl", "org.apache.iceberg.aws.s3.S3FileIO")
Note: For your own code, you will need to customize both the URL and warehouse to reflect the AWS region that you are working in. You will also need to specify the region when doing the authentication setup (code below). There is a list of regions available  from here.  For the warehouse, you will also need to specify the Amazon S3 Tables bucket you created in your environment.
These configuration settings are the foundation of our integration with S3 Tables. We're registering a new catalog named "s3tables" that Spark can query using standard SQL. The catalog uses Apache Iceberg's REST API to communicate with S3 Tables rather than accessing S3 directly, which provides better abstraction and management capabilities.
The URI points to the S3 Tables REST API endpoint in the ap-southeast-2 region, while the warehouse ARN identifies the specific S3 Tables bucket where our data will be physically stored. Finally, we specify that Iceberg should use AWS S3-optimized file operations through the  S3FileIO  implementation, which provides better performance for S3 interactions.

Authentication Setup

Python
spark.conf.set("spark.sql.catalog.s3tables.rest.sigv4-enabled", "true")
spark.conf.set("spark.sql.catalog.s3tables.rest.signing-region", "ap-southeast-2")
spark.conf.set("spark.sql.catalog.s3tables.rest.signing-name", "s3tables")
AWS Signature Version 4 (SigV4) is AWS's authentication protocol, and these settings enable automatic credential signing for all REST API calls to S3 Tables. The beauty of this approach is that it uses the IAM role attached to your Glue job, meaning you don't need to manage or hardcode any credentials. The authentication happens seamlessly in the background, using the permissions already granted to your Glue job's execution role.

Step 2: Import Dependencies

Python
from faker import Faker
from pyspark.sql.functions import col
from pyspark.sql.types import (
    StructType, StructField,
    StringType, TimestampType, DoubleType, LongType
)
We're importing the essential tools for our data generation pipeline. Faker is a Python library that generates realistic fake data like names, addresses, and company names. The PySpark functions and types allow us to transform DataFrames and explicitly define our data schema, which is critical for both performance and data quality in production systems.

Step 3: Generating Synthetic Transaction Data

Configuration and Constants

Python
num_rows = 1_000_000
num_partitions = 200

merchant_categories = [
    "Grocery", "Fuel", "Online Retail", "Restaurants",
    "Travel", "Subscriptions", "Electronics", "Fashion"
]
currencies = ["AUD", "USD", "EUR", "GBP"]
card_types = ["VISA", "MASTERCARD", "AMEX", "DEBIT"]
The configuration variables control the scale of our data generation. We start with 10,000 rows for testing, but this can easily scale to millions for production use cases. The number of partitions is set to 200 to parallelize the work across your Glue cluster, and you should adjust this based on your DPU allocation. The constant lists define realistic business domains for our synthetic data, ensuring that the generated transactions look like real-world financial data.
These categories and types aren't arbitrary. They're chosen to create interesting query patterns that will help us explore S3 Tables features. The eight merchant categories provide enough cardinality to demonstrate partition pruning without creating an unwieldy number of partitions. The four currencies represent common international transactions. Together, these create 32 distinct partitions (8 categories × 4 currencies), which is a realistic number for demonstrating how Iceberg's hidden partitioning improves query performance.

Creating the Base DataFrame

Python
fake = Faker()
Faker.seed(42)

base_df = (
    spark.range(0, num_rows)
         .repartition(num_partitions)
         .withColumnRenamed("id", "row_id")
)
We create a foundation DataFrame using Spark's range function, which generates a distributed DataFrame with sequential IDs from 0 to num_rows minus one. The repartition operation splits this data across 200 partitions for parallel processing, allowing multiple workers to generate data simultaneously. Setting the seed to 42 ensures reproducibility, meaning the same seed will always generate the same "random" data, which is invaluable for testing and debugging.

The Transaction Generator

Python
def generate_transaction(row_id):
    """Generate a single synthetic transaction record"""
    random.seed(42 + row_id)
    fake_local = Faker()
    Faker.seed(42 + row_id)
    
    customer_id = random.randint(1, 50_000)
    
    # Generate timestamp within last 90 days
    end = datetime.utcnow()
    start = end - timedelta(days=90)
    delta_seconds = int((end - start).total_seconds())
    random_second = random.randint(0, delta_seconds)
    tx_timestamp = start + timedelta(seconds=random_second)
    
    amount = round(random.uniform(1.0, 1000.0), 2)
    currency = random.choice(currencies)
    merchant_category = random.choice(merchant_categories)
    merchant_name = f"{merchant_category} - {fake_local.company()}"
    country = fake_local.country_code()
    card_type = random.choice(card_types)
    transaction_id = f"TX-{row_id:010d}"
    
    return (...)
The transaction generator function is where Faker's capabilities really shine. Each transaction gets a unique but deterministic seed by adding the row_id to our base seed of 42. This design ensures that running the code twice produces identical results while preventing parallel workers from interfering with each other's random number generation.
The function creates realistic transaction data with careful attention to business logic. Customer IDs range from 1 to 50,000, simulating a customer base of that size. This creates realistic patterns where some customers appear multiple times in the dataset, just as they would in real transaction data. Transactions are randomly distributed across the last 90 days, mimicking real-world temporal patterns rather than clustering at a single point in time. This temporal distribution is particularly useful for testing time-based queries and exploring Iceberg's time travel features.
Transaction amounts range from $1 to $1,000 with realistic decimal precision. The merchant names combine the category with a fake company name generated by Faker, creating entries like "Grocery - Anderson and Sons" or "Travel - Global Airways Inc." This level of realism is important when you're building demo environments or testing systems that display data to end users. The country codes generated by Faker are valid ISO codes, and the combination of all these elements creates a dataset that looks and behaves like real financial transaction data.

Schema Definition

Python
transaction_schema = StructType([
    StructField("transaction_id", StringType(), False),
    StructField("customer_id", LongType(), False),
    StructField("transaction_ts", TimestampType(), False),
    StructField("amount", DoubleType(), False),
    StructField("currency", StringType(), False),
    StructField("merchant_name", StringType(), False),
    StructField("merchant_category", StringType(), False),
    StructField("country", StringType(), False),
    StructField("card_type", StringType(), False)
])
Defining an explicit schema provides several critical benefits. From a performance perspective, Spark doesn't need to infer types by scanning the data, which can be expensive for large datasets. The schema also serves as data validation, with the  False  parameter indicating that no nulls are allowed in any field. This ensures data quality and consistency every time the pipeline runs. When you're generating synthetic data for testing, having a strict schema helps catch bugs in your data generation logic before they cause problems downstream.

Efficient Data Generation with mapPartitions

Python
def generate_partition(iterator):
    """Generate transactions for a partition"""
    for row in iterator:
        row_id = row.row_id
        yield generate_transaction(row_id)

transactions_df = (
    base_df
    .rdd
    .mapPartitions(generate_partition)
    .toDF(transaction_schema)
)
The  mapPartitions  approach is a critical performance optimization that makes the difference between generating thousands of rows per second versus millions. Traditional UDFs in PySpark serialize data between the JVM and Python for every single row, creating massive overhead that can slow down processing by orders of magnitude. Instead,  mapPartitions  processes entire partitions at once, maintaining one Python context per partition instead of per row. This dramatically reduces serialization overhead while maintaining full parallelism, as each of the 200 partitions processes independently across your cluster.
The flow converts our DataFrame to an RDD, applies the generation function to each partition, and then converts back to a DataFrame with our explicit schema. This pattern is commonly used in PySpark when you need to perform complex row-level operations efficiently. For synthetic data generation at scale, this optimization is essential. Without it, generating a billion rows might take hours. With it, the same task completes in minutes.

Caching and Verification

Python
transactions_df.cache()
print(f"Generated {transactions_df.count():,} transactions")
transactions_df.show(5)
Calling  cache()  stores the computed DataFrame in memory, which is important because we'll use this data multiple times for counting, displaying, and writing. Without caching, Spark would regenerate the entire dataset each time we reference it, wasting computational resources. The verification steps confirm that our data generation worked correctly and give us a preview of what the transactions look like. This is particularly important when you're iterating on your data generation logic to ensure the output matches your expectations.

Step 4: Creating the Iceberg Table

Database Creation

Python
database_name = "synthetic_finance_db"
table_name = "transactions"

spark.sql(f"CREATE NAMESPACE IF NOT EXISTS s3tables.{database_name}")

In Iceberg terminology, a namespace is equivalent to a database. The  IF NOT EXISTS  clause makes this operation idempotent, meaning it's safe to run multiple times without errors. In the code below, the CREATE OR REPLACE table statement ensures we start with a clean slate for testing, though you'd want to remove this in a production environment where preserving existing data is important. For our synthetic data exploration recreating the table allows us to experiment with different table configurations and partitioning strategies.

Table Definition

Python
create_table_sql = f"""
CREATE OR REPLACE TABLE s3tables.{database_name}.{table_name} (
    transaction_id      STRING,
    customer_id         BIGINT,
    transaction_ts      TIMESTAMP,
    amount              DOUBLE,
    merchant_name       STRING,
    country             STRING,
    card_type           STRING,
    currency            STRING,
    merchant_category   STRING
)
USING ICEBERG
PARTITIONED BY (currency, merchant_category)
"""

spark.sql(create_table_sql)
print(f"Created table: s3tables.{database_name}.{table_name}")


Step 5: Writing Data to S3 Tables

The Write Operation

Python
# Write data using DataFrame API
(
    transactions_df
    .writeTo(f"s3tables.{database_name}.{table_name}")
    .overwritePartitions()
)

print(f"Successfully wrote {num_rows:,} transactions to S3 Tables")
This is where our synthetic data actually gets persisted to S3 Tables. The  writeTo()  method is part of Iceberg's DataFrame writer API, which provides more sophisticated capabilities than Spark's traditional  write()  method. This API is specifically designed to work with Iceberg's advanced table management features and provides better integration with the catalog system.
The  overwritePartitions()  operation is particularly intelligent. Rather than replacing the entire table, it dynamically determines which partitions are affected by the incoming data and overwrites only those specific partitions. In our case, since we're writing to a fresh table, all partitions will be created. However, if you were to run this code again with a subset of data (say, only USD transactions in the Grocery category), only that specific partition would be overwritten while all other partitions remain untouched. This is an atomic operation, meaning either all the data writes successfully or none of it does, preventing partial failures that could corrupt your data or leave it in an inconsistent state.
Under the hood, several things are happening simultaneously. Iceberg writes the actual transaction data as Parquet files to S3, taking advantage of Parquet's columnar format for efficient storage and query performance. The system creates metadata files that track all the data files, their locations, and their statistics. Partition pruning metadata is automatically maintained, recording which values exist in which files so that future queries can skip irrelevant data. Statistics are computed for each column, including min/max values, null counts, and value counts, which the query optimizer uses to create efficient execution plans. All of this happens transparently without requiring manual intervention or configuration.

Verification Steps

Python
# Verify the data
result_df = spark.sql(f"SELECT * FROM s3tables.{database_name}.{table_name} LIMIT 10")
result_df.show()

# Show table statistics
spark.sql(f"DESCRIBE EXTENDED s3tables.{database_name}.{table_name}").show(truncate=False)
The verification steps serve as critical quality checks to ensure everything worked as expected. The sample query confirms that data is not only written but also readable and properly formatted. By limiting the results to 10 rows, we get a quick preview without overwhelming the output. This is your first opportunity to visually inspect the synthetic data and confirm that it matches your expectations in terms of format, data types, and realistic values.
The  DESCRIBE EXTENDED  command provides a comprehensive view of the table's metadata and configuration. This output includes the complete schema with all column names and data types, the partitioning specification showing how data is organized by currency and merchant category, the physical storage location in S3 where your data files reside, table properties including Iceberg-specific settings and configurations, and detailed statistics about the data distribution and file organization.
This information is invaluable for troubleshooting performance issues, understanding how your data is physically organized, and verifying that your partitioning strategy is working as intended. When you're experimenting with different table configurations or trying to optimize query performance, this metadata view helps you understand exactly what Iceberg has created and how it will behave under different query patterns.
Using the default Glue Studio settings (Glue 5.0, 5 DPUs, G1.X instance) you can generation 1 million rows in around 4 minutes. When you run the notebook successfully, your synthetic table will be created, and you will see the details below:


Summary

We've built a complete end-to-end pipeline that demonstrates the power of combining synthetic data generation with modern cloud data architecture. Starting with the Python Faker library, we created a scalable system capable of generating millions of realistic financial transactions with proper seeding for reproducibility. By leveraging AWS Glue's distributed computing capabilities and the  mapPartitions  optimization pattern, we achieved performance levels that make it practical to generate billion-row datasets in minutes rather than hours.
The integration with S3 Tables and Apache Iceberg brings enterprise-grade data management capabilities to our synthetic data. We've created a partitioned table that automatically organizes data by currency and merchant category, setting the foundation for efficient queries. The Iceberg table format provides ACID transactions, schema evolution, and time travel capabilities that transform S3 from simple object storage into a sophisticated data lakehouse platform.
This pipeline demonstrates several production best practices that extend beyond synthetic data generation. The architecture is highly scalable, allowing you to adjust from thousands to billions of rows simply by changing configuration parameters. The deterministic seeding ensures reproducibility, which is essential for testing and debugging complex data pipelines. The AWS integration using SigV4 authentication and IAM roles eliminates credential management headaches and follows security best practices. Most importantly, this approach enables privacy-compliant development since you never need to use real customer data for testing, training, or demonstration purposes.
Related Resources:
-  A Beginner’s Guide to Apache Parquet  -  Inside AWS Glue: Understanding the Spark Engine and Using the Spark UI for Troubleshooting  -  A Beginner’s Guide to Orchestrating AWS Glue Jobs with Amazon Managed Workflows for Apache Airflow (MWAA)  -  Connecting to Salesforce Data Using AWS Glue  -  Troubleshooting AWS Glue Jobs  -  How to Ingest Excel Files into a Data Lake Using AWS Glue 
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article