AWS Builder Center

Builder Center

AWS Data Engineering in October 2026: 5 Trends to Know

AWS Data Engineering in October 2026: 5 Trends to Know

Iceberg V3 on S3 Tables, Redshift Iceberg views, Aurora querying the lake and agents on AWS: what changed and what data engineers should learn next.

The last few weeks brought a wave of AWS launches for data engineers. Taken together, they point in one clear direction: open tables in the middle, fewer copy jobs around them, and AI agents on top.
Here are the five trends I think matter most, what changed, and what to learn next.
Architecture diagram of the Iceberg-centred AWS data stack in October 2026: ingest, S3 Tables storage, query engines, agents and governance
The Iceberg-centred AWS data stack after the latest launches

1. Apache Iceberg V3 arrives on Amazon S3 Tables

On 30 September 2026, Amazon S3 Tables added support for all Apache Iceberg V3 data types. This is the biggest change for the lakehouse this quarter.
  • New data types: Variant (for semi-structured JSON-like data), geometry, geography, nanosecond timestamps and unknown.
  • Row lineage: every row gets _row_id and _last_updated_sequence_number. Pipelines can find changed rows without scanning the whole table.
  • Deletion vectors: a compact format for deletes. This means less delete-file overhead and faster compaction.
Row lineage is the quiet hero. Incremental processing becomes the default, not a custom project.
Before and after comparison: a full-table compare scans every row, while Iceberg V3 row lineage reads only the changed rows
Row lineage lets downstream jobs read only what changed
A simple incremental read in Spark SQL could look like this:
1
2
3
4
-- read only rows changed since the last successful run
SELECT *
FROM s3tablescatalog.sales.orders
WHERE _last_updated_sequence_number > ${last_processed_seq};
Things to check first:
  • The new types need Spark 4.0+ (for example AWS Glue 6.0+ or Amazon EMR 8.1+).
  • They work with Parquet files only.
  • Upgrading a table from V2 to V3 is one-way. Make sure every engine you use supports V3 before you upgrade.

2. Amazon Redshift writes materialized views as Iceberg

Amazon Redshift can now build materialized views that are stored as Apache Iceberg tables in Amazon S3, registered in the AWS Glue Data Catalog.
Why this matters: your pre-computed joins and aggregates are no longer locked inside the warehouse. Athena, Spark or any Iceberg-aware engine can read the same results. The warehouse becomes one more engine on top of an open lake.

3. Zero-ETL works in both directions

For years, zero-ETL meant moving operational data into analytics. Now it also goes the other way. Amazon Aurora PostgreSQL can query Apache Iceberg and Parquet data directly, using normal PostgreSQL tools, with no ETL pipeline.
An application can join live orders in Aurora with historical data in the lake in a single query. That removes a whole class of "copy the lake back to the database" jobs.

4. Agents move into the data stack

Agentic AI is no longer just a chatbot on the side. Several October launches put agents close to your data and your infrastructure:
  • Amazon Bedrock Managed Agents powered by OpenAI (preview) run inside your AWS account, under your existing IAM identities and controls.
  • Kiro workflows coordinate several agents to finish multi-step tasks with less supervision.
  • AWS Well-Architected Agent (preview) reviews your environment and suggests cost, security, performance and resilience improvements.
  • Claude Haiku 5.5 on Amazon Bedrock and Strands Agents give fast, low-cost building blocks.
The lesson for data engineers: agents are only as good as the data they read. Clean, well-documented, governed tables are now an AI requirement, not just a reporting one.

5. Security and streaming get tighter

  • Amazon GuardDuty RDS Protection now looks at data activity. It can detect data exfiltration and destruction on Aurora PostgreSQL and RDS for PostgreSQL by analysing query behaviour, with no agents to install.
  • AWS Lambda can now use OAuth (for example Amazon Cognito or Okta) to authenticate to self-managed Apache Kafka event sources. Streaming ingestion fits better with central identity.
Scale is real, too. During Prime Day 2026, Amazon DynamoDB handled more than 59 trillion requests over four days, peaking at 192 million requests per second.

What I would learn next

  1. Apache Iceberg basics: snapshots, partitions, compaction, and now V3 row lineage.
  2. Incremental pipelines: process only what changed, using sequence numbers instead of full reloads.
  3. One catalog: AWS Glue Data Catalog and the Iceberg REST Catalog API as the shared layer for every engine.
  4. Data for agents: clear table and column descriptions, quality checks and access controls.
  5. Cost awareness: columnar formats, partitions and compaction still decide your bill.

Sources

🙌 A small request

I'm working towards my AWS Builder Center badges, and your support really helps. If you found this useful:
Thanks for reading. Happy building! 🚀
# data-engineering# amazon-redshift# amazon-s3# agentic-ai# amazon-aurora
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article