We Built Our Streaming Pipeline on Kinesis, Then Migrated to SQS. Here's the Honest Breakdown.

We made a reasonable call. Our pipeline needed to handle hundreds of thousands of events per hour, preserve ordering, and fan out to multiple consumers. Kinesis Data Streams checked every box on the whiteboard. So we built on it.

Two years and one painful migration later, I want to share what we actually learned. Not the AWS comparison table. The production reality.

The pitch is compelling: managed sharding, 24-hour retention, multiple consumers reading the same stream, ordering within a shard. If you've read anything about AWS streaming, Kinesis gets positioned as the "serious" option for high-throughput workloads.

What the docs don't front-load: Kinesis pricing is based on shard-hours, not consumption. You pay whether your shards are busy or not. And shard management -- scaling up when throughput spikes, scaling down when it drops -- is entirely manual unless you bolt on auto-scaling logic you'll need to build and operate yourself.

Our workload was bursty. Heavy on weekday afternoons, nearly silent on weekends. We were paying for weekend capacity we barely used, and manual shard management was causing us to either over-shard (wasted spend) or under-shard (throttled writes showing up as ProvisionedThroughputExceededException that we caught late).

The real problem wasn't any single incident. It was operational friction that compounded. Every time traffic patterns shifted, we had a task. Consumer code had to track shard iterators. When a consumer fell behind, we had to reason about which specific shard had the backlog. Kinesis does not self-heal the way SQS does.

Three things improved immediately:

Zero shard management. SQS scales automatically. Nothing to provision, nothing to watch.

Pay-per-message pricing. Our weekend infrastructure costs dropped noticeably without any application changes.

Dead-letter queues with no ceremony. Setting up DLQs in Kinesis required Lambda event source mapping configuration and careful retention window math. In SQS it is two fields in the console.

We did lose something real: guaranteed ordering per partition. If your consumers genuinely need strict sequencing -- financial ledger entries, audit logs where order is contractual -- SQS FIFO with message group IDs gives you ordering within a group, but it caps throughput at 3,000 messages per second with batching. For our workload, that was fine.

This is what a simplified Kinesis consumer looks like:

python
shard_iterator = kinesis.get_shard_iterator(
StreamName='my-stream',
ShardId='shardId-000000000000',
ShardIteratorType='LATEST'
)['ShardIterator']

while True:
response = kinesis.get_records(ShardIterator=shard_iterator, Limit=100)
records = response['Records']
shard_iterator = response['NextShardIterator']
for record in records:
process(record['Data'])

You are managing iterator state. Every shard needs its own loop. If you add a shard, you add a consumer thread or process.

The equivalent SQS consumer:

python
response = sqs.receive_message(
QueueUrl=queue_url,
MaxNumberOfMessages=10,
WaitTimeSeconds=20
)
for msg in response.get('Messages', []):
process(msg['Body'])
sqs.delete_message(
QueueUrl=queue_url,
ReceiptHandle=msg['ReceiptHandle']
)

No iterator state. No shard coordination. You poll, process, delete. Multiple consumers just poll the same queue independently.

Before defaulting to either, I now ask four questions:

Do you need to replay events hours or days later? Kinesis has built-in retention and replay. SQS does not. If replay is a hard requirement, Kinesis or Kafka is the right answer.

Do multiple independent systems consume the same event stream? Kinesis fan-out works cleanly here. SNS-to-SQS fan-out also works but adds a component. For pure fan-out scenarios, Kinesis is a reasonable default.

Is your throughput steady and predictable? Kinesis fixed shard costs make more sense when you can right-size upfront. Bursty or unpredictable traffic patterns favor SQS.

Do you need strict ordering at high throughput? Neither service is perfect here. You are choosing between Kinesis (shard-level ordering, manual scaling) and SQS FIFO (group-level ordering, hard throughput ceiling).

The hardest part of the migration was not the infrastructure. It was re-educating the team. We had engineers who assumed Kinesis was the correct answer for any "serious" streaming workload, and that using SQS was somehow less scalable or less mature. That assumption cost us two years of operational overhead.

SQS handles enormous throughput with significantly less operational surface area. Choose Kinesis when you specifically need replay, extended retention, or tight integration with Kinesis Data Analytics. Choose SQS when you need cost efficiency on variable workloads, simpler consumer code, and DLQ ergonomics that work out of the box.

For event-driven architectures where events carry semantic meaning and you are routing on content, EventBridge is often the better answer than either -- but that is a different conversation.

Have you migrated between Kinesis and SQS in production? What drove the decision -- and if you stayed on Kinesis, what made it worth the operational overhead?