Building a Real-Time M-Pesa Transaction Pipeline: Kafka, PyFlink, and Terraform on Kubernetes

## Why Streaming, Not Batch?

When most people start learning data engineering, they learn batch processing first: collect data all day, then run one big job at midnight to process it. That works fine for things like daily sales reports.

But some problems can't wait until midnight. If M-Pesa sends a transaction event, a fintech system needs to know about it **immediately** — to update a balance, flag potential fraud, or trigger a notification. That's where streaming data engineering comes in.

| Batch Processing | Streaming Processing |
|---|---|
| Runs on a schedule (hourly/daily) | Reacts the moment data arrives |
| Simpler to build | More moving parts, more complexity |
| Fine for reports, dashboards | Needed for real-time decisions |

I built a real-time M-Pesa transaction pipeline to learn these concepts hands-on. Here's what I learned, tool by tool.

## The Core Building Blocks

### 1. Webhook Receiver (Flask)

Every pipeline needs a "front door" — somewhere external systems send data. M-Pesa's Daraja API sends transaction events as webhooks, so I built a lightweight Flask endpoint to receive them.

```python
@app.route('/webhook/mpesa', methods=['POST'])
def receive_transaction():
payload = request.get_json()
producer.send('mpesa-transactions', value=payload)
return jsonify({"status": "received"}), 200