Writing Cloud-Agnostic Queue Integration Code with Kiro
Keep your application independent of Azure Service Bus, Google Cloud Pub/Sub or RabbitMQ, and test it without any of them running, using code Kiro writes for you.
The problem: one import, one architectural decision
Most message-queue code starts like this:
1
2
3
4
5
6
from azure.servicebus import ServiceBusClient, ServiceBusMessage
def place_order(order):
with ServiceBusClient.from_connection_string(CONN_STR) as client:
with client.get_queue_sender("orders") as sender:
sender.send_messages(ServiceBusMessage(json.dumps(order)))It works, but that first import defines the order logic and now depends on Azure Service Bus. Moving to Pub/Sub or RabbitMQ means editing every function that sends or receives a message. Testing
place_order means a connection string, network access and a live namespace.The usual fix is to put a small interface between your code and the broker, with one adapter per broker behind it. In this post I'll show how I had Kiro write that interface, a RabbitMQ adapter, an in-memory adapter for tests, and a check that keeps the boundary in place.
Step 1: Describe the abstraction to Kiro
I started a spec in Kiro rather than a chat, because I wanted the constraints written down, not just implied. My starting prompt:
> I want a Python producer and consumer code that works with Azure Service Bus, GCP Pub/Sub or RabbitMQ behind one interface. The interface should only have publish and subscribe. Application code must never import a broker SDK directly; it should get a queue from a factory. Include adapters for all three brokers.
Kiro turned that into three files:
- requirements, with acceptance criteria in EARS notation, for example: "WHEN the developer requests an unknown broker, THE factory SHALL raise a ValueError listing the allowed values."
- design, with the module layout, the interface and the factory.
- tasks, an ordered checklist Kiro then worked through.
Writing "application code must never import a broker SDK" as a requirement turned out to matter. It's the reason Kiro later wrote a test to enforce it, which I'll come back to.
Step 2: The interface Kiro generated
The whole contract is a message type and two methods:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
from abc import ABC, abstractmethod
from dataclasses import dataclass, field
from typing import Iterator
class Message:
body: bytes
headers: dict[str, str] = field(default_factory=dict)
class QueueInterface(ABC):
def publish(self, queue_name: str, message: Message) -\> None:
"""Publish a message to the named queue."""
def subscribe(self, queue_name: str) -\> Iterator[Message]:
"""Yield messages from the named queue in FIFO order."""No broker types appear anywhere in it. The body is bytes and the headers are strings, which every broker can carry. That narrowness is what lets three very different services fit behind it.
Step 3: One concrete adapter
Each adapter is the only module allowed to import its broker's SDK. Here's the RabbitMQ adapter Kiro wrote, using the
pika client:1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
from typing import Iterator
import pika
from .interface import Message, QueueInterface
class RabbitMqAdapter(QueueInterface):
def __init__(self, url: str) -\> None:
if not url:
raise ValueError("url must be a non-empty AMQP URL")
self._params = pika.URLParameters(url)
def publish(self, queue_name: str, message: Message) -\> None:
conn = pika.BlockingConnection(self._params)
try:
ch = conn.channel()
ch.queue_declare(queue=queue_name, durable=True)
ch.basic_publish(
exchange="",
routing_key=queue_name,
body=message.body,
properties=pika.BasicProperties(headers=dict(message.headers)),
)
finally:
conn.close()
def subscribe(self, queue_name: str) -\> Iterator[Message]:
conn = pika.BlockingConnection(self._params)
ch = conn.channel()
ch.queue_declare(queue=queue_name, durable=True)
try:
for method, props, body in ch.consume(queue_name, inactivity_timeout=5.0):
if method is None: # no message within the timeout
return
yield Message(body=body, headers=dict((props.headers or {}) if props else {}))
ch.basic_ack(method.delivery_tag)
finally:
try:
ch.cancel()
finally:
conn.close()The Azure Service Bus and Pub/Sub adapters have the same shape with their own SDKs. Your application never sees any of them.
Kiro also generated a factory,
create_queue(broker, **config), as the single place adapters are chosen. Each broker's SDK is an optional install (pip install my_queues[rabbitmq]), so a team using one broker doesn't install the others. If you pick a broker whose SDK is missing, the factory tells you the exact pip install command.Step 4: Write a producer and a consumer against the interface
This is where the abstraction pays off. Here's an order service written only against
QueueInterface:1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
import json
from my_queues import Message, QueueInterface
class OrderProducer:
def __init__(self, queue: QueueInterface) -\> None:
self._queue = queue
def order_placed(self, order_id: str, total: float) -\> None:
body = json.dumps({"order_id": order_id, "total": total}).encode()
self._queue.publish("orders", Message(body=body, headers={"type": "order.placed"}))
class OrderConsumer:
def __init__(self, queue: QueueInterface) -\> None:
self._queue = queue
def process(self) -\> list[str]:
processed = []
for msg in self._queue.subscribe("orders"):
if msg.headers.get("type") == "order.placed":
processed.append(json.loads(msg.body)["order_id"])
return processedNeither class imports
pika, azure or google. They take a queue as a constructor argument, and the broker is chosen once, where the application starts up:1
2
3
4
5
from my_queues import create_queue
queue = create_queue("rabbitmq", url=os.environ["RABBITMQ_URL"])
producer = OrderProducer(queue)
consumer = OrderConsumer(queue)Step 5: Swap brokers in one line
Moving from RabbitMQ to Azure Service Bus changes only that startup line:
1
2
3
4
queue = create_queue(
"azure-service-bus",
connection_string=os.environ["AZURE_SERVICE_BUS_CONNECTION_STRING"],
)OrderProducer and OrderConsumer don't change. That's what portability means in practice: the broker is a deployment decision, not something spread through your business logic.Step 6: Ask Kiro for an in-memory implementation
Now the testability side. With the interface in place, I asked Kiro:
> Add an in-memory implementation of QueueInterface for local testing. Use only the standard library, keep FIFO order, make it thread-safe, and make sure later changes to a published message don't affect what subscribers receive.
Kiro produced this:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
import copy
import threading
from collections import defaultdict, deque
from typing import Iterator
from .interface import Message, QueueInterface
class InMemoryAdapter(QueueInterface):
def __init__(self) -\> None:
self._queues: dict[str, deque[Message]] = defaultdict(deque)
self._lock = threading.Lock()
def publish(self, queue_name: str, message: Message) -\> None:
with self._lock:
self._queues[queue_name].append(copy.deepcopy(message))
def subscribe(self, queue_name: str) -\> Iterator[Message]:
while True:
with self._lock:
if not self._queues[queue_name]:
return
yield self._queues[queue_name].popleft()The
deepcopy came from the last sentence of my prompt. Without it, a producer that changes a message's headers after publishing would also change the copy waiting in the queue. A real broker can't do that, because the message has already left your process, so a good test double shouldn't either.Kiro registered it in the factory as
"inmemory". Testing the order service now needs no broker, no credentials and no network:1
2
3
4
5
6
7
8
from my_queues import create_queue
def test_consumer_processes_placed_orders():
queue = create_queue("inmemory")
OrderProducer(queue).order_placed("A-100", 25.00)
OrderProducer(queue).order_placed("A-101", 40.00)
assert OrderConsumer(queue).process() == ["A-100", "A-101"]It runs in milliseconds and exercises the same interface production uses. The generated test suite for the package itself, covering round trips, ordering, isolation between queues and the deep-copy behavior, ran in 0.04 seconds.
Step 7: Have Kiro enforce the boundary
An abstraction only helps if people keep using it. Six months from now, someone in a hurry will type
import pika in a handler. Because the spec said application code must not import a broker SDK, Kiro generated a test that fails when that happens:1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
FORBIDDEN_PREFIXES = ("azure.servicebus", "google.cloud.pubsub", "pika")
violations = []
for py_path in PACKAGE_ROOT.rglob("*.py"):
if py_path.name in EXEMPT_FILENAMES: # the adapters and the factory
continue
tree = ast.parse(py_path.read_text())
for node in ast.walk(tree):
if isinstance(node, ast.Import):
names = [a.name for a in node.names]
elif isinstance(node, ast.ImportFrom) and node.module:
names = [node.module]
else:
continue
violations += [f"{py_path}: {n}" for n in names if n.startswith(FORBIDDEN_PREFIXES)]
assert not violations, "Use create_queue() instead of importing a broker SDK."It parses the code with
ast instead of searching text. A text search can't tell an import from a mention, so it would flag docstrings and comments that name the SDKs. Parsing finds every real import, including ones inside functions, and nothing else. Run it in CI and the boundary is checked on every change.The trade-offs
Keep the interface small on purpose. I wanted to add batching, delays and message expiry. Each one works on some brokers and not others, and each would push broker-specific behavior back into the interface. Two methods cover most producer and consumer code. If one service needs a broker-specific feature, use that adapter directly in that one place.
Every abstraction has a seam, so document it. Google Cloud Pub/Sub separates topics, where you publish, from subscriptions, where you consume. This interface has one queue name, so the Pub/Sub adapter treats it as the topic when publishing and as the subscription when subscribing. That's fine for simple point-to-point messaging. If you need fan-out to several subscriptions, use the Pub/Sub adapter directly. The adapter's docstring says so, which is better than discovering it in production.
Adding a broker is a contained change. Supporting Amazon SQS, for example, would mean a new adapter module, a new factory option, a new optional install and a new smoke test.
OrderProducer and OrderConsumer would not change.Going further: make it repeatable
Once this worked, I packaged it as a custom Kiro Power, so anyone on a team can ask Kiro for "a broker-agnostic queue package" and get the interface, all four adapters, the factory and the tests in one step. If you want to do the same, the Kiro docs on creating Powers are the place to start.
You don't need a Power to use the approach, though. In your next spec, tell Kiro:
- The two operations your code actually needs.
- The brokers you want to support.
- That application code must go through a factory and never import an SDK.
- That you want an in-memory implementation for tests.
Then ask Kiro to write a test that enforces point 3. That last step is what keeps the abstraction in place after the first commit.
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article