
Building an asynchronous AI image generation pipeline with BullMQ and Redis
A practical overview of designing an asynchronous AI image generation pipeline with NestJS, BullMQ, Redis, provider adapters, and S3-compatible object storage.
Why asynchronous generation matters
AI image generation can take several seconds or longer. Treating it like a normal synchronous HTTP request creates a poor user experience: requests may time out, servers remain occupied, and it becomes difficult to retry failed work safely.
While building GENZAI, an AI multimedia platform, I designed an asynchronous pipeline to process image generation jobs reliably. The core idea is simple: the API accepts a request quickly, a background worker performs the expensive generation task, and the user can track progress without keeping the original HTTP request open.
High-level architecture
The pipeline consists of the following components:
- Next.js frontend for prompt input, job tracking, and displaying generated assets.
- NestJS API for authentication, validation, credit checks, and job creation.
- Redis and BullMQ for durable background-job queues.
- Worker service for calling AI providers and processing results.
- PostgreSQL for job status, assets, and credit-ledger records.
- S3-compatible object storage for generated images and thumbnails.
- Provider adapters to keep integrations with AI models interchangeable.
The request flow is:
User request → API validation and authorization → create RenderJob record → enqueue BullMQ job → worker calls an AI provider → store output in object storage → update job status and asset record → frontend retrieves the completed result
Job lifecycle
Each generation request is represented by a
RenderJob . The job progresses through predictable states:QUEUED → RUNNING → SUCCEEDED ↘ FAILED
The API creates the job with the
QUEUED status and returns its identifier immediately. A BullMQ worker consumes the job and changes the status to RUNNING . Once the provider returns a valid image and the asset is stored, the job becomes SUCCEEDED . If a recoverable error occurs, BullMQ retries the job using exponential backoff. If the maximum retry count is reached, the job is marked as FAILED .This state model makes the system easier to observe, debug, and scale.
Why BullMQ and Redis?
BullMQ provides several features that are useful for long-running AI workloads:
- Separate API response time from generation time.
- Configurable concurrency for image and video workloads.
- Retry policies with exponential backoff.
- Progress reporting for the frontend.
- Queue monitoring and failure inspection.
- Horizontal scaling by running additional workers.
Redis acts as the queue backend. The API and workers can therefore scale independently. For example, the API can handle many users submitting prompts while a smaller pool of workers processes generation jobs at a controlled rate.
Provider adapters
AI providers have different request formats, response structures, latency, and error behavior. To avoid coupling the application to one provider, I use an adapter layer.
A simplified interface looks like this:
generateImage(input) → standardized result
The rest of the system does not need to know whether the result came from a managed cloud model, a local model, or a fallback provider. This approach makes it easier to add providers, introduce fallback behavior, and compare cost or latency later.
Storage and asset management
Generated files should not be stored directly in the database. Instead, the worker uploads the output to S3-compatible object storage and writes metadata to PostgreSQL.
Typical metadata includes:
- Asset type, such as image or video
- Storage URL and thumbnail URL
- Width and height
- Associated project and user
- Creation time
- Related generation job
This keeps the relational database focused on transactional data while object storage handles large media files efficiently.
Cost control and reliability
AI generation has a direct cost, so the job lifecycle is connected to a credit ledger:
- Validate the user quota or available credits.
- Place a temporary credit hold when the job is accepted.
- Convert the hold to usage after successful generation.
- Refund the hold if the job fails permanently.
A prompt cache can also reduce repeated cost. If a normalized prompt and configuration already produced an asset recently, the system can return the cached result instead of submitting another generation job.
Lessons learned
The most important lesson is that asynchronous processing is not only about performance. It is also about building a system that is observable, recoverable, and cost-aware.
By combining a queue, explicit job states, provider adapters, object storage, and a transactional credit flow, an AI generation platform can handle slow and variable workloads without making the user wait on a single HTTP request.
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article