GreenQueue : Carbon Aware Workload Scheduler
GreenQueue is a carbon-aware job scheduler built on AWS for the Environmental Hacks. Users submit a job with a deadline, and the system uses live grid carbon data for Northern India to run it at a cleaner time. It uses Lambda, DynamoDB, SQS, EventBridge, and EC2 Auto Scaling (scaling from zero), is deployed with Terraform, and shows estimated energy and CO2 avoided in a Grafana dashboard.
GreenQueue: Running Cloud Jobs When the Grid Is Cleaner
Built for the WeMakeDevs Bharat Builds Tour - Environmental Hacks (AWS)
The problem
Cloud teams run many jobs that are not urgent: builds, batch processing, backups, model training. These jobs use electricity, and the carbon in that electricity changes during the day. The same job can create more or less CO2 depending on when it runs. Most teams cannot see this, and they run jobs at whatever time is convenient.
What GreenQueue does
A user submits a job with a deadline. GreenQueue then:
- Reads live grid carbon intensity and forecast data for Northern India (from Electricity Maps).
- Holds the job if a cleaner time is coming before the deadline.
- Releases the job when the grid is clean enough, or when the deadline requires it.
- Runs the job on workers that scale up on demand and back to zero when idle.
- Shows estimated energy and CO2, including how much was avoided compared with running the job immediately.
How I built it on AWS
Flow: Frontend -> Lambda API -> DynamoDB -> scheduler -> SQS -> Auto Scaling workers -> DynamoDB -> Grafana
- AWS Lambda (Function URL): three functions: the job API, the scheduler (every minute), and an hourly ingest of grid carbon data.
- Amazon DynamoDB: two tables, one for jobs and results, one for grid carbon data (live values and forecast).
- Amazon EventBridge: triggers the scheduler every minute and the ingest every hour.
- Amazon SQS (with a dead-letter queue): the scheduler adds a job ID only when the job is allowed to run, which keeps control logic and workers separate.
- Amazon EC2 Auto Scaling + CloudWatch: alarms on queue depth scale workers from 0 to N, and back to 0 when idle, so there is no cost for idle servers.
- Amazon S3: static website hosting for the frontend.
- AWS Systems Manager: Parameter Store (SecureString) holds the Electricity Maps token, which the ingest Lambda reads at runtime so it never appears in the website. Session Manager gives access to servers without SSH keys or open ports.
- AWS IAM: separate roles for the Lambdas, the ingest function, the workers, and monitoring.
- Terraform: all infrastructure is in modules. I destroyed and rebuilt the whole stack with one command to check it is reproducible.
- Prometheus and Grafana (open source): a small exporter reads DynamoDB and SQS, and Grafana shows job status, queue depth, estimated energy, and CO2.
How CO2 is estimated
The worker measures CPU use while the job runs and converts it to power with a simple model (minimum and maximum watts per vCPU).
energy (kWh) = power (kW) x time (hours)
CO2 = energy x grid carbon intensity
"Run now" and "run later" use the same energy, so the savings come only from running at a cleaner time. These are estimates, not meter readings.
Demo
I added a switch to simulate a dirty grid (800 gCO2/kWh) and a clean grid (300 gCO2/kWh). On the dirty grid, new jobs wait and no workers run. When I switch to clean, the scheduler releases the jobs, workers start, and the dashboard shows the CO2 avoided. In this simulated switch, the same job produces about 62% less CO2. This is a simulated example, not a measured result.
What I learned
- Lambda Function URLs only show "Internal Server Error" when code fails. The real error is in CloudWatch Logs.
- DynamoDB is simple once you understand keys, but Python returns Decimal values that need conversion before JSON.
- With Auto Scaling on SQS, messages become "in flight" while a job runs, so the visible queue looks empty. I used scale-in protection so a busy worker is not terminated.
- IAM is easier with one role per component, but some actions (like ec2:DescribeInstances) cannot be limited to one resource.
- Terraform modules made it safe to rebuild and fix things quickly.
Limits
- Energy and CO2 values are estimates from a CPU power model.
- One job saves only a small amount. The value is in the pattern, applied to many jobs.
- Live grid data depends on the Electricity Maps API. If it is unavailable, the scheduler uses a built-in hourly profile.
- The demo workload is a timed CPU job, not a real user workload yet.
- The demo API is public and unauthenticated, and the frontend uses the S3 HTTP website endpoint. It is a demo, not production-ready.
Next steps
- Real workload types (for example CI builds or ML inference) in Docker images.
- API authentication, and CloudFront with HTTPS for the frontend.
- Tracking forecast accuracy (predicted vs actual intensity).
- More regions.
I used AI assistants (GitHub Copilot and Claude) for code suggestions and debugging. The architecture, deployment, and testing were done by me.
Tags: aws, serverless, terraform, sustainability, sqs, dynamodb, auto-scaling
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article