Building AIDataGate — a DataHub‑powered Operations Copilot (Architecture, Observability, and Deployment)
A concise, hands-on walkthrough that guides developers through building, testing, and running the AIDataGate FastAPI app locally and in containers, covering core features like caching, session management, observability, and automated tests.
Building AIDataGate — a DataHub‑powered Operations Copilot
AIDataGate is a metadata‑aware operations copilot that reads DataHub lineage, ownership, and quality context and returns prioritized, explainable recommendations for data teams. This post walks through the architecture, key implementation choices, observability setup, testing strategy, and deployment steps used to produce a reliable, demo‑ready service.
Why this matters
- Metadata grounding makes recommendations actionable (ownership, downstream impact)
- Observability and testing make agent deployments safe and debuggable
- Resilient caches and graceful degradation reduce outage-induced surprises
Architecture overview
- FastAPI backend that orchestrates services
- DataHub client (async + cache + 2‑hop lineage)
- RecommendationEngine: insight extraction + confidence scoring
- RenderingService: Jinja2 templates, role personalization
- SessionService: JSON session context persisted to DB + Redis cache
- AuditService + Feedback API persisted to Postgres
- Observability: OpenTelemetry → OTEL Collector → Jaeger / Prometheus → Grafana
- Docker Compose for local infra (Postgres, Redis, Prometheus, Grafana, Jaeger)
Diagram: see the cover image at the top of this post.
Quick start (local)
Prerequisites: Docker, Docker Compose, Python 3.14 (recommended), git
Clone and run:
git clone https://github.com/\<org\>/aidatagate-agent-copilot.git
cd aidatagate-agent-copilot
cp .env.example .env
docker compose up -d
# create venv and install deps
python -m venv .venv
source .venv/bin/activate # or .\.venv\Scripts\Activate.ps1 on Windows
pip install -r requirements.txt
python main.pyVerify:
curl http://localhost:8000/health
# Prometheus: http://localhost:9090
# Grafana: http://localhost:3000
# Jaeger: http://localhost:16686Core implementation patterns (with snippets)
1) Safe Redis caching (serialize non‑primitive values)
Don't pass Python dicts directly to redis-py. Serialize and deserialize consistently.
import json
def cache_set(client, key, value, ttl=3600):
if not isinstance(value, (str, bytes, int, float)):
store = json.dumps(value)
else:
store = value
await client.set(key, store, ex=ttl)
def cache_get(client, key):
stored = await client.get(key)
if stored is None:
return None
try:
return json.loads(stored)
except Exception:
return storedThis avoids redis.exceptions.DataError and keeps cached objects portable.
2) SQLAlchemy JSON mutation detection
SQLAlchemy may not persist in‑place changes to JSON columns unless flagged as modified. Use flag_modified after updating the dict.
from sqlalchemy.orm.attributes import flag_modified
session.context.update(new_context)
flag_modified(session, 'context')
db.commit()3) Idempotent Prometheus registry in tests
When the app is started multiple times (TestClient lifecycles), registering metrics into the default registry can cause DuplicateTimeseries. Create a dedicated CollectorRegistry per app instance or guard against double registration.
from prometheus_client import CollectorRegistry
registry = CollectorRegistry()
hist = Histogram('http_request_duration_seconds', '...', registry=registry)Observability & error handling
- Use OpenTelemetry to trace the full orchestrator flow (DataHub calls, rendering, DB/cache interactions).
- Metrics: request latencies, cache hits/misses, endpoint error counters.
- Error categories: validation, authentication, service, internal — emit labeled metrics so SLOs can be tracked.
Middleware pattern (error handling):
try:
response = await call_next(request)
if response.status_code \>= 400:
# ensure request_id present in body
return JSONResponse(...)
except Exception as exc:
# log, categorize, emit metric, invoke app handler if present
return JSONResponse(status_code=500, content={...})Testing strategy
- pytest + pytest-asyncio for async tests
- coverage collection during CI: pytest --cov=app tests/
- focused tests: recommendation engine, ownership service, rendering templates, session handling, observability middleware
- avoid registering global metrics during tests — use a per-app registry (see above)
CI & Docker
Sample GitHub Actions (summary):
name: CI
on: [push]
jobs:
test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- name: Set up Python
uses: actions/setup-python@v4
with: python-version: '3.14'
- run: python -m pip install -r requirements.txt
- run: pytest --cov=app tests/
build:
needs: test
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- name: Build and push Docker
uses: docker/build-push-action@v4
with:
push: true
tags: ${{ secrets.REGISTRY }}/aidatagate:latestLessons learned & gotchas
- Prometheus duplicate registration is a common pitfall when creating metrics in module scope — prefer a per-app registry or idempotent setup.
- Always serialize complex cache values before storing in Redis.
- Mark SQLAlchemy JSON fields as modified after in‑place updates.
Where to find the code
- Repository:
https://github.com/abdelrazekrizk/aidatagate-agent-copilot
- Demo & submission assets:
Screenshots and assets
- Include the following in the repo under
- prometheus.png (Prometheus query page)
- grafana.png (Dashboard)
- jaeger.png (trace view)
- health.png (API health response)
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article