Using AgenticAI to power the data platform and dataOps
Watch how a simple natural‑language request transforms into complex queries running seamlessly on massive Iceberg datasets across Redshift, Athena, and other cutting‑edge data services. Explore our next‑generation data platform architecture powered by agentic AI, and see how strong DataOps practices lay the foundation for scalable, intelligent systems that drive real‑world innovation.
This episode of the Let's Talk About Data Show, hosted by Ibrahim Emara (Database Specialist Solutions Architect at AWS), featured guests Sid (Principal Technical Account Manager) and Vlad (Senior Technical Account Manager). The discussion centred on a common challenge facing enterprise customers: data scattered across multiple platforms (relational databases, S3 buckets, data lakes) and the difficulty of extracting insights efficiently. The team presented a solution they built that uses agentic AI to allow analysts to query massive datasets using natural language, without needing to know where the data resides or how to write optimised SQL.
The demo showcased a gaming dataset with over 158 million rows stored on Apache Iceberg via S3 Tables. A main orchestrator agent receives natural language queries, determines query complexity, and routes requests to the appropriate compute engine — Athena for simple ad-hoc queries or Redshift for complex multi-join operations. Sub-agents convert natural language to SQL, execute queries via Lambda, and a "probe agent" then generates human-readable insights, executive summaries, and suggested follow-up questions. The architecture also includes a semantic layer (OpenSearch) that caches vectorised queries, enabling the system to bypass the full agent chain for previously-asked similar questions — reducing latency and cost.
Vlad introduced the concept of DataOps as a strategic framework with five pillars (architecture, cost/observability, security, governance, and automation/testing) that keeps data platforms clean, reliable, and AI-ready. He also demonstrated a Redshift MCP (Model Context Protocol) that allows agents to run operational health reviews using natural language. The team emphasised governance throughout — every agent interaction is logged in DynamoDB for full traceability, and security is enforced through both Lake Formation permissions and Bedrock Guardrails to block access to sensitive data like credit card details.
Bullet Point Highlights
- Multi-agent architecture — A main orchestrator agent routes queries to specialised sub-agents (Athena agent, Redshift agent, Customer 360 agent, Probe agent, Governance agent)
- Natural language to SQL — Analysts query data in plain English; agents handle SQL generation and execution
- Smart compute routing — The system automatically selects Athena (simple/ad-hoc) or Redshift (complex/multi-join) based on query complexity, optimising both cost and performance
- S3 Tables with Apache Iceberg — Used as the single source of truth, providing flexibility, cross-region resilience, and openness to any compute engine
- Semantic caching layer — OpenSearch stores vectorised queries; similar questions bypass the agent chain entirely, reducing latency and token costs
- Probe agent insights — Goes beyond raw results to provide executive summaries, pattern analysis, prioritised actions, and follow-up question suggestions
- Full governance & audit trail — All interactions logged in DynamoDB (who asked, what was asked, which SQL ran, what results returned)
- Security — Dual-layer protection via Lake Formation (column masking/blocking) and Bedrock Guardrails (blocking PII/credit card queries)
- Model flexibility — Configurable model selection (demonstrated with Claude Opus 4.8); different models tested for consistency before production rollout
- DataOps framework — Five pillars to strengthen data foundations in parallel with building agentic applications
- Redshift MCP — Enables natural language operational reviews (health checks, slow query identification, storage utilisation)
- UI built with Kiro — Flask application using Python APIs; GitHub repo planned after internal security review
- Extensible design — Can add Snowflake or other engines by adding corresponding agents with access to the Iceberg data layer
Hosts of the show 🎤
Ibrahim Emara - Database Specialist SA @AWS
Guests 🎤
Sidhanth Muralidhar - Cloud, Data & AI | Principal TAM @AWS
Vladyslav Siniavin - Cloud & AI Strategy | Sr. TAM @AWS
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article