
Production-Ready Real-Time Monitoring Solution for LLMs on Amazon SageMaker AI Endpoint inference
Building a comprehensive LLM monitoring solution presents unique challenges: high-volume inference data, evaluating complex model outputs, & maintaining real-time visibility—all while keeping costs under control. Here is a cost-effective monitoring solution using AWS first-party (1P) services. By leveraging Amazon SageMaker's built-in data capture capabilities, AWS Step Functions, AWS Lambda, we create an automated solution that monitors LLM inferences in near-real-time with minimal operational overhead.
As organizations deploy Large Language Models (LLMs) into production, monitoring their performance, quality, and behavior becomes critical for maintaining reliable AI systems. LLMs change behavior with prompt drift, new user cohorts, and updated system prompts, so you need continuous visibility into quality, latency, and safety signals instead of one‑off offline tests. For generative workloads, monitoring also underpins responsible AI: you can track toxicity, hallucination rates, and policy violations, and feed those metrics back into fine‑tuning, prompt engineering, or guardrail updates. Unmonitored endpoints risk silent failures such as growing latency, degraded answer quality, and escalating inference spend when traffic or prompts change in production. Unlike traditional ML models, LLMs present unique monitoring challenges:
- Output variability: LLMs generate free-form text that's difficult to validate with traditional metrics
- Prompt sensitivity: Small changes in input prompts can dramatically affect outputs
- Hallucinations and factual accuracy: Models may generate plausible but incorrect information
- Safety and toxicity: Outputs must be monitored for harmful or biased content
- Cost and latency: Token usage and response times directly impact user experience and operational costs
Without proper monitoring, organizations risk deploying models that degrade over time, produce unsafe outputs, or fail to meet business requirements. A managed monitoring stack helps teams standardize evaluation, centralize traces, and create auditable evidence for compliance and incident reviews.
In this blog post, we'll demonstrate how to build a scalable, cost-effective monitoring solution using AWS first-party (1P) services. By leveraging Amazon SageMaker AI Endpoint built-in data capture capabilities, SageMaker AI MLflow Apps, AWS Step Functions, AWS Lambda, and Amazon Bedrock, we create an automated pipeline that monitors LLM inferences in near-real-time with minimal operational overhead.
Architecture using 1P SageMaker AI features
The solution monitors an Amazon SageMaker real-time endpoint with Data Capture functionality enabled, which automatically logs the SageMaker real-time endpoint request and response payloads with metadata to Amazon S3 bucket in near real time as JSON Lines (.jsonl) files, as illustrated in the architecture diagram below.

ML Engineers and infrastructure admins configure Data Capture once in the SageMaker endpoint configuration, specifying capture mode (request, response, or both), the S3 destination, and sampling rate.
Each captured file (Endpoint data capture .jsonl file) is written to a configured S3 prefix within seconds of the endpoint invocation, containing records with the full input, model output, timestamps, and endpoint metadata. An S3 event notification or EventBridge rule then triggers an AWS Step Functions state machine whenever a new data capture jsonl object is inserted, turning SageMaker endpoint data capture traffic into an automated near‑real‑time monitoring pipeline.
Within Step Functions using its Lambda function, it parses the SageMaker Data Capture JSONL file, batches records, and prepares them for MLflow tracing and MLflow LLM evaluation. This workflow subsequently logs the traces (SageMaker endpoint request-response) and evaluation to the Amazon SageMaker AI MLflow App, as seen in the screenshot below.

The SageMaker AI MLflow LLM Evaluations run asynchronously after traces are logged, preventing bottlenecks with results are stored alongside the originating traces in MLflow for unified analysis, as shown below for illustration.

SageMaker AI MLflow App provides a fully managed tracking app in Amazon SageMaker AI Studio, allowing users to log traces and LLM evaluation without operating their own monitoring MLflow infrastructure, as shown in expanded view of a processed SageMaker Endpoint request-response.

The custom monitoring solution can create an MLflow trace per batch or per endpoint, attach tags such as endpoint name, model version, and timestamp, and log metrics like toxicity scores, hallucination rates, latency percentiles, and any user‑defined business KPIs, as shown below of a SageMaker data capture MLflow trace attribute with custom metadata.

These MLflow trace (trace with LLM evaluations) are visible in the SageMaker AI MLflow App UI or can be accessed through APIs, where teams can use MLflow filters to filter by endpoint, time window, or version, and compare evaluation scores across deployments or configuration changes, enabling deeper debugging, scenario replay and root‑cause analysis, as shown below a request that SageMaker Endpoint LLM has failed to be serve with ERROR response status code 424 .

Scaling this pattern to production
SageMaker Data Capture configuration can be integrated with any SageMaker AI real-time endpoint, and the same pattern can scale to many LLMs or many tenants by configuring different S3 prefixes and routing the data capture processing events to per‑endpoint or shared Step Functions workflows. The processed data can be logged to dedicated MLflow experiments for clear organization or logged to dedicated SageMaker MLflow App to help maintain isolation for multi‑team or multi‑tenant environments. For operational visibility, you can surface the captured key metrics such as evaluation failure rates, toxicity distributions, and latency trends in CloudWatch or existing observability dashboard, giving personas like site-reliability engineer (SREs) and data science teams a unified view of LLM endpoint health. The monitored traces can be used in the MLflow App for data science analysis to understand the behavior of LLMs at scale as show in the scatter plot of over 1,000 processed traces.

To scale this pattern to production across multiple accounts, you can deploy SageMaker LLM endpoints in individual project or tenant accounts while routing their data capture events to a centralized, shared‑services ML monitoring account, shown in the below architecture. In this setup, each production account publishes S3 data capture events via EventBridge to a central event bus in the ML account, where Step Functions, Lambda, and MLflow orchestrate trace ingestion, evaluation, and monitoring. This hub‑and‑spoke, event‑driven architecture enables millisecond‑latency telemetry ingestion and millions of events per second (demonstrated in the blog ), all while keeping monitoring logic, models, and governance artifacts isolated and centrally managed.

AWS Step Functions and Lambda both scale horizontally with traffic as the S3 data capture events increase, so this monitoring system naturally scales with higher endpoint request-response volumes without manual capacity management. Furthermore, users can leverage AWS Step Functions with Distributed Map (blog) , with this pattern to further scale efficiently across thousands of items in parallel, automatically distributing workloads across multiple executions. Step Function’s Distributed Map enables near‑linear scaling for large JSONL files, so evaluation and transformation workloads can keep pace with growing data capture volumes from SageMaker AI endpoints.
Often production LLM use-case requires unique metrics to be monitored and with this solution both the trace ingestion and the LLM evaluation workflows are highly customizable allowing data science users to introduce custom code to monitor LLM endpoints based on their LLM use-case and governance requirements.
Together, administrators can have centralized LLM safety and quality alerting policies by encoding them as MLflow evaluation metrics (e.g., minimum answer relevance, maximum toxicity) and using them to trigger alerts when thresholds are violated. See the experimentation jupyter notebook for insights into the concepts involved and various customization options available.
Cost considerations and optimization strategies
The SageMaker AI endpoint Data Capture itself is a configuration on existing real-time endpoint, so its incremental cost comes primarily from additional S3 storage for captured payloads and downstream compute for Step Functions and Lambda Function executions. You can control the overall solution downstream S3, processing compute and LLM evaluation costs at source by setting the Data Capture sampling percentage to a value less than 100% which reduces the number of logged requests while still maintaining representative coverage, (e.g., start with 10-20% sampling for baseline monitoring and increase to 100% only for critical production deployments).
At a component level, you can further optimize for cost in Amazon S3 bucket through automatic storage class transitions and data expiration. Use lifecycle rules to transition objects to lower-cost storage classes (e.g., from S3 Standard to S3 Standard-IA after 30 days) or delete them when no longer needed (e.g., after the SageMaker endpoint LLM data capture jsonl file is processed it can be deleted to optimize for cost).
For the compute components - Step Functions & lambda functions – the charges are per state transition and Lambda per invocation duration, so batching multiple captured records per evaluation run keeps overhead low. For example, transition to a batch processing by configuring EventBridge to trigger on file size thresholds rather than every file or process multiple DataCapture files in a single Lambda invocation, reducing the Lambda function invocations and Step Functions state transitions.
Finally, for the SageMaker MLflow LLM evaluations component, selecting lightweight LLM‑as‑judge model (trade off accuracy vs. cost) provides viable options in controlling the LLM invocation costs during MLflow LLM evaluations. Furthermore, you can use lightweight metrics (toxicity, readability, and heuristic) for continuous monitoring and avoid running expensive LLM based evaluations on every execution.
Conclusion
As Large Language Models (LLMs) increasingly power mission-critical applications, monitoring their performance, safety, and quality in production has become essential. Monitoring LLMs in production is complex, but AWS first-party services dramatically simplify the challenge. With this event-driven architecture you can use production-ready monitoring solution that:
- Deploys in minutes with infrastructure-as-code.
- Scales automatically with your inference volume.
- Provides real-time insights into LLM behavior.
- Optimizes costs through intelligent sampling and serverless architecture.
- Integrates seamlessly with existing SageMaker workflows and requires minimal operational overhead.
Ready to implement the production-ready LLM monitoring solution? Check out our GitHub repository for complete code examples and infrastructure-as-code CDK templates to get started in minutes.
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article