
From Notebook to Production: Building a Real-Time Fraud Detection MLOps Pipeline on AWS!
Wiring up SageMaker AI, EventBridge, and GitHub Actions so a fraud model trains, ships, and retrains itself, with a person signing off on every release.
Why I wrote this (and why you should finish reading it)
You’ve probably been here. The model works in your notebook, the accuracy number looks great, and then someone asks how it gets to real users. Silence.
Most tutorials end at
model.fit() . The part nobody shows you is everything after it: getting the model behind an API, retraining it when the data shifts, and making sure someone notices before it quietly starts letting fraud through.
That part is MLOps, and it’s what this post is about. We’ll build one project from start to finish, the kind you can talk through in an interview without hand-waving: a real-time fraud detection pipeline on AWS.
I picked fraud on purpose. There’s real money behind it. Nasdaq Verafin put global losses from fraud scams and bank fraud at about $485.6 billion for 2023.
The good news is that ML is winning real battles. In Mastercard’s 2025 survey of 300 payments executives, run with FT Longitude, 42% of card issuers and 26% of acquirers said AI had saved them more than $5 million in fraud attempts over two years.
Banks and payment companies run systems like this every day. Ours will be small, but every piece of it is real.
🏭 The Analogy: Think of MLOps as a Car Factory, Not a Garage
Before touching AWS, one picture helps.
- A notebook is a mechanic building one car by hand in a garage. It can be a lovely car. But it’s slow, nobody else can rebuild it exactly, and if the mechanic is off sick, nothing ships.
- MLOps is the assembly line. Parts are standard, each station checks the work, and a bad part gets pulled before a customer ever sees it. Anyone on the team can build the same car again tomorrow.
Each piece of our setup has a station on that line:

🎯 Project Goal
Build an end-to-end, automated MLOps pipeline on AWS that:
- Load the Kaggle transactions into SageMaker Feature Store.
- Train an XGBoost model inside a SageMaker Pipeline.
- Score it on held-out data, and stop right there if it isn’t good enough.
- Park every model that passes in the Model Registry until a person approves it.
- Ship approved models to a real-time endpoint, with nobody running a deploy script by hand.
- Put an API (API Gateway and Lambda) in front so other apps can call it.
- Watch incoming data for drift.
- Retrain every night, and sooner if drift shows up.
So the model can retrain on its own, but it can’t promote itself. A human still decides what goes live, which is the right call for anything that touches people’s money.
🧰 Architecture Overview
Here’s the full picture, assembly-line style:

It’s the same shape AWS describes in its own SageMaker MLOps guidance. Pipelines run the workflow, the Model Registry keeps track of versions and approvals, and CI/CD handles promotion.
✅ What you’ll need
- An AWS account where you can use SageMaker AI, S3, Lambda, EventBridge, API Gateway, Firehose, and CloudWatch, and pass IAM roles (
iam:PassRole).
- Python 3.10 or newer, and basic Git and GitHub.
- The Kaggle “Credit Card Fraud Detection” dataset (
creditcard.csv). It has 284,807 anonymized card transactions, and only 492 of them are fraud. Keep that ratio in mind, because it drives almost every decision below.
# Pin SDK v2: v3 removed the Estimator/Model classes used belowpip install "sagemaker>=2.200,<3" boto3 pandas numpy scikit-learn xgboostOne thing that will trip you up: version 3 of the SageMaker Python SDK changed a lot and dropped the classes this post uses. I’ve pinned v2 on purpose, since most examples and most production code out there still run on it.
Step 1: S3 bucket and IAM role
Nothing runs without somewhere to put the data and a role that’s allowed to touch it.
export BUCKET=fraud-detection-mlops-<your-unique-id>aws s3 mb s3://$BUCKETaws s3 cp creditcard.csv s3://$BUCKET/raw-data/creditcard.csv
You don’t need to create empty “folders” in S3. They’re only key prefixes, so uploading the file creates the path.

For the execution role, attach these managed policies and give it read and write access to your bucket (via console or CLI). That’s fine for a tutorial.
-
AmazonSageMakerFullAccess
-
AmazonS3FullAccess(scope this down to your bucket in production)
-
CloudWatchFullAccess
💡 Real-world tip: In production, never useFullAccesspolicies. Scope IAM permissions to exact resource ARNs. This is one of the first things a senior engineer checks in a code review.
AWS’s own Well-Architected guidance backs this up: define your ML workflow architecture by mapping out your end-to-end ML workflow, identify which steps can be automated and which require human intervention, and determine the appropriate level of separation between development, testing, and production environments.
Step 2: Data preparation and the Feature Store
Back to the factory for a second. The Feature Store is the shared parts bin: every station takes the same versioned parts from it instead of machining its own. It has an online store for fast lookups while you score a transaction, and an offline store (Parquet files in S3) for training and audits.
import timeimport pandas as pdimport sagemakerfrom sagemaker.feature_store.feature_group import FeatureGroup
session = sagemaker.Session()role = "arn:aws:iam::<account-id>:role/SageMakerExecutionRole"bucket = "fraud-detection-mlops-<your-unique-id>"
df = pd.read_csv("./creditcard.csv").rename(columns={"Class": "is_fraud"})df["transaction_id"] = df.index.astype(str).astype("string") # object dtype is rejecteddf["event_time"] = float(round(time.time()))
feature_group = FeatureGroup(name="fraud-detection-features", sagemaker_session=session)feature_group.load_feature_definitions(data_frame=df)feature_group.create( s3_uri=f"s3://{bucket}/feature-store", record_identifier_name="transaction_id", event_time_feature_name="event_time", role_arn=role, enable_online_store=True,)
# create() returns immediately; wait before ingestingwhile feature_group.describe()["FeatureGroupStatus"] == "Creating": time.sleep(10)feature_group.ingest(data_frame=df, max_workers=4, wait=True)print("Features ingested into Feature Store")
What this code actually creates.
create() registers a real feature group in SageMaker AI, and the data lives in two places:- An online store that AWS manages for you. It’s built for millisecond lookups by
transaction_id(get_record), and you won't see it as files anywhere.
- An offline store in your bucket under
feature-store/, written as Parquet files partitioned by year, month, day, and hour. SageMaker also registers a Glue Data Catalog table for it, so you can query it with Athena.
To see it, open SageMaker Studio from the SageMaker AI console then choose Data → Feature Store in the left menu. From a terminal,
aws sagemaker list-feature-groups works too.
Be patient with
ingest() . It sends every row through the PutRecord API, so all 284,807 rows can take a while. Plan on roughly 15 to 30 minutes with four workers, depending on where you run it from, and give the Parquet files a few more minutes to show up in S3. For a first run, try a sample such as df.sample(20_000, random_state=42) .

When you’re done,
feature_group.delete() removes the feature group and its online store. It leaves the Parquet files in S3, so delete that prefix yourself.
A word on data preparation. With this dataset there isn’t much to do, because the columns V1 to V28 were already run through a PCA transform before Kaggle published them. In a real bank, this is where the heavy lifting happens: cleaning raw transactions, checking for missing or impossible values, and building features such as how many purchases a card made in the last hour or how far the merchant is from the cardholder’s usual location. Run that as a SageMaker Processing job and write the results to the Feature Store, so training and real-time scoring use exactly the same features.
A confession: to keep this post a sane length, the pipeline below reads the raw CSV straight from S3. In production, training would query the offline store through Athena (feature_group.athena_query()), and the inference Lambda would pull account-level features, such as spend velocity or merchant history, from the online store withget_record. That shared source is what actually keeps training and serving from drifting apart.
⭐️ Create the Model package group in AWS Sagemaker which provides the capability of version control for models, comparing its metrics and then approving (approval gate), & even the deployment trigger.
aws sagemaker create-model-package-group model-package-group-name FraudDetectionModelGroup model-package-group-description "Fraud detection models"

Step 3: The training pipeline
This is where the notebook turns into something you can repeat. Preprocessing, training, evaluation and the quality check all live in code, so a run looks the same whether you started it, the nightly schedule did, or a drift alert did.
First,preprocessing.py. I split by time rather than at random, so the model is always tested on transactions that came after the ones it learned from. A random split leaks the future into training, and the scores end up looking better than they will in real life.
import osimport pandas as pd
df = pd.read_csv("/opt/ml/processing/input/creditcard.csv").sort_values("Time")label = df.pop("Class")features = df.drop(columns=["Time"]) # V1..V28 + Amount = 29 featuresdata = pd.concat([label, features], axis=1) # built-in XGBoost wants the label first
n = len(data)splits = { "train": data.iloc[: int(0.70 * n)], "validation": data.iloc[int(0.70 * n) : int(0.85 * n)], "test": data.iloc[int(0.85 * n) :],}for name, part in splits.items(): os.makedirs(f"/opt/ml/processing/{name}", exist_ok=True) part.to_csv(f"/opt/ml/processing/{name}/{name}.csv", header=False, index=False)
Next, “evaluate.py”. It reports PR-AUC, which is the number to watch when fraud is 0.17% of your data.
import json, os, tarfileimport pandas as pdimport xgboost as xgbfrom sklearn.metrics import average_precision_score, roc_auc_score
with tarfile.open("/opt/ml/processing/model/model.tar.gz") as tar: tar.extractall(path=".")booster = xgb.Booster()booster.load_model("xgboost-model")
test = pd.read_csv("/opt/ml/processing/test/test.csv", header=None)y_true = test.iloc[:, 0]y_score = booster.predict(xgb.DMatrix(test.iloc[:, 1:].values))
report = { "binary_classification_metrics": { "pr_auc": {"value": float(average_precision_score(y_true, y_score))}, "roc_auc": {"value": float(roc_auc_score(y_true, y_score))}, }}os.makedirs("/opt/ml/processing/evaluation", exist_ok=True)with open("/opt/ml/processing/evaluation/evaluation.json", "w") as f: json.dump(report, f)
And the pipeline itself, “pipeline.py”:
from sagemaker import image_urisfrom sagemaker.estimator import Estimatorfrom sagemaker.inputs import TrainingInputfrom sagemaker.model import Modelfrom sagemaker.model_metrics import MetricsSource, ModelMetricsfrom sagemaker.processing import ProcessingInput, ProcessingOutput, ScriptProcessorfrom sagemaker.sklearn.processing import SKLearnProcessorfrom sagemaker.workflow.condition_step import ConditionStepfrom sagemaker.workflow.conditions import ConditionGreaterThanOrEqualTofrom sagemaker.workflow.fail_step import FailStepfrom sagemaker.workflow.functions import Join, JsonGetfrom sagemaker.workflow.model_step import ModelStepfrom sagemaker.workflow.parameters import ParameterFloat, ParameterStringfrom sagemaker.workflow.pipeline import Pipelinefrom sagemaker.workflow.pipeline_context import PipelineSessionfrom sagemaker.workflow.properties import PropertyFilefrom sagemaker.workflow.steps import ProcessingStep, TrainingStep
role = "arn:aws:iam::976765432158:role/Sagemaker_S3_Cloudwatch_Full_Access"bucket = "fraud-detection-mlops-by-harshitdawar"
pipeline_session = PipelineSession()region = pipeline_session.boto_region_namexgb_image = image_uris.retrieve("xgboost", region, "1.7-1")
input_data = ParameterString( "InputData", default_value=f"s3://{bucket}/raw-data/creditcard.csv")min_pr_auc = ParameterFloat("MinPrAuc", default_value=0.80)
# --- 1. Preprocess ---sklearn_processor = SKLearnProcessor( framework_version="1.2-1", role=role, instance_type="ml.m5.xlarge", instance_count=1, sagemaker_session=pipeline_session,)step_process = ProcessingStep( name="PreprocessFraudData", step_args=sklearn_processor.run( inputs=[ ProcessingInput(source=input_data, destination="/opt/ml/processing/input") ], outputs=[ ProcessingOutput(output_name=s, source=f"/opt/ml/processing/{s}") for s in ("train", "validation", "test") ], code="preprocessing.py", ),)outs = step_process.properties.ProcessingOutputConfig.Outputs
# --- 2. Train: built-in XGBoost on Spot capacity ---xgb = Estimator( image_uri=xgb_image, role=role, instance_type="ml.m5.xlarge", instance_count=1, output_path=f"s3://{bucket}/model-artifacts", use_spot_instances=True, max_run=3600, max_wait=7200, sagemaker_session=pipeline_session,)xgb.set_hyperparameters( objective="binary:logistic", eval_metric="aucpr", num_round=300, max_depth=5, eta=0.1, early_stopping_rounds=20, scale_pos_weight=578, # ≈ legitimate / fraud ratio in this dataset)step_train = TrainingStep( name="TrainFraudModel", step_args=xgb.fit( inputs={ "train": TrainingInput( outs["train"].S3Output.S3Uri, content_type="text/csv" ), "validation": TrainingInput( outs["validation"].S3Output.S3Uri, content_type="text/csv" ), } ),)
# --- 3. Evaluate: XGBoost image, so xgboost is installed ---eval_processor = ScriptProcessor( image_uri=xgb_image, command=["python3"], role=role, instance_type="ml.m5.xlarge", instance_count=1, sagemaker_session=pipeline_session,)evaluation_report = PropertyFile( name="EvaluationReport", output_name="evaluation", path="evaluation.json")step_eval = ProcessingStep( name="EvaluateFraudModel", step_args=eval_processor.run( inputs=[ ProcessingInput( source=step_train.properties.ModelArtifacts.S3ModelArtifacts, destination="/opt/ml/processing/model", ), ProcessingInput( source=outs["test"].S3Output.S3Uri, destination="/opt/ml/processing/test", ), ], outputs=[ ProcessingOutput( output_name="evaluation", source="/opt/ml/processing/evaluation" ) ], code="evaluate.py", ), property_files=[evaluation_report],)
# --- 4. Quality gate + registration ---model = Model( image_uri=xgb_image, role=role, sagemaker_session=pipeline_session, model_data=step_train.properties.ModelArtifacts.S3ModelArtifacts,)model_metrics = ModelMetrics( model_statistics=MetricsSource( s3_uri=Join( on="/", values=[ step_eval.properties.ProcessingOutputConfig.Outputs[ "evaluation" ].S3Output.S3Uri, "evaluation.json", ], ), # pipeline variables can't go in f-strings content_type="application/json", ))step_register = ModelStep( name="RegisterFraudModel", step_args=model.register( content_types=["text/csv"], response_types=["text/csv"], inference_instances=["ml.m5.large"], transform_instances=["ml.m5.large"], model_package_group_name="FraudDetectionModelGroup", approval_status="PendingManualApproval", model_metrics=model_metrics, ),)step_fail = FailStep( name="PrAucTooLow", error_message=Join(on=" ", values=["PR-AUC below threshold", min_pr_auc]),)step_gate = ConditionStep( name="CheckPrAuc", conditions=[ ConditionGreaterThanOrEqualTo( left=JsonGet( step_name=step_eval.name, property_file=evaluation_report, json_path="binary_classification_metrics.pr_auc.value", ), right=min_pr_auc, ) ], if_steps=[step_register], else_steps=[step_fail],)
pipeline = Pipeline( name="FraudDetectionPipeline", parameters=[input_data, min_pr_auc], steps=[step_process, step_train, step_eval, step_gate], sagemaker_session=pipeline_session,)pipeline.upsert(role_arn=role)execution = pipeline.start()print("Pipeline execution started:", execution.arn)
Run the "pipeline.py" first by keeping all 3 of these scripts in the same folder because this script will upload the other 2 to the default S3 bucket (created by this pipeline script) for SageMaker.




What just happened (in plain English):
- It prepares the data: it drops the raw Time column, puts the label first (the format built-in XGBoost expects), and splits by time, so the test set only holds “future” transactions.
- It trains XGBoost on Spot capacity and gives the rare fraud cases extra weight with
scale_pos_weight.
- It scores the model with PR-AUC instead of accuracy. Only about 0.17% of these transactions are fraud, so a model that always answers “not fraud” would be 99.8% accurate while catching zero fraud. PR-AUC measures how well the model finds the rare fraud cases, which is the whole point.
- It registers the model only if PR-AUC clears the bar. If it doesn’t, the run fails on purpose, so you see it instead of wondering why no new model turned up.
That’s the order AWS recommends for SageMaker Pipelines: data preparation, then training, then evaluation, and registration only when the metrics pass.
The 0.80 bar is a starting point, it’s a pipeline parameter, so you can raise it as your baseline improves without touching code.
Step 4: The Model Registry and the Quality Stamp of Approval
Every model that passes lands in the registry as
PendingManualApproval , with its evaluation metrics attached. You looks at the numbers, either in SageMaker Studio or with a few lines of boto3, and approves or rejects it:Example Code to accept:
import boto3
sm = boto3.client("sagemaker")latest = sm.list_model_packages( ModelPackageGroupName="FraudDetectionModelGroup", SortBy="CreationTime", SortOrder="Descending", MaxResults=1,)["ModelPackageSummaryList"][0]
sm.update_model_package( ModelPackageArn=latest["ModelPackageArn"], ModelApprovalStatus="Approved", # or "Rejected" ApprovalDescription="PR-AUC reviewed against current production model",)print("Model approved for deployment")
🧠 Best practice: Only ever deploy Approved packages, and let the approval itself trigger the deploy. That’s Step 5.
Step 5: Deploy on approval (EventBridge and GitHub Actions)
Once a model is approved, nobody should have to open a terminal to ship it. The EventBridge rule below listens for approvals, and only in our model group:
EventBridge Rule (triggers on approval):
{ "source": ["aws.sagemaker"], "detail-type": ["SageMaker Model Package State Change"], "detail": { "ModelPackageGroupName": ["FraudDetectionModelGroup"], "ModelApprovalStatus": ["Approved"] }}
There’s a catch, though. EventBridge can’t call GitHub’s
repository_dispatch API by itself. So the rule points at a small Lambda that forwards the approved model's ARN to GitHub. The GitHub token sits in Secrets Manager, not in an environment variable.notify_github.py , the Lambda:
import json, os, urllib.requestimport boto3
TOKEN = boto3.client("secretsmanager").get_secret_value( SecretId=os.environ["GITHUB_TOKEN_SECRET"])["SecretString"]
def lambda_handler(event, context): body = { "event_type": "model-approved", "client_payload": {"model_package_arn": event["detail"]["ModelPackageArn"]}, } req = urllib.request.Request( f"<https://api.github.com/repos/{os.environ['GITHUB_REPO']}/dispatches>", data=json.dumps(body).encode(), method="POST", headers={ "Authorization": f"Bearer {TOKEN}", "Accept": "application/vnd.github+json", }, ) urllib.request.urlopen(req)
GitHub Actions workflow (
deploy.yml ):name: Deploy Approved Fraud Modelon: repository_dispatch: types: [model-approved]permissions: id-token: write # required for OIDC role assumption contents: readjobs: deploy: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - uses: actions/setup-python@v5 with: python-version: "3.11" - run: pip install boto3 - uses: aws-actions/configure-aws-credentials@v4 with: role-to-assume: arn:aws:iam::<account-id>:role/GitHubActionsDeployRole aws-region: us-east-1 - name: Deploy to SageMaker endpoint run: python deploy_endpoint.py "${{ github.event.client_payload.model_package_arn }}"
deploy_endpoint.py deploys the exact package that was approved. The first run creates the endpoint. After that, every run does a blue/green update, so traffic only moves once the new instances are healthy:
deploy_endpoint.py :import sys, timeimport boto3
sm = boto3.client("sagemaker")package_arn = sys.argv[1]role = "arn:aws:iam::<account-id>:role/SageMakerExecutionRole"endpoint = "fraud-detection-endpoint"stamp = time.strftime("%Y%m%d-%H%M%S")sm.create_model( ModelName=f"fraud-model-{stamp}", ExecutionRoleArn=role, Containers=[{"ModelPackageName": package_arn}],)sm.create_endpoint_config( EndpointConfigName=f"fraud-config-{stamp}", ProductionVariants=[ { "VariantName": "AllTraffic", "ModelName": f"fraud-model-{stamp}", "InitialInstanceCount": 1, "InstanceType": "ml.m5.large", } ],)exists = any( e["EndpointName"] == endpoint for e in sm.list_endpoints(NameContains=endpoint)["Endpoints"])if exists: # Blue/green: new fleet is provisioned before traffic moves over sm.update_endpoint( EndpointName=endpoint, EndpointConfigName=f"fraud-config-{stamp}" )else: sm.create_endpoint( EndpointName=endpoint, EndpointConfigName=f"fraud-config-{stamp}" )sm.get_waiter("endpoint_in_service").wait(EndpointName=endpoint)print("Model deployed to real-time endpoint")
If you want to go a step further, pass a
DeploymentConfig to update_endpoint with canary traffic shifting and a CloudWatch alarm. SageMaker will then roll back by itself if the new model misbehaves.
Step 6: An API in front of the model
Checkout apps don’t talk to SageMaker endpoints directly, they call an API. This Lambda checks the input, gets a score, applies a threshold you can change without redeploying, and logs each request so we have live data for the drift checks later.
import json, osimport boto3
runtime = boto3.client("sagemaker-runtime")firehose = boto3.client("firehose")ENDPOINT = os.environ.get("ENDPOINT_NAME", "fraud-detection-endpoint")THRESHOLD = float( os.environ.get("FRAUD_THRESHOLD", "0.5")) # tune by cost, not by habitCAPTURE_STREAM = os.environ.get("CAPTURE_STREAM") # Firehose → S3N_FEATURES = 29 # V1..V28 + Amount
def lambda_handler(event, context): try: features = [float(v) for v in json.loads(event["body"])["features"]] if len(features) != N_FEATURES: raise ValueError(f"expected {N_FEATURES} features, got {len(features)}") except (KeyError, TypeError, ValueError) as err: return {"statusCode": 400, "body": json.dumps({"error": str(err)})} payload = ",".join(str(v) for v in features) response = runtime.invoke_endpoint( EndpointName=ENDPOINT, ContentType="text/csv", Body=payload ) score = float(response["Body"].read().decode().strip()) if CAPTURE_STREAM: firehose.put_record( DeliveryStreamName=CAPTURE_STREAM, Record={"Data": (payload + "\n").encode()}, ) return { "statusCode": 200, "body": json.dumps( {"fraud_probability": score, "is_fraud": score >= THRESHOLD} ), }
Hook it up to API Gateway and give the Lambda’s role
sagemaker:InvokeEndpoint and firehose:PutRecord . Then lock the API down with IAM auth, Cognito or API keys behind AWS WAF. If only internal services call it, make it a private API.About that threshold: with fraud this rare, 0.5 is almost never the right cut-off. Pick it on your validation set by asking what a missed fraud costs you compared with blocking a real customer’s card.
Step 7: Monitoring for drift
Shipping a model without monitoring is like shipping a car with no dashboard. You only find out it was overheating once it’s stuck on the hard shoulder.
A recent change matters here. Amazon SageMaker Model Monitor stopped taking new customers on July 30, 2026. If you already use it, it keeps working. If you’re starting fresh, as we are, you’ll need your own drift checks in CloudWatch. Clarify and Debugger went into maintenance at the same time.
The drift job below compares recent requests (from the Firehose logs) with the training data, one feature at a time, using the Population Stability Index (PSI). I prefer PSI to a KS-test p-value for this. With thousands of rows, p-values flag tiny shifts that don’t matter, while PSI tells you how far a distribution actually moved.
import jsonimport boto3import numpy as np
cloudwatch = boto3.client("cloudwatch")events = boto3.client("events")
def psi(expected, actual, bins=10): edges = np.unique(np.quantile(expected, np.linspace(0, 1, bins + 1))) e = np.histogram(expected, edges)[0] / len(expected) a = np.histogram(np.clip(actual, edges[0], edges[-1]), edges)[0] / len(actual) e, a = np.clip(e, 1e-6, None), np.clip(a, 1e-6, None) return float(np.sum((a - e) * np.log(a / e)))
def check_drift(baseline_df, live_df, features, threshold=0.2): drifted = {} for feature in features: score = psi(baseline_df[feature].values, live_df[feature].values) cloudwatch.put_metric_data( Namespace="FraudModel/Drift", MetricData=[ { "MetricName": "PSI", "Value": score, "Unit": "None", "Dimensions": [{"Name": "Feature", "Value": feature}], } ], ) if score > threshold: drifted[feature] = round(score, 3) if drifted: # this event is what Step 8 listens for events.put_events( Entries=[ { "Source": "custom.fraudmodel", "DetailType": "DriftDetected", "Detail": json.dumps({"features": drifted}), } ] ) return drifted
Schedule it hourly or daily with EventBridge Scheduler. Then add a CloudWatch alarm so a person hears about it as well:
aws cloudwatch put-metric-alarm \ --alarm-name "FraudModel-AmountDrift" \ --namespace "FraudModel/Drift" --metric-name "PSI" \ --dimensions Name=Feature,Value=Amount \ --statistic Maximum --period 3600 --evaluation-periods 1 \ --threshold 0.2 --comparison-operator GreaterThanThreshold \ --treat-missing-data notBreaching \ --alarm-actions arn:aws:sns:us-east-1:<account-id>:fraud-model-alertsDrift and decay aren’t the same thing. Drift means the incoming data changed. Knowing whether the model actually got worse needs labels, and fraud labels (chargebacks) can take weeks to arrive. Join those late labels back to your logged predictions and track precision and recall over time as well.
Step 8: Automated Retraining
Last piece. EventBridge can start a SageMaker pipeline directly, so this needs no extra Lambda. One rule fires on drift and another fires every night.
import jsonimport boto3
events = boto3.client("events")pipeline_arn = boto3.client("sagemaker").describe_pipeline( PipelineName="FraudDetectionPipeline")["PipelineArn"]target = { "Arn": pipeline_arn, "RoleArn": "arn:aws:iam::<account-id>:role/EventBridgeStartPipelineRole", "SageMakerPipelineParameters": {"PipelineParameterList": []},}# 1. Retrain when drift is detectedevents.put_rule( Name="FraudModelDriftRetrain", State="ENABLED", EventPattern=json.dumps( {"source": ["custom.fraudmodel"], "detail-type": ["DriftDetected"]} ),)events.put_targets(Rule="FraudModelDriftRetrain", Targets=[{"Id": "Retrain", **target}])# 2. Retrain nightly on fresh dataevents.put_rule( Name="FraudModelNightlyRetrain", State="ENABLED", ScheduleExpression="cron(0 2 * * ? *)",)events.put_targets( Rule="FraudModelNightlyRetrain", Targets=[{"Id": "Nightly", **target}]
The
EventBridgeStartPipelineRole needs sagemaker:StartPipelineExecution on the pipeline. And because every retrained model still lands in the registry as PendingManualApproval , the loop runs on its own right up to the point where you decides what goes live.
🔐 Security & Governance, Don’t Skip This
This system handles card data, so none of this is optional:
- Encrypt S3 with SSE-KMS, use KMS keys for training, processing and endpoints, and keep TLS on for everything in transit.
- Run training and inference in a private VPC, with VPC endpoints for S3 and SageMaker.
- Give every Lambda, pipeline step and CI role only the permissions it needs.
- Never expose the scoring API without authentication.
- Keep the audit trail. Pipelines records each step’s inputs and outputs, and the registry records who approved which model and why.
- Keep both gates: the automatic PR-AUC check and the human approval.
💰 Cost Optimization Tips
- Spot training is already on in Step 3. AWS quotes savings of up to 90% compared with on-demand.
- If traffic is low or comes in bursts, Serverless Inference can cost less than an always-on instance.
- Add S3 lifecycle rules to the offline store and the Firehose logs so old data moves to cheaper storage.
- Shut dev endpoints down every night with a scheduled Lambda.
Cleaning up
An idle endpoint still bills by the hour, so tear things down when you’re done:
aws sagemaker delete-endpoint --endpoint-name fraud-detection-endpointaws sagemaker delete-feature-group --feature-group-name fraud-detection-featuresaws events remove-targets --rule FraudModelNightlyRetrain --ids Nightlyaws events delete-rule --name FraudModelNightlyRetrainDo the same for the drift rule, and empty the bucket if you don’t need the data any more.
🏆 What You Just Achieved
Pause for a second. Look at what you built:
- ✅ A fully automated, reproducible training pipeline (no more “it worked on my laptop”).
- ✅ A quality gate that mathematically guarantees bad models never reach production.
- ✅ A governed model registry with full audit lineage.
- ✅ A CI/CD system that deploys models the moment they’re approved, zero manual deployment steps.
- ✅ A real-time, internet-facing inference API.
- ✅ A self-monitoring system that detects drift and heals itself by retraining automatically.
This is not a toy project. This is the exact architecture pattern used by banks, fintechs, and payment processors protecting billions of dollars in transactions every single day. You didn’t just “learn MLOps”, you built a miniature version of the systems that companies like Mastercard and major banks rely on, where 83% of industry leaders say AI has reduced false positives and churn, marking a new era in fraud prevention.
Banks and payment companies run far bigger versions of this, but the shape is the same. If you can explain your choices in an interview (why PR-AUC, why two gates, why drift isn’t decay), you’ll stand out.
One last thing
Learning every AWS service won’t make you good at MLOps. Building ML systems you can trust will. The whole idea fits on one line:
Data → Features → Training → Registry → Deployment → Monitoring → Retraining, with every arrow automated, tested and logged.
Don’t try to build all of it in one weekend. You can get one pipeline running end to end, then add CI/CD, then monitoring, then retraining. The same skeleton works for churn, recommendations or credit scoring, mostly you just swap the model.
⭐️ Final takeaway
MLOps isn’t about knowing every AWS service — it’s about building trustworthy, self-sustaining ML systems instead of one-off notebooks. The core idea to remember:
**Data → Features → Training → Registry → Deployment → Monitoring → Retraining, **and every arrow between these stages should be automated, tested, and logged.
Start small: get one pipeline working end-to-end (even with a toy dataset), then layer in CI/CD, monitoring, and auto-retraining one piece at a time. The fraud detection project in this blog gives you a real, industry-relevant blueprint — reuse this same skeleton for churn prediction, recommendation systems, credit scoring, or any other high-stakes ML use case. The pattern doesn’t change, only the model does.
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article