AWS Builder Center

Build a Metric-Driven ML Pipeline with Amazon S3, EC2, and SageMaker AI

Learn a reusable pattern for storing durable ML state in Amazon S3, running preparation on Amazon EC2, training with Amazon SageMaker AI, and starting or stopping compute safely from operational, model, and cost metrics.

Machine-learning infrastructure does not need to run continuously.
A practical AWS workflow can keep data and artifacts in durable storage, create compute only when work is ready, and stop that compute as soon as the work is complete or a safety threshold is crossed. This pattern applies to computer vision, natural-language processing, forecasting, recommendation systems, and many other workloads.
The core idea is simple:
Keep state in Amazon S3, treat Amazon EC2 and Amazon SageMaker AI compute as disposable, and make start/stop decisions from explicit metrics.
This article presents a reusable architecture, the metrics that drive it, and the guardrails that make automation safe.

The reference architecture

A small set of AWS services is enough to build the workflow:
AWS service
Responsibility
Amazon S3
Durable datasets, manifests, checkpoints, metrics, and final artifacts
Amazon EC2
CPU-, memory-, network-, or storage-heavy data preparation
Amazon SageMaker AI
Managed model training on CPU or GPU instances
Amazon CloudWatch
Logs, native metrics, custom metrics, alarms, and dashboards
Amazon EventBridge
Schedules and reactions to service or alarm events
AWS Lambda
Lightweight control logic for start, stop, and validation decisions
AWS Step Functions
Optional orchestration for multi-stage workflows
AWS Systems Manager
Secure EC2 commands and Automation runbooks
AWS IAM and AWS KMS
Least-privilege access and encryption
AWS Budgets
Cost visibility and alerts
A typical flow looks like this:
Data or work request
|
v
S3 manifest or queue
|
v
Launch gate: budget, quota, inputs, concurrency
|
+-------------------+
| |
v v
EC2 preparation SageMaker training
| |
+---------+---------+
|
v
CloudWatch logs and metrics
|
v
Continue, stop, retry, or promote
|
v
Durable results in S3
Amazon S3 is the system of record. EC2 instances and SageMaker training jobs can disappear without losing the workflow's durable state.

1. Organize Amazon S3 as a contract

Avoid using one bucket prefix as an unstructured file share. Separate immutable inputs from generated outputs and mutable coordination state.
s3://example-ml-platform/
├── source/<git-commit>/
├── raw/<dataset-version>/
├── prepared/<preprocessing-version>/
├── manifests/
├── checkpoints/<job-name>/
├── experiments/<job-name>/
└── promoted/<model-version>/
Each run should record enough information to reproduce it:
  • source-code commit;
  • container image digest;
  • dataset and preprocessing versions;
  • hyperparameters;
  • input object checksums;
  • instance type;
  • start and completion timestamps;
  • validation metrics; and
  • output object checksums.
Use S3 versioning where appropriate, server-side encryption, lifecycle policies, and IAM permissions scoped to the prefixes each component needs.
The important design rule is that a worker uploads durable output before it stops. Local instance storage is working space, not the only copy of a checkpoint or processed dataset.

2. Use Amazon EC2 for preparation work

Some tasks do not benefit from a GPU: downloading data, decompressing archives, parsing files, joining tables, feature generation, sharding, and calculating checksums. Amazon EC2 is a flexible fit for this stage.
Choose the instance family from measurements instead of habit:
  • a compute-optimized instance for sustained CPU work;
  • a memory-optimized instance for large in-memory transformations;
  • a storage-optimized instance for intensive local I/O; or
  • a general-purpose instance for balanced pipelines.
The worker can be launched by an EventBridge schedule, a Step Functions workflow, or a Lambda function reacting to a work signal. Pass only a manifest URI and run identifier. The instance retrieves its configuration and inputs from S3 instead of embedding large scripts or credentials in user data.
A robust EC2 worker follows this sequence:
  1. Validate its account, Region, run identifier, and input manifest.
  1. Download or stream the required inputs.
  1. Perform the transformation.
  1. Upload outputs, checksums, logs, and a terminal status to S3.
  1. Publish a completion or failure metric.
  1. Stop or terminate itself.
Use an instance profile rather than long-lived access keys. AWS Systems Manager can provide shell access and execute commands without opening inbound SSH.

3. Use SageMaker AI for bounded training jobs

SageMaker training jobs are a natural fit for disposable model training. A job starts the requested infrastructure, runs the training container, uploads model artifacts, and releases the training infrastructure when the job reaches a terminal state.
Treat each job as immutable. Give it versioned input channels, a pinned container image, an S3 output prefix, and a maximum runtime. For interruption-tolerant workloads, Managed Spot Training can be combined with checkpointing to S3.
Before creating a job, use a fail-closed launch gate. At minimum, verify:
  • the input manifest exists and passes validation;
  • the requested instance quota is available;
  • no conflicting job is already active;
  • the estimated worst-case job cost fits the remaining envelope;
  • the output prefix does not overwrite another run;
  • the training image is versioned; and
  • the maximum runtime and checkpoint configuration are present.
A dry run should print the resolved request and estimated ceiling without launching paid compute. Require an explicit execution flag for the real job.

4. Choose the metrics that control the workflow

Automation is only as good as its signals. Separate metrics into four groups.

Work metrics

These answer, "Is there work to do?"
Examples include:
  • pending manifest count;
  • queue depth;
  • number of unprocessed S3 objects;
  • scheduled retraining due;
  • new approved dataset version; and
  • data-drift threshold crossed.
A positive work signal may start an EC2 worker or submit a SageMaker training job.

Health metrics

These answer, "Is the worker making progress?"
Useful signals include:
  • heartbeat age;
  • CPU, memory, disk, and network utilization;
  • processed items per minute;
  • time since the last checkpoint;
  • error count; and
  • retry count.
Publish workload-specific custom metrics to CloudWatch. Low CPU alone is not a reliable idle signal: a healthy worker may be waiting for network or storage. Combine infrastructure metrics with an application heartbeat or progress counter.

Model metrics

These answer, "Should training continue?"
Examples include:
  • validation loss;
  • accuracy, F1, AUC, or a domain-specific metric;
  • improvement since the best checkpoint;
  • number of epochs without improvement;
  • calibration error; and
  • performance relative to the current production model.
Early stopping belongs inside the training program because it has immediate access to epoch-level metrics. The program should save its best checkpoint, write the decision to the run metadata, and exit cleanly. SageMaker then uploads the final artifacts and releases the training infrastructure.

Cost and time metrics

These answer, "Are we still inside the operating envelope?"
Track:
  • elapsed runtime;
  • estimated job cost;
  • month-to-date actual cost;
  • forecast cost;
  • remaining project allocation;
  • active instance count; and
  • concurrent training-job count.
AWS Budgets is valuable for visibility and alerts, but an alert should not be treated as an instantaneous hard-stop mechanism. Enforce maximum runtime in the job configuration and perform a cost check before launch. Preserve a contingency reserve for delayed billing data and retries.

5. Start compute from explicit demand

There are two common patterns.

Event-driven start

An application writes an approved manifest to S3 or sends a message to a queue. EventBridge or Lambda receives the event and runs the launch gate. If every check passes, it starts an existing stopped EC2 instance, launches a new instance, or creates a SageMaker training job.
Use idempotency. Store a run identifier in DynamoDB, S3, or the job tags so duplicate events cannot launch duplicate compute.

Scheduled evaluation

An EventBridge schedule invokes a controller every few minutes or hours. The controller reads work, health, and budget metrics. It starts compute only when all required conditions are true.
Scheduling is useful when immediate reaction is unnecessary or several signals must be evaluated together. It also provides a predictable rate limit for control-plane actions.
A generic decision can be expressed as:
START when
pending_work > 0
AND budget_gate = PASS
AND active_workers = 0
AND inputs_valid = true
AND cooldown_expired = true

6. Stop compute safely

Stopping is more sensitive than starting because an incorrect decision can interrupt useful work. Define both a graceful path and an emergency path.

Graceful completion

The application finishes its current unit of work, writes outputs and a terminal manifest to S3, publishes a success metric, and requests shutdown. For a one-shot EC2 worker, termination is often better than stopping because it avoids retaining unnecessary attached resources. Set temporary EBS volumes to delete on termination where appropriate.
SageMaker training infrastructure is released automatically when the training job completes, fails, or reaches its configured stopping condition.

Idle shutdown

Use a compound rule such as:
STOP when
pending_work = 0
AND heartbeat_is_healthy = true
AND progress_delta = 0
AND cpu_utilization < threshold
FOR the complete idle window
The controller should first ask the worker to checkpoint and exit. If the worker does not reach a terminal state before a second timeout, an emergency automation can stop or terminate it.

Metric-based experiment stopping

For multi-run experiments, use a decision gate after each job:
CONTINUE when
artifact_integrity = PASS
AND validation_metric >= minimum_threshold
AND improvement >= minimum_delta
AND remaining_budget >= next_job_ceiling

STOP otherwise
This prevents a weak or corrupt first run from automatically triggering an expensive sequence. Preserve the failed run's artifacts for analysis; stop further spending, not the evidence.

Alarm-driven actions

CloudWatch alarms can react to native or custom metrics. For general workflows, route an alarm state change through EventBridge to Lambda, Step Functions, or a Systems Manager Automation runbook. The controller can then verify current state, checkpoint safely, and perform the appropriate EC2 or SageMaker action.
Avoid an alarm that terminates compute from a single noisy data point. Require multiple evaluation periods, treat missing data deliberately, add a cooldown, and verify tags or run identifiers before acting.

7. Make the controller fail closed

The controller should decline an action when required information is missing or ambiguous.
Before starting compute, confirm the exact account, Region, project tag, resource identifier, input version, budget state, and concurrency state. Before stopping compute, confirm that the resource belongs to the workflow and that its latest heartbeat, checkpoint, and work metrics support the decision.
A simplified controller might look like this:
def decide(metrics, run):
if not run.inputs_valid:
return "BLOCK"

if metrics.emergency_cost_limit_crossed:
return "CHECKPOINT_AND_STOP"

if run.active:
if run.completed:
return "FINALIZE_AND_STOP"
if metrics.stalled_for_too_long:
return "CHECKPOINT_AND_STOP"
if metrics.epochs_without_improvement >= run.patience:
return "EARLY_STOP"
return "CONTINUE"

if (
metrics.pending_work > 0
and metrics.active_workers == 0
and run.budget_gate_passed
and run.cooldown_expired
):
return "START"

return "NO_ACTION"
Keep the decision function separate from the code that calls AWS APIs. This makes it easy to unit-test edge cases without launching or stopping real resources.

8. Add observability before automation

Create one CloudWatch dashboard that answers:
  • What is running now?
  • What work is pending?
  • Is progress being made?
  • What was the latest validation metric?
  • When was the latest checkpoint uploaded?
  • How long has each resource been active?
  • What decision did the controller make, and why?
  • How much budget remains?
Log every control decision as structured data. Include evaluated metric values, thresholds, run identifier, resource identifier, and the selected action. An operator should be able to explain why a worker started or stopped without reconstructing the event from multiple consoles.
Add alarms for missing heartbeats, repeated failures, runtime backstops, stalled progress, unexpectedly high concurrency, and budget notifications.

9. Test the lifecycle in stages

Start with a dry-run controller that only records recommended actions. Then test with a small EC2 instance and a short SageMaker smoke job.
A practical rollout sequence is:
  1. Validate S3 manifests and IAM permissions.
  1. Publish synthetic work, health, and model metrics.
  1. Verify dashboards and alarm transitions.
  1. Run the controller without AWS mutations.
  1. Enable start actions for one tagged test resource.
  1. Verify graceful checkpoint and shutdown.
  1. Test duplicate events and controller retries.
  1. Test missing metrics and stale heartbeats.
  1. Test the maximum-runtime backstop.
  1. Expand to production-sized compute only after the lifecycle is reliable.

Final takeaway

The most reusable AWS machine-learning pattern is not a particular instance type or model framework. It is the separation of durable state from disposable compute.
Amazon S3 preserves inputs, checkpoints, and evidence. Amazon EC2 handles flexible preparation work. Amazon SageMaker AI provides bounded managed training. CloudWatch, EventBridge, Lambda, and Systems Manager turn operational, model, and cost metrics into controlled lifecycle decisions.
Start compute only when verified work exists. Stop it after durable output is written, when progress stalls, when model evidence says not to continue, or when a time or cost guardrail is reached.
That pattern makes ML infrastructure easier to reproduce, easier to audit, and safer for teams of any size to operate.
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article