AWS Builder Center

Builder Center

Running GPU workloads with Amazon ECS and AWS Cloud Development Kit

A Cloud Development Kit demo app that shows how to run an application that depends on GPU resources.

Nathan Peck

About

This pattern shows how to setup a fleet of GPU instances and use Amazon ECS to launch GPU enabled tasks across the cluster. You can use this pattern as the basis for setting up your own GPU accelerated machine learning workload orchestrated through Amazon ECS.

Setup Cloud Development Kit

To use this pattern you need TypeScript and Node. First, ensure that you have Node.js installed on your development machine. Then create the following files:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
// File: package.json
{
"name": "ecs-cluster",
"version": "1.0.0",
"description": "ECS GPU Cluster and Task",
"private": true,
"scripts": {
"build": "tsc",
"watch": "tsc -w",
"cdk": "cdk"
},
"author": {
"name": "Amazon Web Services",
"url": "https://aws.amazon.com",
"organization": true
},
"license": "Apache-2.0",
"devDependencies": {
"@types/node": "^8.10.38",
"aws-cdk": "2.102.0",
"typescript": "~4.6.0",
"ts-node": "^10.9.1"
},
"dependencies": {
"aws-cdk-lib": "2.102.0",
"constructs": "^10.0.0"
}
}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
// File: package.json
{
"name": "ecs-cluster",
"version": "1.0.0",
"description": "ECS GPU Cluster and Task",
"private": true,
"scripts": {
"build": "tsc",
"watch": "tsc -w",
"cdk": "cdk"
},
"author": {
"name": "Amazon Web Services",
"url": "https://aws.amazon.com",
"organization": true
},
"license": "Apache-2.0",
"devDependencies": {
"@types/node": "^8.10.38",
"aws-cdk": "2.102.0",
"typescript": "~4.6.0",
"ts-node": "^10.9.1"
},
"dependencies": {
"aws-cdk-lib": "2.102.0",
"constructs": "^10.0.0"
}
}
The files above serve the following purpose:
  • package.json - This file is used by NPM or Yarn to identify and install all the required dependencies:
  • tsconfig.json - Configures the TypeScript settings for the project:
  • cdk.json - Tells CDK what command to run, and provides a place to pass other contextual settings to CDK.
Last but not least run the following commands to install dependencies and setup your AWS account for the deployment:
1
2
npm install
npm run-script cdk bootstrap

Create the CDK application

Now create the following file to define the CDK application itself:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
// File: index.ts
import autoscaling = require("aws-cdk-lib/aws-autoscaling");
import ec2 = require("aws-cdk-lib/aws-ec2");
import ecs = require("aws-cdk-lib/aws-ecs");
import cdk = require("aws-cdk-lib");

class ECSCluster extends cdk.Stack {
constructor(scope: cdk.App, id: string, props?: cdk.StackProps) {
super(scope, id, props);

const vpc = new ec2.Vpc(this, "MyVpc", { maxAzs: 2 });

// Autoscaling group that will launch a fleet of instances that have GPU's
const asg = new autoscaling.AutoScalingGroup(this, "MyFleet", {
instanceType: ec2.InstanceType.of(
ec2.InstanceClass.G4DN,
ec2.InstanceSize.XLARGE
),
machineImage: ec2.MachineImage.fromSsmParameter(
"/aws/service/ecs/optimized-ami/amazon-linux-2023/gpu/recommended/image_id"
),
vpc,
maxCapacity: 10,
});

// Attach the fleet to an ECS cluster with a capacity provider.
// This capacity provider will automatically scale up the ASG
// to launch more GPU instances when GPU tasks need them.
const cluster = new ecs.Cluster(this, "EcsCluster", { vpc });
const capacityProvider = new ecs.AsgCapacityProvider(
this,
"AsgCapacityProvider",
{ autoScalingGroup: asg }
);
cluster.addAsgCapacityProvider(capacityProvider);

// Define a task that requires GPU. In this case we just run
// nvidia-smi to verify that the task is able to reach the GPU
const gpuTaskDefinition = new ecs.Ec2TaskDefinition(this, "gpu-task");
gpuTaskDefinition.addContainer("gpu", {
essential: true,
image: ecs.ContainerImage.fromRegistry("public.ecr.aws/deep-learning-containers/base:runtime-cu129-amzn2023"),
memoryLimitMiB: 2048,
cpu: 256,
gpuCount: 1,
command: ["sh", "-c", "nvidia-smi && sleep 3600"],
logging: new ecs.AwsLogDriver({
streamPrefix: "gpu-service",
logRetention: 1,
}),
});

// Request ECS to launch the task onto the fleet
new ecs.Ec2Service(this, "gpu-service", {
cluster,
desiredCount: 2,
// Service will automatically request capacity from the
// capacity provider
capacityProviderStrategies: [
{
capacityProvider: capacityProvider.capacityProviderName,
base: 0,
weight: 1,
},
],
taskDefinition: gpuTaskDefinition,
});
}
}

const app = new cdk.App();

new ECSCluster(app, "GpuTask");

app.synth();
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
// File: index.ts
import autoscaling = require("aws-cdk-lib/aws-autoscaling");
import ec2 = require("aws-cdk-lib/aws-ec2");
import ecs = require("aws-cdk-lib/aws-ecs");
import cdk = require("aws-cdk-lib");

class ECSCluster extends cdk.Stack {
constructor(scope: cdk.App, id: string, props?: cdk.StackProps) {
super(scope, id, props);

const vpc = new ec2.Vpc(this, "MyVpc", { maxAzs: 2 });

// Autoscaling group that will launch a fleet of instances that have GPU's
const asg = new autoscaling.AutoScalingGroup(this, "MyFleet", {
instanceType: ec2.InstanceType.of(
ec2.InstanceClass.G4DN,
ec2.InstanceSize.XLARGE
),
machineImage: ec2.MachineImage.fromSsmParameter(
"/aws/service/ecs/optimized-ami/amazon-linux-2023/gpu/recommended/image_id"
),
vpc,
maxCapacity: 10,
});

// Attach the fleet to an ECS cluster with a capacity provider.
// This capacity provider will automatically scale up the ASG
// to launch more GPU instances when GPU tasks need them.
const cluster = new ecs.Cluster(this, "EcsCluster", { vpc });
const capacityProvider = new ecs.AsgCapacityProvider(
this,
"AsgCapacityProvider",
{ autoScalingGroup: asg }
);
cluster.addAsgCapacityProvider(capacityProvider);

// Define a task that requires GPU. In this case we just run
// nvidia-smi to verify that the task is able to reach the GPU
const gpuTaskDefinition = new ecs.Ec2TaskDefinition(this, "gpu-task");
gpuTaskDefinition.addContainer("gpu", {
essential: true,
image: ecs.ContainerImage.fromRegistry("public.ecr.aws/deep-learning-containers/base:runtime-cu129-amzn2023"),
memoryLimitMiB: 2048,
cpu: 256,
gpuCount: 1,
command: ["sh", "-c", "nvidia-smi && sleep 3600"],
logging: new ecs.AwsLogDriver({
streamPrefix: "gpu-service",
logRetention: 1,
}),
});

// Request ECS to launch the task onto the fleet
new ecs.Ec2Service(this, "gpu-service", {
cluster,
desiredCount: 2,
// Service will automatically request capacity from the
// capacity provider
capacityProviderStrategies: [
{
capacityProvider: capacityProvider.capacityProviderName,
base: 0,
weight: 1,
},
],
taskDefinition: gpuTaskDefinition,
});
}
}

const app = new cdk.App();

new ECSCluster(app, "GpuTask");

app.synth();
Some things to note in this application:
  • ec2.MachineImage.fromSsmParameter('/aws/service/ecs/optimized-ami/amazon-linux-2023/gpu/recommended/image_id') - The CDK application uses this SSM Parameter to automatically launch the latest version of the ECS Optimized AMI with built-in GPU support.
  • gpuCount: 1 - The ECS task is configured to require one GPU core.
  • The G4dn instance type provides NVIDIA T4 GPUs with CUDA 12 support.

Build and deploy the CDK app

Preview what will be deployed:
1
npm run-script cdk diff
Deploy the GPU cluster and task:
1
npm run-script cdk deploy

Make sure it worked

Open up the Amazon ECS console and find the cluster that deployed.
Within that cluster you will see the GPU enabled service has launched
two tasks. Select one of the tasks to view it's details and click on the "Logs"
tab to see its output. You should see something similar to this:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
---------------------------------------------------------------------------------------------------
| timestamp | message |
|---------------|---------------------------------------------------------------------------------|
| 1752574414000 | Wed Jul 15 11:46:58 2026 |
| 1752574414000 | +-----------------------------------------------------------------------------------------+ |
| 1752574414000 | | NVIDIA-SMI 580.159.03 Driver Version: 580.159.03 CUDA Version: 13.0 | |
| 1752574414000 | +-----------------------------------------+------------------------+----------------------+ |
| 1752574414000 | | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | |
| 1752574414000 | | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | |
| 1752574414000 | | | | MIG M. | |
| 1752574414000 | |=========================================+========================+======================| |
| 1752574414000 | | 0 Tesla T4 On | 00000000:00:1E.0 Off | 0 | |
| 1752574414000 | | N/A 29C P8 9W / 70W | 0MiB / 15360MiB | 0% Default | |
| 1752574414000 | | | | N/A | |
| 1752574414000 | +-----------------------------------------+------------------------+----------------------+ |
| 1752574414000 | |
| 1752574414000 | +-----------------------------------------------------------------------------------------+ |
| 1752574414000 | | Processes: | |
| 1752574414000 | | GPU GI CI PID Type Process name GPU Memory | |
| 1752574414000 | | ID ID Usage | |
| 1752574414000 | |=========================================================================================| |
| 1752574414000 | | No running processes found | |
| 1752574414000 | +-----------------------------------------------------------------------------------------+ |
---------------------------------------------------------------------------------------------------

Clean Up

You can tear down the example stack when you are done with it by running:
1
npm run-script cdk destroy
# containers# aws-cdk# machine-learning
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article