AWS Builder Center

Builder Center

Building a Custom Amazon SageMaker Ground Truth Labeling Job for Multi-Label Pixel Segmentation

Learn how to build a custom Amazon SageMaker Ground Truth labeling job using a pre-annotation AWS Lambda function, a Crowd HTML worker template, and a post-annotation function, with a worked example that captures overlapping, hierarchical, pixel-level labels the built-in task types can't represent.

Michaelangelo Battaglia

This post covers building a custom labeling job end to end using three services. Amazon SageMaker Ground Truth (a data labeling capability of Amazon SageMaker AI) orchestrates the job, AWS Lambda runs the pre- and post-processing functions, and Amazon Simple Storage Service (Amazon S3) holds the data and results. The worked example is pixel-level segmentation where a single pixel can carry more than one label and labels are organized in a hierarchy. The built-in semantic segmentation tool stores exactly one class per pixel, so it cannot represent a pixel that is both Category A and Category B. That single constraint is what requires a custom job, and it makes a useful vehicle for learning how custom jobs actually work.

Introduction

An Amazon SageMaker Ground Truth custom labeling workflow has three custom ingredients that you supply, sitting around the managed service that orchestrates everything.
  • Pre-annotation AWS Lambda function — This function runs once per data object, before a worker sees it. It takes a line from your input manifest and produces the taskInput payload that your worker template renders.
  • Custom worker task template — This is an HTML file, using AWS Crowd HTML Elements (a library of pre-built user interface components for labeling tasks) and the Liquid templating language, that renders the task and collects the annotation. This is where the task-specific user interface (UI) lives.
  • Post-annotation AWS Lambda function — Also called the consolidation function, this runs after workers submit and merges or normalizes their answers into your final output.
The managed service handles everything else, such as distributing tasks, hosting the worker portal, authenticating workers, and storing results. You provide those three pieces plus the supporting resources, which are an Amazon S3 layout, an AWS Identity and Access Management (IAM) role, and the CreateLabelingJob call.
Here is the data flow.
The diagram shows the per-object flow through the job.
  1. Amazon SageMaker Ground Truth reads one object (one image) from the input manifest in Amazon S3.
  2. It invokes the pre-annotation AWS Lambda function, which returns the taskInput payload (the image reference plus the label taxonomy).
  3. The custom worker template renders that payload in the worker portal, and the worker paints labels on the image.
  4. On submit, the worker's answer (a JSON string) is sent to the post-annotation AWS Lambda function.
  5. The post-annotation function consolidates the answer and writes the result to the Amazon S3 output location.
The key idea for anything custom in Amazon SageMaker Ground Truth is that the pre-annotation function decides what your UI receives, the template decides what the worker does and what JSON comes back, and the post-annotation function decides what gets stored. Once you understand those three contracts, you can build a labeling tool for almost any data type.
Let's get building!

1 — Define a shared label taxonomy

Both the pre-annotation AWS Lambda function and the worker template need to agree on the label set. Keep it in one place, a single JSON file, so they never drift apart. We use a small hierarchical taxonomy where parent nodes are organizational and only leaf nodes are paintable. Two leaves can overlap on the same pixels.
1
2
3
4
5
6
7
8
[
{ "id": "vehicle", "name": "Vehicle", "color": "#1f77b4", "children": [
{ "id": "vehicle.car", "name": "Car", "color": "#2ca02c", "children": [] },
{ "id": "vehicle.truck", "name": "Truck", "color": "#8c564b", "children": [] }
]},
{ "id": "shadow", "name": "Shadow", "color": "#d62728", "children": [] },
{ "id": "occluded", "name": "Occluded", "color": "#9467bd", "children": [] }
]
To adapt this for a different domain, you edit only this file, with no code changes.

2 — The pre-annotation AWS Lambda function (custom piece #1)

Amazon SageMaker Ground Truth invokes this function once per manifest object. The contract has two parts. The function receives an event containing a dataObject, and it must return an object with a taskInput key. Whatever you place in taskInput becomes available as task.input inside your Liquid template.
The inbound event looks like this.
1
2
3
4
5
{
"version": "2018-10-16",
"labelingJobArn": "arn:aws:sagemaker:<region>:<acct>:labeling-job/...",
"dataObject": { "source-ref": "s3://<bucket>/input/img_000.png" }
}
Here is the complete function.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
from category_tree import CATEGORY_TREE # the JSON from step 1, loaded once

SOURCE_REF_FIELD = "source-ref"

def handler(event, context=None):
data_object = event.get("dataObject") or {}
source_ref = data_object.get(SOURCE_REF_FIELD)

# Signal failure by RAISING. Ground Truth then marks the task failed.
if not source_ref:
raise ValueError(f"Required field '{SOURCE_REF_FIELD}' is missing or empty")

# Whatever we return under taskInput becomes `task.input` in the template.
return {
"taskInput": {
"sourceRef": source_ref,
"categories": CATEGORY_TREE,
}
}
Two details cause trouble for people building this the first time.
  • Two-argument handler — The handler must accept two arguments, handler(event, context). The AWS Lambda runtime always passes a context object, so a one-argument handler runs locally but crashes at runtime.
  • Raise to fail a task — To fail a task, raise an exception rather than return an error object. Raising tells Amazon SageMaker Ground Truth to record the task as failed, whereas returning a dictionary makes it treat your taskInput as successful.
This function is also the natural hook for model-assisted pre-labeling. You can call a model here and fold its predictions into taskInput so workers correct a draft instead of starting from a blank canvas. For the full event and response contract, see the Amazon SageMaker Ground Truth pre-annotation Lambda documentation .

3 — The custom worker template (custom piece #2)

This is the heart of a custom job. It is a single HTML file built from three things. A <crowd-form> wrapper captures everything inside it on submit, the Liquid templating language injects your taskInput, and the rest is the HTML and JavaScript you need for the task itself.
We render the hierarchical label menu with Liquid. We loop over task.input.categories and render each leaf as a selectable button carrying its id and parent id.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
<script src="https://assets.crowd.aws/crowd-html-elements.js"></script>

<crowd-form>
<div id="category-tree">
{% for cat in task.input.categories %}
<div class="cat-parent">{{ cat.name }}
{% for child in cat.children %}
<button type="button" class="cat-leaf"
data-cat-id="{{ child.id }}"
data-parent-id="{{ cat.id }}">{{ child.name }}</button>
{% endfor %}
</div>
{% endfor %}
</div>

<!-- The source image. grant_read_access signs the S3 URL so the browser can load it. -->
<div id="stage" style="position:relative">
<img id="src-image" src="{{ task.input.sourceRef | grant_read_access }}"/>
<canvas id="overlay"></canvas>
</div>

<!-- The hidden field that carries our answer out on submit. -->
<input type="hidden" name="annotationData" id="annotation-data"/>
</crowd-form>
Two Liquid-specific notes are worth calling out.
  • grant_read_access — This is an Amazon SageMaker Ground Truth Liquid filter that signs your Amazon S3 URI so the worker's browser can fetch a private object. Use it for any Amazon S3 asset. See the Crowd HTML Elements and Liquid filter reference .
  • No crossorigin attribute — Do not add crossorigin to the <img> element unless you actually read the image's pixels back off a canvas. It forces a Cross-Origin Resource Sharing (CORS) preflight against the signed URL and breaks image loading in the worker portal.
Now for the part that makes overlap possible. The technique is to keep one independent mask layer per leaf category. In JavaScript, that is a map of categoryId to a Set of pixel indices, where pixelIndex = y * width + x. Painting one category only ever touches its own set, so a pixel can belong to two categories at once.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
// One independent layer per category. This is what a single-class
// segmentation map cannot do.
const layers = new Map(); // categoryId -> Set<number>
function layerFor(id) {
if (!layers.has(id)) layers.set(id, new Set());
return layers.get(id);
}

let activeCategory = null;
document.querySelectorAll(".cat-leaf").forEach(btn => {
btn.onclick = () => { activeCategory = btn.dataset.catId; };
});

// Paint a brush disk into the ACTIVE layer only.
function paintAt(x, y, radius) {
if (!activeCategory) return;
const set = layerFor(activeCategory);
for (let dy = -radius; dy <= radius; dy++) {
for (let dx = -radius; dx <= radius; dx++) {
if (dx*dx + dy*dy <= radius*radius) {
set.add((y + dy) * imageWidth + (x + dx)); // overlap is automatic
}
}
}
}
On submit, we serialize to JSON and write it into the hidden field. We use run-length encoding, which stores a mask as alternating counts of unpainted and painted pixels rather than a value per pixel, so the payload stays compact.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
function encodeRle(pixelSet, total) {
const ordered = [...pixelSet].sort((a, b) => a - b);
const rle = []; let cursor = 0, i = 0;
while (i < ordered.length) {
const start = ordered[i];
let end = start; i++;
while (i < ordered.length && ordered[i] === end + 1) { end = ordered[i]; i++; }
rle.push(start - cursor, end - start + 1); // [skip, fill, ...]
cursor = end + 1;
}
if (cursor < total) rle.push(total - cursor);
return rle.length ? rle : [total];
}

document.querySelector("crowd-form").addEventListener("submit", () => {
const total = imageWidth * imageHeight;
const categories = [];
document.querySelectorAll(".cat-leaf").forEach(btn => {
const id = btn.dataset.catId, set = layers.get(id);
if (set && set.size) categories.push({
categoryId: id,
parentId: btn.dataset.parentId || null,
rle: encodeRle(set, total),
});
});
document.getElementById("annotation-data").value =
JSON.stringify({ imageWidth, imageHeight, categories });
});
Tip: validate without launching a job. The RenderUiTemplate API, or the console worker-template Preview, renders your template against a sample taskInput and reports template errors. It needs no workforce, so you can iterate on the UI in seconds. See the RenderUiTemplate API reference .

4 — The output JSON, and how to read it

When a worker submits, the value of annotationData is the JSON your template produced.
1
2
3
4
5
6
7
8
{
"imageWidth": 768,
"imageHeight": 512,
"categories": [
{ "categoryId": "vehicle.car", "parentId": "vehicle", "rle": [67322, 205, 559, 231, ...] },
{ "categoryId": "shadow", "parentId": null, "rle": [625, 19, 748, 21, ...] }
]
}
The rle array is an alternating list of run lengths over the flattened image, starting with a skip, in the form [skip, fill, skip, fill, ...], and the runs always sum to width × height. To turn a category back into a usable mask, decode it as follows.
1
2
3
4
5
6
7
8
def decode_rle(rle, total_pixels):
pixels, cursor, is_fill = set(), 0, False # first run is a skip
for run in rle:
if is_fill:
pixels.update(range(cursor, min(cursor + run, total_pixels)))
cursor += run
is_fill = not is_fill
return pixels # set of flat pixel indices for this category
Two categories overlap wherever their decoded pixel sets intersect, and parentId preserves the hierarchy. That is the payoff of the custom approach expressed in one data structure.

5 — The post-annotation AWS Lambda function (custom piece #3)

After workers submit, Amazon SageMaker Ground Truth calls this function with a pointer to the collected answers and expects back one consolidated entry per data object. For a single-annotator job, consolidation is a structural pass-through that preserves your payload. With multiple annotators you would merge their answers here.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
import json

def handler(event, context=None):
# event["payload"]["s3Uri"] points to a JSON array of per-object answers.
dataset_objects = read_json_from_s3(event["payload"]["s3Uri"])
label_attr = event.get("labelAttributeName", "my-seg")
consolidated = []

for obj in dataset_objects:
answer = json.loads(obj["annotations"][0]["annotationData"]["content"])
consolidated.append({
"datasetObjectId": obj["datasetObjectId"],
"consolidatedAnnotation": {
# Pass the segmentation payload through intact. Never collapse
# the per-category layers, or you lose overlap and hierarchy.
"content": { label_attr: answer }
},
})
return consolidated # Ground Truth merges this into the output manifest
The rule that matters is to avoid flattening the per-category layers during consolidation, because doing so discards the overlap and hierarchy you worked to capture. For the consolidation event and response contract, see the Amazon SageMaker Ground Truth post-annotation Lambda documentation .

6 — Wire it together with CreateLabelingJob

You select the custom task type by providing both AWS Lambda functions plus a UiTemplateS3Uri, instead of a built-in task type. Here is the AWS SDK for Python (Boto3) call.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
import boto3
sm = boto3.client("sagemaker", region_name="us-east-1")

sm.create_labeling_job(
LabelingJobName="custom-seg-demo",
LabelAttributeName="my-seg",
InputConfig={"DataSource": {"S3DataSource": {
"ManifestS3Uri": "s3://<bucket>/manifest/input.manifest"}}},
OutputConfig={"S3OutputPath": "s3://<bucket>/output/"},
RoleArn="arn:aws:iam::<acct>:role/<gt-execution-role>",
HumanTaskConfig={
"WorkteamArn": "arn:aws:sagemaker:<region>:<acct>:workteam/private-crowd/<team>",
"PreHumanTaskLambdaArn": "arn:aws:lambda:<region>:<acct>:function:<pre-lambda>",
"AnnotationConsolidationConfig": {
"AnnotationConsolidationLambdaArn":
"arn:aws:lambda:<region>:<acct>:function:<post-lambda>"},
"UiConfig": {"UiTemplateS3Uri": "s3://<bucket>/template/worker.liquid.html"},
"TaskTitle": "Multi-label pixel segmentation",
"TaskDescription": "Paint overlapping categories at the pixel level",
"NumberOfHumanWorkersPerDataObject": 1,
"TaskTimeLimitInSeconds": 3600,
},
)
Three details are worth knowing before you run this. See the CreateLabelingJob API reference  for the full parameter set.
  • WorkteamArn is required — A labeling job cannot be created without a workteam, even a custom one. To validate only the UI, use RenderUiTemplate instead, which needs no job and no workteam. To set up a private workforce, see the Amazon SageMaker Ground Truth private workforce documentation .
  • AWS Lambda function naming — The pre- and post-annotation function names must contain one of SageMaker, Sagemaker, sagemaker, or LabelingFunction, otherwise CreateLabelingJob rejects them.
  • IAM least-privilege, with attention to the Amazon S3 actions — The execution role needs s3:GetObject and s3:PutObject across the bucket, because Amazon SageMaker Ground Truth writes intermediate manifests under your output prefix and reads them back. It also needs unconditional s3:ListBucket on the bucket. A prefix-conditioned s3:ListBucket trips an existence check during the job and surfaces as a confusing AccessDenied error, which can cost you several failed jobs before you find it.

The output manifest

When the job finishes, Amazon SageMaker Ground Truth writes an output manifest, one JSON line per image, at s3://<bucket>/output/<job-name>/manifests/output/output.manifest.
1
2
3
4
5
6
{
"source-ref": "s3://<bucket>/input/img_000.png",
"my-seg": { "imageWidth": 768, "imageHeight": 512, "categories": [ ... ] },
"my-seg-metadata": { "type": "groundtruth/custom", "human-annotated": "yes",
"job-name": "custom-seg-demo", "creation-date": "..." }
}
Read it line by line, run decode_rle on each category, and rasterize to one binary mask channel per category, which supports overlapping labels and parent/child grouping for training.

Wrapping up

That is a complete custom Amazon SageMaker Ground Truth job. The managed service did the heavy lifting, such as task distribution, the worker portal, authentication, and storage, and we supplied the three custom pieces that gave us a capability no built-in task type offers.
  • Pre-annotation AWS Lambda function — Shapes each task's input.
  • Custom worker template — Its one-layer-per-category data model makes overlapping, hierarchical, pixel-level labels possible.
  • Post-annotation AWS Lambda function — Preserves that structure into the output.
Once you have built one custom job, the approach generalizes. You can swap the template for a different data type, adjust the two AWS Lambda functions' contracts, and label almost anything. The built-in task types are a useful on-ramp, and custom jobs are where Amazon SageMaker Ground Truth becomes genuinely flexible. To go deeper, see the Amazon SageMaker Ground Truth custom labeling workflows documentation .
# amazon-sagemaker-ground-truth# amazon-sagemaker-ai# augmenting-image-data# computer-vision# tutorials
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article