Sub-clipping in Vision Data Engineering Using MediaConvert
In vision data engineering, we often need to sub-clip source videos with frame accuracy from multiple synchronized cameras. MediaConvert supports the so-called assembly workflow configuration which can be used to perform sub-clipping with frame accuracy. In this document we share certain design considerations and implementation details of such sub-clipping.
Co-author: Britt Evans, Biocore
Introduction
In vision data engineering, we often need to perform sub-clipping of source videos with frame accuracy. Cameras could be computer vision cameras supporting GenICam (Generic Interface for Cameras) interface or cinematic cameras. More often these cameras are synchronized thru either HW or SW to the accuracy at millisecond or even nanosecond level.
MediaConvert's assembly workflow allows us to perform sub-clipping with frame accuracy. The purpose of this post is to document some design considerations and implementation notes in building a workflow involving such sub-clipping using MediaConvert and other AWS resources.
MediaConvert Job Settings
Job settings is a JSON structure you pass to MediaConvert CreateJob API. It has three main sections:
- Inputs — Source files, each with optional clipping, audio selectors, and caption selectors
- Output Groups — Where and how to deliver outputs (File group for S3, HLS group for streaming, DASH, etc.)
- Outputs — Within each output group, defines the encoding settings (codec, resolution, bitrate, quality)
Other key settings:
- Role — IAM role MediaConvert assumes to read inputs and write outputs
- Queue — Which processing queue to use (default or custom)
- UserMetadata — Your custom pass-through data (max 10 key-value pairs)
- Tags — For cost allocation and resource management
The following job settings is for sub-clipping a high resolution source video in H265 with MP4 media container and outputting two versions of sub-clips:
- Original resolution with H.265 codec
- HD resolution with H.264 codec
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
{
"Queue": default_queue_arn,
"UserMetadata": user_metadata,
"Role": role_arn,
"Settings": {
"TimecodeConfig": {
"Source": "ZEROBASED"
},
"OutputGroups": [
{
"CustomName": "filegroup-hevc",
"Name": "File Group",
"Outputs": [
{
"ContainerSettings": {
"Container": "MP4",
"Mp4Settings": {}
},
"VideoDescription": {
"TimecodeInsertion": "PIC_TIMING_SEI",
"CodecSettings": {
"Codec": "H_265",
"H265Settings": {
"FramerateDenominator": 1,
"MaxBitrate": 50000000,
"FramerateControl": "SPECIFIED",
"RateControlMode": "QVBR",
"FramerateNumerator": framerate,
"SceneChangeDetect": "TRANSITION_DETECTION",
"QualityTuningLevel": "SINGLE_PASS"
}
},
"DropFrameTimecode": "ENABLED"
},
"AudioDescriptions": [
{
"AudioSourceName": "Audio Selector 1",
"CodecSettings": {
"Codec": "AAC",
"AacSettings": {
"Bitrate": 96000,
"CodingMode": "CODING_MODE_2_0",
"SampleRate": 48000
}
}
}
]
},
# HD/H264 output
{
"ContainerSettings": {
"Container": "MP4",
"Mp4Settings": {}
},
"VideoDescription": {
"Width": 1280,
"Height": 720,
"CodecSettings": {
"Codec": "H_264",
"H264Settings": {
"FramerateDenominator": 1000,
"MaxBitrate": 5000000, # increased from 4MBit
"FramerateControl": "SPECIFIED",
"RateControlMode": "QVBR",
"FramerateNumerator": 60000,
"SceneChangeDetect": "TRANSITION_DETECTION"
}
}
},
"AudioDescriptions": [
{
"CodecSettings": {
"Codec": "AAC",
"AacSettings": {
"Bitrate": 96000,
"CodingMode": "CODING_MODE_2_0",
"SampleRate": 48000
}
}
}
],
"NameModifier": "_hd"
}
],
"OutputGroupSettings": {
"Type": "FILE_GROUP_SETTINGS",
"FileGroupSettings": {
"Destination": output_path,
"DestinationSettings": {
"S3Settings": {
"StorageClass": "STANDARD",
'Encryption': {
'EncryptionType': 'SERVER_SIDE_ENCRYPTION_KMS',
'KmsKeyArn': kms_key_arn
}
}
}
}
}
}
],
"FollowSource": 1,
"Inputs": [
{
"FileInput": input_url,
"InputClippings": [
{
"StartTimecode": start_timecode,
"EndTimecode": end_timecode
}
],
"AudioSelectors": {
"Audio Selector 1": {
"DefaultSelection": "DEFAULT"
}
},
"VideoSelector": {
"Rotate": rotate
},
"TimecodeSource": "ZEROBASED",
}
]
},
"BillingTagsSource": "JOB",
"AccelerationSettings": {
"Mode": "DISABLED"
},
"StatusUpdateInterval": "SECONDS_60",
"Priority": 0,
"Tags": {
"workstream": "video-processing"
}
}Next we will discuss a few design considerations and technical notes which will refer back to different sections of the above job settings.
Timecode and its Limitations
To specify start and end times (start_timecode and end_timecode variables in the above job settings), we need to use timecode in the format: HH:MM:SS:FF, where FF is a 2-digit frame number.
2-digit frame number is sufficient for cinematic videos since for human perception, frame rate above 100fps is generally not needed. For vision data engineering, frame rate could be higher, well above 100fps. In this case, 2-digit frame number is no longer enough. This is a current limitation.
The timecode format with 2-digit frame number, or specifically SMPTE timecode - is a legacy standard that dates back to the late 1960s. It was designed for cinematic videos such as television and broadcasting industries. It becomes insufficient for vision data engineering.
For given source video, if we know its frame rate, source video start timestamp in UTC and any given point in the source video (such as either start or end time for sub-clipping) in UTC, the following Python method constructs the timecode for the given point:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
def utc_to_mediaconvert_timecode(self, video_start_utc: datetime, clip_time_utc: datetime, frame_rate: float):
"""
Convert UTC datetime in source to MediaConvert timecode format (HH:mm:ss:ff)
Args:
video_start_utc: datetime object - when the video recording started
clip_time_utc: datetime object - the specific time point to convert
frame_rate: float - video frame rate
Returns:
str: Timecode in HH:mm:ss:ff format
"""
# Convert to total frames first to lock in precision
total_frames = int(round((clip_time_utc - video_start_utc).total_seconds() * frame_rate))
# Derive HH:MM:SS:FF from total_frames
# NOTE: only 2 digits is allowed for frame number. Error introduced if frame rate is above 99.
ff = min(total_frames % frame_rate, 99)
ss = (total_frames // frame_rate) % 60
mm = (total_frames // (frame_rate * 60)) % 60
hh = (total_frames // (frame_rate * 3600))
# Format as HH:mm:ss:ff (MediaConvert NDF format)
return f"{hh:02d}:{mm:02d}:{ss:02d}:{ff:02d}"As pointed out in the comment, if source video frame rate is above 99fps, error will be introduced due to 2-digit limitation of SMPTE timecode.
UTC vs TAI (International Atomic Time)
Vision data engineers working on cameras, camera synchronization and recording prefer to use TAI instead of UTC since TAI is always continuous and monotonic - it never has leap seconds. UTC occasionally (no matter how rare) inserts leap seconds to stay aligned with Earth's rotation, which means UTC can have a 61-second minute. Obviously leap seconds in theory could cause problems if UTC is used. For camera synchronization, regardless which scheme or which camera interface, vision data engineers prefer to use TAI in camera synchronization setup even though both TAI and UTC are available in a grand master.
However, for video metadata stored in data store, e.g. start/end times of sub-clipping, is more likely in UTC since UTC is almost a default choice for data storage, and almost all services in AWS (DynamoDB, CloudWatch, EventBridge, S3 timestamps and so on).
In video processing workflow, we should carefully choose to use a time standard which is convenient for the task.
Another issue to watch is: there are quite some libraries for TAI-UTC conversion. They more often require two things:
- Internet access: so that it can pull down historical leap second table for time conversion
- Local disk access: so that it can cache the table
If your TAI-UTC conversion happens in a serverless runtime such as lambdas, you need some gymnastics to meet the above requirements.
Another option is to roll your own custom code for TAI-UTC conversion, with hardcoded leap second table. You have to maintain the table when next leap second occurs. The International Earth Rotation and Reference Systems Service (IERS), based in Paris, monitors Earth's rotation and issue a bulletin (Bulletin C) every six months announcing whether a leap second will be added.
Here is a custom utility in Python for UTC-TAI conversion with hardcoded leap second table.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
from datetime import datetime, timedelta, timezone
import bisect
from typing import Final
'''
The leap-second data comes directly from the public-domain IERS/IANA leap-seconds.list (last updated through the latest Bulletin
C, expires December 2026). No new leap seconds have been announced since 2016. If IERS ever announces a new one (they give ~6
months notice), you just add one line to the lists below.
If a new leap second is ever announced, just append the new Unix timestamp and the new offset (e.g. ... , new_unix_time) and ... , 38)
to the two tuples and redeploy. You can get the exact values from https://data.iana.org/time-zones/data/leap-seconds.list.
'''
class TimeConverter:
# Official leap-second table (Unix/POSIX timestamps of each change + the TAI-UTC offset after that change)
# First entry sets the base offset to +10 s (1972 definition); each subsequent entry adds +1 s.
LEAP_TIMES: Final = (63072000, 78796800, 94694400, 126230400, 157766400, 189302400,
220924800, 252460800, 283996800, 315532800, 362793600, 394329600,
425865600, 489024000, 567993600, 631152000, 662688000, 709948800,
741484800, 773020800, 820454400, 867715200, 915148800, 1136073600,
1230768000, 1341100800, 1435708800, 1483228800)
OFFSETS: Final = (10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25,
26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37)
def _get_tai_utc_offset(posix_seconds: float) -> int:
"""Return TAI - UTC offset (seconds) for a given POSIX timestamp.
Works for all modern dates (post-1972)."""
idx = bisect.bisect_right(TimeConverter.LEAP_TIMES, posix_seconds) - 1
return TimeConverter.OFFSETS[idx] if idx >= 0 else 10 # fallback for very old dates
def utc_to_tai(utc_dt: datetime) -> datetime:
"""Convert a naive datetime (assumed to be UTC) to TAI datetime.
Returns a new naive datetime object."""
# Treat naive datetimes as UTC (standard in Lambda)
epoch = datetime(1970, 1, 1)
posix = (utc_dt - epoch).total_seconds()
offset = TimeConverter._get_tai_utc_offset(posix)
tai_posix = posix + offset
return epoch + timedelta(seconds=tai_posix)
def tai_to_utc(tai_dt: datetime) -> datetime:
"""Convert a naive datetime (assumed to be TAI) to UTC datetime.
Returns a new naive datetime object."""
epoch = datetime(1970, 1, 1)
tai_posix = (tai_dt - epoch).total_seconds()
offset = TimeConverter._get_tai_utc_offset(tai_posix) # safe approximation (leaps are years apart)
utc_posix = tai_posix - offset
return epoch + timedelta(seconds=utc_posix)
def tai_micros_to_utc_micros(tai_micros: int) -> int:
"""
Convert TAI time (microseconds since 1970-01-01 00:00:00 UTC)
to UTC time (microseconds since the same epoch).
Input and output are integers in microseconds to preserve full precision.
Uses the same hardcoded leap-second table.
"""
# Convert microseconds → floating-point POSIX seconds
tai_posix = tai_micros / 1_000_000.0
# Find the correct TAI - UTC offset for this TAI instant
offset = TimeConverter._get_tai_utc_offset(tai_posix)
# Subtract the offset to get UTC POSIX seconds
utc_posix = tai_posix - offset
# Convert back to integer microseconds (rounds toward zero, but since leap seconds
# are whole seconds and far apart, fractional part stays accurate)
utc_micros = int(utc_posix * 1_000_000.0)
return utc_micros
def tai_micros_to_utc_datetime(tai_micros: int) -> datetime:
"""
timezone-aware version
Convert TAI microseconds to a timezone-aware UTC datetime.
"""
# 1. Estimate UTC seconds to find the correct offset index
tai_seconds = tai_micros / 1_000_000.0
approx_utc = tai_seconds - TimeConverter._get_tai_utc_offset(tai_seconds)
offset = TimeConverter._get_tai_utc_offset(approx_utc)
# 2. Calculate the exact UTC POSIX timestamp (seconds since 1970-01-01 00:00:00 UTC)
# subtracting whole-second leap offsets from the TAI micros
utc_seconds = (tai_micros - (offset * 1_000_000)) / 1_000_000.0
# 3. Create an aware datetime directly from the timestamp
# Use datetime.UTC for Python 3.11+; otherwise use timezone.utc
return datetime.fromtimestamp(utc_seconds, tz=timezone.utc)
Worth noting — in 2022, the General Conference on Weights and Measures (CGPM) voted to abolish leap seconds by 2035. That means after 2035, the difference between UTC and TAI will be frozen for large number of years and UTC will no longer be in sync with Earth rotation. This means if you use custom code approach for UTC-TAI conversion, at most you only need to maintain that code until 2035 if leap second happens between now and 2035.
MediaConvert Job Submission Pattern
MediaConvert has high level of job concurrency, with the default max being 200 concurrent jobs per account per region for on-demand queues. However, there is a soft quota on job submission rate: the default is 20 TPS (transactions per second). In a sense, MediaConvert is a high concurrency, high output system with constrained input. For a scalable workflow design, we need to take this into consideration to stay within the job submission quota to avoid losing jobs.
As an example, suppose we use a lambda function to submit MediaConvert jobs when a source video arrives in an input bucket. If we use S3 event (ObjectCreated) -> EventBridge -> lambda pattern, we run the risk of exceeding the job submission quota if it happens that data objects arriving in the bucket in high enough rate.
To "regulate" job submission rate and handle failure cases, we can use S3 event -> EventBridge -> SQS queue -> lambda pattern. Adding SQS queue provides two advantages:
- If the lambda fails to submit a job, the job is returned to the queue for later processing. The queue is also backed by DLQ.
- It allows us to regulate how fast a lambda reads from the queue thru max_concurrency setting, as shown below. Suppose lambda_subclip is a lambda for submitting MediaConvert jobs, the following CDK snippet illustrates how job submission is "regulated" via the queue (self.sub_clipping_queue).
1
2
3
4
5
6
7
lambda_subclip.add_event_source(
event_sources.SqsEventSource(
self.sub_clipping_queue,
batch_size=1, # 1 lambda event message/sqs message
max_concurrency=18, # keeps under MediaConvert's ~20 TPS limit
)
)Output Destination and its Options
In the OutputGroupSettings section of a MediaConvert job settings,
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
"OutputGroupSettings": {
"Type": "FILE_GROUP_SETTINGS",
"FileGroupSettings": {
"Destination": output_path,
"DestinationSettings": {
"S3Settings": {
"StorageClass": "STANDARD",
'Encryption': {
'EncryptionType': 'SERVER_SIDE_ENCRYPTION_KMS',
'KmsKeyArn': kms_key_arn
}
}
}
}
}we need to carefully define the destination so that the output file name meets your requirements. These are the options in defining destination name and format:
- Folder Path (Default Behavior): If the string provided ends with a forward slash (/), MediaConvert interprets it as a destination folder. It then automatically names the output files using the input file's base name and NamedModifier setting in the output settings.
- Specific File Name (Overriding Behavior): If the string provided does not end with a forward slash, MediaConvert interprets the entire string as the precise, desired filename for that specific output group. In such case, do not give suffix such as .mp4 which will be added by MediaConvert. In such case, "NameModifier" also applies/works.
User Metadata and its Limit
In a MediaConvert job settings, you can use UserMetadata to carry custom data. It must be a list of key-value pairs with both key and value being of string type. It is a useful mechanism to carry custom data from one step to the next in a video processing workflow such as metadata, debug info, or tracking info and so on. UserMetadata is not used by MediaConvert in its job processing.
We should be aware that the number of such key-value pairs in UserMetadata is limited to 10. To work around this limit, we can lump some data into a JSON string as the value of one of the key-value pairs. However, we need to work against another limit: the maximum length of each value is 256 Unicode characters.
In summary, UserMetadata has the following three limitations:
- value must of string type
- number of key-value pairs is limited to 10
- length of each value is limited to 256 Unicode characters
MediaConvert Output Encryption
If the default SSE-S3 bucket encryption is used for the output bucket, there is no additional job settings config needed. If your requirements include SSE-KMS encryption for MediaConvert outputs, the following two steps are required.
First in job settings, you need to provide the ARN of the KMS key in OutputGroupSettings section and specify Encryption, as shown below:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
"OutputGroupSettings": {
"Type": "FILE_GROUP_SETTINGS",
"FileGroupSettings": {
"Destination": output_path,
"DestinationSettings": {
"S3Settings": {
"StorageClass": "STANDARD",
'Encryption': {
'EncryptionType': 'SERVER_SIDE_ENCRYPTION_KMS',
'KmsKeyArn': kms_key_arn
}
}
}
}
}The next is to grant KMS permissions for the MediaConvert role , which is used by MediaConvert to run your job under. A MediaConvert role is always required and can be specified in job settings when submitting a MediaConvert job.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
role.add_to_policy(
iam.PolicyStatement(
actions=[
"kms:GenerateDataKey",
"kms:Encrypt",
"kms:Decrypt"
],
resources=["*"],
conditions={
"StringEquals": {
"kms:ViaService": "s3.*.amazonaws.com"
}
},
effect=iam.Effect.ALLOW
)
)Summary
We presented a complete job settings for MediaConvert sub-clipping and shared the following design considerations/implementation notes in video sub-clipping:
- Timecode used for specifying clip start/end times and its limitations
- UTC used in metadata vs TAI used in video and camera (International Atomic Time)
- A MediaConvert job submission design pattern to "regulate" job submission rate against the soft TPS quota
- Options to specify job output location and format
- User metadata and its constraints
- How to encrypt output assets with SSE-KMS
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article