How to Perform MediaConvert Sub-clipping for High Frame Rate Videos
MediaConvert sub-clipping requires clipping start and end to be specified with SMPTE timecode: HH:MM:SS:FF where FF is a 2-digit frame number. In vision data engineering for computer vision (CV)/machine learning (ML) scenarios, video frame rates could exceed 99fps. In such case, how can we perform sub-clipping with MediaConvert to meet CV/ML needs?
Introduction
MediaConvert's assembly workflow allows us to perform sub-clipping with frame accuracy. To specify start and end times, it is required to use SMPTE timecode format:
HH:MM:SS:FF (hours, minutes, seconds, frames) for non-drop frame timecodes, or HH:MM:SS;FF for drop frame timecodes. For a detailed article of using MediaConvert for sub-clipping in the context of vision data engineering, please see the author's article: Sub-clipping in Vision Data Engineering Using MediaConvert .In this article, we focus on the following problem: for videos with frame rates higher than 99fps, SMPTE timecode format with 2-digit frame number no longer works for frame numbers requiring 3 digits - one of the gaps between cinematic videos/TV era and vision data in CV/ML era.
Since SMPTE ST 12-1 standard is followed by video software and hardware vendor ecosystem, simply adding a 3rd frame number digit would be a long term evolution even if possible.
The Approach
Video sub-clipping always start and end on actual frames of the input video based on target start/end times given. The start/end times are specified in video timeline instead of wall clock. This is true regardless the tools we use, such as MediaConvert or FFMPEG. Even if using FFMPEG which allows us to specify start and end time in standard time HH:MM:SS.fff with millisecond accuracy, the actual sub-clipping always lands on the nearest frames instead of the specified video times.
A video has two time domains: wall clock and video time. For vision data engineering, wall clock is particularly important since wall clock is used to correlate video data with external metadata captured via various means, most likely with data timestamps in UTC. For CV/ML what matters is to know the precise wall clock times of the actual sub-clipping start and end frames while how close they are to the specified/target start/end times is secondary.
Suppose source video start wall clock is provided. If source video has frame rate higher than 99fps, we set FF = 0 in sub-clipping start/end timecodes. This will indeed result in deviation from the target/specified start/end times due to setting FF = 0. Then based on this rule, we can calculate the precise wall clock and provide it as output metadata.
The following method converts a given wall clock time into SMPTE timecode for an input video whose frame rate could be either smaller or larger than 99fps. It also outputs the actual wall clock time of the actual clipping frame, which could deviate from the input target/specified time (play_time_utc).
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
def utc_to_smpte_timecode(self, video_start_utc: datetime, target_time_utc: datetime, frame_rate: float) -> tuple[str, datetime]:
"""
Given video_start_utc and a time point inside video: target_time_utc - either the target clip start or clip end
Find MediaConvert input timecode in SMPTE timecode format (HH:mm:ss:FF)
Args:
video_start_utc: datetime object - when the video recording started
target_time_utc: datetime object - the specific time point in the clip
frame_rate: float - video frame rate (e.g., 29.97, 30, 59.94, 60, 120)
Returns:
str: SMPTE timecode (with only 2 :FF digit)
the actual clip time which must lands on the nearest frame which may not be identical to the input datetime (target_time_utc) even for 60fps
"""
nominal_fps = int(round(frame_rate))
total_frames = int(round((target_time_utc - video_start_utc).total_seconds() * frame_rate))
# for 3-digit frame rates, we ensure FF=0 by moving timecode earlier than theoretical, but provides the actual timestamp in UTC in wall clock
if frame_rate > 99:
total_frames = (total_frames // nominal_fps) * nominal_fps
# exact UTC of the frame boundary, via integer microsecond math (no float error)
offset_us = (total_frames * 1_000_000) // nominal_fps
actual_time_utc = video_start_utc + timedelta(microseconds=offset_us)
ff = int(total_frames % nominal_fps)
ss = int((total_frames // nominal_fps) % 60)
mm = int((total_frames // (nominal_fps * 60)) % 60)
hh = int(total_frames // (nominal_fps * 3600))
if ff > 99:
log_event("ERROR", "issue_detected", "subclip",
msg=f"FF={ff} at {frame_rate}fps cannot be represented in SMPTE timecode")
return f"{hh:02d}:{mm:02d}:{ss:02d}:{ff:02d}", actual_time_utcThe Difference between Specified Clip Time and Actual Clip Time
Since we know that sub-clipping always lands on a frame instead of specified time, there is almost always a difference between target/specified wall clock time and actual clipping wall clock time, hence certainly in video time domain too. The next table provides the difference boundaries and example differences in actual MediaConvert sub-clipping for three input video frame rates.
| Frame Rate of Input Video (fps) | Difference between Actual and Target Times (ms) | Example: Target | Example: Actual |
|---|---|---|---|
| 120 | 0<=target - actual < 1000 | 2026-08-14T02:29:54.610000+00:00 | 2026-08-14T02:29:54.132772+00:00 |
| 60 | | target - actual | <= 16.66/2 | 2026-08-14T01:40:16.815000+00:00 | 2026-08-14T01:40:16.809367+00:00 |
| 10 | | target - actual | <= 100.0/2 | 2026-08-14T02:23:31.977000+00:00 | 2026-08-14T02:23:31.938890+00:00 |
Conclusion
We provided an approach in addressing a limitation in SMPTE timecode standard for vision data engineering - one of the gaps between specs/standards for cinematic videos and today's CV/ML needs.
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article