Media containers keep time in ways that are easy to get subtly wrong—for example, negative timestamps or audio and video streams that don't start together. After an off-the-shelf decoder silently misaligned audio and video in our training data, we wrote our own. Today we're open-sourcing it: AVTensor, a Rust media decoder that demuxes both streams in a single pass and improved our training MFU by 1.8 percentage points.

Training audio-visual models requires audio and video in sync. A pipeline that decodes a clip implicitly assumes frame i of the video corresponds to sample window i of the audio. A few tens of milliseconds of drift is enough to notice, lip sync especially—yet containers don't encode that alignment in any straightforward way.

To see why, look at what a video file actually is. An MP4 or MOV is a container holding several independent streams: usually one video stream, one or more audio streams and sometimes subtitles. Each stream is stored as a sequence of compressed packets and keeps its own clock. When you play the file, a demuxer pulls the packets apart and hands them to codecs, which decode them into frames of video and samples of audio.

Three properties of an ordinary MP4 cause most of the trouble:

• Presentation order isn't decode order. Codecs don't store every frame as a complete image. An I-frame is a full picture; a P-frame stores only what changed since the previous frame; a B-frame stores changes relative to both the previous frame and the next one. The decoder must decode that future frame first, so the container stores packets out of playback order and each frame carries two timestamps: a decode timestamp (DTS) and a presentation timestamp (PTS). Only PTS reflects what a viewer sees. That's also why you can't "seek to frame 500": you seek to the nearest I-frame before the point you want and decode forward.
• Timestamps aren't frame counts. Containers express time as integer ticks over a stream-specific time base (1/24000, 1/90000, …), a convention that dates to 1953, when American color TV shifted the broadcast framerate from 30 to 30/1.001 (~29.97) to fit the color signal and cinema's 24fps became 24/1.001 for broadcast. "Frame number × frame duration" has been a floating-point footgun ever since. Variable-framerate streams abandon uniform spacing entirely: an encoder can drop a near-identical frame and extend the previous one; a screen-recording stream might have two frames a minute at inconsistent times.
• Streams don't have to start at zero—or together. Video frames and audio samples often use different clocks and offsets.

We hit assets whose first video packet had a PTS of −3.337 seconds while the first audio packet sat at −0.023 seconds. Most players present this plausibly (applying container timing metadata like edit lists), which is why it goes unnoticed—in datasets and in some of the tools we used to review them.

At the time, our pipeline decoded video with torchcodec and audio with a different library entirely—torchaudio's StreamingMediaDecoder—then stitched the results together. On assets with negative PTS, torchcodec's first frame appeared at 1.6s (it dropped everything before zero), while the audio reader started from a different origin. The first frame stayed frozen for over a second while audio played and the rest of the clip ran desynced.

It's a silent failure: the training pipeline runs fine. But the model learns that lips and speech are only loosely coupled and you start hunting for the problem in the architecture—because the first instinct is that it's always a RoPE issue, right?

That's why we keep repeating the mantra: always look at the data. We traced quality problems in audio-visual training back to decoded video and audio returning different effective lengths and origins. Our stopgap was defensive cropping (cut both streams to the shorter one and assert they agree within 10 milliseconds), which treated the symptom without fixing the decoder.

torchcodec, PyTorch's official media decoder, is good at what it's built for: random-access video decode with per-frame timestamps. But when we evaluated it (v0.11 at the time), its audio decoder was still early—upstream reworked the seeking logic in later releases—so our audio path ran through torchaudio's StreamingMediaDecoder instead. Neither library was designed to decode audio and video together on a single timeline and on negative-PTS assets they disagreed about where time zero was. That disagreement is the frozen first frame. So we wrote our own.

AVTensor

AVTensor is a Rust library that talks to FFmpeg's C API directly and exposes one call to Python: give it a URI and a time range, get back a video tensor and an audio tensor that share an origin.

We've been moving foundational infrastructure at Runway to Rust. Rust isn't the obvious language for a decoder, since the media and ML ecosystem is C and C++ and every call into FFmpeg or libtorch crosses an FFI boundary. In practice the overhead is negligible: the hot loops run inside FFmpeg either way and tch-rs wraps libtorch so well that decoded frames go straight into torch tensors. We chose Rust because this kind of code—hand-managed FFmpeg buffers and lifetimes—is exactly where C++ projects rot and the Rust compiler catches memory and threading mistakes at build time instead of in production.

The demuxer splits the container's bytestream into packets; we route each packet to the correct codec (or drop it if it falls outside the requested range). Decoded frames then pass through a filter graph that normalizes framerate, resolution and pixel format for video and sample rate, channel layout and loudness for audio. The filtered frames land in output tensors. Because both streams come from a single demux pass over one file on a shared timeline, audio and video agree on time zero by construction.

Allocation happens once, up front. AVTensor computes the output shapes—[frames, height, width, 3] for video, [channels, samples] for audio—then writes decoded frames straight into their slices, so samples stay contiguous and nothing reallocates mid-decode. The tradeoff is that output lengths are a best guess from stream timestamps and container metadata, which can be wrong on malformed assets. The whole decode also runs with Python's Global Interpreter Lock released, which matters because our dataloaders are thread-based pipelines (built on SPDL) rather than pools of worker processes: Python threads keep working while native code decodes.

One constraint we kept on purpose: the API seeks by timestamp, not frame index. Demuxers are built for time-based seeking—the only notion that spans both streams—while "frame 500" is a video-only concept that can require scanning the bitstream to resolve. Pipelines that think in frame indices (shot boundaries, caption alignments) precompute a frame-to-PTS mapping once and everything downstream uses PTS.

AVTensor also streams directly from object storage. The pipeline previously began with a download stage that pulled each asset into memory and handed the bytes to the decoder—an extra copy of every video on the Python heap and a full download even when training wanted five seconds of one.

FFmpeg has a native escape hatch that the CLI doesn't expose: a custom AVIOContext, which lets you provide the read and seek callbacks the demuxer uses as its data source. AVTensor implements one backed by concurrent byte-range requests to GCS or S3 (plain HTTP(S) URLs go through FFmpeg's own protocol layer), so the demuxer reads from object storage as if it were a local file.

We logged every callback invocation to see how FFmpeg reads an MP4 in practice: it reads the first 4 KiB for the header, asks for the file size, then—on files not encoded with faststart—jumps straight to the tail of the file, because that's where the moov atom that holds the seek index lives. From there, it lands near the requested timestamp and, during decode, bounces between nearby byte ranges as it alternates between interleaved audio and video data. Sequential streaming handles this pattern poorly, so the reader fetches large segments (64 MiB), buffers previously fetched ranges for short back-jumps and on a far seek races a fresh range request against readers already in flight.

Seeking into the middle of a large asset now downloads the header, the index and the bytes around the target—instead of the whole file. That also changes what we store: extracting a shot no longer requires cutting it into its own object first, so the pipeline can train on shots straight from full clips instead of paying to store every asset twice.

In a training dataloading benchmark, replacing the download-then-decode pipeline with direct object storage streaming cut 100 batches from 176 seconds to 122 (about 30%) and eliminated the download stage and its heap copies. On a production audio-visual training run, the switch was worth about 1.8 percentage points of MFU over the torchcodec-based pipeline. MFU measures the fraction of a GPU fleet's theoretical compute doing useful work, so at the scale of a training cluster even tenths of a point are material.

And the sync bug is gone. We took the adversarial asset from the incident and no longer have the out of sync issue.

The performance against torchcodec (0.14 at the time of writing) is comparable as shown in the diagram. The main difference shows up at decode-time resize, which is common for training dataloaders that downscale during decoding. Downscaling 1080p to 256×144 runs 1.8× faster in AVTensor with one FFmpeg thread and about 6× faster with FFmpeg's automatic threading, because the filter graph's scaling overlaps with decoding and the output writes are smaller—while torchcodec's resize transform runs on the decode thread.

AVTensor is now open source. It's the default video path in our data processing stack, feeding preprocessing and training across our audio-visual models. We welcome feedback and contributions.