Video processing with WebCodecs

Manipulating video stream components.

Eugene Zemtsov

François Beaufort

Modern web technologies provide ample ways to work with video. Media Stream API, Media Recording API, Media Source API, and WebRTC API add up to a rich tool set for recording, transferring, and playing video streams. While solving certain high-level tasks, these APIs don't let web programmers work with individual components of a video stream such as frames and unmuxed chunks of encoded video or audio. To get low-level access to these basic components, developers have been using WebAssembly to bring video and audio codecs into the browser. But given that modern browsers already ship with a variety of codecs (which are often accelerated by hardware), repackaging them as WebAssembly seems like a waste of human and computer resources.

WebCodecs API eliminates this inefficiency by giving programmers a way to use media components that are already present in the browser. Specifically:

• Video and audio decoders
• Video and audio encoders
• Raw video frames
• Image decoders

The WebCodecs API is useful for web applications that require full control over the way media content is processed, such as video editors, video conferencing, video streaming, etc.

Video processing workflow

Frames are the centerpiece in video processing. Thus in WebCodecs most classes either consume or produce frames. Video encoders convert frames into encoded chunks. Video decoders do the opposite.

Also VideoFrame plays nicely with other Web APIs by being a CanvasImageSource and having a constructor that accepts CanvasImageSource. So it can be used in functions like drawImage() andtexImage2D(). Also it can be constructed from canvases, bitmaps, video elements and other video frames.

WebCodecs API works well in tandem with the classes from Insertable Streams API which connect WebCodecs to media stream tracks.

• MediaStreamTrackProcessor breaks media tracks into individual frames.
• MediaStreamTrackGenerator creates a media track from a stream of frames.

WebCodecs and web workers

By design WebCodecs API does all the heavy lifting asynchronously and off the main thread. But since frame and chunk callbacks can often be called multiple times a second, they might clutter the main thread and thus make the website less responsive. Therefore it is preferable to move handling of individual frames and encoded chunks into a web worker.

To help with that, ReadableStream provides a convenient way to automatically transfer all frames coming from a media track to the worker. For example, MediaStreamTrackProcessor can be used to obtain a ReadableStream for a media stream track coming from the web camera. After that the stream is transferred to a web worker where frames are read one by one and queued into a VideoEncoder.

With HTMLCanvasElement.transferControlToOffscreen even rendering can be done off the main thread. But if all the high level tools turned out to be inconvenient, VideoFrame itself is transferable and may be moved between workers.

WebCodecs in action

Encoding

It all starts with a VideoFrame. There are three ways to construct video frames.

• From an image source like a canvas, an image bitmap, or a video element.

  const canvas = document.createElement("canvas");
  // Draw something on the canvas...
  
  const frameFromCanvas = new VideoFrame(canvas, { timestamp: 0 });
  

• Use MediaStreamTrackProcessor to pull frames from a MediaStreamTrack

  const stream = await navigator.mediaDevices.getUserMedia({…});
  const track = stream.getTracks()[0];
  
  const trackProcessor = new MediaStreamTrackProcessor(track);
  
  const reader = trackProcessor.readable.getReader();
  while (true) {
    const result = await reader.read();
    if (result.done) break;
    const frameFromCamera = result.value;
  }
  

• Create a frame from its binary pixel representation in a BufferSource

  const pixelSize = 4;
  const init = {
    timestamp: 0,
    codedWidth: