zinc:video

Video playback for hosts with an OS and some memory: a Player decodes a playlist on its own thread (FFmpeg on the desktop and the Pi, the platform’s decoders on phones and webOS) and is shown as a runtime image. Plugin: plugins/video/ (module zinc:video).

import * as gfx from 'zinc:gfx';
import { Player, EXTENSIONS, RANDOM } from 'zinc:video';

const p = new Player(gfx.width(), gfx.height());  // image size; 0 x 0 = size of the first video
p.addFolder('media', EXTENSIONS);                  // or p.add('clip.mp4') per file
p.order = RANDOM;                                  // SEQUENTIAL (default), RANDOM, RANDOM_UNIQUE
p.play();                                          // loops forever (repeat = LOOP); STOP, ONE_SHOT
gfx.onFrame(() => p.draw(0, 0, gfx.width(), gfx.height()));

Player: add, addFolder, play, stop, pause(on), next, previous, jump(i), close; setters order, repeat, background; getters image, width, height, index, count, position, duration, loops, playing, paused, decoder, frames, dropped. Several players run at once (up to 16).

Support matrix

Using zinc:video on an unlisted target is error Z5003 at build time.

How it works

• Worker thread per player: demux + decode (FFmpeg), then swscale converts to 0x00RRGGBB straight at the player’s size, aspect ratio kept (sample aspect honoured), bars filled with background. Hardware frames are copied back with av_hwframe_transfer_data (NV12) first. Decode at the size you draw: a 1:1 drawImage is a row copy. A V4L2 decoder’s frame is copied into ordinary memory before the conversion: its buffer is mapped without a cache, and swscale read it at 85 ms a picture on a Pi 3B+ (854x480), against 10 ms with the copy. The conversion takes the colour range the stream names (a V4L2 decoder calls every frame limited range). Numbers: docs/reports/raspberry-pi.md.
• Zero copy on a KMS screen (Linux, ZN-731): when the display driver has hardware layers (display-fbdev on a Raspberry Pi, docs/plugins/display-fbdev.md), a V4L2 decoder is asked for its frames as dmabufs (AV_PIX_FMT_DRM_PRIME). A slot then holds the decoder’s frame and no pixels; the player’s image gets a raster::Picture (the buffers’ descriptors, the colour range and matrix, where the picture lies in the image) and the display shows the buffer on a plane of its own, scaled and converted there. player.decoder reads h264_v4l2m2m (dmabuf). A frame that draws the image another way (turned, blended, rounded, a part of it, into an image) asks for the pixels: the picture on screen is mapped, copied and converted at that moment on the main thread, and the decode thread converts the following ones, as long as frames draw them so. The frame is the same either way. ZINC_VIDEO_PRIME=0 turns the dmabufs off. Measured on a Pi 3B+: 3.4 to 4.8 ms of CPU a picture for the whole program against 35 to 38 (docs/reports/raspberry-pi.md).
• Hand-over: a ring of buffers slots (plugin option, default 4). The worker only writes free slots; the main thread (a zrt::Poller, once per loop iteration, before onFrame) picks the newest frame whose timestamp is due, frees older ones (dropped), and publishes it with raster::dyn_update. The player’s image id never changes: gfx.drawImage sees a new version every frame, and GPU compositors read the pixels with raster::dyn_view.
• Pacing: a player clock starts when two frames are in hand (a V4L2 decoder hands its frames in bursts: the second one came 70 ms after the first and was dropped as late) and follows hal_time_us; frames are shown when due (within half a 60 Hz refresh). When the decoder falls behind, the clock waits for it (at most 100 ms ahead) instead of skipping.
• Gapless looping: one continuous timeline per playlist pass — each file starts exactly where the previous one ended (last PTS + frame duration). The next file (demuxer + decoder) is opened as soon as the current one shows its first frame, so a switch costs one decode, never an open. One file loops the same way (a second instance of it is pre-opened). ZINC_VIDEO_LOG=1 prints one line per shown frame with entry, position, timeline and clock, and one per converted frame with the time of its conversion and the decoder (“kept … as a dmabuf” for a frame that is not converted); when a player closes, how many of its pictures the CPU never read. Measured on macOS (M1 Pro, VideoToolbox) over 3.5 passes of a 3-file playlist with 30 and 25 fps clips: the first frame of each file is shown one frame period after the last frame of the previous one, within 3 ms of its due time, no frame dropped at any loop point (one dropped at startup).
• stop() shows the background colour; repeat = ONE_SHOT stops after each file (then play() starts the next one), STOP after the whole playlist.

Phones

zinc export --target android|ios compiles the plugin’s phone backend into the app (plugin.json targets.android, targets.ios, see docs/plugins.md “Phones”); the shared part (player.h: playlists, gapless loops, the ring, the pacing) is the desktop’s. Paths are relative to the app’s working directory: the project’s assets/ travel in the app (assets/clip.mp4), and a file the app did not ship goes in its files directory (Android files/, iOS Library/Application Support/zinc/; examples/hero reads media/bbb.mp4 there).

• Android (video.android.cpp): AMediaExtractor over the file’s descriptor, AMediaCodec for the first video track. The decoder renders into an AImageReader of YUV_420_888, the layout every decoder must give there (CTS checks it), with each plane’s row and pixel strides and the crop: no vendor colour format, stride or slice height is guessed. A decoder that refuses the reader’s window hands its pictures in ByteBuffers instead (player.decoder ends in mediacodec, buffers): planar or semi-planar YUV 4:2:0 with the stride and slice height of the output format (a slice height of 0, seen on old devices, is the height); a tiled vendor format is refused in the log. ZINC_VIDEO_BYTEBUFFER=1 takes that path on any device. Either way the picture is scaled to the player’s box (nearest sample) and converted to 0x00RRGGBB (BT.601, BT.709 from 720 lines up, the range the format names) on the decode thread, as on webOS.
• iOS (video.ios.mm): AVAssetReader with an AVAssetReaderTrackOutput in 32BGRA: the system decodes (VideoToolbox) and converts, and the picture is only scaled to the box (a row copy at 1:1).
• No audio on phones either (the follow-up below). Seeking inside a file is not in the API on any target: jump(i) and previous() restart an entry (hero’s Restart button).

Options (zinc.json plugins.video)

Raspberry Pi notes

• Build: zinc build <app> --target rpi1 installs ffmpeg-dev into a derived SDK image (Alpine armhf, ARMv6 + VFP). The binary links FFmpeg’s shared libraries: run it on Alpine for Raspberry Pi (apk add ffmpeg-libs), or in a container, or rebuild against the distribution’s FFmpeg on the device.
• Hardware decode needs the bcm2835-codec V4L2 driver (/dev/video10, Raspberry Pi kernels with the default vc4-kms-v3d/fkms overlays, Pi 1..4) and enough gpu_mem. When it cannot be opened the player falls back to software decoding silently (player.decoder tells which one runs).
• Performance on a Pi 3B+ (64-bit OS, linux target) is measured in docs/reports/raspberry-pi.md. On rpi1 it is not measured (no hardware here). Expectations: a Pi 1/Zero (ARMv6, no NEON) is limited by the YUV to RGB conversion and memory bandwidth, so keep clips at 480p..720p and the surface equal to the framebuffer size; Pi 3/4 should handle 1080p H.264 through V4L2. Four simultaneous players (quad) need small clips on a Pi 1.
• Pi 5 has no H.264 decoder block (software only) and is a 64-bit target; use the linux target there.

Follow-ups

• Audio (SDL3 audio on macOS, ALSA on Linux) with the video clock slaved to the audio clock; wait_time pauses and one-shot already work without it.
• Zero-copy paths on the other targets: a VideoToolbox CVPixelBuffer (an IOSurface) to the window’s compositor on macOS, VA-API surfaces as dmabufs on a Linux desktop, an EGLImage in display-gl for a film that is turned or blended. The contract is the one the KMS planes use (HalDisplay.layers, raster::Picture).
• Seeking (seek(seconds)), playback rate, frame-accurate sync between players or machines.

Edit this page on GitHub ↗