zinc:mapping

Video mapping on the GPU: layers of video, images or generated content, each warped onto a physical surface (corner pin, mesh warp), masked, edge-blended and colour-corrected in shaders, and controlled over OSC. Plugin: plugins/mapping/ (module zinc:mapping). It draws through display-gl, so the project needs "display": "gl". Example: examples/video/mapper/ with its web companion.

import * as gfx from 'zinc:gfx';
import * as mapping from 'zinc:mapping';
import { Player, EXTENSIONS } from 'zinc:video';

const p = new Player(640, 360);
p.addFolder('media', EXTENSIONS);
p.play();
if (!mapping.load('mapping.json')) {                   // previous setup, else a fresh one
  const l = mapping.addLayer('pattern', 'facade');     // test card until the video has a frame
  mapping.set(l, 'corners', [0.1, 0.1, 0.9, 0.05, 0.95, 0.9, 0.05, 0.95]);
}
mapping.listen(9000);                                  // OSC server: the whole address space below
gfx.onFrame(() => { if (p.image >= 0) mapping.setImage(0, p.image); gfx.clear(0); });

API: command(address, numbers, strings), addLayer(source, name), set(layer, key, numbers), setImage(layer, imageId), layerCount(), toJson(), fromJson(json), save(path), load(path), listen(port).

Support matrix

Coordinates

Everything is normalized. Output: (0,0) top-left to (1,1) bottom-right of the screen, whatever its resolution. Surface: a layer’s own unit square (the source image spans it); masks, crop and edge blend live there, so they follow the warp. Per vertex: surface → mesh warp → corner pin → transform → output.

OSC address space

UDP port given to listen (9000 in the example). Numbers may be i, f or d; strings s. <n> is the layer index (0 = bottom, drawn first).

The saved JSON uses the same vocabulary: {"version":1,"layers":[{"corners":[...],"mask/0":[...],...}]}, one key per /layer/<n>/<key> command with its arguments, so a file can be edited by hand and replayed as OSC. Keys unknown to an older version are ignored. Image ids are saved as numbers: they refer to the same player/image when the program creates its images in the same order (the example does).

Incoming OSC datagrams are limited to 2048 bytes (zinc:osc), so a 16×16 mesh cannot be sent in one /mesh message: send /mesh cols rows then /point per point (the companion does).

Rendering

• Geometry: the mesh is a Catmull-Rom surface through the control points (a cubic Bézier patch per cell whose handles come from the neighbours), tessellated on the CPU into 32×32 quads only when it changes. Corner pin (square→quad homography), transform and NDC are one 3×3 matrix applied in the vertex shader with w, so the GPU interpolates texture coordinates perspective-correctly.
• Masks: polygons (signed distance to the outline, smoothstep over the feather), crop, edge blending (smoothstep^(1/gamma), so two overlapping projectors sum to constant light) and the image mask are rendered into a 512×512 mask texture per layer, only when one of them changes; drawing a layer then costs one texture fetch.
• Colour: gains, hue rotation, saturation, contrast and brightness are affine, combined on the CPU into one 4×4 matrix, then gamma. Output is premultiplied; blend modes are GL blend functions.
• Sources: one program per source kind (#define SRC), compiled on first use. Runtime images are uploaded only when image_version changes.
• UI: the program’s zinc:gfx frame is the overlay; gfx.clear(0) leaves it transparent (display-gl key colour).

Companion

examples/video/mapper/companion/: node server.mjs --app <pi-ip>:9000 serves the editor on http://<this machine>:8080 and relays it to the app as OSC; no npm dependencies. Layer list, drag corner pins or mesh points over the output preview, masks/crop/edge blend on the surface view, colour and transform sliders, save/load. node demo.mjs <url> drives a running app through the same HTTP API and checks the state it reports back. The server listens on 127.0.0.1 (--host 0.0.0.0 for a tablet on the LAN) and its API needs the token it prints in the editor URL (http://host:8080/?token=..., header x-zinc-token; --token or ZINC_COMPANION_TOKEN to fix one), which also keeps other web pages open in the same browser from driving the app.

The app’s OSC port has no authentication (UDP senders can be spoofed): listen(port) binds every interface unless ZINC_BIND=127.0.0.1 restricts it (the companion then runs on the same machine). Over OSC, /save and /load take a plain file name in the working directory (show.json, no directories), and /sync answers at most 10 times per second. save(path) / load(path) from the program itself accept any path.

Not done

• Source crop (showing part of a video on a layer) and per-layer content transform; soft edge on the mesh border (use crop feather).
• Masks are 8-bit: very wide, very soft edge blends can band.
• Multiple outputs/projectors from one program; output colour calibration (LUT).

Edit this page on GitHub ↗