Assets that exist exactly once: audit and design

Research report, 2026-10-18. No code changed. Code claims cite file:line read on this date; web claims cite a URL (all fetched or seen in search results on 2026-10-18; items marked (unverified) were not confirmed on the page). Paths are under the repository root; next/ is Zinc Next. Not redone here: the OTA design (docs/reports/zinc-app-ota-updates.md), the museum measurements (docs/reports/zinc-next-assets-outside-binary.md, 2026-10-12), the pipeline design (docs/reports/games/toolchain-assets-loading.md).

Goal: an asset (image, font, audio, video, model, data, shader) is stored once, where it suits the target and its size, mapped or loaded lazily, identified by content hash, so OTA and dev reload move only what changed.

1. Exists

2. Partial (works, but a second copy is possible or the check is missing)

Inventory of what lives where today, then every duplicate I found.

Duplicates, ordered by size of the effect:

1. Baked RGBA and the source file. The .app/dist keeps assets/photo.jpg and the baked RGBA of it in the executable. Measured on the museum build: executable 93.67 MB of which 87.35 MB is the ZRS1 blob; 15 images become 75.60 MB of RGBA from 5.71 MB of JPEG; the .app holds both (docs/reports/zinc-next-assets-outside-binary.md, “Ce que contient actuellement le build”). Cause: exportNative always copies the whole assets/ (cli_core.cpp:1763) and the baker bakes every image of it (main.cpp:419-428 skips only texture-rule images).
2. Baked blob embedded and then copied to the heap. The blob lives in the binary’s data segment and installResources mallocs and memcpys every image and font (resources.cpp:83-84,94-95): two copies in RAM terms (file-backed pages plus heap). The museum report counts 75.60 MB images + 11.61 MB fonts copied at start.
3. .zapp unpacked into a cache. zinc run app.zapp and a fused exe unpack all members to ~/.zinc/cache/zapp/<id> (main.cpp:1124-1132); the archive and the cache hold the data twice on disk. The docs promise “read in place from the member” (docs/zpk.md, 1st paragraph) but the code unpacks; recorded as open in backlog/later.md:43 (ZN-430). A fused exe also contains the whole engine (15 MB for a hello, docs/zapp.md).
4. .zapp containing loose assets/ and the packs. pack adds every file of assets/ as a member and, when assets.packed(), also the packs/*.zpk built from them (main.cpp:1760-1769). A packed project therefore ships each source and its pack entry.
5. Android: APK assets/ then copy to app storage. The shell copies zinc/assets/* to disk once per version (cli_core.cpp:1404-1414, src/app/shell_sdl.cpp:90-95): APK copy plus filesystem copy. Compression is not controlled (no noCompress/androidResources found in cli_core.cpp or templates/), so an asset read through the APK cannot be mapped without inflating it.
6. Web: app.res baked images and the loose assets/ copy (cli_core.cpp:1650,1680), and the page splash as a data URL in index.html (D96). Same file twice in the download; the 40 MB download budget (capabilities.json:14) does not see it.
7. Texture-compressed and source image. Avoided only if a texture rule exists; without it (the museum case, simple "assets": "assets") no rule applies, so the RGBA bake and the file coexist (museum report: “cette exclusion n’a pas été appliquée”).
8. Fonts. Subsets of lib/fonts Inter/JetBrains Mono are baked into every program (3 subsets 0.068 MB) plus 58 rasterized variants 11.69 MB in the museum build; with "text": "shaped" whole TTFs are embedded (res.h:19) and the system font fallback may load the same family from /System/Library/Fonts (shaped_gfx.cpp:38). Not a duplicate by design, but nothing checks it.
9. texture + leave is by exact rule match only (main.cpp:427): an image referenced from code that is also in a group via a second rule can land in two packs. No cross-group duplicate detection exists (compare Unity, section 5).
10. ZPK entries are deduplicated by name only. Two names with the same bytes in two groups are stored twice; the importer cache dedups work, not storage (assets.cpp keys by content, zpk.cpp has no content-addressed table).
11. Native export may ship no packs at all (unverified). export builds release packs into build/assets/<target> (cli_core.cpp:1753) but I found no copy of them into dist/<name>-<target>/; zrt_program_packs_dir looks for <base>/assets/<target>/.packs (zrt_assets_dir.h:26-33). A packed project exported natively would then run from the loose-file path or fail. Needs one run to confirm.

Other gaps in the pieces that exist:

• zinc assets implements build|info only (cli_core.cpp:99); ls, report, check, clean, verify are task ZN-437, status Backlog (backlog/tasks/zn-437*.md). The budgets in capabilities.json are therefore not enforced by any command.
• The pack codec per profile is not wired into release packs (backlog/later.md:77, ZN-435); assets.dir is ignored by the baker, export and zinc:assets (later.md:76).
• report.json has per-asset rows (assets.h:58-60) but no refcount, no location column and no duplicate section.

3. Missing

• A single rule for where each asset lives per target, enforced by the exporters. Today every exporter decides on its own (table in section 2).
• A resolver that returns a pointer into a pack instead of a copy for the baked fonts and images (the baker’s blob is a private format, res.h:7, not a ZPK).
• Content-hash identity above the entry: pack file names include no hash (the research report proposes level1.3f9a….zpk for the browser, toolchain-assets-loading.md:482-486), so no cache, CDN or OTA step can say “same bytes”.
• Build-time duplicate and dead-asset report (--report-assets), and unreferenced-asset stripping (references from code literals are known to the baker; dynamic names such as project.json entries need a manifest, as the museum report notes).
• Android noCompress for packs, and Play Asset Delivery / iOS ODR mapping of groups.
• ESP32 data partition for packs and esp_partition_mmap reader (design only, toolchain-assets-loading.md:402-415; nothing in firmware/esp32).
• Pack-level delta for OTA: no per-pack hash is exposed in a manifest (only per-file SHA-256 in .zapp, docs/zapp.md).

4. How others do it (sources)

Not searched or not confirmed this session (no claim made): Flutter assets, Git LFS, IPFS naming, WOFF2 subsetting numbers, Opus/ADPCM sizes, OSTree. Treat any design point resting on them as unsourced.

Two things follow from the sources without a web claim of their own: pages of a read-only mapped file are page cache, shared by processes and evictable, so a mapped pack is the one zero-copy form (that is how assets_host.cpp:138 already behaves); and compression and mmap conflict (a compressed entry must be decoded into a new buffer), so the pack codec must be none for anything the consumer can use as stored (GPU block textures, Opus, glTF buffers, ZBC) and compression is for the transfer only.

5. Proposed

5.1 One rule: where each asset lives

Rule for the exporter: embed in the binary only when fs is false in capabilities.json or for the boot set; never embed anything that is also in an external pack. The baker’s blob becomes a ZPK group (boot for the engine fonts and the splash, ui for baked images), so one reader serves it; fonts and images are then pointers into the mapping instead of mallocs (fixes duplicate 2).

5.2 Identity by content hash

• Pack file name = <group>.<first 12 hex of indexHash()>.zpk; a small signed assets.manifest lists name -> pack, indexHash, size and, per entry, the 16-byte hash already in each TOC row (docs/zpk.md, TOC row offset 48). No new hash is needed; indexHash() is already the value a signed manifest should record.
• Dedup at write time: zpk::write keeps a table of entry hashes within a pack and stores identical bytes once, with two TOC rows pointing at the same offset (the format already has a data offset per row; no format change). Across packs, the build rejects or warns (5.3) instead of silently storing twice.
• The same hash drives: dev hot reload (already by entry hash, D97), OTA (send packs or entries whose hash changed; reuse .zapp per-file SHA-256 idea; casync-style chunking is not needed for assets that are already entry-sized), and the browser cache name.
• Variants (same texture at several sizes): produce them from one source in the import graph (cache key already includes options), store only the variants a target’s profile asks for (maxSize in capabilities.json), never the full-size source next to a downsized texture.

5.3 zinc build --report-assets (also zinc assets report, ZN-437)

Output, text and report.json (extends assets.h:58-60):

asset                    kind    bytes   location               refs  hash
photos/irises.jpg        image   0.31MB  ui.3f9a1c.zpk          1     b41c...
photos/irises.jpg        image   4.06MB  <executable: baked>    1     e90d...   DUPLICATE of ui.3f9a1c.zpk
sfx/click.opus           audio   12KB    boot.77aa02.zpk        3     12ef...
totals: pack 38.2MB, executable-embedded 87.3MB (limit 0.25MB), duplicates 2 (4.06MB), unreferenced 3 (0.9MB)

• Location: pack file and offset, <executable: baked>, <loose file>, or <system font>.
• Refs: number of program references (literals, atlas members, group lists, manifest entries); an entry with 0 is “unreferenced” (dead-asset candidate); entries named only by a dynamic manifest are marked dynamic.
• Duplicates: (a) same content hash in two locations; (b) source file shipped loose and also in a pack or baked; (c) a baked image and a texture-rule image from the same source.
• Gate: error when anything is embedded beyond boot budget on a target with fs: true, when a duplicate exists, or when capabilities.json budgets (boot, ram, gpu, download, flash) are crossed; warning for unreferenced. CI uses the exit code (ZN-437’s check).

5.4 Dev loop versus release

Today the dev run bakes images into memory (zinc run: baked in memory, res.h:5) and the release embeds them; moving both to the ui ZPK group removes this difference as well.

5.5 Ordered work items

Order logic: 1 and 2 make the problem visible and safe to fix; 3 and 4 give the large wins on desktop; 5 to 8 finish the other delivery forms; 9 and 10 are tuning and a board-bound item. Items 3 to 8 each ship with the check of item 2 failing before and passing after.

5.6 Risks and open questions

• Item 4 changes the pixel path of every demo that draws baked images; the 3-engine comparison (tools/diff-matrix) and the proto pixel goldens must stay identical (images are byte copies, so they should).
• Dynamic references (names built at run time, project.json lists) defeat dead-asset stripping; the report must mark them, not delete them.
• iOS App Store rule on downloaded code is unrelated to data packs, but a pack that carries ZBC counts as code (OTA report section 4).
• Pack alignment: Android mapping through the APK needs the entry’s offset aligned for mmap (page offset); set ZPK alignment to 4096 for that target (header field exists, docs/zpk.md).

Edit this page on GitHub ↗