av-tools/README.md

# Audio & Video Tools

av-tools is a standalone, local-first browser application for converting and
lightly editing audio/video. Media is processed by self-hosted FFmpeg
WebAssembly in a browser worker: there is no upload API, server transcoder,
telemetry, runtime CDN, remote-media fetch or third-party script.

It is also a Toolbox SDK application. The same static build runs directly or
under Toolbox Portal with shared shell, theme, help and app switching.

> **Release status:** 0.2.1 is licensed GPL-3.0-or-later. Its canonical
> release consists of the exact reviewed `v0.2.1` source, production
> ZIP/checksum and Corresponding Source bundle published together. See
> [Licensing](#licensing).

## Product scope

Two workflows keep the product intentionally smaller than a nonlinear editor:

- **Quick Convert:** drop/import media, inspect streams, choose a
  capability-checked preset/streams, convert, preview when the browser can, and
  download.
- **Edit:** a media bin and sequential non-overlapping clip timeline with
  non-destructive settings, one export at a time.

The 0.2 release covers typed/capability-gated conversion and remux,
fast/accurate trim, split, compatible/normalized concatenate, reorder, crop,
resize, frame-rate change, audio extraction/stream removal, waveform,
peak/two-pass loudness normalization, gain, audio/video fades, metadata,
chapters, SRT/WebVTT/ASS text subtitles, thumbnails, contact sheets, presets,
versioned project save/load, reattachment, progress, cancellation/recovery,
diagnostics and explicit browser export.

The Edit workspace now exposes structural exports directly: fast stream-copy
or preset-backed accurate trim uses the selected clip's timeline range, while
fast concatenate requires explicit compatibility diagnostics and normalized
concatenate requires a preset plus an explicit missing-audio policy. Normalized
joins can apply a uniform, capability-gated 0.04–10 second audio/video
crossfade at every edit, with clip-material checks before queueing.
Operations enter a bounded, in-memory application queue with at most eight
scheduled operations and one FFmpeg execution at a time.

Thumbnail series and contact sheets support even, interval, chapter, marker or
bounded scene-change selection. Scene analysis records exact source
presentation timestamps before extraction. Contact-sheet labels use only the
bundled, SHA-256-verified DejaVu Sans 2.37 file and remain disabled unless the
active core exposes `drawtext`; there is no remote/system-font fallback.

When a source cannot be played directly, the preview panel can create a
disposable H.264/AAC MP4 proxy. The user can bound it to 1–300 seconds and
choose a maximum width from 480–1920 pixels. A proxy is reused from the local
derivative cache only when the source identity and settings match.

A control is not considered complete merely because its UI or command builder
exists. The exact milestone status is the tested matrix in the final release
report; [CHANGELOG.md](CHANGELOG.md) lists current blockers.

## Engine and format model

Exact runtime packages:

- `@ffmpeg/ffmpeg` 0.12.15;
- `@ffmpeg/util` 0.12.2;
- reviewed FFmpeg core build `0.12.10-reviewed-stack5m.1` in single-thread and
  multithread profiles.

The reviewed cores contain **FFmpeg n5.1.4** (libavcodec 59.37.100), built
with x264, x265, libvpx, LAME, Ogg/Theora/Opus/Vorbis, zlib, libwebp,
FreeType/FriBidi/HarfBuzz/libass and zimg. Both ~32 MiB WASM cores and their
JS/worker files are verified against a tracked build lock, unpacked from a
deterministic repository artifact and served from the app's own origin.
`npm run build` does not download them.

Every engine load queries actual demuxers, muxers, decoders, encoders and
filters. Presets remain disabled when requirements are absent; an extension or
FFmpeg build flag is not a compatibility promise.

To bound native heap retained by heterogeneous long-running sessions, the app
refreshes an idle core before its seventeenth media execution and before a
fourth contact sheet uses the pinned core's retained `tile`-filter state.
Queue state, sources and completed results stay in application memory; only
the worker is recreated. A bare core abort also triggers recovery for a retry.

Generated fixtures cover MP4/H.264/AAC, WebM/VP8/Vorbis, WAV/PCM, MP3, FLAC,
metadata, chapters, attached cover art, SRT, WebVTT, ASS and malformed input.
All 15 fixtures were generated/re-probed with native FFmpeg 7.1.3 for test
preparation and pass the repository checksum/probe verifier. Chromium
automation uses the pinned ST core for a broad operation matrix and separately
proves a real MT H.264/AAC conversion. That does not establish cross-browser
support or exhaustive MT parity, and fixture generation is not evidence that
“all formats” work. See
[format support](docs/FORMAT_SUPPORT.md) and
[fixture provenance](tests/fixtures/README.md).

The recorded pinned-core capability inventory contains 341 demuxers, 173
muxers, 515 decoders, 204 encoders and 465 filters (1,698 entries total).
These are discovered capabilities, not product or browser-playback guarantees.

Browser playback is independent of FFmpeg readability. MP4/H.264/AAC,
WebM/VP8/Vorbis, MP3 and WAV are common preview candidates, but actual codec,
browser and platform support is checked at runtime. Matroska and many other
valid outputs may be download-only. A preview proxy may require re-encoding.

The 2026-07-26 Chromium regression harness ran the real built-in `audio-opus`
plan against both reviewed cores. Each selected the requested ST/MT mode,
produced non-empty Opus and passed codec/duration re-probing. Opus presets are
therefore enabled when runtime capability discovery also succeeds. Exact
commands and evidence boundaries are in
[FFmpeg build](docs/FFMPEG_BUILD.md).

## Execution modes

- **Multithread** is selected only in secure, cross-origin-isolated contexts
  with `SharedArrayBuffer`.
- **Single-thread compatibility mode** is the automatic fallback when
  isolation is unavailable or MT loading fails. It remains fully worker-based
  but may be slower.

MT requires every response in the page to support:

```text
Cross-Origin-Opener-Policy: same-origin
Cross-Origin-Embedder-Policy: require-corp
Cross-Origin-Resource-Policy: same-origin
```

Serve `.wasm` as `application/wasm`, keep all core files same-origin and allow
`'wasm-unsafe-eval'` plus workers in CSP. Blob media preview additionally needs
`media-src 'self' blob:`. The manifest intentionally does **not** require
cross-origin isolation because ST is supported.

## Limits, storage and privacy

ffmpeg.wasm documents a hard **2 GiB input ceiling**. Real browser limits are
often lower because sources, decoded frames, filters and outputs coexist. The
initial application policy warns at 256 MiB input, blocks a single or total
input above 1.5 GiB, warns at a 512 MiB output estimate and blocks an estimated
output above 1 GiB. The in-memory queue accepts at most eight scheduled
operations and runs only one FFmpeg operation at a time. Other defaults bound
clips, streams, dimensions/pixels, thumbnails, contact-sheet cells and runtime.
Refreshing or closing the page discards queued work.

Estimates are not guarantees. Lower resolution, duration, frame rate, pass
count and output count can reduce failure risk. WebAssembly does not offer
desktop FFmpeg performance or hardware encoding.

Source `File` objects and preview/result Blob URLs are temporary and object URLs
are revoked on replacement/cleanup. Per-job FFmpeg files/mounts are deleted;
cancellation discards the engine filesystem. The mounted application currently
uses IndexedDB for validated user presets; project JSON is explicitly
saved/loaded by the user and the operation queue remains in memory. Optional
OPFS caching is limited to regenerable derivatives and is clearable; source
bytes are **not persisted by default**. OPFS belongs to the current origin, so
changing scheme/host/port does not move cached data. See
[storage](docs/STORAGE.md).

Imported media, names, tags and subtitles are untrusted. They are rendered as
text, paths are generated, operations build argument arrays rather than shell
commands, logs are bounded/redacted, and extracted URLs are inert. See
[security](docs/SECURITY.md).

## Development

Requirements: Node.js 22+, npm matching the lockfile, and a modern browser.
Native FFmpeg is a development/test dependency for fixture regeneration and
the conditional native command-plan integration suite; it is not a production
runtime dependency.

```sh
git clone git@git.add-ideas.de:lotobo/av-tools.git
cd av-tools
npm ci
npm run dev
```

The token-free `.npmrc` selects the published Toolbox SDK scope. Do not add
credentials to repository configuration.

Core assets are generated local build inputs:

```sh
npm run vendor:ffmpeg
npm run vendor:verify-runtime
```

The runtime verifier checks the tracked deterministic core artifact, source
lock, build driver and generated same-origin files. The stricter
`npm run vendor:verify` publication gate additionally requires and hashes the
ignored Corresponding Source archive generated by `npm run source:package`.

Regenerate or verify the tiny synthetic fixtures:

```sh
npm run fixtures:generate
npm run fixtures:verify
```

## Checks

```sh
npm run manifest:check
npm run vendor:verify-runtime
npm run typecheck
npm run lint
npm run format:check
npm test
npm run build
npm run toolbox:check
npm run test:browser
npm run evidence:large-preview # set AV_TOOLS_LARGE_MEDIA first
npm run evidence:opus
```

`npm run check` combines the non-browser production checks. Browser tests cover
the real pinned cores, MT-to-ST fallback, cancellation/recovery, generated
fixtures, nested deployment, Toolbox context and the absence of runtime CDN
traffic.

`npm run evidence:opus` independently contains each ST/MT browser process,
prints one JSON report and exits normally even when the result is unsafe.
`npm run test:opus-regression` runs the same evidence as the positive Opus
enablement gate and exits nonzero unless both modes produce non-empty,
re-probeable Opus. It is a mandatory release gate for the reviewed core
profile.

`AV_TOOLS_LARGE_MEDIA=/absolute/video.mp4 npm run evidence:large-preview`
performs the opt-in large-file regression: the native video element must mount
and advance before bounded inspection finishes, remain the same element after
the authoritative probe completes and produce no browser error.

The latest recorded local Vitest run on 2026-07-26 completed **313 passing
tests and two conditional skips**. It includes broad structured-command
execution against native FFmpeg 7.1.3; the skipped native subtitle burn-in and
labeled contact-sheet cases require `subtitles` and `drawtext` filters that are
unavailable in that host build.

The Chromium suite exercises the pinned ST core across conversion,
trim/concat/split, bounded crossfades, analysis, waveform, scene-selected image
generation, bundled-font contact-sheet labels, metadata, chapter, subtitle,
queue and recovery workflows. The MT core is directly initialized and queried,
and completes a real non-empty, re-probeable H.264/AAC conversion; the same
path is benchmarked with 2-second 160×90 and 6-second 640×360 inputs. Nested
static/Toolbox-context loading is also covered. This evidence is
Chromium-specific: Firefox, Safari, mobile and exhaustive MT operation parity
remain unrecorded. The exact one-run performance record and its limitations are
in [performance observations](docs/PERFORMANCE.md).

## Deployment

Deploy the contents of `dist/` beneath any static path; URLs are base-relative
and no server route fallback is required. Production examples:

- [Nginx](deployment/nginx.example.conf)
- [Caddy](deployment/caddy.example)
- [generic static headers](deployment/headers.md)

The policy permits same-origin WebAssembly and Blob workers/media without
broad `'unsafe-eval'`. Keep stable entry/manifest/notices revalidated. Exact
versioned core files and Vite content-hashed assets may be cached immutable.
When isolation headers cannot be configured, ST remains the supported
fallback.

## Toolbox integration

The one typed `de.add-ideas.av-tools` manifest generates
`public/toolbox-app.json`. `AppShell` receives the relative
`./toolbox-app.json` URL and falls back to standalone behavior when Toolbox
context is absent or invalid. av-tools does not duplicate suite
personalization or app switching.

The manifest uses `entry: "./"`, relative icon/assets, top-level navigation,
local processing/no upload/no telemetry, and advertises workers, secure
context and IndexedDB. Its `source` object identifies the canonical av-tools
repository and the `GPL-3.0-or-later` SPDX licence.

Portal must consume an independently built ZIP; it must not build or import
av-tools source. Immutable Portal revision
`5c052bc6d1073a66600cfddc44c0e16da34bede8` (`v0.6.0`) provides the required
isolation, narrow WebAssembly CSP, Blob media policy and explicit WASM MIME
mapping. The active Portal worktree additionally extends immutable caching to
the exact reviewed FFmpeg build profile without accepting mutable-looking
suffixes. Its exact reviewed and staged
boundaries are documented in
[Portal requirements](docs/PORTAL_REQUIREMENTS.md).

## Release

After all checks, normal packaging would produce deterministic:

```text
release/av-tools-<version>.zip
release/av-tools-<version>.zip.sha256
```

The ZIP root contains `index.html`, `toolbox-app.json`, `CHANGELOG.md`,
`LICENSES/`, `SOURCE.md`, `THIRD_PARTY_NOTICES.md`, `assets/` and all
`vendor/` core files. It excludes `node_modules`, sources, source maps and
fixtures. Paths are sorted, timestamps/modes normalized and the SHA-256
sidecar is verified.

Normal packaging requires the root GPL licence and matching package/manifest
metadata:

```sh
npm run package:release
npm run portal:smoke -- \
  --portal-root ../toolbox-portal
```

The smoke harness clones the selected Portal `HEAD`; it never consumes
uncommitted Portal files silently. Full commands and safety behavior are in
[release documentation](docs/RELEASE.md).

## Licensing

av-tools-owned code and documentation are Copyright © 2026 Albrecht Degering
and licensed **GPL-3.0-or-later**. See [LICENSE](LICENSE). The application
provides a visible licence, source and no-warranty notice in its Help dialog.
Third-party components retain their copyright and licence terms.

The wrapper/util are MIT. The reviewed 0.12.10 cores statically include
GPL-enabled FFmpeg/external codec code with additional terms. The branch-based
x264/LAME inputs have been resolved to immutable commits in the source record.
A binary publication must include equivalent access to the reviewed
Corresponding Source and notices. See:

- [third-party notices](THIRD_PARTY_NOTICES.md)
- [source record](SOURCE.md)
- [licence texts/status](LICENSES/README.md)

The complete distributed application must not be called MIT-only. Publishing a
binary or hosted JS/WASM build requires equivalent access to its corresponding
source; linking only to a component homepage is not asserted to satisfy that
obligation. Codec patent requirements are separate from copyright licensing.

## Known limitations

The auditable list is in
[known limitations](docs/KNOWN_LIMITATIONS.md). In summary:

- No server transcoder, remote URL input, live capture, DRM or encrypted media.
- No overlapping video tracks, arbitrary compositing, collaboration or
  professional nonlinear editing.
- No “all formats,” unlimited file size, desktop performance or hardware
  encoding guarantee.
- Stream-copy trim is fast but not promised frame-accurate.
- Re-encoding is not lossless; lossless codec output cannot restore discarded
  source information.
- Metadata removal is best effort, not an anonymity guarantee.
- Browser preview supports fewer combinations than FFmpeg.
- MT depends on isolation; ST can still hit memory/time limits.
- Opus enablement is supported by Chromium ST/MT fixture evidence, not a
  cross-browser or arbitrary-input guarantee.
- Projects do not contain source bytes by default and require reattachment.
- Cross-browser/mobile and exhaustive MT performance matrices remain
  unrecorded; published support claims must stay within the tested evidence.

## Roadmap

1. Extend the evidence matrix beyond Chromium to Firefox, Safari and mobile;
   broaden MT operation parity and replace the one-run performance observation
   with repeated cold/warm measurements.
2. Evaluate optional editing breadth without weakening local-first guarantees:
   additional reviewed transition shapes, richer scene-ranking controls and
   explicit transition audio-curve choices beyond the current bounded uniform
   crossfade.