lotobo/regex-tools
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 6 Wiki Activity
Files
7079cde15f7647160c5aaa133b0ebed9a7c08208
regex-tools/docs/ENGINE_BUILDS.md

7.1 KiB
Raw Blame History

Engine builds

The production application uses the browser's native RegExp for ECMAScript and bundles verified PCRE2, Python and Java runtime packs. Normal application builds never download or compile an engine: reviewed files under public/engines/ are copied into dist/ and verified there.

Rebuilding PCRE2

The reproducible build accepts only:

  • PCRE2 tag pcre2-10.47, signed tag object cd007b4466798f66d479d1442a407099e7c40050, peeled commit f454e231fe5006dd7ff8f4693fd2b8eb94333429 and tree 81a83a3552bd68d0ea7b7004f8bb6e7892f583ba;
  • Emscripten 6.0.4, compiler revision fe5be6afdff43ad58860d821fcc8572a23f92d19, from emsdk commit 224ec5f9f2f72f09f9ce0e26d66bae7dbd8b692f;
  • CMake 4.3.4 and Ninja 1.13.2;
  • 8-bit PCRE2 with Unicode enabled and JIT, threads and filesystem disabled.

The source and compiler checkouts must already exist locally and be completely clean. No command clones, downloads or updates them:

npm run engines:pcre2:build -- \
  --source-dir /absolute/path/to/pcre2 \
  --emcc /absolute/path/to/emscripten/emcc
npm run engines:pcre2:verify
npm run engines:pcre2:install

Use --force only to replace the exact ignored .engine-build/pcre2 output. The explicit install command first re-verifies that pack and then atomically replaces only public/engines/pcre2.

The builder verifies Git objects and critical source/compiler hashes, sanitizes build flags, sets SOURCE_DATE_EPOCH=1760997684, builds serially and emits exactly:

  • pcre2.mjs and pcre2.wasm;
  • deterministic engine-metadata.json;
  • the exact upstream LICENSE.txt;
  • SHA256SUMS.

The verifier checks the closed file set, metadata, source bridge hashes and all checksums. It validates and instantiates WebAssembly, checks ABI/engine/config identity, then executes native Unicode compile, bounded match and bounded substitution smoke cases plus normal and cap-stopped automatic-callout traces.

The immutable signed-tag object is pinned. OpenPGP trust validation remains a release-operator step because the offline builder does not install or trust a key.

Installing the Python pack

The Python pack is derived from the exact pyodide@314.0.3 npm package locked with integrity:

sha512-sK40My6m8tmBUYtYH9au9rXUeh9x0wfahtHdOlGmJxZDsKBGKtP6KznyFB2+u/klbQTdDionR0uaVd176zVQzQ==

That Pyodide release is tag/commit ac57031be7564f864d061cb37c5c152e59f83ad4. Direct executable identity verification reports CPython 3.14.2; the shipped pyodide-lock.json independently reports info.python as 3.14.0, and the verifier records rather than conflates those two upstream identities. The pack copies only the reviewed loader, WebAssembly runtime, Python standard library and lock metadata required for a local CPython re worker. It adds deterministic engine metadata, a closed-file-set checksum manifest and exact Pyodide/CPython licence material.

The install operation reads from the already installed pinned npm package. It does not contact a CDN or package index. It verifies source package identity, file hashes, runtime metadata and licence inputs before atomically replacing only public/engines/python/. The distribution verifier checks the same closed set under dist/, validates WebAssembly magic and confirms that all runtime references remain relative and self-hosted.

node scripts/python-engine-pack.mjs \
  node_modules/pyodide \
  .engine-build/python
node scripts/install-python-engine.mjs .engine-build/python

The installed closed set is:

  • pyodide.mjs, pyodide.asm.mjs and pyodide.asm.wasm;
  • python_stdlib.zip and pyodide-lock.json;
  • engine-metadata.json and SHA256SUMS; and
  • LICENSE.pyodide.txt and LICENSE.cpython.txt.

Building the Java pack

The Java bridge under engines/java/ wraps TeaVM 0.15.0 java.util.regex.Pattern and Matcher. TeaVM tag/commit ee91b03e616c4b45401cd11fb0cd7eb0daf6649b is pinned through the Maven build. The build produces a minified ES2015 module without source maps, source files, remote imports or an OpenJDK runtime.

The pack gate verifies the Maven coordinates, bridge source identity, TeaVM engine identity and self-test, generated module, deterministic metadata, checksums, Apache-2.0 licence and TeaVM NOTICE. Only the verified pack is installed under public/engines/java/.

node scripts/java-engine-pack.mjs
node scripts/install-java-engine.mjs

The pack command performs two clean offline Maven builds and requires identical output. The installed closed set is java-regex.mjs, engine-metadata.json, SHA256SUMS, LICENSE.txt and NOTICE.txt.

TeaVM supplies its own Apache Harmony-derived class-library implementation. The resulting engine is therefore identified as TeaVM java.util.regex 0.15.0, not OpenJDK. TeaVM 0.15.0 does not implement OpenJDK's Pattern.UNICODE_CHARACTER_CLASS; the bridge treats the application U flag as a compatibility request, implies Unicode case folding and reports that predefined-character classes and word boundaries retain TeaVM behaviour.

Runtime boundaries

PCRE2

ABI version 3 has no retained native handles. Each call compiles, obtains capture names, matches/substitutes or traces and releases its code, match data and match context before returning. The caller supplies fixed-capacity result/name/output/event/mark buffers. Hard maxima are 1 MiB pattern, 16 MiB subject, 1,000 captures, 10,000 matches, 100,000 capture rows, 50,000 trace events, 10 MiB serialized trace data and 128 MiB PCRE2 heap limit; the workbench uses stricter pattern and replacement input limits where applicable.

PCRE2 always runs with UTF and UCP. Native UTF-8 byte offsets are retained in DTOs and converted only at valid code-point boundaries to half-open editor UTF-16 ranges. Lone UTF-16 surrogates are rejected before encoding.

Normal execution lives only in pcre2.worker. Instrumented PCRE2_AUTO_CALLOUT collection is a separate ABI operation loaded only by pcre2-trace.worker; normal match, replacement, tests, corpus and benchmarks never call it. Each supervisor terminates its worker on timeout, crash, cancellation or supersession and lazily creates a clean generation for the next request.

Python

The Python worker loads the self-hosted Pyodide pack and verifies Pyodide, CPython and re identities before accepting work. The application bridge is evaluated inside that interpreter and exposes bounded compile, match and native replacement operations. It reports CPython's Unicode code-point positions; the TypeScript adapter retains them and converts only valid boundaries to half-open editor UTF-16 ranges. Runtime load has a separate startup deadline. Execution timeout, cancellation or crash terminates the worker and its complete Python/WebAssembly heap.

Java

The Java worker imports the self-hosted TeaVM ES module and checks the bridge ABI, declared engine identity and compiled self-test before accepting work. Every request compiles a Pattern, obtains bounded Matcher results or native replacement output, and returns native half-open UTF-16 ranges. No Java virtual machine, OpenJDK classes, remote service or ECMAScript regex fallback is involved. Runtime load has a separate startup deadline; termination discards the complete worker module state.