regex-tools/docs/ENGINE_BUILDS.md

Engine builds

The production application uses the browser's native RegExp for ECMAScript and bundles ten verified runtime packs for standalone PCRE2, PHP, Perl, Python, Ruby, Java, C++, Go, Rust and .NET. The Scala/JVM compatibility profile shares the Java pack and contains no Scala runtime. Normal application builds never download or compile an engine: reviewed files under public/engines/ are copied into dist/ and verified there.

Rebuilding PCRE2

The reproducible build accepts only:

• PCRE2 tag pcre2-10.47, signed tag object cd007b4466798f66d479d1442a407099e7c40050, peeled commit f454e231fe5006dd7ff8f4693fd2b8eb94333429 and tree 81a83a3552bd68d0ea7b7004f8bb6e7892f583ba;
• Emscripten 6.0.4, compiler revision fe5be6afdff43ad58860d821fcc8572a23f92d19, from emsdk commit 224ec5f9f2f72f09f9ce0e26d66bae7dbd8b692f;
• CMake 4.3.4 and Ninja 1.13.2;
• 8-bit PCRE2 with Unicode enabled and JIT, threads and filesystem disabled.

The source and compiler checkouts must already exist locally and be completely clean. No command clones, downloads or updates them:

npm run engines:pcre2:build -- \
  --source-dir /absolute/path/to/pcre2 \
  --emcc /absolute/path/to/emscripten/emcc
npm run engines:pcre2:verify
npm run engines:pcre2:install

Use --force only to replace the exact ignored .engine-build/pcre2 output. The explicit install command first re-verifies that pack and then atomically replaces only public/engines/pcre2.

The builder verifies Git objects and critical source/compiler hashes, sanitizes build flags, sets SOURCE_DATE_EPOCH=1760997684, builds serially and emits exactly:

• pcre2.mjs and pcre2.wasm;
• deterministic engine-metadata.json;
• the exact upstream LICENSE.txt;
• SHA256SUMS.

The verifier checks the closed file set, metadata, source bridge hashes and all checksums. It validates and instantiates WebAssembly, checks ABI/engine/config identity, then executes native Unicode compile, bounded match and bounded substitution smoke cases plus normal and cap-stopped automatic-callout traces.

The immutable signed-tag object is pinned. OpenPGP trust validation remains a release-operator step because the offline builder does not install or trust a key.

Installing the Python pack

The Python pack is derived from the exact pyodide@314.0.3 npm package locked with integrity:

sha512-sK40My6m8tmBUYtYH9au9rXUeh9x0wfahtHdOlGmJxZDsKBGKtP6KznyFB2+u/klbQTdDionR0uaVd176zVQzQ==

That Pyodide release is tag/commit ac57031be7564f864d061cb37c5c152e59f83ad4. Direct executable identity verification reports CPython 3.14.2; the shipped pyodide-lock.json independently reports info.python as 3.14.0, and the verifier records rather than conflates those two upstream identities. The pack copies only the reviewed loader, WebAssembly runtime, Python standard library and lock metadata required for a local CPython re worker. It adds deterministic engine metadata, a closed-file-set checksum manifest and a complete SOURCE-MANIFEST.json.

The source manifest records immutable Git commit/tree identities or primary release-archive SHA-256 values for Pyodide, its pyodide-build submodule, its vendored Error Stack Parser 2.1.4 and StackFrame 1.3.4 ports, CPython, libffi, Hiwire, XZ/liblzma, Zstandard, SQLite, the Emscripten bzip2 and zlib ports, Emscripten itself and HACL* preferred source. It hashes both vendored JavaScript ports, all nine CPython patches, all five Emscripten patches, and the Pyodide Makefiles that select those inputs. That is the preferred-form/patch route for the upstream MPL source; the exact npm package remains the binary-distribution reference, not a claim that arbitrary future host SDK packaging will reproduce identical bytes.

The linked-component inventory is derived from the exact Pyodide recipe and checked against executable identities where the libraries expose one. Besides Pyodide and CPython, the base module contains CPython-vendored Expat 2.7.3, libmpdec 2.5.1 and HACL*, plus libffi, Hiwire 1.0.1, liblzma 5.2.2, Zstandard 1.5.7, SQLite 3.39.0, bzip2 1.0.6 and zlib 1.3.1. The conservative Emscripten 5.0.3 runtime notice closure covers musl, compiler-rt, libc++, libc++abi, libunwind, dlmalloc and the MiniLZ4 source selected by the pinned link flags. The upstream stripped module contains no link map, so retaining the complete potentially selected Emscripten runtime notice set is intentional.

The install operation reads from the already installed pinned npm package. It does not contact a CDN or package index. It verifies source package identity, file hashes, runtime metadata, every independently fetched legal-source hash, the generated source manifest and licence inputs before atomically replacing only public/engines/python/. The distribution verifier checks the same closed set under dist/, validates WebAssembly and native component identities and confirms that all runtime references remain relative and self-hosted.

node scripts/python-engine-pack.mjs \
  node_modules/pyodide \
  .engine-build/python
node scripts/install-python-engine.mjs .engine-build/python

The installed closed set is:

• pyodide.mjs, pyodide.asm.mjs and pyodide.asm.wasm;
• python_stdlib.zip and pyodide-lock.json;
• engine-metadata.json, SOURCE-MANIFEST.json and SHA256SUMS; and
• 22 exact LICENSE.*/NOTICE.* files enumerated and source-anchored by that manifest.

pyodide-lock.json is the upstream catalogue of optional packages. Regex Tools does not ship the catalogue's wheels; externalPackagesIncluded: false and the closed pack file set make that boundary machine-verifiable.

Building the Java pack

The Java bridge under engines/java/ wraps TeaVM 0.15.0 java.util.regex.Pattern and Matcher. TeaVM tag/commit ee91b03e616c4b45401cd11fb0cd7eb0daf6649b is pinned through the Maven build. The build produces a minified ES2015 module without source maps, source files, remote imports or an OpenJDK runtime.

The pack gate verifies the Maven coordinates, bridge source identity, TeaVM engine identity and self-test, generated module, deterministic metadata, checksums, Apache-2.0 licence and TeaVM NOTICE. Only the verified pack is installed under public/engines/java/.

node scripts/java-engine-pack.mjs
node scripts/install-java-engine.mjs

The pack command performs two clean offline Maven builds and requires identical output. The installed closed set is java-regex.mjs, engine-metadata.json, SHA256SUMS, LICENSE.txt and NOTICE.txt.

TeaVM supplies its own Apache Harmony-derived class-library implementation. The resulting engine is therefore identified as TeaVM java.util.regex 0.15.0, not OpenJDK. TeaVM 0.15.0 does not implement OpenJDK's Pattern.UNICODE_CHARACTER_CLASS; the bridge treats the application U flag as a compatibility request, implies Unicode case folding and reports that predefined-character classes and word boundaries retain TeaVM behaviour.

Building the PHP pack

The PHP bridge under engines/php/ executes actual PHP 8.5.8 preg_*. @php-wasm/web-8-5 and @php-wasm/universal 3.1.46 plus their pinned support packages supply the Asyncify WebAssembly runtime and host API. Direct identity verification reports PCRE2 10.44; this pack is not the standalone PCRE2 10.47 build.

node scripts/php-engine-pack.mjs
node scripts/install-php-engine.mjs

The builder copies only the pinned npm runtime files, changes the bundler-specific WebAssembly import into a same-directory URL, records the original and transformed hashes, and runs real identity, match, bounded replacement and native-component positive/negative audits before installation. native-components.json binds the exact WebAssembly hash to WordPress Playground v3.1.46 commit 581c7c172428159eb4e6c5309054a568cd39a97a, PHP tag php-8.5.8 commit 26b97507444c4fbda072f57dda1820f7b7d5e467, Emscripten 4.0.19 commit 08e2de1031913e4ba7963b1c56f35f036a7d4d56 and all 32 linked native/source surfaces. It carries exact archive hashes or Git identities where upstream recorded them and honestly bounds the one unpinned Oniguruma clone using the committed archive/header identities and upstream source interval.

The pack elects PHP License version 4 (BSD-3-Clause) for PHP 8.5.8 through PHP's later-version option and includes the complete PCRE2 and PHP-vendored, external-library, patent and Emscripten/LLVM/musl legal set. This includes the exact Zend Engine licence, official php-src binary redistribution notice, CLI HTTP parser MIT terms, and the public-domain, CC0 and FNV permission notices for linked hash implementations. The WordPress Playground @php-wasm packages retain GPL-2.0-or-later; JavaScript support-package licences remain separate. All 42 legal assets and the native inventory are independently SHA-256-pinned by the builder, then covered again by the closed file set, SHA256SUMS, release requirements and metadata.

Building the legacy Perl pack

The Perl bridge under engines/perl/ uses the unmodified WebPerl 0.09-beta prebuilt archive, SHA-256 5f441249217e90ab378c666f473d4206ab4f44907f6bb0aa8d70834bc38c40dc. That 2019 beta contains Perl 5.28.1 and Emscripten 1.38.28 output. It is intentionally a legacy compatibility target, not a current Perl distribution.

The builder accepts only a local copy of that exact official archive; it never downloads. An already extracted release root can be supplied with --prebuilt instead of --archive.

node scripts/perl-engine-pack.mjs \
  --archive /absolute/path/webperl_prebuilt_v0.09-beta.zip
node scripts/perl-engine-pack.mjs --verify .engine-build/perl
node scripts/install-perl-engine.mjs .engine-build/perl

The installed pack retains upstream emperl.js, emperl.wasm and emperl.data byte-for-byte. Application-owned fixed worker and Perl bridge files carry values through an exact-key, byte-bounded JSON request path; they do not splice patterns or replacements into source. The metadata response remains bounded JSON, while successful replacement UTF-8 bytes stream into a separate fixed MEMFS binary file and receive exact worker-side byte, size and encoding validation. Exact WebPerl/Perl, Emscripten and bundled-CPAN notices travel with the pack.

The legacy generated loader contains one eval(code) site in an optional Perl-to-JavaScript interoperability import. The application bridge never loads that facility. Its browser CSP smoke passes with 'wasm-unsafe-eval' and without broad 'unsafe-eval'; that dormant upstream site remains documented rather than removed from the checksummed asset.

Building the Ruby pack

The Ruby pack copies the minimal runtime from pinned @ruby/4.0-wasm-wasi@2.9.3-2.9.4 and uses @ruby/wasm-wasi@2.9.3-2.9.4 as its host. It verifies the CRuby 4.0.0 wasm32-wasi identity and actual Regexp/replacement behavior before installation.

node scripts/ruby-engine-pack.mjs
node scripts/install-ruby-engine.mjs

The closed set contains the runtime WebAssembly file, deterministic metadata and checksums, ruby.wasm licence/complete NOTICE and the exact browser WASI shim and tslib licence texts needed by the host bundle. The production worker passes each request through a bounded length-prefixed file in its private WASI memory filesystem. It receives bounded inert JSON metadata and reads bounded UTF-8 replacement output from a separate private file, rather than amplifying that output through JSON or using RubyVM.wrap/toJS and their JavaScript-eval conversion path. Ruby cold startup passes the release's strict-CSP browser gate.

Building the C++ pack

The C++ bridge under engines/cpp/ is compiled with Emscripten 6.0.4 at compiler revision fe5be6afdff43ad58860d821fcc8572a23f92d19. It uses libc++ std::wregex, 32-bit wchar_t, fixed structured Embind entry points and -sDYNAMIC_EXECUTION=0.

node scripts/cpp-engine-pack.mjs --empp /absolute/path/to/em++
node scripts/install-cpp-engine.mjs

The verifier checks exact Emscripten, emsdk and LLVM identities, bridge hashes, WebAssembly/module structure, absence of dynamic-execution constructs in the generated module, runtime identity, code-point ranges and bounded replacement parity. The build also enables wasm-ld --trace and accepts only the reviewed linked archive set: Emscripten runtime support, musl libc, emmalloc, compiler-rt builtins, libc++ and libc++abi.

SOURCE-MANIFEST.json ties the exact module hash to Emscripten tag/commit 6.0.4/fe5be6afdff43ad58860d821fcc8572a23f92d19, emsdk commit 224ec5f9f2f72f09f9ce0e26d66bae7dbd8b692f, the official release commit and LLVM compiler commit. It records preferred-source paths and independently pinned legal files for musl 1.2.6, emmalloc, compiler-rt 22.1.8, libc++ 21.1.8 and libc++abi 21.1.8. The current JavaScript-exception link selects no libunwind.a member; libunwind 22.1.8 and its legal text remain as an explicit audited exclusion. Any added, removed or newly selected archive fails the pack build.

Building the Go pack

Go 1.26.5 standard-library regexp is built with the official linux/amd64 archive, SHA-256 5c2c3b16caefa1d968a94c1daca04a7ca301a496d9b086e17ad77bb81393f053, for GOOS=js GOARCH=wasm.

node scripts/build-go-engine.mjs /absolute/path/to/go
node scripts/install-go-engine.mjs
node scripts/verify-go-engine.mjs

The pack pins the source commit, official archive, wasm_exec.mjs, bridge source, build flags and Go licence. Verification executes real Unicode matching and regexp.ExpandString cases.

Building the Rust pack

The Rust bridge pins regex crate 1.13.1 and its complete Cargo graph, rustc 1.97.1, the matching wasm32-unknown-unknown standard library and wasm-bindgen 0.2.126.

node scripts/build-rust-engine.mjs /absolute/path/to/rust-toolchain
node scripts/install-rust-engine.mjs
node scripts/verify-rust-engine.mjs

The pack records the distribution manifest/archive hashes, crate checksums, compiler and LLVM identities, bridge sources, exact bindings and complete Rust/crate licence material. Runtime verification covers identity, Unicode byte ranges, limits and Captures::expand.

Building the .NET pack

The fixed [JSExport] bridge under engines/dotnet/ is published by .NET SDK 10.0.302 with runtime 10.0.10 and its wasm-tools workload.

node scripts/dotnet-engine-pack.mjs --dotnet /absolute/path/to/dotnet
node scripts/install-dotnet-engine.mjs

The pack gate checks SDK/runtime/workload and bridge identities, removes compressed/debug outputs, records every _framework file, and includes the .NET MIT licence and third-party notices. The release browser gate executes the browser-WASM identity, Unicode-range and native-replacement smoke. There is no separate Scala pack.

Runtime boundaries

PCRE2

ABI version 3 has no retained native handles. Each call compiles, obtains capture names, matches/substitutes or traces and releases its code, match data and match context before returning. The caller supplies fixed-capacity result/name/output/event/mark buffers. Hard maxima are 1 MiB pattern, 16 MiB subject, 1,000 captures, 10,000 matches, 100,000 capture rows, 50,000 trace events, 10 MiB serialized trace data and 128 MiB PCRE2 heap limit; the workbench uses stricter pattern and replacement input limits where applicable.

PCRE2 always runs with UTF and UCP. Native UTF-8 byte offsets are retained in DTOs and converted only at valid code-point boundaries to half-open editor UTF-16 ranges. Lone UTF-16 surrogates are rejected before encoding.

Normal execution lives only in pcre2.worker. Instrumented PCRE2_AUTO_CALLOUT collection is a separate ABI operation loaded only by pcre2-trace.worker; normal match, replacement, tests, corpus and benchmarks never call it. Each supervisor terminates its worker on timeout, crash, cancellation or supersession and lazily creates a clean generation for the next request.

Python

The Python worker loads the self-hosted Pyodide pack and verifies Pyodide, CPython and re identities before accepting work. The application bridge is evaluated inside that interpreter and exposes bounded compile, match and native replacement operations. It reports CPython's Unicode code-point positions; the TypeScript adapter retains them and converts only valid boundaries to half-open editor UTF-16 ranges. Replacement output is byte-bounded in Python, encoded as canonical base64 in the JSON envelope and decoded only after its declared size, request cap and UTF-8 are validated. Runtime load has a separate startup deadline. Execution timeout, cancellation or crash terminates the worker and its complete Python/WebAssembly heap.

Java

The Java worker imports the self-hosted TeaVM ES module and checks the bridge ABI, declared engine identity and compiled self-test before accepting work. Every request compiles a Pattern, obtains bounded Matcher results or native replacement output, and returns native half-open UTF-16 ranges. No Java virtual machine, OpenJDK classes, remote service or ECMAScript regex fallback is involved. Runtime load has a separate startup deadline; termination discards the complete worker module state.

PHP

The PHP worker starts the same-origin Asyncify runtime, installs the fixed bridge.php in its private virtual filesystem and verifies PHP 8.5.8 plus PCRE2 10.44 before accepting work. User values cross only as JSON data. The bridge selects its own safe delimiter, permits only registered modifiers and calls native preg_match(). PREG_OFFSET_CAPTURE byte offsets are retained; byte matches that split an encoded character are rejected because they cannot map losslessly to a browser string. Bounded replacement mirrors PHP 8.5.8's preg_get_backref() syntax over those native records and clips every append before allocation crosses the requested byte cap. preg_replace() is used only by the fixed identity parity self-test. Replacement bytes cross the JSON envelope as size-predictable base64 and are decoded with a fatal UTF-8 check.

Perl

The supervised module worker owns a nested classic worker required by the legacy WebPerl loader. Terminating the supervisor also terminates that descendant and the complete Perl heap. The fixed bridge uses no re 'eval', rejects code assertions independently and exposes only literal text, $$, greedy $n and ${name} replacement expansion. It pre-tokenizes that grammar, streams small bounded UTF-8 pieces from native spans without constructing a complete expansion or JSON output, and preserves the unmatched tail when the match/capture-row bound stops replacement. Fixed named and numbered fixtures are checked against native Perl s/// during the identity self-test. Perl character offsets are returned as Unicode code points. This containment does not make WebPerl current software; the UI continues to report its legacy beta identity.

Ruby

The Ruby worker starts the minimal WASI runtime, evaluates only the fixed application bridge and verifies CRuby 4.0.0 identity. User values travel as bounded length-prefixed data in the worker-local memory filesystem; the bridge returns inert metadata JSON and never sends a user-controlled value through ruby.wasm's JavaScript-eval object-conversion path. CRuby compiles and matches. Native String#gsub/String#sub block iteration selects replacement matches; a fixed streaming implementation of CRuby 4.0.0's rb_reg_regsub grammar writes only the configured UTF-8 prefix to a separate private file. A load-time parity check compares the fixed expander with CRuby on bounded fixtures. Native character offsets are retained as Unicode code points. Worker termination discards the Ruby VM and both private files.

C++

The C++ worker loads fixed Embind exports whose generated module was built with dynamic execution disabled. Each request constructs std::wregex, iterates under central result limits and streams a fixed implementation of libc++'s match_results::format default grammar through the output byte cap. Load-time fixtures compare it with the native formatter. The adapter exposes 32-bit wchar_t positions as Unicode code points. The default grammar is C++'s modified ECMAScript grammar, not the browser's current ECMAScript grammar.

Go and Rust

The Go and Rust workers load their own fixed JSON bridges and verify the exact runtime/crate identities before work. Both return UTF-8 byte ranges and reject lone surrogates before encoding. Go uses standard-library regexp and ExpandString; Rust uses regex crate iteration and Captures::expand. Their fixed streaming expanders reproduce those native replacement grammars under the output byte cap and compare bounded fixtures with the native APIs at load. The adapters measure exact request JSON bytes, while matching native caps include the worst-case sixfold escaping of every accepted string field. Output bytes use fixed 4:3 base64 JSON envelopes instead of content-dependent JSON escaping. Neither runtime supports look-around or backreferences. Each module also enforces its own pattern, result and output bounds; Rust adds compiled-regex and DFA-size limits.

.NET

The .NET worker loads the browser-WASM runtime and calls only fixed [JSExport] methods with bounded JSON. It verifies runtime 10.0.10, System.Text.RegularExpressions, UTF-16 offsets and invariant culture. A nine-second native regex timeout complements the shorter supervisor deadline; terminating the worker discards the managed runtime. The adapter currently returns the final retained capture only, not .NET capture history. Replacement templates are parsed once by a fixed streaming implementation of .NET's numbered, named and special substitution tokens. A load-time oracle compares the fixed implementation with native Match.Result; bounded UTF-8 output is transported as canonical base64 rather than JSON-escaped user text.

Scala/JVM compatibility

Scala requests deliberately delegate to the same verified TeaVM adapter and module described under Java. Result metadata says “no Scala runtime”. The profile covers Java-compatible pattern/replacement semantics only and does not claim Scala Regex wrappers, extractors, compiler/runtime code or OpenJDK behavior.