Files
regex-tools/docs/ENGINE_BUILDS.md

459 lines
22 KiB
Markdown

# Engine builds
The production application uses the browser's native `RegExp` for ECMAScript
and bundles ten verified runtime packs for standalone PCRE2, PHP, Perl, Python,
Ruby, Java, C++, Go, Rust and .NET. The Scala/JVM compatibility profile shares
the Java pack and contains no Scala runtime. Normal application builds never
download or compile an engine: reviewed files under `public/engines/` are
copied into `dist/` and verified there.
## Rebuilding PCRE2
The reproducible build accepts only:
- PCRE2 tag `pcre2-10.47`, signed tag object
`cd007b4466798f66d479d1442a407099e7c40050`, peeled commit
`f454e231fe5006dd7ff8f4693fd2b8eb94333429` and tree
`81a83a3552bd68d0ea7b7004f8bb6e7892f583ba`;
- Emscripten 6.0.4, compiler revision
`fe5be6afdff43ad58860d821fcc8572a23f92d19`, from emsdk commit
`224ec5f9f2f72f09f9ce0e26d66bae7dbd8b692f`;
- CMake 4.3.4 and Ninja 1.13.2;
- 8-bit PCRE2 with Unicode enabled and JIT, threads and filesystem disabled.
The source and compiler checkouts must already exist locally and be completely
clean. No command clones, downloads or updates them:
```sh
npm run engines:pcre2:build -- \
--source-dir /absolute/path/to/pcre2 \
--emcc /absolute/path/to/emscripten/emcc
npm run engines:pcre2:verify
npm run engines:pcre2:install
```
Use `--force` only to replace the exact ignored `.engine-build/pcre2` output.
The explicit install command first re-verifies that pack and then atomically
replaces only `public/engines/pcre2`.
The builder verifies Git objects and critical source/compiler hashes,
sanitizes build flags, sets `SOURCE_DATE_EPOCH=1760997684`, builds serially and
emits exactly:
- `pcre2.mjs` and `pcre2.wasm`;
- deterministic `engine-metadata.json`;
- the exact upstream `LICENSE.txt`;
- `SHA256SUMS`.
The verifier checks the closed file set, metadata, source bridge hashes and all
checksums. It validates and instantiates WebAssembly, checks ABI/engine/config
identity, then executes native Unicode compile, bounded match and bounded
substitution smoke cases plus normal and cap-stopped automatic-callout traces.
The immutable signed-tag object is pinned. OpenPGP trust validation remains a
release-operator step because the offline builder does not install or trust a
key.
## Installing the Python pack
The Python pack is derived from the exact `pyodide@314.0.3` npm package locked
with integrity:
```text
sha512-sK40My6m8tmBUYtYH9au9rXUeh9x0wfahtHdOlGmJxZDsKBGKtP6KznyFB2+u/klbQTdDionR0uaVd176zVQzQ==
```
That Pyodide release is tag/commit
`ac57031be7564f864d061cb37c5c152e59f83ad4`. Direct executable identity
verification reports CPython 3.14.2; the shipped `pyodide-lock.json`
independently reports `info.python` as 3.14.0, and the verifier records rather
than conflates those two upstream identities. The pack copies only the reviewed
loader, WebAssembly runtime, Python standard library and lock metadata required
for a local CPython `re` worker. It adds deterministic engine metadata, a
closed-file-set checksum manifest and a complete `SOURCE-MANIFEST.json`.
The source manifest records immutable Git commit/tree identities or primary
release-archive SHA-256 values for Pyodide, its `pyodide-build` submodule, its
vendored Error Stack Parser 2.1.4 and StackFrame 1.3.4 ports, CPython, libffi,
Hiwire, XZ/liblzma, Zstandard, SQLite, the Emscripten bzip2 and zlib ports,
Emscripten itself and HACL* preferred source. It hashes both vendored
JavaScript ports, all nine CPython patches, all five Emscripten patches, and
the Pyodide Makefiles that select those inputs. That is the
preferred-form/patch route for the upstream MPL source; the exact npm package
remains the binary-distribution reference, not a claim that arbitrary future
host SDK packaging will reproduce identical bytes.
The linked-component inventory is derived from the exact Pyodide recipe and
checked against executable identities where the libraries expose one. Besides
Pyodide and CPython, the base module contains CPython-vendored Expat 2.7.3,
libmpdec 2.5.1 and HACL*, plus libffi, Hiwire 1.0.1, liblzma 5.2.2,
Zstandard 1.5.7, SQLite 3.39.0, bzip2 1.0.6 and zlib 1.3.1. The conservative
Emscripten 5.0.3 runtime notice closure covers musl, compiler-rt, libc++,
libc++abi, libunwind, dlmalloc and the MiniLZ4 source selected by the pinned
link flags. The upstream stripped module contains no link map, so retaining the
complete potentially selected Emscripten runtime notice set is intentional.
The install operation reads from the already installed pinned npm package. It
does not contact a CDN or package index. It verifies source package identity,
file hashes, runtime metadata, every independently fetched legal-source hash,
the generated source manifest and licence inputs before atomically replacing
only `public/engines/python/`. The distribution verifier checks the same closed
set under `dist/`, validates WebAssembly and native component identities and
confirms that all runtime references remain relative and self-hosted.
```sh
node scripts/python-engine-pack.mjs \
node_modules/pyodide \
.engine-build/python
node scripts/install-python-engine.mjs .engine-build/python
```
The installed closed set is:
- `pyodide.mjs`, `pyodide.asm.mjs` and `pyodide.asm.wasm`;
- `python_stdlib.zip` and `pyodide-lock.json`;
- `engine-metadata.json`, `SOURCE-MANIFEST.json` and `SHA256SUMS`; and
- 22 exact `LICENSE.*`/`NOTICE.*` files enumerated and source-anchored by that
manifest.
`pyodide-lock.json` is the upstream catalogue of optional packages. Regex
Tools does not ship the catalogue's wheels; `externalPackagesIncluded: false`
and the closed pack file set make that boundary machine-verifiable.
## Building the Java pack
The Java bridge under `engines/java/` wraps TeaVM 0.15.0
`java.util.regex.Pattern` and `Matcher`. TeaVM tag/commit
`ee91b03e616c4b45401cd11fb0cd7eb0daf6649b` is pinned through the Maven build.
The build produces a minified ES2015 module without source maps, source files,
remote imports or an OpenJDK runtime.
The pack gate verifies the Maven coordinates, bridge source identity, TeaVM
engine identity and self-test, generated module, deterministic metadata,
checksums, Apache-2.0 licence and TeaVM NOTICE. Only the verified pack is
installed under `public/engines/java/`.
```sh
node scripts/java-engine-pack.mjs
node scripts/install-java-engine.mjs
```
The pack command performs two clean offline Maven builds and requires identical
output. The installed closed set is `java-regex.mjs`,
`engine-metadata.json`, `SHA256SUMS`, `LICENSE.txt` and `NOTICE.txt`.
TeaVM supplies its own Apache Harmony-derived class-library implementation.
The resulting engine is therefore identified as
`TeaVM java.util.regex 0.15.0`, not OpenJDK. TeaVM 0.15.0 does not implement
OpenJDK's `Pattern.UNICODE_CHARACTER_CLASS`; the bridge treats the application
`U` flag as a compatibility request, implies Unicode case folding and reports
that predefined-character classes and word boundaries retain TeaVM behaviour.
## Building the PHP pack
The PHP bridge under `engines/php/` executes actual PHP 8.5.8 `preg_*`.
`@php-wasm/web-8-5` and `@php-wasm/universal` 3.1.46 plus their pinned support
packages supply the Asyncify WebAssembly runtime and host API. Direct identity
verification reports PCRE2 10.44; this pack is not the standalone PCRE2 10.47
build.
```sh
node scripts/php-engine-pack.mjs
node scripts/install-php-engine.mjs
```
The builder copies only the pinned npm runtime files, changes the
bundler-specific WebAssembly import into a same-directory URL, records the
original and transformed hashes, and runs real identity, match, bounded
replacement and native-component positive/negative audits before installation.
`native-components.json` binds the exact WebAssembly hash to WordPress
Playground `v3.1.46` commit
`581c7c172428159eb4e6c5309054a568cd39a97a`, PHP tag `php-8.5.8` commit
`26b97507444c4fbda072f57dda1820f7b7d5e467`, Emscripten 4.0.19 commit
`08e2de1031913e4ba7963b1c56f35f036a7d4d56` and all 32 linked native/source
surfaces. It carries exact archive hashes or Git identities where upstream
recorded them and honestly bounds the one unpinned Oniguruma clone using the
committed archive/header identities and upstream source interval.
The pack elects PHP License version 4 (`BSD-3-Clause`) for PHP 8.5.8 through
PHP's later-version option and includes the complete PCRE2 and PHP-vendored,
external-library, patent and Emscripten/LLVM/musl legal set. This includes the
exact Zend Engine licence, official php-src binary redistribution notice, CLI
HTTP parser MIT terms, and the public-domain, CC0 and FNV permission notices for
linked hash implementations. The WordPress Playground `@php-wasm` packages
retain GPL-2.0-or-later; JavaScript support-package licences remain separate.
All 42 legal assets and the native inventory are independently SHA-256-pinned
by the builder, then covered again by the closed file set, `SHA256SUMS`, release
requirements and metadata.
## Building the legacy Perl pack
The Perl bridge under `engines/perl/` uses the unmodified WebPerl 0.09-beta
prebuilt archive, SHA-256
`5f441249217e90ab378c666f473d4206ab4f44907f6bb0aa8d70834bc38c40dc`.
That 2019 beta contains Perl 5.28.1 and Emscripten 1.38.28 output. It is
intentionally a legacy compatibility target, not a current Perl distribution.
The builder accepts only a local copy of that exact official archive; it never
downloads. An already extracted release root can be supplied with `--prebuilt`
instead of `--archive`.
```sh
node scripts/perl-engine-pack.mjs \
--archive /absolute/path/webperl_prebuilt_v0.09-beta.zip
node scripts/perl-engine-pack.mjs --verify .engine-build/perl
node scripts/install-perl-engine.mjs .engine-build/perl
```
The installed pack retains upstream `emperl.js`, `emperl.wasm` and
`emperl.data` byte-for-byte. Application-owned fixed worker and Perl bridge
files carry values through an exact-key, byte-bounded JSON request path; they do
not splice patterns or replacements into source. The metadata response remains
bounded JSON, while successful replacement UTF-8 bytes stream into a separate
fixed MEMFS binary file and receive exact worker-side byte, size and encoding
validation. Exact WebPerl/Perl, Emscripten and bundled-CPAN notices travel with
the pack.
The legacy generated loader contains one `eval(code)` site in an optional
Perl-to-JavaScript interoperability import. The application bridge never loads
that facility. Its browser CSP smoke passes with `'wasm-unsafe-eval'` and
without broad `'unsafe-eval'`; that dormant upstream site remains documented
rather than removed from the checksummed asset.
## Building the Ruby pack
The Ruby pack copies the minimal runtime from pinned
`@ruby/4.0-wasm-wasi@2.9.3-2.9.4` and uses
`@ruby/wasm-wasi@2.9.3-2.9.4` as its host. It verifies the CRuby 4.0.0
`wasm32-wasi` identity and actual `Regexp`/replacement behavior before
installation.
```sh
node scripts/ruby-engine-pack.mjs
node scripts/install-ruby-engine.mjs
```
The closed set contains the runtime WebAssembly file, deterministic metadata
and checksums, ruby.wasm licence/complete NOTICE and the exact browser WASI
shim and `tslib` licence texts needed by the host bundle. The production worker
passes each request through a bounded length-prefixed file in its private WASI
memory filesystem. It receives bounded inert JSON metadata and reads bounded
UTF-8 replacement output from a separate private file, rather than amplifying
that output through JSON or using `RubyVM.wrap`/`toJS` and their
JavaScript-eval conversion path. Ruby cold startup passes the release's
strict-CSP browser gate.
## Building the C++ pack
The C++ bridge under `engines/cpp/` is compiled with Emscripten 6.0.4 at
compiler revision `fe5be6afdff43ad58860d821fcc8572a23f92d19`. It uses libc++
`std::wregex`, 32-bit `wchar_t`, fixed structured Embind entry points and
`-sDYNAMIC_EXECUTION=0`.
```sh
node scripts/cpp-engine-pack.mjs --empp /absolute/path/to/em++
node scripts/install-cpp-engine.mjs
```
The verifier checks exact Emscripten, emsdk and LLVM identities, bridge hashes,
WebAssembly/module structure, absence of dynamic-execution constructs in the
generated module, runtime identity, code-point ranges and bounded replacement
parity. The build also enables `wasm-ld --trace` and accepts only the reviewed
linked archive set: Emscripten runtime support, musl `libc`, emmalloc,
compiler-rt builtins, libc++ and libc++abi.
`SOURCE-MANIFEST.json` ties the exact module hash to Emscripten tag/commit
`6.0.4`/`fe5be6afdff43ad58860d821fcc8572a23f92d19`, emsdk commit
`224ec5f9f2f72f09f9ce0e26d66bae7dbd8b692f`, the official release commit and
LLVM compiler commit. It records preferred-source paths and independently
pinned legal files for musl 1.2.6, emmalloc, compiler-rt 22.1.8, libc++ 21.1.8
and libc++abi 21.1.8. The current JavaScript-exception link selects no
`libunwind.a` member; libunwind 22.1.8 and its legal text remain as an explicit
audited exclusion. Any added, removed or newly selected archive fails the pack
build.
## Building the Go pack
Go 1.26.5 standard-library `regexp` is built with the official linux/amd64
archive, SHA-256
`5c2c3b16caefa1d968a94c1daca04a7ca301a496d9b086e17ad77bb81393f053`,
for `GOOS=js GOARCH=wasm`.
```sh
node scripts/build-go-engine.mjs /absolute/path/to/go
node scripts/install-go-engine.mjs
node scripts/verify-go-engine.mjs
```
The pack pins the source commit, official archive, `wasm_exec.mjs`, bridge
source, build flags and Go licence. Verification executes real Unicode
matching and `regexp.ExpandString` cases.
## Building the Rust pack
The Rust bridge pins `regex` crate 1.13.1 and its complete Cargo graph, rustc
1.97.1, the matching `wasm32-unknown-unknown` standard library and wasm-bindgen
0.2.126.
```sh
node scripts/build-rust-engine.mjs /absolute/path/to/rust-toolchain
node scripts/install-rust-engine.mjs
node scripts/verify-rust-engine.mjs
```
The pack records the distribution manifest/archive hashes, crate checksums,
compiler and LLVM identities, bridge sources, exact bindings and complete
Rust/crate licence material. Runtime verification covers identity, Unicode
byte ranges, limits and `Captures::expand`.
## Building the .NET pack
The fixed `[JSExport]` bridge under `engines/dotnet/` is published by .NET SDK
10.0.302 with runtime 10.0.10 and its `wasm-tools` workload.
```sh
node scripts/dotnet-engine-pack.mjs --dotnet /absolute/path/to/dotnet
node scripts/install-dotnet-engine.mjs
```
The pack gate checks SDK/runtime/workload and bridge identities, removes
compressed/debug outputs, records every `_framework` file, and includes the
.NET MIT licence and third-party notices. The release browser gate executes
the browser-WASM identity, Unicode-range and native-replacement smoke. There
is no separate Scala pack.
## Runtime boundaries
### PCRE2
ABI version 3 has no retained native handles. Each call compiles, obtains
capture names, matches/substitutes or traces and releases its code, match data
and match context before returning. The caller supplies fixed-capacity
result/name/output/event/mark buffers. Hard maxima are 1 MiB pattern, 16 MiB
subject, 1,000 captures, 10,000 matches, 100,000 capture rows, 50,000 trace
events, 10 MiB serialized trace data and 128 MiB PCRE2 heap limit; the
workbench uses stricter pattern and replacement input limits where applicable.
PCRE2 always runs with UTF and UCP. Native UTF-8 byte offsets are retained in
DTOs and converted only at valid code-point boundaries to half-open editor
UTF-16 ranges. Lone UTF-16 surrogates are rejected before encoding.
Normal execution lives only in `pcre2.worker`. Instrumented
`PCRE2_AUTO_CALLOUT` collection is a separate ABI operation loaded only by
`pcre2-trace.worker`; normal match, replacement, tests, corpus and benchmarks
never call it. Each supervisor terminates its worker on timeout, crash,
cancellation or supersession and lazily creates a clean generation for the
next request.
### Python
The Python worker loads the self-hosted Pyodide pack and verifies Pyodide,
CPython and `re` identities before accepting work. The application bridge is
evaluated inside that interpreter and exposes bounded compile, match and native
replacement operations. It reports CPython's Unicode code-point positions;
the TypeScript adapter retains them and converts only valid boundaries to
half-open editor UTF-16 ranges. Replacement output is byte-bounded in Python,
encoded as canonical base64 in the JSON envelope and decoded only after its
declared size, request cap and UTF-8 are validated. Runtime load has a separate
startup deadline. Execution timeout, cancellation or crash terminates the
worker and its complete Python/WebAssembly heap.
### Java
The Java worker imports the self-hosted TeaVM ES module and checks the bridge
ABI, declared engine identity and compiled self-test before accepting work.
Every request compiles a `Pattern`, obtains bounded `Matcher` results or native
replacement output, and returns native half-open UTF-16 ranges. No Java virtual
machine, OpenJDK classes, remote service or ECMAScript regex fallback is
involved. Runtime load has a separate startup deadline; termination discards
the complete worker module state.
### PHP
The PHP worker starts the same-origin Asyncify runtime, installs the fixed
`bridge.php` in its private virtual filesystem and verifies PHP 8.5.8 plus
PCRE2 10.44 before accepting work. User values cross only as JSON data. The
bridge selects its own safe delimiter, permits only registered modifiers and
calls native `preg_match()`. `PREG_OFFSET_CAPTURE` byte offsets are retained;
byte matches that split an encoded character are rejected because they cannot
map losslessly to a browser string. Bounded replacement mirrors PHP 8.5.8's
`preg_get_backref()` syntax over those native records and clips every append
before allocation crosses the requested byte cap. `preg_replace()` is used
only by the fixed identity parity self-test. Replacement bytes cross the JSON
envelope as size-predictable base64 and are decoded with a fatal UTF-8 check.
### Perl
The supervised module worker owns a nested classic worker required by the
legacy WebPerl loader. Terminating the supervisor also terminates that
descendant and the complete Perl heap. The fixed bridge uses `no re 'eval'`,
rejects code assertions independently and exposes only literal text, `$$`,
greedy `$n` and `${name}` replacement expansion. It pre-tokenizes that grammar,
streams small bounded UTF-8 pieces from native spans without constructing a
complete expansion or JSON output, and preserves the unmatched tail when the
match/capture-row bound stops replacement. Fixed named and numbered fixtures
are checked against native Perl `s///` during the identity self-test. Perl
character offsets are returned as Unicode code points. This containment does
not make WebPerl current software; the UI continues to report its legacy beta
identity.
### Ruby
The Ruby worker starts the minimal WASI runtime, evaluates only the fixed
application bridge and verifies CRuby 4.0.0 identity. User values travel as
bounded length-prefixed data in the worker-local memory filesystem; the bridge
returns inert metadata JSON and never sends a user-controlled value through
ruby.wasm's JavaScript-eval object-conversion path. CRuby compiles and matches.
Native `String#gsub`/`String#sub` block iteration selects replacement matches;
a fixed streaming implementation of CRuby 4.0.0's `rb_reg_regsub` grammar
writes only the configured UTF-8 prefix to a separate private file. A
load-time parity check compares the fixed expander with CRuby on bounded
fixtures. Native character offsets are retained as Unicode code points.
Worker termination discards the Ruby VM and both private files.
### C++
The C++ worker loads fixed Embind exports whose generated module was built with
dynamic execution disabled. Each request constructs `std::wregex`, iterates
under central result limits and streams a fixed implementation of libc++'s
`match_results::format` default grammar through the output byte cap. Load-time
fixtures compare it with the native formatter. The adapter exposes 32-bit
`wchar_t` positions as Unicode code points. The default grammar is C++'s
modified ECMAScript grammar, not the browser's current ECMAScript grammar.
### Go and Rust
The Go and Rust workers load their own fixed JSON bridges and verify the exact
runtime/crate identities before work. Both return UTF-8 byte ranges and reject
lone surrogates before encoding. Go uses standard-library `regexp` and
`ExpandString`; Rust uses `regex` crate iteration and `Captures::expand`.
Their fixed streaming expanders reproduce those native replacement grammars
under the output byte cap and compare bounded fixtures with the native APIs at
load. The adapters measure exact request JSON bytes, while matching native caps
include the worst-case sixfold escaping of every accepted string field. Output
bytes use fixed 4:3 base64 JSON envelopes instead of content-dependent JSON
escaping. Neither runtime supports look-around or backreferences. Each module
also enforces its own pattern, result and output bounds; Rust adds
compiled-regex and DFA-size limits.
### .NET
The .NET worker loads the browser-WASM runtime and calls only fixed
`[JSExport]` methods with bounded JSON. It verifies runtime 10.0.10,
`System.Text.RegularExpressions`, UTF-16 offsets and invariant culture. A
nine-second native regex timeout complements the shorter supervisor deadline;
terminating the worker discards the managed runtime. The adapter currently
returns the final retained capture only, not .NET capture history. Replacement
templates are parsed once by a fixed streaming implementation of .NET's
numbered, named and special substitution tokens. A load-time oracle compares
the fixed implementation with native `Match.Result`; bounded UTF-8 output is
transported as canonical base64 rather than JSON-escaped user text.
### Scala/JVM compatibility
Scala requests deliberately delegate to the same verified TeaVM adapter and
module described under Java. Result metadata says “no Scala runtime”. The
profile covers Java-compatible pattern/replacement semantics only and does not
claim Scala `Regex` wrappers, extractors, compiler/runtime code or OpenJDK
behavior.