lotobo/regex-tools
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 6 Wiki Activity

feat: add multi-engine regex flavour support

Browse Source
This commit is contained in:
Albrecht Degering
2026-07-27 11:43:51 +02:00
parent 7079cde15f
commit 7f3a91ad37
340 changed files with 643286 additions and 483 deletions
Download Patch File Download Diff File Expand all files Collapse all files

317
docs/ENGINE_BUILDS.md
View File

@@ -1,9 +1,11 @@
# Engine builds
The production application uses the browser's native `RegExp` for ECMAScript
and bundles verified PCRE2, Python and Java runtime packs. Normal application
builds never download or compile an engine: reviewed files under
`public/engines/` are copied into `dist/` and verified there.
and bundles ten verified runtime packs for standalone PCRE2, PHP, Perl, Python,
Ruby, Java, C++, Go, Rust and .NET. The Scala/JVM compatibility profile shares
the Java pack and contains no Scala runtime. Normal application builds never
download or compile an engine: reviewed files under `public/engines/` are
copied into `dist/` and verified there.
## Rebuilding PCRE2
@@ -68,14 +70,36 @@ independently reports `info.python` as 3.14.0, and the verifier records rather
than conflates those two upstream identities. The pack copies only the reviewed
loader, WebAssembly runtime, Python standard library and lock metadata required
for a local CPython `re` worker. It adds deterministic engine metadata, a
closed-file-set checksum manifest and exact Pyodide/CPython licence material.
closed-file-set checksum manifest and a complete `SOURCE-MANIFEST.json`.
The source manifest records immutable Git commit/tree identities or primary
release-archive SHA-256 values for Pyodide, its `pyodide-build` submodule, its
vendored Error Stack Parser 2.1.4 and StackFrame 1.3.4 ports, CPython, libffi,
Hiwire, XZ/liblzma, Zstandard, SQLite, the Emscripten bzip2 and zlib ports,
Emscripten itself and HACL* preferred source. It hashes both vendored
JavaScript ports, all nine CPython patches, all five Emscripten patches, and
the Pyodide Makefiles that select those inputs. That is the
preferred-form/patch route for the upstream MPL source; the exact npm package
remains the binary-distribution reference, not a claim that arbitrary future
host SDK packaging will reproduce identical bytes.
The linked-component inventory is derived from the exact Pyodide recipe and
checked against executable identities where the libraries expose one. Besides
Pyodide and CPython, the base module contains CPython-vendored Expat 2.7.3,
libmpdec 2.5.1 and HACL*, plus libffi, Hiwire 1.0.1, liblzma 5.2.2,
Zstandard 1.5.7, SQLite 3.39.0, bzip2 1.0.6 and zlib 1.3.1. The conservative
Emscripten 5.0.3 runtime notice closure covers musl, compiler-rt, libc++,
libc++abi, libunwind, dlmalloc and the MiniLZ4 source selected by the pinned
link flags. The upstream stripped module contains no link map, so retaining the
complete potentially selected Emscripten runtime notice set is intentional.
The install operation reads from the already installed pinned npm package. It
does not contact a CDN or package index. It verifies source package identity,
file hashes, runtime metadata and licence inputs before atomically replacing
file hashes, runtime metadata, every independently fetched legal-source hash,
the generated source manifest and licence inputs before atomically replacing
only `public/engines/python/`. The distribution verifier checks the same closed
set under `dist/`, validates WebAssembly magic and confirms that all runtime
references remain relative and self-hosted.
set under `dist/`, validates WebAssembly and native component identities and
confirms that all runtime references remain relative and self-hosted.
```sh
node scripts/python-engine-pack.mjs \
@@ -88,8 +112,13 @@ The installed closed set is:
- `pyodide.mjs`, `pyodide.asm.mjs` and `pyodide.asm.wasm`;
- `python_stdlib.zip` and `pyodide-lock.json`;
- `engine-metadata.json` and `SHA256SUMS`; and
- `LICENSE.pyodide.txt` and `LICENSE.cpython.txt`.
- `engine-metadata.json`, `SOURCE-MANIFEST.json` and `SHA256SUMS`; and
- 22 exact `LICENSE.*`/`NOTICE.*` files enumerated and source-anchored by that
manifest.
`pyodide-lock.json` is the upstream catalogue of optional packages. Regex
Tools does not ship the catalogue's wheels; `externalPackagesIncluded: false`
and the closed pack file set make that boundary machine-verifiable.
## Building the Java pack
@@ -120,6 +149,179 @@ OpenJDK's `Pattern.UNICODE_CHARACTER_CLASS`; the bridge treats the application
`U` flag as a compatibility request, implies Unicode case folding and reports
that predefined-character classes and word boundaries retain TeaVM behaviour.
## Building the PHP pack
The PHP bridge under `engines/php/` executes actual PHP 8.5.8 `preg_*`.
`@php-wasm/web-8-5` and `@php-wasm/universal` 3.1.46 plus their pinned support
packages supply the Asyncify WebAssembly runtime and host API. Direct identity
verification reports PCRE2 10.44; this pack is not the standalone PCRE2 10.47
build.
```sh
node scripts/php-engine-pack.mjs
node scripts/install-php-engine.mjs
```
The builder copies only the pinned npm runtime files, changes the
bundler-specific WebAssembly import into a same-directory URL, records the
original and transformed hashes, and runs real identity, match, bounded
replacement and native-component positive/negative audits before installation.
`native-components.json` binds the exact WebAssembly hash to WordPress
Playground `v3.1.46` commit
`581c7c172428159eb4e6c5309054a568cd39a97a`, PHP tag `php-8.5.8` commit
`26b97507444c4fbda072f57dda1820f7b7d5e467`, Emscripten 4.0.19 commit
`08e2de1031913e4ba7963b1c56f35f036a7d4d56` and all 32 linked native/source
surfaces. It carries exact archive hashes or Git identities where upstream
recorded them and honestly bounds the one unpinned Oniguruma clone using the
committed archive/header identities and upstream source interval.
The pack elects PHP License version 4 (`BSD-3-Clause`) for PHP 8.5.8 through
PHP's later-version option and includes the complete PCRE2 and PHP-vendored,
external-library, patent and Emscripten/LLVM/musl legal set. This includes the
exact Zend Engine licence, official php-src binary redistribution notice, CLI
HTTP parser MIT terms, and the public-domain, CC0 and FNV permission notices for
linked hash implementations. The WordPress Playground `@php-wasm` packages
retain GPL-2.0-or-later; JavaScript support-package licences remain separate.
All 42 legal assets and the native inventory are independently SHA-256-pinned
by the builder, then covered again by the closed file set, `SHA256SUMS`, release
requirements and metadata.
## Building the legacy Perl pack
The Perl bridge under `engines/perl/` uses the unmodified WebPerl 0.09-beta
prebuilt archive, SHA-256
`5f441249217e90ab378c666f473d4206ab4f44907f6bb0aa8d70834bc38c40dc`.
That 2019 beta contains Perl 5.28.1 and Emscripten 1.38.28 output. It is
intentionally a legacy compatibility target, not a current Perl distribution.
The builder accepts only a local copy of that exact official archive; it never
downloads. An already extracted release root can be supplied with `--prebuilt`
instead of `--archive`.
```sh
node scripts/perl-engine-pack.mjs \
--archive /absolute/path/webperl_prebuilt_v0.09-beta.zip
node scripts/perl-engine-pack.mjs --verify .engine-build/perl
node scripts/install-perl-engine.mjs .engine-build/perl
```
The installed pack retains upstream `emperl.js`, `emperl.wasm` and
`emperl.data` byte-for-byte. Application-owned fixed worker and Perl bridge
files carry values through an exact-key, byte-bounded JSON request path; they do
not splice patterns or replacements into source. The metadata response remains
bounded JSON, while successful replacement UTF-8 bytes stream into a separate
fixed MEMFS binary file and receive exact worker-side byte, size and encoding
validation. Exact WebPerl/Perl, Emscripten and bundled-CPAN notices travel with
the pack.
The legacy generated loader contains one `eval(code)` site in an optional
Perl-to-JavaScript interoperability import. The application bridge never loads
that facility. Its browser CSP smoke passes with `'wasm-unsafe-eval'` and
without broad `'unsafe-eval'`; that dormant upstream site remains documented
rather than removed from the checksummed asset.
## Building the Ruby pack
The Ruby pack copies the minimal runtime from pinned
`@ruby/4.0-wasm-wasi@2.9.3-2.9.4` and uses
`@ruby/wasm-wasi@2.9.3-2.9.4` as its host. It verifies the CRuby 4.0.0
`wasm32-wasi` identity and actual `Regexp`/replacement behavior before
installation.
```sh
node scripts/ruby-engine-pack.mjs
node scripts/install-ruby-engine.mjs
```
The closed set contains the runtime WebAssembly file, deterministic metadata
and checksums, ruby.wasm licence/complete NOTICE and the exact browser WASI
shim and `tslib` licence texts needed by the host bundle. The production worker
passes each request through a bounded length-prefixed file in its private WASI
memory filesystem. It receives bounded inert JSON metadata and reads bounded
UTF-8 replacement output from a separate private file, rather than amplifying
that output through JSON or using `RubyVM.wrap`/`toJS` and their
JavaScript-eval conversion path. Ruby cold startup passes the release's
strict-CSP browser gate.
## Building the C++ pack
The C++ bridge under `engines/cpp/` is compiled with Emscripten 6.0.4 at
compiler revision `fe5be6afdff43ad58860d821fcc8572a23f92d19`. It uses libc++
`std::wregex`, 32-bit `wchar_t`, fixed structured Embind entry points and
`-sDYNAMIC_EXECUTION=0`.
```sh
node scripts/cpp-engine-pack.mjs --empp /absolute/path/to/em++
node scripts/install-cpp-engine.mjs
```
The verifier checks exact Emscripten, emsdk and LLVM identities, bridge hashes,
WebAssembly/module structure, absence of dynamic-execution constructs in the
generated module, runtime identity, code-point ranges and bounded replacement
parity. The build also enables `wasm-ld --trace` and accepts only the reviewed
linked archive set: Emscripten runtime support, musl `libc`, emmalloc,
compiler-rt builtins, libc++ and libc++abi.
`SOURCE-MANIFEST.json` ties the exact module hash to Emscripten tag/commit
`6.0.4`/`fe5be6afdff43ad58860d821fcc8572a23f92d19`, emsdk commit
`224ec5f9f2f72f09f9ce0e26d66bae7dbd8b692f`, the official release commit and
LLVM compiler commit. It records preferred-source paths and independently
pinned legal files for musl 1.2.6, emmalloc, compiler-rt 22.1.8, libc++ 21.1.8
and libc++abi 21.1.8. The current JavaScript-exception link selects no
`libunwind.a` member; libunwind 22.1.8 and its legal text remain as an explicit
audited exclusion. Any added, removed or newly selected archive fails the pack
build.
## Building the Go pack
Go 1.26.5 standard-library `regexp` is built with the official linux/amd64
archive, SHA-256
`5c2c3b16caefa1d968a94c1daca04a7ca301a496d9b086e17ad77bb81393f053`,
for `GOOS=js GOARCH=wasm`.
```sh
node scripts/build-go-engine.mjs /absolute/path/to/go
node scripts/install-go-engine.mjs
node scripts/verify-go-engine.mjs
```
The pack pins the source commit, official archive, `wasm_exec.mjs`, bridge
source, build flags and Go licence. Verification executes real Unicode
matching and `regexp.ExpandString` cases.
## Building the Rust pack
The Rust bridge pins `regex` crate 1.13.1 and its complete Cargo graph, rustc
1.97.1, the matching `wasm32-unknown-unknown` standard library and wasm-bindgen
0.2.126.
```sh
node scripts/build-rust-engine.mjs /absolute/path/to/rust-toolchain
node scripts/install-rust-engine.mjs
node scripts/verify-rust-engine.mjs
```
The pack records the distribution manifest/archive hashes, crate checksums,
compiler and LLVM identities, bridge sources, exact bindings and complete
Rust/crate licence material. Runtime verification covers identity, Unicode
byte ranges, limits and `Captures::expand`.
## Building the .NET pack
The fixed `[JSExport]` bridge under `engines/dotnet/` is published by .NET SDK
10.0.302 with runtime 10.0.10 and its `wasm-tools` workload.
```sh
node scripts/dotnet-engine-pack.mjs --dotnet /absolute/path/to/dotnet
node scripts/install-dotnet-engine.mjs
```
The pack gate checks SDK/runtime/workload and bridge identities, removes
compressed/debug outputs, records every `_framework` file, and includes the
.NET MIT licence and third-party notices. The release browser gate executes
the browser-WASM identity, Unicode-range and native-replacement smoke. There
is no separate Scala pack.
## Runtime boundaries
### PCRE2
@@ -150,9 +352,11 @@ CPython and `re` identities before accepting work. The application bridge is
evaluated inside that interpreter and exposes bounded compile, match and native
replacement operations. It reports CPython's Unicode code-point positions;
the TypeScript adapter retains them and converts only valid boundaries to
half-open editor UTF-16 ranges. Runtime load has a separate startup deadline.
Execution timeout, cancellation or crash terminates the worker and its complete
Python/WebAssembly heap.
half-open editor UTF-16 ranges. Replacement output is byte-bounded in Python,
encoded as canonical base64 in the JSON envelope and decoded only after its
declared size, request cap and UTF-8 are validated. Runtime load has a separate
startup deadline. Execution timeout, cancellation or crash terminates the
worker and its complete Python/WebAssembly heap.
### Java
@@ -163,3 +367,92 @@ replacement output, and returns native half-open UTF-16 ranges. No Java virtual
machine, OpenJDK classes, remote service or ECMAScript regex fallback is
involved. Runtime load has a separate startup deadline; termination discards
the complete worker module state.
### PHP
The PHP worker starts the same-origin Asyncify runtime, installs the fixed
`bridge.php` in its private virtual filesystem and verifies PHP 8.5.8 plus
PCRE2 10.44 before accepting work. User values cross only as JSON data. The
bridge selects its own safe delimiter, permits only registered modifiers and
calls native `preg_match()`. `PREG_OFFSET_CAPTURE` byte offsets are retained;
byte matches that split an encoded character are rejected because they cannot
map losslessly to a browser string. Bounded replacement mirrors PHP 8.5.8's
`preg_get_backref()` syntax over those native records and clips every append
before allocation crosses the requested byte cap. `preg_replace()` is used
only by the fixed identity parity self-test. Replacement bytes cross the JSON
envelope as size-predictable base64 and are decoded with a fatal UTF-8 check.
### Perl
The supervised module worker owns a nested classic worker required by the
legacy WebPerl loader. Terminating the supervisor also terminates that
descendant and the complete Perl heap. The fixed bridge uses `no re 'eval'`,
rejects code assertions independently and exposes only literal text, `$$`,
greedy `$n` and `${name}` replacement expansion. It pre-tokenizes that grammar,
streams small bounded UTF-8 pieces from native spans without constructing a
complete expansion or JSON output, and preserves the unmatched tail when the
match/capture-row bound stops replacement. Fixed named and numbered fixtures
are checked against native Perl `s///` during the identity self-test. Perl
character offsets are returned as Unicode code points. This containment does
not make WebPerl current software; the UI continues to report its legacy beta
identity.
### Ruby
The Ruby worker starts the minimal WASI runtime, evaluates only the fixed
application bridge and verifies CRuby 4.0.0 identity. User values travel as
bounded length-prefixed data in the worker-local memory filesystem; the bridge
returns inert metadata JSON and never sends a user-controlled value through
ruby.wasm's JavaScript-eval object-conversion path. CRuby compiles and matches.
Native `String#gsub`/`String#sub` block iteration selects replacement matches;
a fixed streaming implementation of CRuby 4.0.0's `rb_reg_regsub` grammar
writes only the configured UTF-8 prefix to a separate private file. A
load-time parity check compares the fixed expander with CRuby on bounded
fixtures. Native character offsets are retained as Unicode code points.
Worker termination discards the Ruby VM and both private files.
### C++
The C++ worker loads fixed Embind exports whose generated module was built with
dynamic execution disabled. Each request constructs `std::wregex`, iterates
under central result limits and streams a fixed implementation of libc++'s
`match_results::format` default grammar through the output byte cap. Load-time
fixtures compare it with the native formatter. The adapter exposes 32-bit
`wchar_t` positions as Unicode code points. The default grammar is C++'s
modified ECMAScript grammar, not the browser's current ECMAScript grammar.
### Go and Rust
The Go and Rust workers load their own fixed JSON bridges and verify the exact
runtime/crate identities before work. Both return UTF-8 byte ranges and reject
lone surrogates before encoding. Go uses standard-library `regexp` and
`ExpandString`; Rust uses `regex` crate iteration and `Captures::expand`.
Their fixed streaming expanders reproduce those native replacement grammars
under the output byte cap and compare bounded fixtures with the native APIs at
load. The adapters measure exact request JSON bytes, while matching native caps
include the worst-case sixfold escaping of every accepted string field. Output
bytes use fixed 4:3 base64 JSON envelopes instead of content-dependent JSON
escaping. Neither runtime supports look-around or backreferences. Each module
also enforces its own pattern, result and output bounds; Rust adds
compiled-regex and DFA-size limits.
### .NET
The .NET worker loads the browser-WASM runtime and calls only fixed
`[JSExport]` methods with bounded JSON. It verifies runtime 10.0.10,
`System.Text.RegularExpressions`, UTF-16 offsets and invariant culture. A
nine-second native regex timeout complements the shorter supervisor deadline;
terminating the worker discards the managed runtime. The adapter currently
returns the final retained capture only, not .NET capture history. Replacement
templates are parsed once by a fixed streaming implementation of .NET's
numbered, named and special substitution tokens. A load-time oracle compares
the fixed implementation with native `Match.Result`; bounded UTF-8 output is
transported as canonical base64 rather than JSON-escaped user text.
### Scala/JVM compatibility
Scala requests deliberately delegate to the same verified TeaVM adapter and
module described under Java. Result metadata says “no Scala runtime”. The
profile covers Java-compatible pattern/replacement semantics only and does not
claim Scala `Regex` wrappers, extractors, compiler/runtime code or OpenJDK
behavior.