lotobo/regex-tools
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 6 Wiki Activity

feat: add multi-engine regex flavour support

Browse Source
This commit is contained in:
Albrecht Degering
2026-07-27 11:43:51 +02:00
parent 7079cde15f
commit 7f3a91ad37
340 changed files with 643286 additions and 483 deletions
Download Patch File Download Diff File Expand all files Collapse all files

53
docs/ARCHITECTURE.md
View File

@@ -9,13 +9,22 @@ React workbench
│ ├─ Regexpp ECMAScript provider → normalized AST
│ ├─ PCRE2 lexical provider → partial normalized structure
│ ├─ Python lexical provider → partial normalized structure
│ └─ Java lexical provider → partial normalized structure
│ ├─ Java lexical provider → partial normalized structure
│ └─ PHP/Perl/Ruby/C++/Go/Rust/.NET/Scala lexical providers → partial structures
├─ Replacement SyntaxSupervisor → syntax.worker → typed replacement tokens
├─ EngineSupervisor
│ ├─ ecmascript.worker → native RegExp → match DTOs
│ ├─ pcre2.worker → PCRE2 10.47 WASM ABI → match DTOs
│ ├─ php.worker → PHP 8.5.8 preg / PCRE2 10.44 → match DTOs
│ ├─ perl.worker → legacy WebPerl 0.09-beta / Perl 5.28.1 → match DTOs
│ ├─ python.worker → Pyodide 314.0.3 → CPython 3.14.2 re → match DTOs
│ └─ java.worker → TeaVM 0.15.0 java.util.regex module → match DTOs
│ ├─ ruby.worker → ruby.wasm → CRuby 4.0.0 Regexp → match DTOs
│ ├─ java.worker → TeaVM 0.15.0 java.util.regex module → match DTOs
│ ├─ cpp.worker → Emscripten 6.0.4 libc++ std::wregex → match DTOs
│ ├─ go.worker → Go 1.26.5 standard-library regexp → match DTOs
│ ├─ rust.worker → Rust regex crate 1.13.1 → match DTOs
│ ├─ dotnet.worker → .NET 10.0.10 System.Text.RegularExpressions → match DTOs
│ └─ scala.worker → explicit compatibility adapter → TeaVM Java module
├─ Pcre2TraceSupervisor → pcre2-trace.worker
│ └─ separate ABI-v3 PCRE2_AUTO_CALLOUT operation → bounded trace DTOs
├─ AnalysisPanel
@@ -50,6 +59,9 @@ a new generation. Timeout is a distinct error state. A separate idempotent
load operation verifies a worker's runtime identity before execution; its
flavour-specific startup deadline is not charged to the 250 ms live or selected
manual execution deadline. A killed generation must load and verify again.
The supervisor has a two-worker retention target; before creating another
profile worker it evicts the least recently used idle generation. Active work
is never evicted and may temporarily exceed that target.
The Python worker imports only the self-hosted Pyodide loader and pack, then
evaluates the application-owned bounded bridge in CPython. The bridge returns
@@ -59,6 +71,22 @@ ES2015 module. That module executes TeaVM's own `java.util.regex.Pattern` and
`Matcher` class-library implementation and returns native UTF-16 ranges; it is
not OpenJDK.
PHP, Perl, Ruby, C++, Go, Rust and .NET each use a fixed bounded bridge to the
named self-hosted runtime. User values cross as structured/JSON data, never as
generated source. Scala intentionally reuses the TeaVM Java module and rewrites
only request/result profile metadata; it contains no Scala runtime. The legacy
Perl topology is unusual: the supervised module worker owns the classic worker
required by WebPerl, so supervisor termination still discards the complete
worker tree.
Ruby's production bridge avoids ruby.wasm's generic JavaScript object
conversion: the worker writes one bounded length-prefixed request into its
private WASI memory filesystem, calls the fixed bridge with no user arguments,
reads one bounded inert metadata JSON string, and obtains bounded UTF-8
replacement output from a separate private file. That path avoids JSON output
amplification and passes the same strict-CSP browser gate as the other
profiles.
PCRE2 tracing is not an `EngineSupervisor` match or replacement operation. Its
own worker compiles with `PCRE2_AUTO_CALLOUT`, copies only complete fixed
records and bounded mark bytes, and stops natively at either trace cap.
@@ -86,9 +114,10 @@ deterministic golden identities and an exact PCRE2 10.47 compile/execute gate.
Subject minimization is a main-thread bounded reducer whose expensive predicate
checks stay in independently terminable ECMAScript or PCRE2 workers. Fixed
pattern syntax is parsed once for capture metadata. Python and Java are blocked
at this feature boundary rather than routed to a different engine. The baseline
and every accepted candidate retain the exact target and engine identity.
pattern syntax is parsed once for capture metadata. Every other profile is
blocked at this feature boundary rather than routed to a different engine. The
baseline and every accepted candidate retain the exact target and engine
identity.
Timeout, crash, cancellation, worker failure and incomplete output are separate
outcomes. Deterministic ddmin deletion is followed by a fixed-point local sweep
over Unicode-scalar deletion and lower-rank canonical replacement; only
@@ -139,13 +168,13 @@ separator during apply. This makes anchor and cross-line behavior deliberately
different rather than silently rewriting the pattern. Match DTOs are reduced to
bounded per-group participation counts and samples before the next document.
Native ranges are retained in the engine's declared unit. ECMAScript and Java
use UTF-16 code units. PCRE2 always uses UTF/UCP and exposes UTF-8 byte ranges;
its ABI copies scalar records and names out of WebAssembly before returning and
retains no code pointer. CPython `re` exposes Unicode code-point indices.
Browser editor ranges are half-open UTF-16 code-unit ranges. Reusable
converters cover UTF-8 byte and Unicode code-point offsets, including
invalid-boundary rejection and lone-surrogate detection.
Native ranges are retained in the engine's declared unit. ECMAScript, Java,
Scala compatibility and .NET use UTF-16 code units. Standalone PCRE2, PHP, Go
and Rust expose UTF-8 byte ranges. CPython, CRuby, libc++ `std::wregex` and
legacy Perl expose Unicode code-point positions. Browser editor ranges are
half-open UTF-16 code-unit ranges. Reusable converters cover UTF-8 byte and
Unicode code-point offsets, including invalid-boundary rejection and
lone-surrogate detection.
The release boundary is `dist/` plus checked-in legal/source documents. The
Portal consumes the resulting immutable ZIP and never imports React source.

16
docs/COMPARISON_AND_CODEGEN.md
View File

@@ -7,8 +7,9 @@ ECMAScript worker and one to the bundled PCRE2 10.47 worker. It does not
translate patterns, flags, options or replacement templates.
This is an explicit two-engine feature, not a generic active-flavour
comparison. Selecting Python or Java for the main workbench does not insert it
into either side or route it through ECMAScript/PCRE2.
comparison. Selecting PHP, Perl, Python, Ruby, Java, C++, Go, Rust, .NET or
Scala compatibility for the main workbench does not insert it into either side
or route it through ECMAScript/PCRE2.
Two pattern models are available:
@@ -55,11 +56,12 @@ result. The panel renders at most 250 differences and 100 alignments at once.
## PCRE2 C17 generator
The first reviewed code-generation target is the PCRE2 10.47 8-bit C API.
Python, Java and other targets remain unavailable until their generators have
equivalent escaping, semantic and named-runtime/toolchain gates. The executable
Python and Java browser flavours do not by themselves imply a code-generation
target.
The first reviewed code-generation target is the standalone PCRE2 10.47 8-bit
C API. Every other target remains unavailable until its generator has
equivalent escaping, semantic and named-runtime/toolchain gates. In particular,
PHP's embedded PCRE2 10.44 is not routed through the standalone-PCRE2
generator; an executable browser profile does not by itself imply a
code-generation target.
The generator consumes a validated PCRE2 snapshot and:

317
docs/ENGINE_BUILDS.md
View File

@@ -1,9 +1,11 @@
# Engine builds
The production application uses the browser's native `RegExp` for ECMAScript
and bundles verified PCRE2, Python and Java runtime packs. Normal application
builds never download or compile an engine: reviewed files under
`public/engines/` are copied into `dist/` and verified there.
and bundles ten verified runtime packs for standalone PCRE2, PHP, Perl, Python,
Ruby, Java, C++, Go, Rust and .NET. The Scala/JVM compatibility profile shares
the Java pack and contains no Scala runtime. Normal application builds never
download or compile an engine: reviewed files under `public/engines/` are
copied into `dist/` and verified there.
## Rebuilding PCRE2
@@ -68,14 +70,36 @@ independently reports `info.python` as 3.14.0, and the verifier records rather
than conflates those two upstream identities. The pack copies only the reviewed
loader, WebAssembly runtime, Python standard library and lock metadata required
for a local CPython `re` worker. It adds deterministic engine metadata, a
closed-file-set checksum manifest and exact Pyodide/CPython licence material.
closed-file-set checksum manifest and a complete `SOURCE-MANIFEST.json`.
The source manifest records immutable Git commit/tree identities or primary
release-archive SHA-256 values for Pyodide, its `pyodide-build` submodule, its
vendored Error Stack Parser 2.1.4 and StackFrame 1.3.4 ports, CPython, libffi,
Hiwire, XZ/liblzma, Zstandard, SQLite, the Emscripten bzip2 and zlib ports,
Emscripten itself and HACL* preferred source. It hashes both vendored
JavaScript ports, all nine CPython patches, all five Emscripten patches, and
the Pyodide Makefiles that select those inputs. That is the
preferred-form/patch route for the upstream MPL source; the exact npm package
remains the binary-distribution reference, not a claim that arbitrary future
host SDK packaging will reproduce identical bytes.
The linked-component inventory is derived from the exact Pyodide recipe and
checked against executable identities where the libraries expose one. Besides
Pyodide and CPython, the base module contains CPython-vendored Expat 2.7.3,
libmpdec 2.5.1 and HACL*, plus libffi, Hiwire 1.0.1, liblzma 5.2.2,
Zstandard 1.5.7, SQLite 3.39.0, bzip2 1.0.6 and zlib 1.3.1. The conservative
Emscripten 5.0.3 runtime notice closure covers musl, compiler-rt, libc++,
libc++abi, libunwind, dlmalloc and the MiniLZ4 source selected by the pinned
link flags. The upstream stripped module contains no link map, so retaining the
complete potentially selected Emscripten runtime notice set is intentional.
The install operation reads from the already installed pinned npm package. It
does not contact a CDN or package index. It verifies source package identity,
file hashes, runtime metadata and licence inputs before atomically replacing
file hashes, runtime metadata, every independently fetched legal-source hash,
the generated source manifest and licence inputs before atomically replacing
only `public/engines/python/`. The distribution verifier checks the same closed
set under `dist/`, validates WebAssembly magic and confirms that all runtime
references remain relative and self-hosted.
set under `dist/`, validates WebAssembly and native component identities and
confirms that all runtime references remain relative and self-hosted.
```sh
node scripts/python-engine-pack.mjs \
@@ -88,8 +112,13 @@ The installed closed set is:
- `pyodide.mjs`, `pyodide.asm.mjs` and `pyodide.asm.wasm`;
- `python_stdlib.zip` and `pyodide-lock.json`;
- `engine-metadata.json` and `SHA256SUMS`; and
- `LICENSE.pyodide.txt` and `LICENSE.cpython.txt`.
- `engine-metadata.json`, `SOURCE-MANIFEST.json` and `SHA256SUMS`; and
- 22 exact `LICENSE.*`/`NOTICE.*` files enumerated and source-anchored by that
manifest.
`pyodide-lock.json` is the upstream catalogue of optional packages. Regex
Tools does not ship the catalogue's wheels; `externalPackagesIncluded: false`
and the closed pack file set make that boundary machine-verifiable.
## Building the Java pack
@@ -120,6 +149,179 @@ OpenJDK's `Pattern.UNICODE_CHARACTER_CLASS`; the bridge treats the application
`U` flag as a compatibility request, implies Unicode case folding and reports
that predefined-character classes and word boundaries retain TeaVM behaviour.
## Building the PHP pack
The PHP bridge under `engines/php/` executes actual PHP 8.5.8 `preg_*`.
`@php-wasm/web-8-5` and `@php-wasm/universal` 3.1.46 plus their pinned support
packages supply the Asyncify WebAssembly runtime and host API. Direct identity
verification reports PCRE2 10.44; this pack is not the standalone PCRE2 10.47
build.
```sh
node scripts/php-engine-pack.mjs
node scripts/install-php-engine.mjs
```
The builder copies only the pinned npm runtime files, changes the
bundler-specific WebAssembly import into a same-directory URL, records the
original and transformed hashes, and runs real identity, match, bounded
replacement and native-component positive/negative audits before installation.
`native-components.json` binds the exact WebAssembly hash to WordPress
Playground `v3.1.46` commit
`581c7c172428159eb4e6c5309054a568cd39a97a`, PHP tag `php-8.5.8` commit
`26b97507444c4fbda072f57dda1820f7b7d5e467`, Emscripten 4.0.19 commit
`08e2de1031913e4ba7963b1c56f35f036a7d4d56` and all 32 linked native/source
surfaces. It carries exact archive hashes or Git identities where upstream
recorded them and honestly bounds the one unpinned Oniguruma clone using the
committed archive/header identities and upstream source interval.
The pack elects PHP License version 4 (`BSD-3-Clause`) for PHP 8.5.8 through
PHP's later-version option and includes the complete PCRE2 and PHP-vendored,
external-library, patent and Emscripten/LLVM/musl legal set. This includes the
exact Zend Engine licence, official php-src binary redistribution notice, CLI
HTTP parser MIT terms, and the public-domain, CC0 and FNV permission notices for
linked hash implementations. The WordPress Playground `@php-wasm` packages
retain GPL-2.0-or-later; JavaScript support-package licences remain separate.
All 42 legal assets and the native inventory are independently SHA-256-pinned
by the builder, then covered again by the closed file set, `SHA256SUMS`, release
requirements and metadata.
## Building the legacy Perl pack
The Perl bridge under `engines/perl/` uses the unmodified WebPerl 0.09-beta
prebuilt archive, SHA-256
`5f441249217e90ab378c666f473d4206ab4f44907f6bb0aa8d70834bc38c40dc`.
That 2019 beta contains Perl 5.28.1 and Emscripten 1.38.28 output. It is
intentionally a legacy compatibility target, not a current Perl distribution.
The builder accepts only a local copy of that exact official archive; it never
downloads. An already extracted release root can be supplied with `--prebuilt`
instead of `--archive`.
```sh
node scripts/perl-engine-pack.mjs \
--archive /absolute/path/webperl_prebuilt_v0.09-beta.zip
node scripts/perl-engine-pack.mjs --verify .engine-build/perl
node scripts/install-perl-engine.mjs .engine-build/perl
```
The installed pack retains upstream `emperl.js`, `emperl.wasm` and
`emperl.data` byte-for-byte. Application-owned fixed worker and Perl bridge
files carry values through an exact-key, byte-bounded JSON request path; they do
not splice patterns or replacements into source. The metadata response remains
bounded JSON, while successful replacement UTF-8 bytes stream into a separate
fixed MEMFS binary file and receive exact worker-side byte, size and encoding
validation. Exact WebPerl/Perl, Emscripten and bundled-CPAN notices travel with
the pack.
The legacy generated loader contains one `eval(code)` site in an optional
Perl-to-JavaScript interoperability import. The application bridge never loads
that facility. Its browser CSP smoke passes with `'wasm-unsafe-eval'` and
without broad `'unsafe-eval'`; that dormant upstream site remains documented
rather than removed from the checksummed asset.
## Building the Ruby pack
The Ruby pack copies the minimal runtime from pinned
`@ruby/4.0-wasm-wasi@2.9.3-2.9.4` and uses
`@ruby/wasm-wasi@2.9.3-2.9.4` as its host. It verifies the CRuby 4.0.0
`wasm32-wasi` identity and actual `Regexp`/replacement behavior before
installation.
```sh
node scripts/ruby-engine-pack.mjs
node scripts/install-ruby-engine.mjs
```
The closed set contains the runtime WebAssembly file, deterministic metadata
and checksums, ruby.wasm licence/complete NOTICE and the exact browser WASI
shim and `tslib` licence texts needed by the host bundle. The production worker
passes each request through a bounded length-prefixed file in its private WASI
memory filesystem. It receives bounded inert JSON metadata and reads bounded
UTF-8 replacement output from a separate private file, rather than amplifying
that output through JSON or using `RubyVM.wrap`/`toJS` and their
JavaScript-eval conversion path. Ruby cold startup passes the release's
strict-CSP browser gate.
## Building the C++ pack
The C++ bridge under `engines/cpp/` is compiled with Emscripten 6.0.4 at
compiler revision `fe5be6afdff43ad58860d821fcc8572a23f92d19`. It uses libc++
`std::wregex`, 32-bit `wchar_t`, fixed structured Embind entry points and
`-sDYNAMIC_EXECUTION=0`.
```sh
node scripts/cpp-engine-pack.mjs --empp /absolute/path/to/em++
node scripts/install-cpp-engine.mjs
```
The verifier checks exact Emscripten, emsdk and LLVM identities, bridge hashes,
WebAssembly/module structure, absence of dynamic-execution constructs in the
generated module, runtime identity, code-point ranges and bounded replacement
parity. The build also enables `wasm-ld --trace` and accepts only the reviewed
linked archive set: Emscripten runtime support, musl `libc`, emmalloc,
compiler-rt builtins, libc++ and libc++abi.
`SOURCE-MANIFEST.json` ties the exact module hash to Emscripten tag/commit
`6.0.4`/`fe5be6afdff43ad58860d821fcc8572a23f92d19`, emsdk commit
`224ec5f9f2f72f09f9ce0e26d66bae7dbd8b692f`, the official release commit and
LLVM compiler commit. It records preferred-source paths and independently
pinned legal files for musl 1.2.6, emmalloc, compiler-rt 22.1.8, libc++ 21.1.8
and libc++abi 21.1.8. The current JavaScript-exception link selects no
`libunwind.a` member; libunwind 22.1.8 and its legal text remain as an explicit
audited exclusion. Any added, removed or newly selected archive fails the pack
build.
## Building the Go pack
Go 1.26.5 standard-library `regexp` is built with the official linux/amd64
archive, SHA-256
`5c2c3b16caefa1d968a94c1daca04a7ca301a496d9b086e17ad77bb81393f053`,
for `GOOS=js GOARCH=wasm`.
```sh
node scripts/build-go-engine.mjs /absolute/path/to/go
node scripts/install-go-engine.mjs
node scripts/verify-go-engine.mjs
```
The pack pins the source commit, official archive, `wasm_exec.mjs`, bridge
source, build flags and Go licence. Verification executes real Unicode
matching and `regexp.ExpandString` cases.
## Building the Rust pack
The Rust bridge pins `regex` crate 1.13.1 and its complete Cargo graph, rustc
1.97.1, the matching `wasm32-unknown-unknown` standard library and wasm-bindgen
0.2.126.
```sh
node scripts/build-rust-engine.mjs /absolute/path/to/rust-toolchain
node scripts/install-rust-engine.mjs
node scripts/verify-rust-engine.mjs
```
The pack records the distribution manifest/archive hashes, crate checksums,
compiler and LLVM identities, bridge sources, exact bindings and complete
Rust/crate licence material. Runtime verification covers identity, Unicode
byte ranges, limits and `Captures::expand`.
## Building the .NET pack
The fixed `[JSExport]` bridge under `engines/dotnet/` is published by .NET SDK
10.0.302 with runtime 10.0.10 and its `wasm-tools` workload.
```sh
node scripts/dotnet-engine-pack.mjs --dotnet /absolute/path/to/dotnet
node scripts/install-dotnet-engine.mjs
```
The pack gate checks SDK/runtime/workload and bridge identities, removes
compressed/debug outputs, records every `_framework` file, and includes the
.NET MIT licence and third-party notices. The release browser gate executes
the browser-WASM identity, Unicode-range and native-replacement smoke. There
is no separate Scala pack.
## Runtime boundaries
### PCRE2
@@ -150,9 +352,11 @@ CPython and `re` identities before accepting work. The application bridge is
evaluated inside that interpreter and exposes bounded compile, match and native
replacement operations. It reports CPython's Unicode code-point positions;
the TypeScript adapter retains them and converts only valid boundaries to
half-open editor UTF-16 ranges. Runtime load has a separate startup deadline.
Execution timeout, cancellation or crash terminates the worker and its complete
Python/WebAssembly heap.
half-open editor UTF-16 ranges. Replacement output is byte-bounded in Python,
encoded as canonical base64 in the JSON envelope and decoded only after its
declared size, request cap and UTF-8 are validated. Runtime load has a separate
startup deadline. Execution timeout, cancellation or crash terminates the
worker and its complete Python/WebAssembly heap.
### Java
@@ -163,3 +367,92 @@ replacement output, and returns native half-open UTF-16 ranges. No Java virtual
machine, OpenJDK classes, remote service or ECMAScript regex fallback is
involved. Runtime load has a separate startup deadline; termination discards
the complete worker module state.
### PHP
The PHP worker starts the same-origin Asyncify runtime, installs the fixed
`bridge.php` in its private virtual filesystem and verifies PHP 8.5.8 plus
PCRE2 10.44 before accepting work. User values cross only as JSON data. The
bridge selects its own safe delimiter, permits only registered modifiers and
calls native `preg_match()`. `PREG_OFFSET_CAPTURE` byte offsets are retained;
byte matches that split an encoded character are rejected because they cannot
map losslessly to a browser string. Bounded replacement mirrors PHP 8.5.8's
`preg_get_backref()` syntax over those native records and clips every append
before allocation crosses the requested byte cap. `preg_replace()` is used
only by the fixed identity parity self-test. Replacement bytes cross the JSON
envelope as size-predictable base64 and are decoded with a fatal UTF-8 check.
### Perl
The supervised module worker owns a nested classic worker required by the
legacy WebPerl loader. Terminating the supervisor also terminates that
descendant and the complete Perl heap. The fixed bridge uses `no re 'eval'`,
rejects code assertions independently and exposes only literal text, `$$`,
greedy `$n` and `${name}` replacement expansion. It pre-tokenizes that grammar,
streams small bounded UTF-8 pieces from native spans without constructing a
complete expansion or JSON output, and preserves the unmatched tail when the
match/capture-row bound stops replacement. Fixed named and numbered fixtures
are checked against native Perl `s///` during the identity self-test. Perl
character offsets are returned as Unicode code points. This containment does
not make WebPerl current software; the UI continues to report its legacy beta
identity.
### Ruby
The Ruby worker starts the minimal WASI runtime, evaluates only the fixed
application bridge and verifies CRuby 4.0.0 identity. User values travel as
bounded length-prefixed data in the worker-local memory filesystem; the bridge
returns inert metadata JSON and never sends a user-controlled value through
ruby.wasm's JavaScript-eval object-conversion path. CRuby compiles and matches.
Native `String#gsub`/`String#sub` block iteration selects replacement matches;
a fixed streaming implementation of CRuby 4.0.0's `rb_reg_regsub` grammar
writes only the configured UTF-8 prefix to a separate private file. A
load-time parity check compares the fixed expander with CRuby on bounded
fixtures. Native character offsets are retained as Unicode code points.
Worker termination discards the Ruby VM and both private files.
### C++
The C++ worker loads fixed Embind exports whose generated module was built with
dynamic execution disabled. Each request constructs `std::wregex`, iterates
under central result limits and streams a fixed implementation of libc++'s
`match_results::format` default grammar through the output byte cap. Load-time
fixtures compare it with the native formatter. The adapter exposes 32-bit
`wchar_t` positions as Unicode code points. The default grammar is C++'s
modified ECMAScript grammar, not the browser's current ECMAScript grammar.
### Go and Rust
The Go and Rust workers load their own fixed JSON bridges and verify the exact
runtime/crate identities before work. Both return UTF-8 byte ranges and reject
lone surrogates before encoding. Go uses standard-library `regexp` and
`ExpandString`; Rust uses `regex` crate iteration and `Captures::expand`.
Their fixed streaming expanders reproduce those native replacement grammars
under the output byte cap and compare bounded fixtures with the native APIs at
load. The adapters measure exact request JSON bytes, while matching native caps
include the worst-case sixfold escaping of every accepted string field. Output
bytes use fixed 4:3 base64 JSON envelopes instead of content-dependent JSON
escaping. Neither runtime supports look-around or backreferences. Each module
also enforces its own pattern, result and output bounds; Rust adds
compiled-regex and DFA-size limits.
### .NET
The .NET worker loads the browser-WASM runtime and calls only fixed
`[JSExport]` methods with bounded JSON. It verifies runtime 10.0.10,
`System.Text.RegularExpressions`, UTF-16 offsets and invariant culture. A
nine-second native regex timeout complements the shorter supervisor deadline;
terminating the worker discards the managed runtime. The adapter currently
returns the final retained capture only, not .NET capture history. Replacement
templates are parsed once by a fixed streaming implementation of .NET's
numbered, named and special substitution tokens. A load-time oracle compares
the fixed implementation with native `Match.Result`; bounded UTF-8 output is
transported as canonical base64 rather than JSON-escaped user text.
### Scala/JVM compatibility
Scala requests deliberately delegate to the same verified TeaVM adapter and
module described under Java. Result metadata says “no Scala runtime”. The
profile covers Java-compatible pattern/replacement semantics only and does not
claim Scala `Regex` wrappers, extractors, compiler/runtime code or OpenJDK
behavior.

7
docs/EXECUTION_TRACES.md
View File

@@ -5,9 +5,10 @@ results, but no V8, SpiderMonkey or JavaScriptCore execution trace. Regex Tools
does not relabel structural explanations, static findings or timing
observations as an ECMAScript engine trace.
The bundled CPython `re` and TeaVM `java.util.regex` bridges likewise expose
compilation, bounded match/capture and replacement results, not internal engine
steps. Regex Tools does not label their partial lexical syntax trees as traces.
The PHP, legacy Perl, CPython, CRuby, TeaVM Java/Scala compatibility, libc++,
Go, Rust and .NET bridges likewise expose compilation, bounded match/capture
and replacement results, not internal engine steps. Regex Tools does not label
their partial lexical syntax trees as traces.
PCRE2 10.47 has a separate actual automatic-callout vertical. ABI version 3
compiles the requested pattern with `PCRE2_AUTO_CALLOUT`, registers an

16
docs/EXPLANATION_MODEL.md
View File

@@ -20,16 +20,18 @@ common capture forms. It does not infer branch-reset numbering or claim full
nesting; authoritative compilation, capture counts, names and result ranges
come from PCRE2 itself.
The Python and Java providers likewise emit deliberately partial lexical trees.
They recognize their documented capture and replacement forms for navigation
and display, but do not claim full grammar acceptance. CPython `re` and TeaVM
`Pattern` remain authoritative for compilation. Java capture names displayed
with native numbered spans come from the bounded lexical metadata; they are not
invented as a TeaVM-reported name table.
The Python, Java, PHP, Perl, Ruby, C++, Go, Rust, .NET and Scala-compatibility
providers likewise emit deliberately partial lexical trees. They recognize
their documented capture and replacement forms for navigation and display,
but do not claim full grammar acceptance. The selected named runtime remains
authoritative for compilation. Java and Scala capture names displayed with
native numbered spans come from bounded lexical metadata; they are not
invented as a TeaVM-reported name table. C++ capture discovery is explicitly
unavailable for alternate grammars or `nosubs`.
The explanation tree is structural. It is never called an actual execution
trace. Selecting a node selects and scrolls its exact editor range; selecting
pattern text chooses the smallest enclosing node. PCRE2’s separate trace viewer
contains actual bounded automatic callouts; its movement labels remain a
distinct derived layer and are never copied into the structural explanation.
Python and Java expose no actual execution trace.
The other eleven profiles expose no actual execution trace.

28
docs/EXTRACTION_MODEL.md
View File

@@ -18,14 +18,22 @@ budget. Consumers must inspect the preview status rather than treating a
preview as complete. In particular, list export is disabled if its template
would consume an incomplete value.
Repeated ECMAScript, PCRE2, Python and Java captures expose only the final
retained capture. The UI labels this limitation and does not invent history.
Later .NET support may add actual engine capture-history nodes without
degrading them to this model.
Repeated captures expose only the final retained value in all twelve v0.4.0
profiles. This includes .NET even though its native API can retain capture
history: the current adapter deliberately normalizes only the final record and
reports `captureHistory: false`. The UI labels the limitation and does not
invent or silently discard advertised history.
The syntactic hierarchy remains useful even if actual spans overlap or fall
outside a parent's geometry; native and normalized ranges are preserved rather
than “fixed”. ECMAScript and Java native offsets are UTF-16 code units, PCRE2
native offsets are UTF-8 bytes and Python native offsets are Unicode code
points. All editor ranges are normalized to UTF-16 without discarding the
native unit.
The partial syntactic hierarchy remains useful when the active provider found
capture relationships, even if actual spans overlap or fall outside a parent's
geometry. For C++ alternate grammars or suppressed submatches, capture
discovery can be explicitly unavailable. Runtime records remain authoritative.
Native and normalized ranges are preserved rather than “fixed”:
- JavaScript, Java, Scala compatibility and .NET use UTF-16 code units;
- standalone PCRE2, PHP, Go and Rust use UTF-8 bytes; and
- Python, Ruby, C++ and legacy Perl use Unicode code points.
All editor ranges are normalized to UTF-16 only after the adapter verifies
valid boundaries. The native unit and original range remain available.

136
docs/FLAVOUR_SUPPORT.md
View File

@@ -1,75 +1,87 @@
# Flavour support
| Flavour | Syntax | Execution | Replacement | Captures | Trace | Analysis | Generated cases | Native offsets |
| ---------------------- | ------------------------------------------------------------ | --------------------------------------------------------- | ---------------------------------------- | -------------------------------------------------- | ------------------------------------------------------------ | ------------------------------------------------------ | ------------------------------------------------------- | -------------- |
| ECMAScript | regexpp 4.12.2; explanations partial/feature-matrix tested | Current browser `RegExp`; conformance tested | Bounded ECMAScript `GetSubstitution` | Named/numbered; final repeated capture; no history | Unavailable | Experimental static heuristics + bounded native probes | Bounded normalized-AST sampling; actual-engine verified | UTF-16 |
| PCRE2 10.47 | Application lexical provider partial; compiler authoritative | Pinned 8-bit WASM; bounded/killable/conformance tested | Native bounded `pcre2_substitute()` loop | Native named/numbered records; no capture history | Bounded reported automatic callouts; derived movement labels | Unavailable; ECMAScript heuristics are never applied | Unavailable; structural syntax provider is partial | UTF-8 bytes |
| Python `re` | Application lexical provider partial; compiler authoritative | CPython 3.14.2 in self-hosted Pyodide 314.0.3 | Native bounded `re` expansion | Native named/numbered spans; no capture history | Unavailable | Unavailable; ECMAScript heuristics are never applied | Unavailable; structural syntax provider is partial | Code points |
| Java `java.util.regex` | Application lexical provider partial; compiler authoritative | TeaVM 0.15.0 class library; bounded/killable; not OpenJDK | TeaVM single-digit `$n`; no `${name}` | Native numbered ranges; lexical names; no history | Unavailable | Unavailable; ECMAScript heuristics are never applied | Unavailable; structural syntax provider is partial | UTF-16 |
| PCRE | Unavailable | Unavailable | Unavailable | Unavailable | Unavailable | Unavailable | Unavailable | Unavailable |
| Go `regexp` | Unavailable | Unavailable | Unavailable | Unavailable | Unavailable | Unavailable | Unavailable | Planned bytes |
| Rust `regex` | Unavailable | Unavailable | Unavailable | Unavailable | Unavailable | Unavailable | Unavailable | Planned bytes |
| .NET | Unavailable | Unavailable | Unavailable | Planned capture history | Unavailable | Unavailable | Unavailable | Planned UTF-16 |
Version 0.4.0 exposes twelve executable profiles. “Partial lexical provider”
means that Regex Tools recognizes common structure and replacement tokens for
explanation only; the named runtime compiler and replacement operation remain
authoritative.
ECMAScript tests cover its complete exposed flag matrix, internal indices,
zero-length iteration, captures, lookbehind, backreferences, Unicode properties
and bounded replacement.
| Profile | Syntax and execution identity | Replacement | Captures and native offsets |
| ----------------------- | ------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------ |
| JavaScript (ECMAScript) | regexpp 4.12.2 with ECMAScript 2025 syntax; current browser `RegExp` executes | Bounded ECMAScript `GetSubstitution` | Named/numbered, final repeated capture; UTF-16 |
| PCRE2 | Partial provider; official standalone PCRE2 10.47 8-bit WASM executes in mandatory UTF/UCP mode | Native bounded `pcre2_substitute()` loop | Native named/numbered records, no history; UTF-8 bytes |
| PHP | Partial provider; actual PHP 8.5.8 `preg_*` in `@php-wasm/web-8-5` 3.1.46, reporting PCRE2 10.44 | Fixed bounded PHP-token expansion; load-time `preg_replace()` parity | Native named/numbered records, no history; UTF-8 bytes |
| Perl | Partial provider; actual Perl 5.28.1 through legacy WebPerl 0.09-beta | Fixed bounded literal/`$$`/`$n`/`${name}` expansion; arbitrary Perl expressions unavailable | Native numbered spans plus bounded name annotations, no history; Unicode code points |
| Python | Partial provider; CPython 3.14.2 `re` in Pyodide 314.0.3 | Native bounded `re` expansion | Native named/numbered spans, no history; Unicode code points |
| Ruby | Partial provider; CRuby 4.0.0 `Regexp` in ruby.wasm 2.9.3-2.9.4 | Native `gsub`/`sub` iteration plus bounded CRuby 4.0.0-compatible expansion | Native named/numbered spans, no history; Unicode code points |
| Java | Partial provider; TeaVM 0.15.0 `java.util.regex`, not OpenJDK | TeaVM single-digit `$n`; no `${name}` | Native numbered ranges plus lexical names, no history; UTF-16 |
| C++ | Partial provider; Emscripten 6.0.4 libc++ `std::wregex`; modified ECMAScript grammar by default | Fixed bounded formatter; load-time libc++ `match.format` parity | Numbered captures only, no history; 32-bit `wchar_t` code points |
| Go | Partial provider; Go 1.26.5 standard-library `regexp` with RE2 syntax | Fixed bounded expansion; load-time `regexp.ExpandString` parity | Native named/numbered records, no history; UTF-8 bytes |
| Rust | Partial provider; Rust `regex` crate 1.13.1, built with rustc 1.97.1 | Fixed bounded expansion; load-time `Captures::expand` parity | Native named/numbered records, no history; UTF-8 bytes |
| .NET | Partial provider; .NET 10.0.10 `System.Text.RegularExpressions`, invariant culture | Fixed streaming .NET-token expansion with load-time native `Match.Result` parity | Named/numbered final records, no capture history in this adapter; UTF-16 |
| Scala/JVM compatibility | Partial provider; explicitly shares TeaVM 0.15.0 `java.util.regex`; no Scala runtime | Same TeaVM single-digit `$n` subset; no `${name}` | Same ranges as the TeaVM Java adapter; UTF-16 |
PCRE2 tests cover `g`, `i`, `m`, `s`, `x`, `U`, `J`, mandatory UTF/UCP,
branch-reset groups, recursion, `\K`, duplicate names, native substitution,
astral byte/UTF-16 ranges, zero-length iteration, compile errors, match limits
and result/output truncation. Trace tests cover reported automatic callouts,
bounded mark copying, native stop at the complete-event cap, compile ranges,
worker loading and browser rendering.
## Flags and options
Python tests cover application-level `g`, CPython flags `a`, `i`, `m`, `s` and
`x`, Python-only named-group/replacement syntax, native `re` substitution,
astral code-point/UTF-16 normalization, zero-length iteration, compile errors,
result limits, runtime identity and worker recovery.
- JavaScript exposes `g`, `i`, `m`, `s`, `u`, `v`, `y` and `d`; `u` and `v`
are mutually exclusive.
- Standalone PCRE2 exposes application iteration `g` plus `i`, `m`, `s`, `x`,
`U` and `J`, with match-step, depth and heap limits. UTF/UCP is mandatory.
- PHP exposes application iteration `g` plus the reviewed `preg` modifiers
`i`, `m`, `s`, `x`, `u`, `U`, `A`, `D`, `J`, `n` and `r`. PHP's `u` is not
conflated with standalone PCRE2's mandatory UTF/UCP configuration.
- Legacy Perl exposes `g`, `i`, `m`, `s`, `x`, `n` and one of `a`, `d`, `l` or
`u`. The two-character `/aa` mode is not exposed.
- Python exposes application iteration `g` plus `a`, `i`, `m`, `s` and `x`.
- Ruby exposes application iteration `g` plus `i`, `m` and `x`; Ruby `m`
includes dot-all behavior.
- Java and Scala compatibility expose application iteration `g` plus TeaVM
`Pattern` flags `i`, `d`, `m`, `s`, `u`, `x` and the documented `U`
compatibility request.
- C++ exposes application iteration `g` plus `i`, `n`, `o` and `c`, and a
required grammar option: `ECMAScript`, `basic`, `extended`, `awk`, `grep` or
`egrep`.
- Go exposes application iteration `g` plus `i`, `m`, `s` and `U`.
- Rust exposes application iteration `g` plus `i`, `m`, `s`, `U`, `u`, `x` and
`R`; Unicode is enabled by default.
- .NET exposes application iteration `g` plus `i`, `m`, `s`, `n`, `x`, `r`,
`c` and `b`. Execution is always culture invariant; `c` makes that fixed
compatibility choice explicit.
Java tests cover application-level `g`, TeaVM `Pattern` flags, Java named-group
syntax, TeaVM's native single-digit `$n` `Matcher` replacement, explicit
rejection of unsupported `${name}`, astral UTF-16 ranges, zero-length
iteration, compile errors, result limits, module identity and worker recovery.
The runtime and UI explicitly report that TeaVM 0.15.0 is not OpenJDK and does
not implement OpenJDK `Pattern.UNICODE_CHARACTER_CLASS`; `U` is a compatibility
request rather than an exact implementation of that flag.
## Verified boundaries
ECMAScript static findings and generated-input timing observations are
advisory. They describe only the selected ECMAScript 2025 pattern, browser
runtime and bounded inputs; they are not proofs of safety or general
complexity.
Every adapter has focused identity, request and result-contract tests.
Conformance and browser suites exercise compilation errors, bounded matching,
zero-length iteration, range normalization, replacement and worker recovery
across the profile set. Pack verifiers additionally check self-hosted assets,
metadata, closed-file-set checksums and applicable licences/notices.
ECMAScript generated-case version 1 uses only a complete accepted normalized
AST, an explicit deterministic seed and bounded sampling strategies. Every
retained positive or negative label is confirmed by the actual selected
browser engine. PCRE2, Python and Java generation are unavailable because
their current lexical providers are intentionally partial; ECMAScript behavior
is never relabelled as another flavour.
TeaVM supplies an Apache Harmony-derived class library. It is not OpenJDK and
does not implement OpenJDK `Pattern.UNICODE_CHARACTER_CLASS`; `U` means the
documented TeaVM compatibility behavior in both the Java and Scala profiles.
The Scala profile also does not implement `scala.util.matching.Regex` wrapper
or extractor APIs.
The PCRE2, Python and Java explanation providers deliberately do not invent
full structural coverage. PCRE2 branch-reset numbering is omitted from the
syntax tree, while actual capture numbers, names and ranges still come from the
selected runtime result.
The Perl target is intentionally legacy. Its unmodified WebPerl loader contains
one `eval(code)` site in an optional Perl-to-JavaScript interoperability import.
The fixed Regex Tools bridge does not load that facility, accepts pattern and
replacement values only as JSON data, uses `no re 'eval'`, rejects `(?{...})`
and `(??{...})`, and passes the documented CSP smoke without broad
`'unsafe-eval'`.
ECMAScript and PCRE2 support exact-request comparison for syntax acceptance,
engine compilation, matches, captures and engine-native replacement. Results
align by normalized UTF-16 ranges while retaining each native offset model.
Timeout, cancellation, rejection and truncation are never presented as
equivalent results.
## Feature scope
Pattern formatting is available only for an exact accepted ECMAScript regexpp
snapshot. It canonicalizes literal slashes and raw control/line-separator
representations, then requires paired actual-engine and applicable exact-test
validation before apply. PCRE2, Python and Java formatting is unavailable
because those providers are deliberately partial; their syntax is never
treated as ECMAScript.
Static findings, bounded growth probes, benchmarks, generated cases and
formatting are ECMAScript-only. Generated-case version 1 requires the complete
accepted regexpp AST and verifies every retained label through the actual
browser engine. No partial provider is passed through that generator or
formatter.
Subject minimization accepts only the verified ECMAScript and PCRE2 paths.
Python and Java selections are disabled rather than being minimized through a
different flavour. Semantic comparison is likewise an explicit
ECMAScript/PCRE2 operation; it is not a generic four-way equivalence checker.
Automatic-callout tracing and reviewed C17 code generation are standalone
PCRE2-only. The PHP profile does not reuse the standalone PCRE2 10.47 trace or
code-generation path because its authoritative engine is PHP `preg` with
PCRE2 10.44.
PCRE2 is the only current code-generation target. The reviewed C17 generator is
pinned to the PCRE2 10.47 8-bit API; other language generators remain
unavailable until they pass equivalent escaping and named-toolchain gates.
Exact-request semantic comparison and subject minimization support only their
reviewed ECMAScript/standalone-PCRE2 paths. Other selections are disabled
rather than routed through another profile. Results never claim generic
twelve-way equivalence.

6
docs/FORMATTING.md
View File

@@ -23,9 +23,9 @@ pretty-printer. The transform is idempotent.
Formatting requires the exact current, accepted, non-recovered syntax snapshot
from the bundled `@eslint-community/regexpp` provider. Transformations are
derived from the provider's literal-token ranges rather than from a heuristic
text scan. A stale parse, a malformed pattern, a partial provider, PCRE2,
Python, Java or any other flavour is reported as unavailable and is never
silently rewritten.
text scan. A stale parse, malformed pattern, partial provider or any
non-ECMAScript profile is reported as unavailable and is never silently
rewritten.
## Mandatory validation before apply

10
docs/GENERATED_CASES.md
View File

@@ -25,11 +25,11 @@ The coverage report says `covered`, `partial`, `unsupported` or
sampling strategy was requested. It does not mean every string in the regular
language was enumerated.
PCRE2, Python and Java generation is unavailable in version 1. Their actual
engines are complete for the advertised execution operations, but their
current structural syntax providers are intentionally partial. Regex Tools
does not feed their syntax to the ECMAScript generator or relabel ECMAScript
behaviour.
Standalone PCRE2, PHP, Perl, Python, Ruby, Java, C++, Go, Rust, .NET and Scala
compatibility generation is unavailable in version 1. Their actual engines are
complete for the advertised execution operations, but their current structural
syntax providers are intentionally partial. Regex Tools does not feed their
syntax to the ECMAScript generator or relabel ECMAScript behaviour.
## Unsupported and partial constructs

7
docs/MINIMIZATION.md
View File

@@ -10,9 +10,10 @@ selected real ECMAScript or PCRE2 worker. It can preserve:
- the same one-sided comparison timeout while the other engine completes with
authoritative, untruncated output.
Python and Java are not supported minimization targets in v0.3.0. The workspace
disables reduction for those active flavours; it does not substitute
ECMAScript, PCRE2 or a lexical provider as an execution oracle.
PHP, Perl, Python, Ruby, Java, C++, Go, Rust, .NET and Scala compatibility are
not supported minimization targets in v0.4.0. The workspace disables reduction
for those active profiles; it does not substitute ECMAScript, PCRE2 or a
lexical provider as an execution oracle.
The fixed pattern, ordered flags, options, replacement and engine versions form
the target identity. Syntax is parsed once to obtain capture metadata. Every

46
docs/PARSER_PROFILES.md
View File

@@ -6,20 +6,38 @@ Regex Tools has one public profile:
community
```
It uses only reviewed open-source dependencies and builds without private
registry credentials. ECMAScript uses regexpp 4.12.2. PCRE2 uses an
application-owned lexical provider for partial explanations and the actual
bundled PCRE2 compiler for authoritative acceptance. Python and Java use
application-owned lexical providers for partial explanations and
flavour-native replacement tokens; bundled CPython `re` and TeaVM
`java.util.regex` respectively remain authoritative for compilation.
The Java provider follows the bundled TeaVM 0.15.0 replacement subset:
single-digit `$n` is executable, `$12` is `$1` followed by literal `2`, and
Java SE `${name}` references are surfaced as a compatibility limitation.
It uses reviewed open-source dependencies and builds without private registry
credentials. No licensed parser is installed, referenced or required.
JavaScript/ECMAScript uses regexpp 4.12.2 for ECMAScript 2025 syntax. That is
the only current complete grammar-backed normalized syntax tree. All other
profiles use application-owned, deliberately partial lexical providers:
| Profile | Authoritative compiler | Lexical-provider boundary |
| ----------------------- | --------------------------------------------- | -------------------------------------------------------------------------------------------------------------------- |
| PCRE2 | Standalone PCRE2 10.47 | Common captures and PCRE2 constructs; no complete structural grammar or branch-reset numbering |
| PHP | PHP 8.5.8 `preg` / PCRE2 10.44 | Common PCRE2/PHP capture forms and numbered replacement references; no delimiter or complete `preg` template model |
| Perl | Perl 5.28.1 via legacy WebPerl 0.09-beta | Common captures and fixed bridge references only; embedded code and complete Perl interpolation are unavailable |
| Python | CPython 3.14.2 `re` | Common Python captures and replacement references |
| Ruby | CRuby 4.0.0 `Regexp` | Common Onigmo capture forms and `String#gsub` references |
| Java | TeaVM 0.15.0 `java.util.regex` | Common Java captures and TeaVM replacement subset; not OpenJDK |
| C++ | Emscripten 6.0.4 libc++ `std::wregex` | Capture discovery only for the default modified ECMAScript grammar; alternate grammars remain compiler-authoritative |
| Go | Go 1.26.5 standard-library `regexp` | Common RE2 capture forms and `regexp.Expand` references |
| Rust | Rust `regex` crate 1.13.1 | Common capture forms and `Captures::expand` references |
| .NET | .NET 10.0.10 `System.Text.RegularExpressions` | Common captures/substitutions; no balancing-group or capture-history structure |
| Scala/JVM compatibility | Same TeaVM 0.15.0 Java engine | Java-compatible syntax only; no Scala runtime, wrapper or extractor model |
The Java and Scala providers follow the bundled TeaVM 0.15.0 replacement
subset: single-digit `$n` is executable, `$12` is `$1` followed by literal
`2`, and `${name}` is surfaced as a compatibility limitation. The Scala
provider deliberately shares that implementation and says so.
Every partial provider exposes its gaps through coverage metadata. Actual
acceptance, capture numbering and replacement output always come from the
selected runtime. ECMAScript analysis, case generation and formatting are not
applied to these partial trees.
The product deliberately does not implement or reference an `@r101/parser`
profile. No commercial package alias, import, tarball, credential, licence key
or private CI path exists. Additional flavour syntax is implemented
incrementally as open-source providers behind the same application-owned
interface. Partial coverage is exposed as metadata and never presented as a
complete parser.
or private CI path exists. Further syntax coverage can grow behind the same
application-owned interface without changing the runtime identity.

23
docs/PERFORMANCE.md
View File

@@ -19,12 +19,23 @@ supervisor wall clock, and cancellation discards the complete worker heap.
Python has a materially larger cold start because its worker loads the
self-hosted Pyodide JavaScript/WebAssembly runtime and Python standard library
before verifying CPython and `re`. Java imports and self-tests its bundled
TeaVM ES module. Engine loading is an explicit operation with a
flavour-specific startup deadline; it is not charged to the 250 ms live or
selected manual execution timeout. Warm requests reuse the verified worker.
Killing a worker for timeout, crash or cancellation discards its complete
runtime, so the next request pays the cold-load cost and verifies identity
again.
TeaVM ES module. Scala compatibility shares that Java module and does not load
a second runtime.
PHP, legacy Perl, Ruby, Go, Rust, C++ and .NET likewise load only when first
selected. Their cold cost differs substantially: PHP, Perl, Ruby and .NET
instantiate larger language/runtime packs; Go includes its standard
WebAssembly runtime; Rust and C++ are comparatively compact native modules.
The UI reports loading separately rather than charging it to the 250 ms live or
selected manual execution timeout.
Warm requests reuse a verified worker. To prevent profile exploration from
leaving every large runtime resident, the supervisor uses a two-worker
retention target and evicts the least recently used idle generation before it
creates another profile worker. An active request is never evicted, so
concurrent active work can temporarily exceed the target. Killing or evicting
a worker discards its complete runtime, so the next request pays the cold-load
cost and verifies identity again.
Automatic-callout tracing is deliberately excluded from every benchmark. It
compiles an independently instrumented PCRE2 pattern in a separate worker and

55
docs/PORTAL_REQUIREMENTS.md
View File

@@ -1,31 +1,46 @@
# Portal requirements
Regex Tools is a static nested-path-safe application.
Regex Tools is a static nested-path-safe application. All engine resources are
self-hosted below the mounted release path; no runtime CDN fallback is allowed.
Required delivery behavior:
- JavaScript workers use a JavaScript MIME type.
- `toolbox-app.json`, HTML and legal/source documents should remain
revalidation/no-cache resources.
- Hashed Vite assets may use immutable long-term caching.
- CSP must allow `worker-src 'self' blob:`.
- `engines/pcre2/pcre2.wasm` and
`engines/python/pyodide.asm.wasm` must use `application/wasm`.
- Engine `.mjs` files must use a JavaScript MIME type;
`engines/python/python_stdlib.zip`, JSON, licence, notice and checksum files
must be delivered without content transformation.
- `script-src` must permit self-hosted WebAssembly compilation, normally with
`'wasm-unsafe-eval'`; broad `'unsafe-eval'` is neither needed nor recommended.
- `connect-src 'self'` must allow the Pyodide worker to fetch its own
same-origin WebAssembly, standard-library and lock files.
- Every file below `engines/pcre2/`, `engines/python/` and `engines/java/`
should use revalidation or a release-scoped immutable cache. These are not
content-hashed filenames and must never drift under one release identity.
- JavaScript, module and worker files (`.js` and `.mjs`) use a JavaScript MIME
type.
- Every `.wasm` file below `engines/` uses `application/wasm`. This includes
standalone PCRE2, PHP, Perl, Python, Ruby, C++, Go, Rust and every .NET
`_framework` WebAssembly file.
- `toolbox-app.json`, HTML and legal/source documents remain
revalidation/no-cache resources. Hashed Vite assets may use immutable
long-term caching.
- CSP permits `worker-src 'self' blob:`. The legacy Perl profile starts one
nested same-origin classic worker inside its supervised module worker.
- CSP permits self-hosted WebAssembly compilation in `script-src`, normally
through `'wasm-unsafe-eval'`. Broad `'unsafe-eval'` must not be enabled to
accommodate a runtime. Ruby cold startup and the fixed legacy Perl path pass
the release's strict-CSP browser gate without broad eval. WebPerl retains a
dormant optional interoperability `eval` site in upstream `emperl.js`, but
the fixed bridge does not load it.
- `connect-src 'self'` permits workers/runtimes to fetch their own same-origin
WebAssembly, Python standard-library, Perl data, .NET framework and metadata
resources.
- Binary/data assets such as `python_stdlib.zip`, `emperl.data`, metadata JSON,
checksum files, licences and notices are delivered without content
transformation.
- Range support is not required, but servers and proxies must not rewrite
engine bytes or apply HTML fallbacks to missing asset paths.
- Every directory below `engines/` uses revalidation or a release-scoped
immutable cache. Most engine filenames are not content-addressed and must
never drift under one release identity.
A suitable CSP is documented in [`SECURITY.md`](SECURITY.md). Deployments
should test both a direct root mount and a nested mount such as
`/apps/regex/`, including cold loading of all twelve profiles.
The Portal must consume the immutable release ZIP, verify its SHA-256, verify
manifest ID `de.add-ideas.regex-tools` and version `0.3.0`, and mount it at a
manifest ID `de.add-ideas.regex-tools` and version `0.4.0`, and mount it at a
relative target such as `apps/regex/`.
The package, manifest, tag, artifact URL and checksum must all use the v0.3.0
The package, manifest, tag, artifact URL and checksum must all use the v0.4.0
identity. Historical artifact coordinates remain immutable and must not be
overwritten or reused.

49
docs/REFERENCE_IMPLEMENTATIONS.md
View File

@@ -2,23 +2,34 @@
Inspection date: 2026-07-27.
| Reference | Exact revision/version | Licence | Use and copying status |
| ------------------------ | ----------------------------------------------------------------------------------------------------------------------- | -------------------------------------- | --------------------------------------------------------------------------------------------------- |
| Regex Tools | Release identity `v0.3.0` | GPL-3.0-or-later | Application source written for this project. |
| Toolbox SDK | `53c40a61ba1581246f65773fcbb1c1cfd31ac98e` / 0.2.2 | Apache-2.0 | Contract, shell and release conventions; package APIs used, no source adapted. |
| Toolbox Portal | `a9c31c8986c40a0097966318e925083302e91e13` / 0.5.0 | AGPL-3.0-only | Assembly/lock conventions inspected; no source copied into the app. |
| regexpp | 4.12.2; npm integrity `sha512-EriSTlt5OC9/7SXkRSCAhfSxxoSUgBm33OH+IkwbdpgoqsSsUg7y3uh+IICI/Qg4BBWr3U2i39RpmycbxMq4ew==` | MIT | Pinned ECMAScript syntax provider. Provider AST is normalized in a worker. |
| PCRE2 | signed tag 10.47; commit `f454e231fe5006dd7ff8f4693fd2b8eb94333429` | BSD-3-Clause WITH PCRE2-exception | Pinned offline build; generated verified WebAssembly pack is shipped, source is not vendored. |
| Pyodide | tag 314.0.3; commit `ac57031be7564f864d061cb37c5c152e59f83ad4`; pinned npm integrity | MPL-2.0 | Reviewed self-hosted runtime pack; no CDN use. |
| CPython | executable runtime identity 3.14.2 in Pyodide 314.0.3 | PSF-2.0 | Actual `re` compiler, matcher and replacement runtime; Python source is not vendored. |
| TeaVM | tag/commit `ee91b03e616c4b45401cd11fb0cd7eb0daf6649b` / 0.15.0 | Apache-2.0 | `java.util.regex` class library compiled into the verified module; not OpenJDK. |
| ECMAScript specification | <https://tc39.es/ecma262/>; living specification inspected 2026-07-24 | ECMA terms | Semantics reference; no prose copied. |
| regex101 | <https://regex101.com/> and public parser documentation | Product/commercial terms | Product reference only. No source, branding, explanations, reference prose or visual design copied. |
| `recheck` | 4.5.0 | MIT | Metadata inspected; not installed or shipped. |
| `regexp-ast-analysis` | 0.7.1 | MIT | Metadata inspected; not installed or shipped. |
| RegexHub | repository revision `d1d9f19d259745dfdd21935b9ef3e747b62b9bfb` | MIT | Potential future adversarial cases; no file copied. |
| RegexLib | Website inspected | Redistribution licence not established | No fixture copied. |
| RGXP.RU | Website inspected | Redistribution licence not established | No fixture copied. |
| Reference | Exact revision/version | Licence | Use and copying status |
| -------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------- |
| Regex Tools | Release identity `v0.4.0` | GPL-3.0-or-later | Application source written for this project. |
| Toolbox SDK | `53c40a61ba1581246f65773fcbb1c1cfd31ac98e` / 0.2.2 | Apache-2.0 | Contract, shell and release conventions; package APIs used, no source adapted. |
| Toolbox Portal | `9347554324299c8ec9dfaec2c9730907736f62c3` / aggregate v0.8.0 / package 0.2.7 | AGPL-3.0-only | Assembly/lock conventions inspected; no source copied into the app. |
| regexpp | 4.12.2; pinned npm integrity in `package-lock.json` | MIT | ECMAScript 2025 syntax provider; normalized in the syntax worker. |
| PCRE2 | Signed tag 10.47; commit `f454e231fe5006dd7ff8f4693fd2b8eb94333429` | BSD-3-Clause WITH PCRE2-exception | Standalone pinned offline build; verified 8-bit WebAssembly pack shipped, source not vendored. |
| PHP | Executable identity 8.5.8; embedded PCRE2 identity 10.44 | PHP License version 4 (BSD-3-Clause), elected through the earlier licence's later-version option | Actual PHP `preg_*` compiler, matcher and replacement runtime; PHP source not vendored. |
| WordPress Playground `@php-wasm` | `web-8-5`, `universal`, `util`, `logger`, `stream-compression` and `progress` 3.1.46; pinned npm integrities | GPL-2.0-or-later | Self-hosted PHP WebAssembly distribution/host packages; their separate GPL notice is included and recorded in the pack metadata. |
| WebPerl | 0.09-beta; commit `6f2173d29a2c2e3536e1de75ff5d291ae96ab348`; prebuilt archive SHA-256 `5f441249217e90ab378c666f473d4206ab4f44907f6bb0aa8d70834bc38c40dc` | GPL-1.0-or-later OR Artistic-1.0-Perl | Unmodified legacy prebuilt pack containing Perl 5.28.1; visibly labelled beta/legacy. |
| Emscripten used by WebPerl | 1.38.28 | MIT OR NCSA | Upstream-generated legacy Perl loader/runtime; applicable licence included. |
| Pyodide | Tag 314.0.3; commit `ac57031be7564f864d061cb37c5c152e59f83ad4`; pinned npm integrity | MPL-2.0 | Reviewed self-hosted runtime pack; no CDN use. |
| CPython | Executable runtime identity 3.14.2 in Pyodide 314.0.3 | PSF-2.0 | Actual `re` compiler, matcher and replacement runtime; Python source not vendored. |
| ruby.wasm / CRuby | `@ruby/4.0-wasm-wasi` and `@ruby/wasm-wasi` 2.9.3-2.9.4; executable CRuby identity 4.0.0 | MIT for ruby.wasm host; Ruby/BSD and component terms in bundled NOTICE | Minimal self-hosted CRuby `Regexp` runtime; pinned npm integrities/notices; fixed file/JSON bridge avoids the JavaScript-eval conversion. |
| TeaVM | Tag/commit `ee91b03e616c4b45401cd11fb0cd7eb0daf6649b` / 0.15.0 | Apache-2.0 | `java.util.regex` class library compiled into the verified Java module; not OpenJDK. Also backs the explicit Scala compatibility profile. |
| Emscripten / libc++ | Emscripten 6.0.4 compiler revision `fe5be6afdff43ad58860d821fcc8572a23f92d19` | MIT OR NCSA; Apache-2.0 WITH LLVM-exception | Compiles the application C++ bridge against libc++ `std::wregex`; toolchain source not vendored. |
| Go | Tag 1.26.5; commit `c19862e5f8415b4f24b189d065ed739517c548ba`; official archive SHA-256 `5c2c3b16caefa1d968a94c1daca04a7ca301a496d9b086e17ad77bb81393f053` | BSD-3-Clause | Standard-library `regexp` compiler/matcher and official `js/wasm` runtime. |
| Rust `regex` | 1.13.1; commit `2b527599eb9eea0dcc288c704584f242f26a5c61`; crate checksum `f020237b6c8eed93db2e2cb53c00c60a8e1bc73da7d073199a1180401450218d` | MIT OR Apache-2.0 | Actual crate compiler/matcher built for WebAssembly with the pinned Cargo graph. |
| Rust toolchain | rustc 1.97.1 commit `8bab26f4f68e0e26f0bb7960be334d5b520ea452`; wasm-bindgen 0.2.126 | Apache-2.0 OR MIT plus bundled component terms | Exact compiler, standard library and binding tool identities recorded in pack metadata. |
| .NET | SDK 10.0.302 / runtime 10.0.10; runtime commit `f7d90799ce` | MIT plus bundled third-party notices | Actual `System.Text.RegularExpressions` browser-WASM runtime. |
| ECMAScript specification | <https://tc39.es/ecma262/>; living specification inspected 2026-07-24 | ECMA terms | Semantics reference; no prose copied. |
| regex101 | <https://regex101.com/> and public parser documentation | Product/commercial terms | Product reference only. No source, branding, explanations, reference prose or visual design copied. |
| `recheck` | 4.5.0 | MIT | Metadata inspected; not installed or shipped. |
| `regexp-ast-analysis` | 0.7.1 | MIT | Metadata inspected; not installed or shipped. |
| RegexHub | Repository revision `d1d9f19d259745dfdd21935b9ef3e747b62b9bfb` | MIT | Potential future adversarial cases; no file copied. |
| RegexLib | Website inspected | Redistribution licence not established | No fixture copied. |
| RGXP.RU | Website inspected | Redistribution licence not established | No fixture copied. |
All v0.1.0, v0.2.0 and v0.3.0 conformance cases are project-authored. See
`tests/fixtures/README.md`.
All v0.1.0 through v0.4.0 conformance cases are project-authored. See
`tests/fixtures/README.md`. Runtime pack metadata and adjacent licence/notice
files are authoritative for the exact binary distribution.

32
docs/RELEASE.md
View File

@@ -18,13 +18,31 @@ on publication failure, replaces an existing exact version only with
`--force`, and produces:
```text
release/regex-tools-0.3.0.zip
release/regex-tools-0.3.0.zip.sha256
release/regex-tools-0.4.0.zip
release/regex-tools-0.4.0.zip.sha256
```
Before packaging, the engine verifier checks the closed PCRE2, Python and Java
file sets in `dist/`, including checksums, metadata, licences/notices, module
and WebAssembly identities and self-hosted relative asset references.
The release scan checks ASCII/UTF-8, UTF-16LE and UTF-16BE content in text,
JavaScript, WebAssembly and PDB-like assets for local Unix/macOS/Windows build
paths. Runtime virtual homes are allowed only for the named generated loaders.
Two byte-exact upstream prebuilts are hash-bound exceptions:
`engines/perl/emperl.data` retains WebPerl's historical build paths and
documentation examples, and `engines/ruby/ruby.wasm` retains ruby.wasm's
upstream build paths. Any byte or path change removes that exception and fails
the release scan; neither artifact is rewritten.
The existing v0.1.0 and v0.2.0 ZIPs and matching sidecars are historical
immutable artifacts. Creating v0.3.0 must not replace, rename or remove them.
Before packaging, the engine verifier checks the closed standalone-PCRE2, PHP,
Perl, Python, Ruby, Java, C++, Go, Rust and .NET file sets in `dist/`,
including checksums, metadata, licences/notices, module and WebAssembly
identities and self-hosted relative asset references. Browser/runtime smoke
tests cover every selectable profile; Scala compatibility must resolve to the
verified TeaVM Java pack while reporting that no Scala runtime is present.
The Python gate additionally requires its complete `SOURCE-MANIFEST.json`,
every preferred-form archive/Git/patch identity and all 22 independently
anchored base-runtime legal files. The C++ gate requires its exact traced
component/source manifest and six independently pinned legal files. The release
packager names every one of those files explicitly.
The existing v0.1.0, v0.2.0 and v0.3.0 ZIPs and matching sidecars are
historical immutable artifacts. Creating v0.4.0 must not replace, rename or
remove them.

81
docs/SECURITY.md
View File

@@ -23,10 +23,67 @@ untrusted.
identities. Pattern, subject, flags and replacement remain string data passed
to the fixed application bridge; no user-supplied Python source is evaluated.
Native code-point boundaries are checked before conversion to editor UTF-16.
Bounded replacement bytes use canonical base64 rather than content-dependent
JSON escaping; declared length, request cap and canonical UTF-8 are validated
before the output is exposed.
- Java starts only after the worker verifies the TeaVM bridge ABI, engine
identity and compiled self-test. It exposes only fixed bounded
`Pattern`/`Matcher` entry points; no JVM, Java class loading or user-supplied
JavaScript is available. The runtime is TeaVM, not OpenJDK.
- PHP starts only after the worker verifies PHP 8.5.8, PCRE2 10.44 and the
fixed bridge ABI. User values are JSON data written to a private virtual
filesystem; the bridge chooses a safe delimiter, allowlists modifiers and
invokes only fixed `preg_*` operations.
- The legacy Perl worker writes exact-key, byte-bounded JSON to a fixed MEMFS
request path and invokes only the constant `RegexTools::run_request()` entry
point. The bridge uses `no re 'eval'`, rejects `(?{...})` and `(??{...})`
outside escaped literals/classes, pre-tokenizes its non-executable
replacement grammar and streams bounded UTF-8 chunks from native match spans.
Successful output uses a separate fixed binary file, never the JSON response;
the worker verifies exact bytes, limit and canonical UTF-8 before decoding
it. Replacement stops at the match/capture-row bound and leaves the
unprocessed subject tail unchanged. WebPerl 0.09-beta's unmodified generated
loader contains one `eval(code)` site in an optional
JavaScript-interoperability import; the fixed bridge never loads or calls it.
The checksummed asset passes the browser CSP smoke without broad
`'unsafe-eval'`.
- Ruby evaluates only the fixed application bridge during worker startup.
Pattern, subject, flags, limits and replacement travel in one bounded,
length-prefixed file in the worker-local WASI memory filesystem. Native
`String#gsub`/`String#sub` block iteration cannot materialize expanded user
output: a fixed CRuby 4.0.0-compatible expander stops at the byte limit and
writes that prefix to a separate bounded file. The inert JSON response
contains metadata only and is assembled without per-code-point arrays. No
user value is interpolated into source or sent through ruby.wasm's
`RubyVM.wrap`/`toJS` JavaScript-eval conversion path. Unpaired browser
surrogates are rejected before UTF-8 encoding. Ruby cold startup passes the
strict-CSP release browser gate.
- C++ accepts only structured fixed Embind calls and is compiled with
Emscripten `-sDYNAMIC_EXECUTION=0`; the generated module is checked for
dynamic-source construction. Its libc++-compatible replacement expander
appends directly through the UTF-8 byte cap and is parity-checked at load.
Embind returns a structured JavaScript string rather than JSON, so there is
no escape amplification; the 64 MiB native buffer and its bounded UTF-16
conversion are the explicit peak-output copies.
- Go and Rust accept JSON data only through fixed exports. Each validates its
request and enforces pattern, subject, result and output caps inside
WebAssembly. Adapters measure exact encoded request bytes; the native request
cap covers worst-case sixfold JSON escaping for every accepted string field,
so control-heavy values retain the documented decoded field limits. Their
native-compatible replacement token streams stop at the output cap without
allocating a whole capture expansion. Bounded UTF-8 output crosses the JSON
bridge as fixed 4:3 base64 rather than content-dependent escaped text. Rust
additionally caps compiled-regex and DFA storage.
- .NET accepts bounded JSON only through fixed `[JSExport]` methods and never
evaluates user input as managed or JavaScript source. A native regex timeout
complements the supervisor timeout. Its fixed replacement parser streams
native-compatible token fragments through the UTF-8 byte cap, verifies
parity with `Match.Result` at load and transports output as canonical base64
to avoid JSON escaping amplification. The shipped .NET host contains runtime
debugging machinery, but normal production startup disables debugging and
is verified under the documented CSP.
- Scala compatibility shares the TeaVM Java bridge. It introduces no Scala
runtime, dynamic compiler or additional execution surface.
- Every bundled engine directory has a closed file set, deterministic
metadata/checksums and licence material verified both before installation and
in `dist/`. Runtime imports and fetches resolve only against the same-origin
@@ -86,12 +143,15 @@ untrusted.
closed. Subjects and results are ephemeral and never uploaded or persisted.
- No backend, uploads, telemetry, remote corpus URL, CDN, remote font,
arbitrary command line or unsafe HTML rendering exists. Application code
does not evaluate user-controlled JavaScript or Python. The generic generated
Pyodide loader contains an Emscripten dynamic-link branch that uses
JavaScript `eval`, but the shipped self-contained CPython/stdlib path does not
exercise it and loads under the CSP below without broad `'unsafe-eval'`.
does not evaluate user-controlled source in any language. The generic
generated Pyodide loader contains an Emscripten dynamic-link branch that
uses JavaScript `eval`, while the legacy WebPerl loader contains the optional
interop site described above. Neither shipped fixed execution path exercises
those branches, and both load under the CSP below without broad
`'unsafe-eval'`.
The engine-enabled build needs a CSP equivalent to:
Every enabled profile, including a cold Ruby start, passes the release CSP
equivalent to:
```text
default-src 'self';
@@ -107,11 +167,12 @@ base-uri 'none';
form-action 'none';
```
The Toolbox shell currently requires inline styles. Corpus ZIP generation,
PCRE2, Pyodide and TeaVM use self-hosted modules/assets and local workers under
the existing `worker-src` and `connect-src` policy. Broad `'unsafe-eval'` is not
required; `'wasm-unsafe-eval'` remains required for the two WebAssembly
runtimes.
The Toolbox shell currently requires inline styles. Corpus ZIP generation and
every engine pack use self-hosted modules/assets and local workers under the
existing `worker-src` and `connect-src` policy. Broad `'unsafe-eval'` must not
be added to accommodate an engine. The release gate blocks any future profile
that fails this strict-CSP browser matrix. `'wasm-unsafe-eval'` remains required
for the WebAssembly runtimes.
Report vulnerabilities privately to the repository owner before public issue
details are posted.