regex-tools/README.md

Regex Tools

Regex Tools is a production-oriented, local-first browser workbench for developing, understanding, testing and applying regular expressions. It is an independent static React/Vite application and a module of add·ideas Toolbox.

Patterns, replacement templates, test text and project files stay in the browser. There is no account, backend, upload, remote execution, telemetry, runtime CDN, remote font, or cloud-save service.

Current release — 0.3.0

The v0.3.0 release contains four executable engine slices:

• the current browser's native ECMAScript RegExp;
• official PCRE2 10.47 as a self-hosted 8-bit WebAssembly pack;
• CPython 3.14.2 re in the self-hosted Pyodide 314.0.3 runtime; and
• TeaVM 0.15.0's java.util.regex class-library implementation as a self-hosted ES2015 module.

The Java flavour is TeaVM class-library code compiled to JavaScript. It is not OpenJDK and is not advertised as an OpenJDK compatibility oracle.

• Syntax acceptance: @eslint-community/regexpp 4.12.2, ECMAScript 2025 grammar; application-owned structural explanations are partial and feature-matrix tested.
• ECMAScript native and editor offsets are UTF-16 code units.
• PCRE2 execution and substitution: pinned official PCRE2 10.47 compiled to a self-hosted 8-bit WebAssembly pack with UTF/UCP always enabled.
• PCRE2 native UTF-8 byte ranges are retained and normalized to exact editor UTF-16 ranges; inputs with lone surrogates are rejected instead of rewritten.
• PCRE2 match-step, depth and heap limits are configurable inside hard caps.
• PCRE2 automatic callouts are copied through a separate ABI operation and displayed from a dedicated killable trace worker under 50,000-event and 10 MiB serialized-data hard caps.
• Python compilation, matching and replacement use the bundled CPython re module. Native code-point ranges are retained and normalized to exact editor UTF-16 ranges.
• Java compilation, matching and replacement use TeaVM's java.util.regex.Pattern and Matcher. Native ranges are UTF-16 code units.
• TeaVM 0.15.0 replacement supports its actual single-digit $n subset. $12 is therefore $1 followed by literal 2; Java SE ${name} replacement is reported as unavailable rather than presented as OpenJDK behavior.
• Python and Java use deliberately partial application-owned lexical syntax providers; their actual runtime compiler remains authoritative.
• Parsing and execution run in separate workers.
• Actual execution can be terminated by killing its worker.
• The worker is recreated after timeout, crash or cancellation.
• ECMAScript capture ranges use the engine's d indices result; the other flavours return their native range records through bounded bridges.
• Named, numbered, optional, empty and repeated-final captures are represented.
• Match, replacement, typed list/extraction, corpus/apply and unit-test modes are available.
• ECMAScript-specific advisory static risk findings, bounded dynamic growth probes and cold/warm benchmarks run locally; timing and growth execution is isolated in a separately killable worker.
• ECMAScript and PCRE2 can be compared through two exact, independently timed requests without translating patterns, flags or replacements. Incomplete or rejected sides are explicitly not comparable.
• The reviewed PCRE2 C17 generator emits exact UTF-8 byte arrays, configured limits and error handling for the PCRE2 10.47 8-bit API.
• A cancellable, deterministic subject minimizer verifies saved-test failures, capture-range failures, exact timeouts and ECMAScript/PCRE2 mismatches against the selected real workers under explicit evaluation and wall budgets.
• Deterministic ECMAScript test-case generation samples the normalized AST from an explicit seed, reports structural coverage and unsupported constructs, and labels cases only after verification by the selected real engine worker.
• Project JSON import/export and optional IndexedDB save are available.
• Interactive test text is not persisted by default.

No licensed parser is used, referenced at runtime, or required to build the project. This is the community build and will grow flavour by flavour using reviewed open-source syntax providers and actual named engines.

The two trees

The explanation tree is a deterministic structural explanation generated from the application-owned normalized syntax tree. It is not an engine trace.

The extraction tree contains actual engine matches and captures. Its hierarchy uses syntactic capture nesting while preserving actual engine ranges. ECMAScript exposes only the final retained capture of a repeated group; Regex Tools says so and never invents capture history.

Browser ECMAScript exposes no actual engine trace. PCRE2 has a separate bounded automatic-callout viewer. Its event fields are reported by PCRE2, while “forward”, “same position” and “apparent backtrack” are visibly derived only from adjacent reported subject positions. The callout stream is not described as a complete record of every internal engine action.

Workbench modes

• Match — live highlighting, extraction tree and bounded capture table.
• Replace — engine-native bounded substitution for all four executable flavours; output completeness is reported separately from bounded per-match original/result, changed-range and token-contribution previews that synchronize the replacement, pattern, subject and output editors.
• List — non-executable typed templates with text, CSV, JSON and NDJSON export; export is disabled when engine collection or bounded rendering is incomplete.
• Corpus / Apply — bounded multi-file UTF-8 and pasted-text input processed sequentially in killable workers. Whole-document and independent-line semantics are explicit; line apply preserves original separators. Progress, cancellation, bounded capture samples, replacement previews, per-document exact/partial summaries, content-free JSON/CSV/NDJSON reports and privacy-gated exact-output ZIP/downloads are available. Original files are never modified.
• Unit tests — editable and cloneable versioned presence, count, full-match, capture, replacement, timing and expected-timeout assertions. Capture a complete current result, select only cases to rerun, filter failures, and import/export a standalone validated JSON suite; execution stays in isolated workers.

The auxiliary PCRE trace, ECMAScript Analysis, Compare & code, Generate cases, Minimize and Format workspaces are separate from normal matching, replacement, tests and corpus execution. Trace collection is never included in benchmark measurements. Comparison retains native ranges while aligning actual results by normalized editor UTF-16 ranges. Agreement is described only as evidence for the current subject, never as proof of pattern equivalence. Case generation is bounded sampling, not a proof of language coverage. Version 1 supports only a complete accepted ECMAScript normalized AST; the partial PCRE2 provider is reported as unavailable rather than passed through or relabeled. See docs/GENERATED_CASES.md. Minimization verifies every accepted candidate against the exact selected worker request and claims only local minimality under its documented Unicode-scalar transforms; see docs/MINIMIZATION.md. Formatting is currently ECMAScript-only and changes only parser-reported literal slash/control representations. Apply requires independent reparse, actual-engine current-subject/replacement comparison, every applicable exact unit test and explicit confirmation; see docs/FORMATTING.md.

Flags and iteration

Pattern and flags are stored separately; delimiters are presentation only. ECMAScript preserves native g and y; PCRE2 offers the application-level g iteration flag plus i, m, s, x, U and J. PCRE2 UTF/UCP mode is mandatory because browser strings are Unicode and partial-byte ranges cannot be represented safely in the editor. Python exposes application-level g plus CPython re flags a, i, m, s and x. Java exposes application-level g plus TeaVM Pattern flags i, d, m, s, u, x and the documented U compatibility request. The optional Scan all action adds only an internal iteration request and does not change saved flags. ECMAScript may add internal d to obtain exact indices without changing match semantics.

Zero-length global iteration advances using the selected Unicode semantics and cannot loop forever.

Limits and timeout behavior

Important defaults:

Limit	Default
Live parse debounce	120 ms
Live execution debounce	220 ms
Live execution timeout	250 ms
Manual execution timeout	2 s
Advanced maximum timeout	10 s
Pattern hard limit	64 Ki UTF-16 units
Interactive subject hard limit	16 MiB
Corpus documents	256
Per corpus document	16 MiB
Aggregate corpus input	256 MiB
Independent corpus lines	100,000
Aggregate corpus matches	100,000
Capture summaries per document	32
Samples per capture	3
Retained applied output	64 MiB
Applied ZIP	68 MiB
Corpus batch wall time	5 min
Replacement template	64 Ki UTF-16 units
List template	16 Ki UTF-16 units
Per-match replacement preview	200 matches
Replacement contributions	2,000
Maximum matches	10,000
Maximum capture groups	1,000
Maximum capture rows	100,000
Replacement preview	64 MiB
Project JSON document	32 MiB UTF-8
Standalone test-suite JSON	32 MiB UTF-8
In-memory unit-test suite	32 MiB UTF-8
Tests per suite	1,000
Retained message per test	2 KiB UTF-8
Unit-test suite wall time	60 s
PCRE2 trace events	50,000
PCRE2 serialized trace	10 MiB
Comparison retained differences	5,000
Comparison retained alignments	2,000
Static analysis traversal	20,000 nodes
Static analysis findings	100
Growth probe steps	24
Analysis aggregate wall time	30 s
Generated verified cases	48
Generated candidate attempts	192
Generation aggregate wall time	15 s
Generation aggregate candidates	1 MiB
Minimizer subject	1 MiB UTF-8
Minimizer candidate evaluations	2,000
Minimizer aggregate wall time	60 s
Formatter validation tests	1,000
Formatter test wall time	60 s
Retained minimizer history	200

Analysis defaults to three warm-up plus 15 measured samples (caps: 100 and 1,000), with a 2-second per-sample worker limit that can be raised to 10 seconds. Growth generation defaults to 1 MiB, preflights the central 16 MiB subject cap and rejects more than 10,000,000 repetitions. Replacement timing is skipped above an 8 Mi UTF-16-unit output estimate.

A timeout terminates actual execution. It is reported as timed out, never as “no match”. The next request creates a fresh worker.

Replacement applies exactly the authoritative matches returned under the match and capture-row limits. If either collection limit is reached, later subject text remains unchanged and the output is explicitly reported as partial. The separate output-byte limit reports a truncated output preview.

Project persistence

Project schema version 1 stores flavour, pattern, flags, options, replacement, list template, unit tests (including explicit scan-all behavior and supported per-test timeouts) and selected UI state. Import rebuilds a validated plain object; it never evaluates code and pauses live execution until review. Workbench-wide manual timeout changes are not persisted, and unsupported project resource overrides are rejected.

Interactive test text is excluded from local save and export unless explicitly enabled. Unit-test definitions are deliberate project data, but their subjects require separate explicit opt-ins for JSON export and IndexedDB save. Standalone test-suite JSON likewise omits subjects by default, is schema- and field-validated on import, and appends at most 1,000 total tests. Add, update, clone and append-import also preflight the complete in-memory suite against a 32 MiB aggregate bound. Added generated tests retain their validated generator version, seed, candidate ID and intended outcome; editing one removes that provenance. Schema v1 can remember that the corpus workspace was selected, but corpus documents, contents, outputs and results are always ephemeral and excluded from JSON and IndexedDB. Imports that try to inject those fields are rejected. Import, export and IndexedDB save enforce one 32 MiB aggregate UTF-8 JSON-document limit in addition to per-field limits. The latest local save is read through an updatedAt index cursor; loading it does not materialize the complete project store.

Development

Requirements: Node.js 22 or later and npm 11.

npm install
npm run dev

Quality and release commands:

npm run typecheck
npm run lint
npm run format:check
npm test
npm run test:conformance
npm run test:browser
npm run engines:build
npm run engines:verify
npm run engines:pcre2:build -- --source-dir /path/to/pcre2 --emcc /path/to/emcc
npm run engines:pcre2:verify
npm run engines:pcre2:install
node scripts/python-engine-pack.mjs node_modules/pyodide .engine-build/python
node scripts/install-python-engine.mjs .engine-build/python
node scripts/java-engine-pack.mjs
node scripts/install-java-engine.mjs
npm run build
npm run toolbox:check
npm run check
npm run release:artifact

The normal build performs no network download. vite.config.ts uses base: "./"; all workers and assets are self-hosted and nested-path safe.

Toolbox SDK and Portal

The application uses Toolbox SDK 0.2.2 AppShell. Its canonical typed/static manifest source is src/toolbox/manifest.source.json; generation fails if its version or source identity drifts.

The deterministic release ZIP can be consumed by Toolbox Portal using an exact artifact URL, manifest ID, version and SHA-256. Portal builds consume the artifact; they do not build Regex Tools source.

The PCRE2 and Python workers load only self-hosted JavaScript, WebAssembly and Python-standard-library assets; Java loads its self-hosted ES module. Deployment must serve .wasm as application/wasm and allow WebAssembly compilation in script-src; see docs/PORTAL_REQUIREMENTS.md.

Flavour support and boundaries

PCRE2's application-owned ABI provides bounded compilation, matching, native substitution and separate automatic-callout tracing, copied caller-owned records, deterministic allocation cleanup and explicit result completeness. The exact source/toolchain build is reproducible offline and the verified pack is bundled with its metadata, checksums and licence.

The Python slice executes actual CPython 3.14.2 re in Pyodide 314.0.3; it does not translate a Python pattern into ECMAScript. Pyodide startup is a separate worker-load operation and is excluded from the normal execution deadline.

The Java slice executes TeaVM 0.15.0's own java.util.regex class library. It does not translate a Java pattern into ECMAScript, but it is also not OpenJDK: in particular, TeaVM 0.15.0 does not implement Pattern.UNICODE_CHARACTER_CLASS. The UI reports the U flag as a compatibility request and preserves TeaVM's predefined-class and word-boundary behaviour.

Generated-case version 1 is ECMAScript-only. It traverses the complete application-owned ECMAScript AST under node, depth, attempt, byte, repetition and wall-time caps, then verifies every retained label through the actual browser RegExp adapter. PCRE2 generation remains unavailable until its syntax provider can expose a complete normalized tree.

Pattern formatting is likewise ECMAScript-only. It consumes the complete regexpp token model and does not reinterpret the partial PCRE2, Python or Java structures. Those flavours remain unavailable until complete grammar-backed implementations pass equivalent idempotence, reparse, engine and test gates.

The first advertised generated-code target is the PCRE2 10.47 8-bit C API. Its deterministic match and replacement fixtures compile and execute against the exact official release. See docs/COMPARISON_AND_CODEGEN.md.

Go, Rust, .NET and legacy PCRE follow only through their actual named runtimes. No flavour is emulated through JavaScript and relabelled.

Licensing

Regex Tools original source is GPL-3.0-or-later, copyright © 2026 Albrecht Degering. Shipped dependencies retain their own compatible licences and notices; see THIRD_PARTY_NOTICES.md and LICENSES/.

No regex101 application source, branding, generated explanation prose, reference text or visual design is copied. RegexLib and RGXP.RU fixtures are not redistributed because a suitable fixture licence was not established. No RegexHub fixture is shipped in v0.3.0.

Known limitations

• ECMAScript syntax coverage follows regexpp's ECMAScript 2025 grammar; a later browser proposal may execute before the provider understands it.
• Exact consumed-length bounds for ECMAScript Unicode properties of strings remain unknown and are reported as such rather than guessed.
• regexpp does not provide tolerant AST recovery. Malformed input receives an exact reported error position as a zero-width range and an application-owned error node, not a fabricated multi-character span or partial provider AST.
• Native JavaScript compile errors do not consistently expose a machine-readable source offset; the provider diagnostic supplies the range.
• Capture history is unavailable. Browser ECMAScript exposes no engine trace; PCRE2 exposes bounded automatic callouts, not a complete internal trace. Movement classifications are derived and labelled as such.
• Static risk heuristics and generated-input observations apply only to ECMAScript 2025 and are advisory. They are never proof of safety or of a pattern’s general time complexity; analysis settings and results are ephemeral.
• PCRE2 structural explanations are deliberately partial. The application recognizes common capture forms and PCRE2-only constructs for reference, but the actual PCRE2 compiler is authoritative. Branch-reset capture numbering is omitted from the explanation rather than guessed.
• Python and Java structural explanations are deliberately partial lexical models. CPython re and TeaVM Pattern respectively remain authoritative for compilation, group count and matching.
• Python reports native Unicode code-point offsets and converts them at valid boundaries to editor UTF-16 ranges. Java and ECMAScript report native UTF-16 offsets; PCRE2 reports native UTF-8 bytes. Native units remain visible.
• TeaVM java.util.regex is not OpenJDK. Its U compatibility request does not add OpenJDK's UNICODE_CHARACTER_CLASS semantics.
• Capture-table rendering is paged at 200 rows while aggregate collection is bounded separately.
• Explanation/extraction trees and editor decorations show bounded 2,000-node previews with explicit totals; normalized results and table/export data stay separate from those render-only caps.
• Extraction-tree values show at most 100 UTF-16 units and capture-table cells at most 512. UI-only shortening shows the exact range length; engine-clipped values are labelled as prefixes of that range. Copy and eligible JSON export continue to use the separately bounded engine value.
• Replacement parsing is complete below its 64 Ki input limit, while editor marks, token rows and diagnostics have explicit presentation caps. List token chips are similarly bounded, and list rendering stops at 250,000 token evaluations.
• Per-match replacement presentation renders at most 200 matches and 2,000 token contributions under a 50,000 token-evaluation budget. Bounded replacement output remains available separately and is labelled partial if match collection reached a limit.
• Result-value previews are capped per value and overall. List export is disabled whenever a bounded preview would otherwise be mistaken for complete data.
• Unit-test failure messages retain at most 2 KiB UTF-8 per case, so 1,000 results stay below the 2 MiB log budget. Oversized exact replacement results offer a lightweight should-match draft when one is valid and fits the suite.
• Corpus input accepts recognized UTF-8 text only. It does not auto-detect legacy encodings or recursively traverse directories.
• Corpus capture extraction is a bounded per-document summary: full match plus at most 31 capture groups, three 160-unit samples each. Content-free reports include counts and clipping metadata, never the samples themselves.
• Generated cases deliberately sample only ECMAScript and report backreferences, lookaround, boundaries, complex Unicode sets and other unsupported constraints instead of claiming exhaustive coverage.
• Pattern formatting is intentionally narrow and ECMAScript-only. Its bounded source/candidate and unit-test validation is evidence for the checked snapshots, not a proof of equivalence over all subjects.
• Cross-flavour comparison, PCRE2 C generation and subject minimization remain scoped to their explicitly supported ECMAScript/PCRE2 paths; selecting Python or Java does not silently substitute one of those engines.

Architecture, security, performance, provenance and flavour details live in docs/.