regex-tools/README.md

Regex Tools

Regex Tools is a production-oriented, local-first browser workbench for developing, understanding, testing and applying regular expressions. It is an independent static React/Vite application and a module of add·ideas Toolbox.

Patterns, replacement templates, test text and project files stay in the browser. There is no account, backend, upload, remote execution, telemetry, runtime CDN, remote font, or cloud-save service.

Current release — 0.2.0

The v0.2.0 release contains two real production engine slices: ECMAScript and PCRE2 10.47.

• Syntax acceptance: @eslint-community/regexpp 4.12.2, ECMAScript 2025 grammar; application-owned structural explanations are partial and feature-matrix tested.
• Execution engine: the current browser's native ECMAScript RegExp.
• Native and editor offsets: UTF-16 code units.
• PCRE2 execution and substitution: pinned official PCRE2 10.47 compiled to a self-hosted 8-bit WebAssembly pack with UTF/UCP always enabled.
• PCRE2 native UTF-8 byte ranges are retained and normalized to exact editor UTF-16 ranges; inputs with lone surrogates are rejected instead of rewritten.
• PCRE2 match-step, depth and heap limits are configurable inside hard caps.
• PCRE2 automatic callouts are copied through a separate ABI operation and displayed from a dedicated killable trace worker under 50,000-event and 10 MiB serialized-data hard caps.
• Parsing and execution run in separate workers.
• Actual execution can be terminated by killing its worker.
• The worker is recreated after timeout, crash or cancellation.
• Capture ranges use the engine's d indices result.
• Named, numbered, optional, empty and repeated-final captures are represented.
• Match, replacement, typed list/extraction, corpus/apply and unit-test modes are available.
• ECMAScript-specific advisory static risk findings, bounded dynamic growth probes and cold/warm benchmarks run locally; timing and growth execution is isolated in a separately killable worker.
• ECMAScript and PCRE2 can be compared through two exact, independently timed requests without translating patterns, flags or replacements. Incomplete or rejected sides are explicitly not comparable.
• The reviewed PCRE2 C17 generator emits exact UTF-8 byte arrays, configured limits and error handling for the PCRE2 10.47 8-bit API.
• A cancellable, deterministic subject minimizer verifies saved-test failures, capture-range failures, exact timeouts and ECMAScript/PCRE2 mismatches against the selected real workers under explicit evaluation and wall budgets.
• Deterministic ECMAScript test-case generation samples the normalized AST from an explicit seed, reports structural coverage and unsupported constructs, and labels cases only after verification by the selected real engine worker.
• Project JSON import/export and optional IndexedDB save are available.
• Interactive test text is not persisted by default.

No licensed parser is used, referenced at runtime, or required to build the project. This is the community build and will grow flavour by flavour using reviewed open-source syntax providers and actual named engines.

The two trees

The explanation tree is a deterministic structural explanation generated from the application-owned normalized syntax tree. It is not an engine trace.

The extraction tree contains actual engine matches and captures. Its hierarchy uses syntactic capture nesting while preserving actual engine ranges. ECMAScript exposes only the final retained capture of a repeated group; Regex Tools says so and never invents capture history.

Browser ECMAScript exposes no actual engine trace. PCRE2 has a separate bounded automatic-callout viewer. Its event fields are reported by PCRE2, while “forward”, “same position” and “apparent backtrack” are visibly derived only from adjacent reported subject positions. The callout stream is not described as a complete record of every internal engine action.

Workbench modes

• Match — live highlighting, extraction tree and bounded capture table.
• Replace — engine-native bounded substitution for ECMAScript and PCRE2; output completeness is reported separately from bounded per-match original/result, changed-range and token-contribution previews that synchronize the replacement, pattern, subject and output editors.
• List — non-executable typed templates with text, CSV, JSON and NDJSON export; export is disabled when engine collection or bounded rendering is incomplete.
• Corpus / Apply — bounded multi-file UTF-8 and pasted-text input processed sequentially in killable workers. Whole-document and independent-line semantics are explicit; line apply preserves original separators. Progress, cancellation, bounded capture samples, replacement previews, per-document exact/partial summaries, content-free JSON/CSV/NDJSON reports and privacy-gated exact-output ZIP/downloads are available. Original files are never modified.
• Unit tests — editable and cloneable versioned presence, count, full-match, capture, replacement, timing and expected-timeout assertions. Capture a complete current result, select only cases to rerun, filter failures, and import/export a standalone validated JSON suite; execution stays in isolated workers.

The auxiliary PCRE trace, ECMAScript Analysis, Compare & code, Generate cases, Minimize and Format workspaces are separate from normal matching, replacement, tests and corpus execution. Trace collection is never included in benchmark measurements. Comparison retains native ranges while aligning actual results by normalized editor UTF-16 ranges. Agreement is described only as evidence for the current subject, never as proof of pattern equivalence. Case generation is bounded sampling, not a proof of language coverage. Version 1 supports only a complete accepted ECMAScript normalized AST; the partial PCRE2 provider is reported as unavailable rather than passed through or relabeled. See docs/GENERATED_CASES.md. Minimization verifies every accepted candidate against the exact selected worker request and claims only local minimality under its documented Unicode-scalar transforms; see docs/MINIMIZATION.md. Formatting is currently ECMAScript-only and changes only parser-reported literal slash/control representations. Apply requires independent reparse, actual-engine current-subject/replacement comparison, every applicable exact unit test and explicit confirmation; see docs/FORMATTING.md.

Flags and iteration

Pattern and flags are stored separately; delimiters are presentation only. ECMAScript preserves native g and y; PCRE2 offers the application-level g iteration flag plus i, m, s, x, U and J. PCRE2 UTF/UCP mode is mandatory because browser strings are Unicode and partial-byte ranges cannot be represented safely in the editor. The optional Scan all action adds only an internal iteration flag to that request and does not change saved flags. ECMAScript may add internal d to obtain exact indices without changing match semantics.

Zero-length global iteration advances using the selected Unicode semantics and cannot loop forever.

Limits and timeout behavior

Important defaults:

Limit	Default
Live parse debounce	120 ms
Live execution debounce	220 ms
Live execution timeout	250 ms
Manual execution timeout	2 s
Advanced maximum timeout	10 s
Pattern hard limit	64 Ki UTF-16 units
Interactive subject hard limit	16 MiB
Corpus documents	256
Per corpus document	16 MiB
Aggregate corpus input	256 MiB
Independent corpus lines	100,000
Aggregate corpus matches	100,000
Capture summaries per document	32
Samples per capture	3
Retained applied output	64 MiB
Applied ZIP	68 MiB
Corpus batch wall time	5 min
Replacement template	64 Ki UTF-16 units
List template	16 Ki UTF-16 units
Per-match replacement preview	200 matches
Replacement contributions	2,000
Maximum matches	10,000
Maximum capture groups	1,000
Maximum capture rows	100,000
Replacement preview	64 MiB
Project JSON document	32 MiB UTF-8
Standalone test-suite JSON	32 MiB UTF-8
In-memory unit-test suite	32 MiB UTF-8
Tests per suite	1,000
Retained message per test	2 KiB UTF-8
Unit-test suite wall time	60 s
PCRE2 trace events	50,000
PCRE2 serialized trace	10 MiB
Comparison retained differences	5,000
Comparison retained alignments	2,000
Static analysis traversal	20,000 nodes
Static analysis findings	100
Growth probe steps	24
Analysis aggregate wall time	30 s
Generated verified cases	48
Generated candidate attempts	192
Generation aggregate wall time	15 s
Generation aggregate candidates	1 MiB
Minimizer subject	1 MiB UTF-8
Minimizer candidate evaluations	2,000
Minimizer aggregate wall time	60 s
Formatter validation tests	1,000
Formatter test wall time	60 s
Retained minimizer history	200

Analysis defaults to three warm-up plus 15 measured samples (caps: 100 and 1,000), with a 2-second per-sample worker limit that can be raised to 10 seconds. Growth generation defaults to 1 MiB, preflights the central 16 MiB subject cap and rejects more than 10,000,000 repetitions. Replacement timing is skipped above an 8 Mi UTF-16-unit output estimate.

A timeout terminates actual execution. It is reported as timed out, never as “no match”. The next request creates a fresh worker.

Replacement applies exactly the authoritative matches returned under the match and capture-row limits. If either collection limit is reached, later subject text remains unchanged and the output is explicitly reported as partial. The separate output-byte limit reports a truncated output preview.

Project persistence

Project schema version 1 stores flavour, pattern, flags, options, replacement, list template, unit tests (including explicit scan-all behavior and supported per-test timeouts) and selected UI state. Import rebuilds a validated plain object; it never evaluates code and pauses live execution until review. Workbench-wide manual timeout changes are not persisted, and unsupported project resource overrides are rejected.

Interactive test text is excluded from local save and export unless explicitly enabled. Unit-test definitions are deliberate project data, but their subjects require separate explicit opt-ins for JSON export and IndexedDB save. Standalone test-suite JSON likewise omits subjects by default, is schema- and field-validated on import, and appends at most 1,000 total tests. Add, update, clone and append-import also preflight the complete in-memory suite against a 32 MiB aggregate bound. Added generated tests retain their validated generator version, seed, candidate ID and intended outcome; editing one removes that provenance. Schema v1 can remember that the corpus workspace was selected, but corpus documents, contents, outputs and results are always ephemeral and excluded from JSON and IndexedDB. Imports that try to inject those fields are rejected. Import, export and IndexedDB save enforce one 32 MiB aggregate UTF-8 JSON-document limit in addition to per-field limits. The latest local save is read through an updatedAt index cursor; loading it does not materialize the complete project store.

Development

Requirements: Node.js 22 or later and npm 11.

npm install
npm run dev

Quality and release commands:

npm run typecheck
npm run lint
npm run format:check
npm test
npm run test:conformance
npm run test:browser
npm run engines:build
npm run engines:verify
npm run engines:pcre2:build -- --source-dir /path/to/pcre2 --emcc /path/to/emcc
npm run engines:pcre2:verify
npm run engines:pcre2:install
npm run build
npm run toolbox:check
npm run check
npm run release:artifact

The normal build performs no network download. vite.config.ts uses base: "./"; all workers and assets are self-hosted and nested-path safe.

Toolbox SDK and Portal

The application uses Toolbox SDK 0.2.2 AppShell. Its canonical typed/static manifest source is src/toolbox/manifest.source.json; generation fails if its version or source identity drifts.

The deterministic release ZIP can be consumed by Toolbox Portal using an exact artifact URL, manifest ID, version and SHA-256. Portal builds consume the artifact; they do not build Regex Tools source.

The PCRE2 worker loads self-hosted JavaScript and WebAssembly only. Deployment must serve .wasm as application/wasm and allow WebAssembly compilation in script-src; see docs/PORTAL_REQUIREMENTS.md.

Flavour roadmap

PCRE2 10.47 is the second implemented flavour. Its application-owned ABI provides bounded compilation, matching, native substitution and separate automatic-callout tracing, copied caller-owned records, deterministic allocation cleanup and explicit result completeness. The exact source/toolchain build is reproducible offline and the verified pack is bundled with its metadata, checksums and licence.

Generated-case version 1 is ECMAScript-only. It traverses the complete application-owned ECMAScript AST under node, depth, attempt, byte, repetition and wall-time caps, then verifies every retained label through the actual browser RegExp adapter. PCRE2 generation remains unavailable until its syntax provider can expose a complete normalized tree.

Pattern formatting is likewise ECMAScript-only. It consumes the complete regexpp token model and does not reinterpret the partial PCRE2 structure. PCRE2 formatting remains unavailable until a complete grammar-backed implementation passes equivalent idempotence, reparse, engine and test gates.

The first advertised generated-code target is the PCRE2 10.47 8-bit C API. Its deterministic match and replacement fixtures compile and execute against the exact official release. See docs/COMPARISON_AND_CODEGEN.md.

Python, Go, Rust, .NET, Java and legacy PCRE follow only through their actual named runtimes. No flavour is emulated through JavaScript and relabelled.

Licensing

Regex Tools original source is GPL-3.0-or-later, copyright © 2026 Albrecht Degering. Shipped dependencies retain their own compatible licences and notices; see THIRD_PARTY_NOTICES.md and LICENSES/.

No regex101 application source, branding, generated explanation prose, reference text or visual design is copied. RegexLib and RGXP.RU fixtures are not redistributed because a suitable fixture licence was not established. No RegexHub fixture is shipped in v0.2.0.

Known limitations

• ECMAScript syntax coverage follows regexpp's ECMAScript 2025 grammar; a later browser proposal may execute before the provider understands it.
• Exact consumed-length bounds for ECMAScript Unicode properties of strings remain unknown and are reported as such rather than guessed.
• regexpp does not provide tolerant AST recovery. Malformed input receives an exact reported error position as a zero-width range and an application-owned error node, not a fabricated multi-character span or partial provider AST.
• Native JavaScript compile errors do not consistently expose a machine-readable source offset; the provider diagnostic supplies the range.
• Capture history is unavailable. Browser ECMAScript exposes no engine trace; PCRE2 exposes bounded automatic callouts, not a complete internal trace. Movement classifications are derived and labelled as such.
• Static risk heuristics and generated-input observations apply only to ECMAScript 2025 and are advisory. They are never proof of safety or of a pattern’s general time complexity; analysis settings and results are ephemeral.
• PCRE2 structural explanations are deliberately partial. The application recognizes common capture forms and PCRE2-only constructs for reference, but the actual PCRE2 compiler is authoritative. Branch-reset capture numbering is omitted from the explanation rather than guessed.
• Capture-table rendering is paged at 200 rows while aggregate collection is bounded separately.
• Explanation/extraction trees and editor decorations show bounded 2,000-node previews with explicit totals; normalized results and table/export data stay separate from those render-only caps.
• Extraction-tree values show at most 100 UTF-16 units and capture-table cells at most 512. UI-only shortening shows the exact range length; engine-clipped values are labelled as prefixes of that range. Copy and eligible JSON export continue to use the separately bounded engine value.
• Replacement parsing is complete below its 64 Ki input limit, while editor marks, token rows and diagnostics have explicit presentation caps. List token chips are similarly bounded, and list rendering stops at 250,000 token evaluations.
• Per-match replacement presentation renders at most 200 matches and 2,000 token contributions under a 50,000 token-evaluation budget. Bounded replacement output remains available separately and is labelled partial if match collection reached a limit.
• Result-value previews are capped per value and overall. List export is disabled whenever a bounded preview would otherwise be mistaken for complete data.
• Unit-test failure messages retain at most 2 KiB UTF-8 per case, so 1,000 results stay below the 2 MiB log budget. Oversized exact replacement results offer a lightweight should-match draft when one is valid and fits the suite.
• Corpus input accepts recognized UTF-8 text only. It does not auto-detect legacy encodings or recursively traverse directories.
• Corpus capture extraction is a bounded per-document summary: full match plus at most 31 capture groups, three 160-unit samples each. Content-free reports include counts and clipping metadata, never the samples themselves.
• Generated cases deliberately sample only ECMAScript and report backreferences, lookaround, boundaries, complex Unicode sets and other unsupported constraints instead of claiming exhaustive coverage.
• Pattern formatting is intentionally narrow and ECMAScript-only. Its bounded source/candidate and unit-test validation is evidence for the checked snapshots, not a proof of equivalence over all subjects.

Architecture, security, performance, provenance and flavour details live in docs/.