372 lines
19 KiB
Markdown
372 lines
19 KiB
Markdown
# Regex Tools
|
||
|
||
Regex Tools is a production-oriented, local-first browser workbench for
|
||
developing, understanding, testing and applying regular expressions. It is an
|
||
independent static React/Vite application and a module of add·ideas Toolbox.
|
||
|
||
Patterns, replacement templates, test text and project files stay in the
|
||
browser. There is no account, backend, upload, remote execution, telemetry,
|
||
runtime CDN, remote font, or cloud-save service.
|
||
|
||
## Current release — 0.2.0
|
||
|
||
The v0.2.0 release contains two real production engine slices: ECMAScript and
|
||
PCRE2 10.47.
|
||
|
||
- Syntax acceptance: `@eslint-community/regexpp` 4.12.2, ECMAScript 2025
|
||
grammar; application-owned structural explanations are partial and
|
||
feature-matrix tested.
|
||
- Execution engine: the current browser's native ECMAScript `RegExp`.
|
||
- Native and editor offsets: UTF-16 code units.
|
||
- PCRE2 execution and substitution: pinned official PCRE2 10.47 compiled to a
|
||
self-hosted 8-bit WebAssembly pack with UTF/UCP always enabled.
|
||
- PCRE2 native UTF-8 byte ranges are retained and normalized to exact editor
|
||
UTF-16 ranges; inputs with lone surrogates are rejected instead of rewritten.
|
||
- PCRE2 match-step, depth and heap limits are configurable inside hard caps.
|
||
- PCRE2 automatic callouts are copied through a separate ABI operation and
|
||
displayed from a dedicated killable trace worker under 50,000-event and
|
||
10 MiB serialized-data hard caps.
|
||
- Parsing and execution run in separate workers.
|
||
- Actual execution can be terminated by killing its worker.
|
||
- The worker is recreated after timeout, crash or cancellation.
|
||
- Capture ranges use the engine's `d` indices result.
|
||
- Named, numbered, optional, empty and repeated-final captures are represented.
|
||
- Match, replacement, typed list/extraction, corpus/apply and unit-test modes
|
||
are available.
|
||
- ECMAScript-specific advisory static risk findings, bounded dynamic growth
|
||
probes and cold/warm benchmarks run locally; timing and growth execution is
|
||
isolated in a separately killable worker.
|
||
- ECMAScript and PCRE2 can be compared through two exact, independently timed
|
||
requests without translating patterns, flags or replacements. Incomplete or
|
||
rejected sides are explicitly not comparable.
|
||
- The reviewed PCRE2 C17 generator emits exact UTF-8 byte arrays, configured
|
||
limits and error handling for the PCRE2 10.47 8-bit API.
|
||
- A cancellable, deterministic subject minimizer verifies saved-test failures,
|
||
capture-range failures, exact timeouts and ECMAScript/PCRE2 mismatches against
|
||
the selected real workers under explicit evaluation and wall budgets.
|
||
- Deterministic ECMAScript test-case generation samples the normalized AST from
|
||
an explicit seed, reports structural coverage and unsupported constructs, and
|
||
labels cases only after verification by the selected real engine worker.
|
||
- Project JSON import/export and optional IndexedDB save are available.
|
||
- Interactive test text is not persisted by default.
|
||
|
||
No licensed parser is used, referenced at runtime, or required to build the
|
||
project. This is the community build and will grow flavour by flavour using
|
||
reviewed open-source syntax providers and actual named engines.
|
||
|
||
## The two trees
|
||
|
||
The **explanation tree** is a deterministic structural explanation generated
|
||
from the application-owned normalized syntax tree. It is not an engine trace.
|
||
|
||
The **extraction tree** contains actual engine matches and captures. Its
|
||
hierarchy uses syntactic capture nesting while preserving actual engine ranges.
|
||
ECMAScript exposes only the final retained capture of a repeated group; Regex
|
||
Tools says so and never invents capture history.
|
||
|
||
Browser ECMAScript exposes no actual engine trace. PCRE2 has a separate bounded
|
||
automatic-callout viewer. Its event fields are reported by PCRE2, while
|
||
“forward”, “same position” and “apparent backtrack” are visibly derived only
|
||
from adjacent reported subject positions. The callout stream is not described
|
||
as a complete record of every internal engine action.
|
||
|
||
## Workbench modes
|
||
|
||
- **Match** — live highlighting, extraction tree and bounded capture table.
|
||
- **Replace** — engine-native bounded substitution for ECMAScript and PCRE2;
|
||
output completeness is reported separately from bounded per-match
|
||
original/result, changed-range and token-contribution previews that
|
||
synchronize the replacement, pattern, subject and output editors.
|
||
- **List** — non-executable typed templates with text, CSV, JSON and NDJSON
|
||
export; export is disabled when engine collection or bounded rendering is
|
||
incomplete.
|
||
- **Corpus / Apply** — bounded multi-file UTF-8 and pasted-text input processed
|
||
sequentially in killable workers. Whole-document and independent-line
|
||
semantics are explicit; line apply preserves original separators. Progress,
|
||
cancellation, bounded capture samples, replacement previews, per-document
|
||
exact/partial summaries, content-free JSON/CSV/NDJSON reports and
|
||
privacy-gated exact-output ZIP/downloads are available. Original files are
|
||
never modified.
|
||
- **Unit tests** — editable and cloneable versioned presence, count, full-match,
|
||
capture, replacement, timing and expected-timeout assertions. Capture a
|
||
complete current result, select only cases to rerun, filter failures, and
|
||
import/export a standalone validated JSON suite; execution stays in isolated
|
||
workers.
|
||
|
||
The auxiliary **PCRE trace**, ECMAScript **Analysis**, **Compare & code**,
|
||
**Generate cases**, **Minimize** and **Format** workspaces are separate from
|
||
normal matching, replacement, tests and corpus execution. Trace collection is
|
||
never included in benchmark measurements.
|
||
Comparison retains native ranges while aligning actual results by normalized
|
||
editor UTF-16 ranges. Agreement is described only as evidence for the current
|
||
subject, never as proof of pattern equivalence.
|
||
Case generation is bounded sampling, not a proof of language coverage. Version
|
||
1 supports only a complete accepted ECMAScript normalized AST; the partial
|
||
PCRE2 provider is reported as unavailable rather than passed through or
|
||
relabeled. See
|
||
[`docs/GENERATED_CASES.md`](docs/GENERATED_CASES.md).
|
||
Minimization verifies every accepted candidate against the exact selected
|
||
worker request and claims only local minimality under its documented
|
||
Unicode-scalar transforms; see
|
||
[`docs/MINIMIZATION.md`](docs/MINIMIZATION.md).
|
||
Formatting is currently ECMAScript-only and changes only parser-reported
|
||
literal slash/control representations. Apply requires independent reparse,
|
||
actual-engine current-subject/replacement comparison, every applicable exact
|
||
unit test and explicit confirmation; see
|
||
[`docs/FORMATTING.md`](docs/FORMATTING.md).
|
||
|
||
## Flags and iteration
|
||
|
||
Pattern and flags are stored separately; delimiters are presentation only.
|
||
ECMAScript preserves native `g` and `y`; PCRE2 offers the application-level
|
||
`g` iteration flag plus `i`, `m`, `s`, `x`, `U` and `J`. PCRE2 UTF/UCP mode is
|
||
mandatory because browser strings are Unicode and partial-byte ranges cannot be
|
||
represented safely in the editor. The optional **Scan all** action adds only an
|
||
internal iteration flag to that request and does not change saved flags.
|
||
ECMAScript may add internal `d` to obtain exact indices without changing match
|
||
semantics.
|
||
|
||
Zero-length global iteration advances using the selected Unicode semantics and
|
||
cannot loop forever.
|
||
|
||
## Limits and timeout behavior
|
||
|
||
Important defaults:
|
||
|
||
| Limit | Default |
|
||
| ------------------------------- | -----------------: |
|
||
| Live parse debounce | 120 ms |
|
||
| Live execution debounce | 220 ms |
|
||
| Live execution timeout | 250 ms |
|
||
| Manual execution timeout | 2 s |
|
||
| Advanced maximum timeout | 10 s |
|
||
| Pattern hard limit | 64 Ki UTF-16 units |
|
||
| Interactive subject hard limit | 16 MiB |
|
||
| Corpus documents | 256 |
|
||
| Per corpus document | 16 MiB |
|
||
| Aggregate corpus input | 256 MiB |
|
||
| Independent corpus lines | 100,000 |
|
||
| Aggregate corpus matches | 100,000 |
|
||
| Capture summaries per document | 32 |
|
||
| Samples per capture | 3 |
|
||
| Retained applied output | 64 MiB |
|
||
| Applied ZIP | 68 MiB |
|
||
| Corpus batch wall time | 5 min |
|
||
| Replacement template | 64 Ki UTF-16 units |
|
||
| List template | 16 Ki UTF-16 units |
|
||
| Per-match replacement preview | 200 matches |
|
||
| Replacement contributions | 2,000 |
|
||
| Maximum matches | 10,000 |
|
||
| Maximum capture groups | 1,000 |
|
||
| Maximum capture rows | 100,000 |
|
||
| Replacement preview | 64 MiB |
|
||
| Project JSON document | 32 MiB UTF-8 |
|
||
| Standalone test-suite JSON | 32 MiB UTF-8 |
|
||
| In-memory unit-test suite | 32 MiB UTF-8 |
|
||
| Tests per suite | 1,000 |
|
||
| Retained message per test | 2 KiB UTF-8 |
|
||
| Unit-test suite wall time | 60 s |
|
||
| PCRE2 trace events | 50,000 |
|
||
| PCRE2 serialized trace | 10 MiB |
|
||
| Comparison retained differences | 5,000 |
|
||
| Comparison retained alignments | 2,000 |
|
||
| Static analysis traversal | 20,000 nodes |
|
||
| Static analysis findings | 100 |
|
||
| Growth probe steps | 24 |
|
||
| Analysis aggregate wall time | 30 s |
|
||
| Generated verified cases | 48 |
|
||
| Generated candidate attempts | 192 |
|
||
| Generation aggregate wall time | 15 s |
|
||
| Generation aggregate candidates | 1 MiB |
|
||
| Minimizer subject | 1 MiB UTF-8 |
|
||
| Minimizer candidate evaluations | 2,000 |
|
||
| Minimizer aggregate wall time | 60 s |
|
||
| Formatter validation tests | 1,000 |
|
||
| Formatter test wall time | 60 s |
|
||
| Retained minimizer history | 200 |
|
||
|
||
Analysis defaults to three warm-up plus 15 measured samples (caps: 100 and
|
||
1,000), with a 2-second per-sample worker limit that can be raised to 10
|
||
seconds. Growth generation defaults to 1 MiB, preflights the central 16 MiB
|
||
subject cap and rejects more than 10,000,000 repetitions. Replacement timing
|
||
is skipped above an 8 Mi UTF-16-unit output estimate.
|
||
|
||
A timeout terminates actual execution. It is reported as **timed out**, never as
|
||
“no match”. The next request creates a fresh worker.
|
||
|
||
Replacement applies exactly the authoritative matches returned under the match
|
||
and capture-row limits. If either collection limit is reached, later subject
|
||
text remains unchanged and the output is explicitly reported as partial. The
|
||
separate output-byte limit reports a truncated output preview.
|
||
|
||
## Project persistence
|
||
|
||
Project schema version 1 stores flavour, pattern, flags, options, replacement,
|
||
list template, unit tests (including explicit scan-all behavior and supported
|
||
per-test timeouts) and selected UI state. Import rebuilds a validated plain
|
||
object; it never evaluates code and pauses live execution until review.
|
||
Workbench-wide manual timeout changes are not persisted, and unsupported
|
||
project resource overrides are rejected.
|
||
|
||
Interactive test text is excluded from local save and export unless explicitly
|
||
enabled. Unit-test definitions are deliberate project data, but their subjects
|
||
require separate explicit opt-ins for JSON export and IndexedDB save.
|
||
Standalone test-suite JSON likewise omits subjects by default, is schema- and
|
||
field-validated on import, and appends at most 1,000 total tests. Add, update,
|
||
clone and append-import also preflight the complete in-memory suite against a
|
||
32 MiB aggregate bound. Added generated tests retain their validated generator
|
||
version, seed, candidate ID and intended outcome; editing one removes that
|
||
provenance. Schema v1 can remember that the corpus workspace was selected, but
|
||
corpus documents, contents, outputs and results are always ephemeral and
|
||
excluded from JSON and IndexedDB. Imports that try to inject those fields are
|
||
rejected. Import, export and IndexedDB save enforce one 32 MiB aggregate UTF-8
|
||
JSON-document limit in addition to per-field limits. The latest local save is
|
||
read through an `updatedAt` index cursor; loading it does not materialize the
|
||
complete project store.
|
||
|
||
## Development
|
||
|
||
Requirements: Node.js 22 or later and npm 11.
|
||
|
||
```sh
|
||
npm install
|
||
npm run dev
|
||
```
|
||
|
||
Quality and release commands:
|
||
|
||
```sh
|
||
npm run typecheck
|
||
npm run lint
|
||
npm run format:check
|
||
npm test
|
||
npm run test:conformance
|
||
npm run test:browser
|
||
npm run engines:build
|
||
npm run engines:verify
|
||
npm run engines:pcre2:build -- --source-dir /path/to/pcre2 --emcc /path/to/emcc
|
||
npm run engines:pcre2:verify
|
||
npm run engines:pcre2:install
|
||
npm run build
|
||
npm run toolbox:check
|
||
npm run check
|
||
npm run release:artifact
|
||
```
|
||
|
||
The normal build performs no network download. `vite.config.ts` uses
|
||
`base: "./"`; all workers and assets are self-hosted and nested-path safe.
|
||
|
||
## Toolbox SDK and Portal
|
||
|
||
The application uses Toolbox SDK 0.2.2 `AppShell`. Its canonical typed/static
|
||
manifest source is `src/toolbox/manifest.source.json`; generation fails if its
|
||
version or source identity drifts.
|
||
|
||
The deterministic release ZIP can be consumed by Toolbox Portal using an exact
|
||
artifact URL, manifest ID, version and SHA-256. Portal builds consume the
|
||
artifact; they do not build Regex Tools source.
|
||
|
||
The PCRE2 worker loads self-hosted JavaScript and WebAssembly only. Deployment
|
||
must serve `.wasm` as `application/wasm` and allow WebAssembly compilation in
|
||
`script-src`; see [`docs/PORTAL_REQUIREMENTS.md`](docs/PORTAL_REQUIREMENTS.md).
|
||
|
||
## Flavour roadmap
|
||
|
||
PCRE2 10.47 is the second implemented flavour. Its application-owned ABI
|
||
provides bounded compilation, matching, native substitution and separate
|
||
automatic-callout tracing, copied caller-owned records, deterministic
|
||
allocation cleanup and explicit result completeness. The exact
|
||
source/toolchain build is reproducible offline and the verified pack is bundled
|
||
with its metadata, checksums and licence.
|
||
|
||
Generated-case version 1 is ECMAScript-only. It traverses the complete
|
||
application-owned ECMAScript AST under node, depth, attempt, byte, repetition
|
||
and wall-time caps, then verifies every retained label through the actual
|
||
browser `RegExp` adapter. PCRE2 generation remains unavailable until its syntax
|
||
provider can expose a complete normalized tree.
|
||
|
||
Pattern formatting is likewise ECMAScript-only. It consumes the complete
|
||
regexpp token model and does not reinterpret the partial PCRE2 structure.
|
||
PCRE2 formatting remains unavailable until a complete grammar-backed
|
||
implementation passes equivalent idempotence, reparse, engine and test gates.
|
||
|
||
The first advertised generated-code target is the PCRE2 10.47 8-bit C API.
|
||
Its deterministic match and replacement fixtures compile and execute against
|
||
the exact official release. See
|
||
[`docs/COMPARISON_AND_CODEGEN.md`](docs/COMPARISON_AND_CODEGEN.md).
|
||
|
||
Python, Go, Rust, .NET, Java and legacy PCRE follow only through their actual
|
||
named runtimes. No flavour is emulated through JavaScript and relabelled.
|
||
|
||
## Licensing
|
||
|
||
Regex Tools original source is GPL-3.0-or-later, copyright © 2026 Albrecht
|
||
Degering. Shipped dependencies retain their own compatible licences and
|
||
notices; see [`THIRD_PARTY_NOTICES.md`](THIRD_PARTY_NOTICES.md) and
|
||
[`LICENSES/`](LICENSES/).
|
||
|
||
No regex101 application source, branding, generated explanation prose,
|
||
reference text or visual design is copied. RegexLib and RGXP.RU fixtures are not
|
||
redistributed because a suitable fixture licence was not established. No
|
||
RegexHub fixture is shipped in v0.2.0.
|
||
|
||
## Known limitations
|
||
|
||
- ECMAScript syntax coverage follows regexpp's ECMAScript 2025 grammar; a later
|
||
browser proposal may execute before the provider understands it.
|
||
- Exact consumed-length bounds for ECMAScript Unicode properties of strings
|
||
remain unknown and are reported as such rather than guessed.
|
||
- regexpp does not provide tolerant AST recovery. Malformed input receives an
|
||
exact reported error position as a zero-width range and an application-owned
|
||
error node, not a fabricated multi-character span or partial provider AST.
|
||
- Native JavaScript compile errors do not consistently expose a machine-readable
|
||
source offset; the provider diagnostic supplies the range.
|
||
- Capture history is unavailable. Browser ECMAScript exposes no engine trace;
|
||
PCRE2 exposes bounded automatic callouts, not a complete internal trace.
|
||
Movement classifications are derived and labelled as such.
|
||
- Static risk heuristics and generated-input observations apply only to
|
||
ECMAScript 2025 and are advisory. They are never proof of safety or of a
|
||
pattern’s general time complexity; analysis settings and results are
|
||
ephemeral.
|
||
- PCRE2 structural explanations are deliberately partial. The application
|
||
recognizes common capture forms and PCRE2-only constructs for reference, but
|
||
the actual PCRE2 compiler is authoritative. Branch-reset capture numbering is
|
||
omitted from the explanation rather than guessed.
|
||
- Capture-table rendering is paged at 200 rows while aggregate collection is
|
||
bounded separately.
|
||
- Explanation/extraction trees and editor decorations show bounded 2,000-node
|
||
previews with explicit totals; normalized results and table/export data stay
|
||
separate from those render-only caps.
|
||
- Extraction-tree values show at most 100 UTF-16 units and capture-table cells
|
||
at most 512. UI-only shortening shows the exact range length; engine-clipped
|
||
values are labelled as prefixes of that range. Copy and eligible JSON export
|
||
continue to use the separately bounded engine value.
|
||
- Replacement parsing is complete below its 64 Ki input limit, while editor
|
||
marks, token rows and diagnostics have explicit presentation caps. List token
|
||
chips are similarly bounded, and list rendering stops at 250,000 token
|
||
evaluations.
|
||
- Per-match replacement presentation renders at most 200 matches and 2,000
|
||
token contributions under a 50,000 token-evaluation budget. Bounded
|
||
replacement output remains available separately and is labelled partial if
|
||
match collection reached a limit.
|
||
- Result-value previews are capped per value and overall. List export is
|
||
disabled whenever a bounded preview would otherwise be mistaken for complete
|
||
data.
|
||
- Unit-test failure messages retain at most 2 KiB UTF-8 per case, so 1,000
|
||
results stay below the 2 MiB log budget. Oversized exact replacement results
|
||
offer a lightweight `should-match` draft when one is valid and fits the suite.
|
||
- Corpus input accepts recognized UTF-8 text only. It does not auto-detect
|
||
legacy encodings or recursively traverse directories.
|
||
- Corpus capture extraction is a bounded per-document summary: full match plus
|
||
at most 31 capture groups, three 160-unit samples each. Content-free reports
|
||
include counts and clipping metadata, never the samples themselves.
|
||
- Generated cases deliberately sample only ECMAScript and report
|
||
backreferences, lookaround, boundaries, complex Unicode sets and other
|
||
unsupported constraints instead of claiming exhaustive coverage.
|
||
- Pattern formatting is intentionally narrow and ECMAScript-only. Its bounded
|
||
source/candidate and unit-test validation is evidence for the checked
|
||
snapshots, not a proof of equivalence over all subjects.
|
||
|
||
Architecture, security, performance, provenance and flavour details live in
|
||
[`docs/`](docs/).
|