lotobo/regex-tools
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 6 Wiki Activity

feat: publish Regex Tools 0.2.0

Browse Source
This commit is contained in:
Albrecht Degering
2026-07-27 01:06:57 +02:00
parent 873b9b218d
commit 4951c966d4
180 changed files with 35344 additions and 503 deletions
Download Patch File Download Diff File Expand all files Collapse all files

241
README.md
View File

@@ -8,21 +8,45 @@ Patterns, replacement templates, test text and project files stay in the
browser. There is no account, backend, upload, remote execution, telemetry,
runtime CDN, remote font, or cloud-save service.
## Current milestone
## Current release — 0.2.0
Version 0.1.0 is the first production flavour slice: ECMAScript.
The v0.2.0 release contains two real production engine slices: ECMAScript and
PCRE2 10.47.
- Syntax acceptance: `@eslint-community/regexpp` 4.12.2, ECMAScript 2025
grammar; application-owned structural explanations are partial and
feature-matrix tested.
- Execution engine: the current browser's native ECMAScript `RegExp`.
- Native and editor offsets: UTF-16 code units.
- PCRE2 execution and substitution: pinned official PCRE2 10.47 compiled to a
self-hosted 8-bit WebAssembly pack with UTF/UCP always enabled.
- PCRE2 native UTF-8 byte ranges are retained and normalized to exact editor
UTF-16 ranges; inputs with lone surrogates are rejected instead of rewritten.
- PCRE2 match-step, depth and heap limits are configurable inside hard caps.
- PCRE2 automatic callouts are copied through a separate ABI operation and
displayed from a dedicated killable trace worker under 50,000-event and
10 MiB serialized-data hard caps.
- Parsing and execution run in separate workers.
- Actual execution can be terminated by killing its worker.
- The worker is recreated after timeout, crash or cancellation.
- Capture ranges use the engine's `d` indices result.
- Named, numbered, optional, empty and repeated-final captures are represented.
- Match, replacement, typed list/extraction and unit-test modes are available.
- Match, replacement, typed list/extraction, corpus/apply and unit-test modes
are available.
- ECMAScript-specific advisory static risk findings, bounded dynamic growth
probes and cold/warm benchmarks run locally; timing and growth execution is
isolated in a separately killable worker.
- ECMAScript and PCRE2 can be compared through two exact, independently timed
requests without translating patterns, flags or replacements. Incomplete or
rejected sides are explicitly not comparable.
- The reviewed PCRE2 C17 generator emits exact UTF-8 byte arrays, configured
limits and error handling for the PCRE2 10.47 8-bit API.
- A cancellable, deterministic subject minimizer verifies saved-test failures,
capture-range failures, exact timeouts and ECMAScript/PCRE2 mismatches against
the selected real workers under explicit evaluation and wall budgets.
- Deterministic ECMAScript test-case generation samples the normalized AST from
an explicit seed, reports structural coverage and unsupported constructs, and
labels cases only after verification by the selected real engine worker.
- Project JSON import/export and optional IndexedDB save are available.
- Interactive test text is not persisted by default.
@@ -40,39 +64,67 @@ hierarchy uses syntactic capture nesting while preserving actual engine ranges.
ECMAScript exposes only the final retained capture of a repeated group; Regex
Tools says so and never invents capture history.
Browser ECMAScript engines do not expose their internal backtracking trace.
Version 0.1.0 therefore presents no ECMAScript execution trace and no
educational trace disguised as one.
Browser ECMAScript exposes no actual engine trace. PCRE2 has a separate bounded
automatic-callout viewer. Its event fields are reported by PCRE2, while
“forward”, “same position” and “apparent backtrack” are visibly derived only
from adjacent reported subject positions. The callout stream is not described
as a complete record of every internal engine action.
## Workbench modes
- **Match** — live highlighting, extraction tree and bounded capture table.
- **Replace** — native ECMAScript `RegExp` matches with bounded,
specification-compatible string substitution; output completeness is
reported separately from bounded per-match original/result, changed-range and
token-contribution previews that synchronize the replacement, pattern,
subject and output editors.
- **Replace** — engine-native bounded substitution for ECMAScript and PCRE2;
output completeness is reported separately from bounded per-match
original/result, changed-range and token-contribution previews that
synchronize the replacement, pattern, subject and output editors.
- **List** — non-executable typed templates with text, CSV, JSON and NDJSON
export; export is disabled when engine collection or bounded rendering is
incomplete.
- **Corpus / Apply** — bounded multi-file UTF-8 and pasted-text input processed
sequentially in killable workers. Whole-document and independent-line
semantics are explicit; line apply preserves original separators. Progress,
cancellation, bounded capture samples, replacement previews, per-document
exact/partial summaries, content-free JSON/CSV/NDJSON reports and
privacy-gated exact-output ZIP/downloads are available. Original files are
never modified.
- **Unit tests** — editable and cloneable versioned presence, count, full-match,
capture, replacement, timing and expected-timeout assertions. Capture a
complete current result, select only cases to rerun, filter failures, and
import/export a standalone validated JSON suite; execution stays in isolated
workers.
Corpus, comparison, debugging, analysis, generation, minimization, formatting
and code generation remain roadmap items. Controls for them are not exposed in
this release.
The auxiliary **PCRE trace**, ECMAScript **Analysis**, **Compare & code**,
**Generate cases**, **Minimize** and **Format** workspaces are separate from
normal matching, replacement, tests and corpus execution. Trace collection is
never included in benchmark measurements.
Comparison retains native ranges while aligning actual results by normalized
editor UTF-16 ranges. Agreement is described only as evidence for the current
subject, never as proof of pattern equivalence.
Case generation is bounded sampling, not a proof of language coverage. Version
1 supports only a complete accepted ECMAScript normalized AST; the partial
PCRE2 provider is reported as unavailable rather than passed through or
relabeled. See
[`docs/GENERATED_CASES.md`](docs/GENERATED_CASES.md).
Minimization verifies every accepted candidate against the exact selected
worker request and claims only local minimality under its documented
Unicode-scalar transforms; see
[`docs/MINIMIZATION.md`](docs/MINIMIZATION.md).
Formatting is currently ECMAScript-only and changes only parser-reported
literal slash/control representations. Apply requires independent reparse,
actual-engine current-subject/replacement comparison, every applicable exact
unit test and explicit confirmation; see
[`docs/FORMATTING.md`](docs/FORMATTING.md).
## Flags and iteration
Pattern and flags are stored separately; delimiters are presentation only.
Native `g` and `y` behavior is preserved. The optional **Scan all** action is
explicit. When neither `g` nor `y` is present it adds `g` only to that execution
request, visibly records the internal flag, and does not change the saved user
flags. The `d` flag may likewise be added internally to obtain exact indices
without changing matching semantics.
ECMAScript preserves native `g` and `y`; PCRE2 offers the application-level
`g` iteration flag plus `i`, `m`, `s`, `x`, `U` and `J`. PCRE2 UTF/UCP mode is
mandatory because browser strings are Unicode and partial-byte ranges cannot be
represented safely in the editor. The optional **Scan all** action adds only an
internal iteration flag to that request and does not change saved flags.
ECMAScript may add internal `d` to obtain exact indices without changing match
semantics.
Zero-length global iteration advances using the selected Unicode semantics and
cannot loop forever.
@@ -81,29 +133,63 @@ cannot loop forever.
Important defaults:
| Limit | Default |
| ------------------------------ | -----------------: |
| Live parse debounce | 120 ms |
| Live execution debounce | 220 ms |
| Live execution timeout | 250 ms |
| Manual execution timeout | 2 s |
| Advanced maximum timeout | 10 s |
| Pattern hard limit | 64 Ki UTF-16 units |
| Interactive subject hard limit | 16 MiB |
| Replacement template | 64 Ki UTF-16 units |
| List template | 16 Ki UTF-16 units |
| Per-match replacement preview | 200 matches |
| Replacement contributions | 2,000 |
| Maximum matches | 10,000 |
| Maximum capture groups | 1,000 |
| Maximum capture rows | 100,000 |
| Replacement preview | 64 MiB |
| Project JSON document | 32 MiB UTF-8 |
| Standalone test-suite JSON | 32 MiB UTF-8 |
| In-memory unit-test suite | 32 MiB UTF-8 |
| Tests per suite | 1,000 |
| Retained message per test | 2 KiB UTF-8 |
| Unit-test suite wall time | 60 s |
| Limit | Default |
| ------------------------------- | -----------------: |
| Live parse debounce | 120 ms |
| Live execution debounce | 220 ms |
| Live execution timeout | 250 ms |
| Manual execution timeout | 2 s |
| Advanced maximum timeout | 10 s |
| Pattern hard limit | 64 Ki UTF-16 units |
| Interactive subject hard limit | 16 MiB |
| Corpus documents | 256 |
| Per corpus document | 16 MiB |
| Aggregate corpus input | 256 MiB |
| Independent corpus lines | 100,000 |
| Aggregate corpus matches | 100,000 |
| Capture summaries per document | 32 |
| Samples per capture | 3 |
| Retained applied output | 64 MiB |
| Applied ZIP | 68 MiB |
| Corpus batch wall time | 5 min |
| Replacement template | 64 Ki UTF-16 units |
| List template | 16 Ki UTF-16 units |
| Per-match replacement preview | 200 matches |
| Replacement contributions | 2,000 |
| Maximum matches | 10,000 |
| Maximum capture groups | 1,000 |
| Maximum capture rows | 100,000 |
| Replacement preview | 64 MiB |
| Project JSON document | 32 MiB UTF-8 |
| Standalone test-suite JSON | 32 MiB UTF-8 |
| In-memory unit-test suite | 32 MiB UTF-8 |
| Tests per suite | 1,000 |
| Retained message per test | 2 KiB UTF-8 |
| Unit-test suite wall time | 60 s |
| PCRE2 trace events | 50,000 |
| PCRE2 serialized trace | 10 MiB |
| Comparison retained differences | 5,000 |
| Comparison retained alignments | 2,000 |
| Static analysis traversal | 20,000 nodes |
| Static analysis findings | 100 |
| Growth probe steps | 24 |
| Analysis aggregate wall time | 30 s |
| Generated verified cases | 48 |
| Generated candidate attempts | 192 |
| Generation aggregate wall time | 15 s |
| Generation aggregate candidates | 1 MiB |
| Minimizer subject | 1 MiB UTF-8 |
| Minimizer candidate evaluations | 2,000 |
| Minimizer aggregate wall time | 60 s |
| Formatter validation tests | 1,000 |
| Formatter test wall time | 60 s |
| Retained minimizer history | 200 |
Analysis defaults to three warm-up plus 15 measured samples (caps: 100 and
1,000), with a 2-second per-sample worker limit that can be raised to 10
seconds. Growth generation defaults to 1 MiB, preflights the central 16 MiB
subject cap and rejects more than 10,000,000 repetitions. Replacement timing
is skipped above an 8 Mi UTF-16-unit output estimate.
A timeout terminates actual execution. It is reported as **timed out**, never as
“no match”. The next request creates a fresh worker.
@@ -128,8 +214,12 @@ require separate explicit opt-ins for JSON export and IndexedDB save.
Standalone test-suite JSON likewise omits subjects by default, is schema- and
field-validated on import, and appends at most 1,000 total tests. Add, update,
clone and append-import also preflight the complete in-memory suite against a
32 MiB aggregate bound. Corpus content is not supported or persisted in this
release. Import, export and IndexedDB save enforce one 32 MiB aggregate UTF-8
32 MiB aggregate bound. Added generated tests retain their validated generator
version, seed, candidate ID and intended outcome; editing one removes that
provenance. Schema v1 can remember that the corpus workspace was selected, but
corpus documents, contents, outputs and results are always ephemeral and
excluded from JSON and IndexedDB. Imports that try to inject those fields are
rejected. Import, export and IndexedDB save enforce one 32 MiB aggregate UTF-8
JSON-document limit in addition to per-field limits. The latest local save is
read through an `updatedAt` index cursor; loading it does not materialize the
complete project store.
@@ -154,6 +244,9 @@ npm run test:conformance
npm run test:browser
npm run engines:build
npm run engines:verify
npm run engines:pcre2:build -- --source-dir /path/to/pcre2 --emcc /path/to/emcc
npm run engines:pcre2:verify
npm run engines:pcre2:install
npm run build
npm run toolbox:check
npm run check
@@ -173,17 +266,34 @@ The deterministic release ZIP can be consumed by Toolbox Portal using an exact
artifact URL, manifest ID, version and SHA-256. Portal builds consume the
artifact; they do not build Regex Tools source.
Version 0.1.0 has no WebAssembly. Its minimum CSP needs self-hosted scripts and
workers, not `wasm-unsafe-eval`. See
[`docs/PORTAL_REQUIREMENTS.md`](docs/PORTAL_REQUIREMENTS.md).
The PCRE2 worker loads self-hosted JavaScript and WebAssembly only. Deployment
must serve `.wasm` as `application/wasm` and allow WebAssembly compilation in
`script-src`; see [`docs/PORTAL_REQUIREMENTS.md`](docs/PORTAL_REQUIREMENTS.md).
## Flavour roadmap
The next flavour is official PCRE2 10.47 WebAssembly. A technical spike from the
signed official tag has already demonstrated Unicode matching, named captures,
substitution, automatic and explicit callouts, and match/depth/heap limits.
PCRE2 is not shipped until the application-owned bridge, byte/UTF-16 offsets,
trace caps, browser tests and source/licence material meet the acceptance gate.
PCRE2 10.47 is the second implemented flavour. Its application-owned ABI
provides bounded compilation, matching, native substitution and separate
automatic-callout tracing, copied caller-owned records, deterministic
allocation cleanup and explicit result completeness. The exact
source/toolchain build is reproducible offline and the verified pack is bundled
with its metadata, checksums and licence.
Generated-case version 1 is ECMAScript-only. It traverses the complete
application-owned ECMAScript AST under node, depth, attempt, byte, repetition
and wall-time caps, then verifies every retained label through the actual
browser `RegExp` adapter. PCRE2 generation remains unavailable until its syntax
provider can expose a complete normalized tree.
Pattern formatting is likewise ECMAScript-only. It consumes the complete
regexpp token model and does not reinterpret the partial PCRE2 structure.
PCRE2 formatting remains unavailable until a complete grammar-backed
implementation passes equivalent idempotence, reparse, engine and test gates.
The first advertised generated-code target is the PCRE2 10.47 8-bit C API.
Its deterministic match and replacement fixtures compile and execute against
the exact official release. See
[`docs/COMPARISON_AND_CODEGEN.md`](docs/COMPARISON_AND_CODEGEN.md).
Python, Go, Rust, .NET, Java and legacy PCRE follow only through their actual
named runtimes. No flavour is emulated through JavaScript and relabelled.
@@ -198,7 +308,7 @@ notices; see [`THIRD_PARTY_NOTICES.md`](THIRD_PARTY_NOTICES.md) and
No regex101 application source, branding, generated explanation prose,
reference text or visual design is copied. RegexLib and RGXP.RU fixtures are not
redistributed because a suitable fixture licence was not established. No
RegexHub fixture is shipped in v0.1.0.
RegexHub fixture is shipped in v0.2.0.
## Known limitations
@@ -211,7 +321,17 @@ RegexHub fixture is shipped in v0.1.0.
error node, not a fabricated multi-character span or partial provider AST.
- Native JavaScript compile errors do not consistently expose a machine-readable
source offset; the provider diagnostic supplies the range.
- Capture history and actual ECMAScript execution tracing are unavailable.
- Capture history is unavailable. Browser ECMAScript exposes no engine trace;
PCRE2 exposes bounded automatic callouts, not a complete internal trace.
Movement classifications are derived and labelled as such.
- Static risk heuristics and generated-input observations apply only to
ECMAScript 2025 and are advisory. They are never proof of safety or of a
pattern’s general time complexity; analysis settings and results are
ephemeral.
- PCRE2 structural explanations are deliberately partial. The application
recognizes common capture forms and PCRE2-only constructs for reference, but
the actual PCRE2 compiler is authoritative. Branch-reset capture numbering is
omitted from the explanation rather than guessed.
- Capture-table rendering is paged at 200 rows while aggregate collection is
bounded separately.
- Explanation/extraction trees and editor decorations show bounded 2,000-node
@@ -235,8 +355,17 @@ RegexHub fixture is shipped in v0.1.0.
- Unit-test failure messages retain at most 2 KiB UTF-8 per case, so 1,000
results stay below the 2 MiB log budget. Oversized exact replacement results
offer a lightweight `should-match` draft when one is valid and fits the suite.
- Benchmark, risk analysis, generated cases, minimization and corpus processing
are not in this release.
- Corpus input accepts recognized UTF-8 text only. It does not auto-detect
legacy encodings or recursively traverse directories.
- Corpus capture extraction is a bounded per-document summary: full match plus
at most 31 capture groups, three 160-unit samples each. Content-free reports
include counts and clipping metadata, never the samples themselves.
- Generated cases deliberately sample only ECMAScript and report
backreferences, lookaround, boundaries, complex Unicode sets and other
unsupported constraints instead of claiming exhaustive coverage.
- Pattern formatting is intentionally narrow and ECMAScript-only. Its bounded
source/candidate and unit-test validation is evidence for the checked
snapshots, not a proof of equivalence over all subjects.
Architecture, security, performance, provenance and flavour details live in
[`docs/`](docs/).