feat: publish Regex Tools 0.2.0
This commit is contained in:
121
docs/ANALYSIS.md
Normal file
121
docs/ANALYSIS.md
Normal file
@@ -0,0 +1,121 @@
|
||||
# Performance and risk analysis
|
||||
|
||||
Regex Tools provides one deliberately scoped analysis implementation:
|
||||
ECMAScript 2025 patterns parsed into the application-owned normalized AST and
|
||||
executed by the browser's native `RegExp` runtime. The analyser does not apply
|
||||
ECMAScript findings to PCRE2 or any other flavour.
|
||||
|
||||
Analysis is advisory. The UI uses “potential risk”, “possibly” and “observed
|
||||
under selected limits”. It never turns an empty finding list into “safe”,
|
||||
“guaranteed linear” or a vulnerability proof.
|
||||
|
||||
## Static analysis
|
||||
|
||||
The application-owned `regex-tools-ecmascript-heuristics` analyser walks at
|
||||
most 20,000 normalized nodes and returns at most 100 findings. Each finding
|
||||
contains:
|
||||
|
||||
- flavour and UTF-16 source range;
|
||||
- stable rule identifier;
|
||||
- static or dynamically observed provenance;
|
||||
- explanation and example risk;
|
||||
- low, medium or high confidence;
|
||||
- limitations;
|
||||
- a suggested investigation.
|
||||
|
||||
The current rules review nested and nullable repetition, ambiguous repeated
|
||||
alternatives, overlapping alternative prefixes, wildcards and backreferences
|
||||
inside repetition, repeated lookaround bodies, unanchored amplification,
|
||||
capture count, nesting depth and replacement-output expansion. These are
|
||||
structural heuristics. They do not model the native engine's complete compiled
|
||||
program, optimizations or calling context.
|
||||
|
||||
## Dynamic growth probes
|
||||
|
||||
The user defines a bounded subject family:
|
||||
|
||||
```text
|
||||
prefix + repeatedFragment.repeat(repetitions) + suffix
|
||||
```
|
||||
|
||||
The byte and UTF-16 size are calculated before the generated string is
|
||||
allocated. The repeated fragment is limited to 4,096 UTF-16 units, each affix
|
||||
to 16,384 units, the run to 24 samples, and repetitions to 10,000,000. The UI
|
||||
defaults to a 1 MiB generated-subject cap even though the central interactive
|
||||
hard cap remains 16 MiB.
|
||||
|
||||
Every probe executes in a dedicated worker. The supervisor terminates and
|
||||
discards that worker on per-sample timeout, cancellation or crash. Runs also
|
||||
stop at:
|
||||
|
||||
- the selected repetition bound;
|
||||
- the selected generated-byte bound;
|
||||
- the selected step bound;
|
||||
- the 30-second aggregate wall bound;
|
||||
- a selected normalized-growth threshold.
|
||||
|
||||
The growth threshold compares the time ratio with the input-size ratio for two
|
||||
adjacent samples. Sub-millisecond measurements below a noise floor do not
|
||||
trigger it. A threshold observation characterizes only those generated inputs
|
||||
and that browser. Timeout, crash, compile rejection and non-match remain
|
||||
distinct states.
|
||||
|
||||
The UI renders time and outcome in separate plots and keeps the exact sample
|
||||
table alongside them.
|
||||
|
||||
## Benchmark method
|
||||
|
||||
A benchmark begins with a fresh analysis worker. Worker creation and its
|
||||
identity response form the separate cold-start measurement. The first native
|
||||
engine use is retained as a cold sample. Configured warm-up samples are
|
||||
discarded; configured measured samples produce the warm statistics.
|
||||
|
||||
Each sample measures these phases separately:
|
||||
|
||||
- `RegExp` construction;
|
||||
- first `exec`;
|
||||
- bounded all-match iteration;
|
||||
- native string replacement when its conservative output bound is acceptable;
|
||||
- all-match throughput and match count.
|
||||
|
||||
All-match iteration retains only counts, not values or captures. It follows the
|
||||
same explicit scan-all, global, sticky and Unicode zero-length advancement
|
||||
semantics as the workbench. The application-added indices flag remains in the
|
||||
effective flags so the measured operation reflects interactive execution.
|
||||
|
||||
Replacement timing is skipped when match collection reaches its bound or a
|
||||
conservative estimate can exceed 8 Mi UTF-16 units. This prevents a benchmark
|
||||
from using an unbounded native replacement merely to obtain a timing.
|
||||
It is also skipped for explicit scan-all with sticky `y`: native
|
||||
`String.replace` processes only one sticky match, while the workbench's
|
||||
explicit scan-all operation can process a contiguous sequence. Reporting that
|
||||
different operation as the same benchmark would be misleading.
|
||||
|
||||
Warm metrics include sample count, minimum, conventional median, nearest-rank
|
||||
p95 and maximum. Milliseconds are shown to two decimal places, with values
|
||||
below 0.01 ms labelled accordingly; this does not imply nanosecond accuracy.
|
||||
The result retains subject size, match count, effective flags, browser runtime
|
||||
identity and engine identity.
|
||||
|
||||
Defaults:
|
||||
|
||||
| Setting | Default | Hard bound |
|
||||
| ----------------------- | ------: | ---------: |
|
||||
| Warm-up samples | 3 | 100 |
|
||||
| Measured samples | 15 | 1,000 |
|
||||
| Per-sample timeout | 2 s | 10 s |
|
||||
| Aggregate wall time | 30 s | 30 s |
|
||||
| Matches per sample | 10,000 | 10,000 |
|
||||
| Replacement output gate | 8 MiU | 8 MiU |
|
||||
|
||||
One subject is not a complete performance characterization. Results from
|
||||
different algorithms, pattern ports, browser engines, WebAssembly runtimes or
|
||||
machines are not directly comparable. Trace-enabled execution is never used
|
||||
for benchmark results.
|
||||
|
||||
## Privacy and persistence
|
||||
|
||||
Patterns, subjects, generated inputs, findings and benchmark samples remain in
|
||||
the browser. No analysis request is uploaded. Analysis settings and results are
|
||||
currently ephemeral and are not added to project JSON or IndexedDB; this also
|
||||
keeps large benchmark subjects out of persisted projects by default.
|
||||
@@ -5,9 +5,32 @@ boundaries.
|
||||
|
||||
```text
|
||||
React workbench
|
||||
├─ Pattern SyntaxSupervisor → syntax.worker → Regexpp provider → normalized AST
|
||||
├─ Pattern SyntaxSupervisor → syntax.worker
|
||||
│ ├─ Regexpp ECMAScript provider → normalized AST
|
||||
│ └─ PCRE2 lexical provider → partial normalized structure
|
||||
├─ Replacement SyntaxSupervisor → syntax.worker → typed replacement tokens
|
||||
├─ EngineSupervisor → ecmascript.worker → native RegExp → match DTOs
|
||||
├─ EngineSupervisor
|
||||
│ ├─ ecmascript.worker → native RegExp → match DTOs
|
||||
│ └─ pcre2.worker → PCRE2 10.47 WASM ABI → match DTOs
|
||||
├─ Pcre2TraceSupervisor → pcre2-trace.worker
|
||||
│ └─ separate ABI-v3 PCRE2_AUTO_CALLOUT operation → bounded trace DTOs
|
||||
├─ AnalysisPanel
|
||||
│ ├─ normalized-AST ECMAScript static heuristics
|
||||
│ └─ AnalysisSupervisor → analysis.worker → native RegExp benchmark/growth probes
|
||||
├─ ComparisonOrchestrator
|
||||
│ ├─ two exact syntax snapshots + independently killable engine workers
|
||||
│ └─ normalized UTF-16 match/capture alignment + explicit incomplete states
|
||||
├─ PCRE2 C17 generator → validated snapshot → exact UTF-8 byte-array source
|
||||
├─ SubjectMinimizer
|
||||
│ ├─ fixed syntax snapshots + exact real-engine failure or comparison oracle
|
||||
│ └─ deterministic Unicode-scalar reducer → bounded local-minimal result
|
||||
├─ CaseGenerationOrchestrator
|
||||
│ ├─ generation.worker → deterministic bounded ECMAScript AST candidates
|
||||
│ └─ EngineSupervisor → selected actual-engine label verification
|
||||
├─ ECMAScript literal/control formatter
|
||||
│ ├─ exact regexpp literal-token ranges → idempotent candidate preview
|
||||
│ └─ PatternFormatValidator → paired syntax + actual-engine workers and exact tests
|
||||
├─ Corpus EngineSupervisor → sequential bounded document jobs → summaries/outputs
|
||||
├─ Test SyntaxSupervisor + EngineSupervisor → isolated cancellable test runs
|
||||
└─ project validator / serializer / IndexedDB persistence
|
||||
```
|
||||
@@ -21,6 +44,59 @@ request ID and worker generation. A supervisor allows one active request,
|
||||
rejects stale responses, terminates on timeout/crash/cancel, and lazily creates
|
||||
a new generation. Timeout is a distinct error state.
|
||||
|
||||
PCRE2 tracing is not an `EngineSupervisor` match or replacement operation. Its
|
||||
own worker compiles with `PCRE2_AUTO_CALLOUT`, copies only complete fixed
|
||||
records and bounded mark bytes, and stops natively at either trace cap.
|
||||
Reported callout fields retain byte positions; movement classifications are a
|
||||
separate derived UI layer. Normal execution, corpus, tests and benchmarks never
|
||||
run through the trace operation.
|
||||
|
||||
Static analysis traverses the normalized ECMAScript tree on the UI side under
|
||||
node/finding caps. Timing and generated-input growth probes use the independent
|
||||
analysis worker. Timeout, cancellation or crash discards that worker; no trace
|
||||
mode is benchmarked.
|
||||
|
||||
Comparison is a separate orchestration layer, not an `EngineSupervisor`
|
||||
operation. It runs explicit ECMAScript and PCRE2 syntax/execution snapshots,
|
||||
retains each engine identity and native range model, and aligns only normalized
|
||||
editor ranges. Timeout, cancellation, compile rejection and truncation remain
|
||||
distinct non-comparable outcomes. Shared patterns are sent unchanged;
|
||||
per-flavour ports are user-owned variants.
|
||||
|
||||
The PCRE2 C generator is application-owned and never implements browser
|
||||
execution. It validates one exact PCRE2 snapshot, emits user strings as
|
||||
independent UTF-8 byte arrays, and states the host-level mappings that the
|
||||
PCRE2 API cannot express directly. Its advertised C17 target is protected by
|
||||
deterministic golden identities and an exact PCRE2 10.47 compile/execute gate.
|
||||
|
||||
Subject minimization is a main-thread bounded reducer whose expensive predicate
|
||||
checks stay in independently terminable engine workers. Fixed pattern syntax
|
||||
is parsed once for capture metadata. The baseline and every accepted candidate
|
||||
retain the exact target and engine identity. Timeout, crash, cancellation,
|
||||
worker failure and incomplete output are separate outcomes. Deterministic
|
||||
ddmin deletion is followed by a fixed-point local sweep over Unicode-scalar
|
||||
deletion and lower-rank canonical replacement; only completion of that sweep
|
||||
permits a transform-local minimality claim.
|
||||
|
||||
Generated-case synthesis and verification are separate trust boundaries. The
|
||||
dedicated generation worker receives only a validated, complete ECMAScript
|
||||
normalized tree plus explicit settings and seed. The main-thread orchestrator
|
||||
then sends each bounded candidate to `EngineSupervisor`; only the selected real
|
||||
adapter can establish a `should-match` or `should-not-match` label. Worker
|
||||
timeout, crash, cancellation, compile rejection, wrong outcome and ordinary
|
||||
failure stay distinct. Settings/results are ephemeral, while explicitly added
|
||||
unit tests retain field-validated generator provenance.
|
||||
|
||||
Formatting is not an engine operation and never rewrites from heuristic text
|
||||
alone. The ECMAScript formatter consumes an exact accepted regexpp snapshot and
|
||||
changes only parser-reported literal/control token ranges. Its validator
|
||||
reparses source and candidate separately, preserves capture shape, runs both in
|
||||
separate actual-engine workers against the current replacement snapshot, then
|
||||
replays every enabled test tied to the exact active pattern/version/flags/options.
|
||||
Any difference, incomplete output, one-sided timeout, identity change or
|
||||
inconclusive worker outcome blocks apply. The preview and observations are
|
||||
ephemeral, and user confirmation is required even after all bounded gates pass.
|
||||
|
||||
Pattern and replacement syntax use independent supervisors. A pattern snapshot
|
||||
is bound to its exact pattern, ordered flags and revision; execution cannot use
|
||||
acceptance or capture metadata from an earlier input. Replacement output is
|
||||
@@ -32,10 +108,27 @@ over those actual matches, the complete replacement-token model and the bounded
|
||||
output. They execute no replacement callback or user code and have separate
|
||||
presentation and token-evaluation limits.
|
||||
|
||||
Native ranges are retained in the engine's declared unit. Browser editor ranges
|
||||
Corpus documents are ephemeral React session state, not project state. A
|
||||
dedicated engine supervisor processes one document at a time and is terminated
|
||||
on cancellation, timeout or workspace exit. Batch orchestration retains only
|
||||
per-document summaries and, for apply runs, bounded outputs. It enforces
|
||||
document-count, per-document byte, aggregate input byte, aggregate match,
|
||||
aggregate output and batch wall-time limits. Exact-output export is unavailable
|
||||
if any document is partial, failed, cancelled or not run.
|
||||
|
||||
Whole-document mode sends each document to the engine once. Independent-line
|
||||
mode segments CRLF, CR, LF, U+2028 and U+2029 without discarding them, executes
|
||||
each logical line as its own subject, and reattaches the exact original
|
||||
separator during apply. This makes anchor and cross-line behavior deliberately
|
||||
different rather than silently rewriting the pattern. Match DTOs are reduced to
|
||||
bounded per-group participation counts and samples before the next document.
|
||||
|
||||
Native ranges are retained in the engine's declared unit. PCRE2 always uses
|
||||
UTF/UCP and exposes UTF-8 byte ranges; its ABI copies scalar records and names
|
||||
out of WebAssembly before returning and retains no code pointer. Browser editor ranges
|
||||
are half-open UTF-16 code-unit ranges. Reusable converters cover UTF-8 byte and
|
||||
Unicode code-point offsets for later flavours, including invalid-boundary
|
||||
rejection and lone-surrogate detection.
|
||||
Unicode code-point offsets, including invalid-boundary rejection and
|
||||
lone-surrogate detection.
|
||||
|
||||
The release boundary is `dist/` plus checked-in legal/source documents. The
|
||||
Portal consumes the resulting immutable ZIP and never imports React source.
|
||||
|
||||
118
docs/COMPARISON_AND_CODEGEN.md
Normal file
118
docs/COMPARISON_AND_CODEGEN.md
Normal file
@@ -0,0 +1,118 @@
|
||||
# ECMAScript/PCRE2 comparison and PCRE2 C generation
|
||||
|
||||
## Comparison contract
|
||||
|
||||
The comparison panel sends two explicit requests: one to the native browser
|
||||
ECMAScript worker and one to the bundled PCRE2 10.47 worker. It does not
|
||||
translate patterns, flags, options or replacement templates.
|
||||
|
||||
Two pattern models are available:
|
||||
|
||||
- **Shared pattern** sends the same string unchanged to both engines.
|
||||
- **Per-flavour variants** retains two explicit ports supplied by the user.
|
||||
|
||||
Flags are selected independently from each flavour's registered allowlist.
|
||||
PCRE2 match, depth and heap bounds are part of its exact request. Both sides
|
||||
share the selected subject, scan-all intent, host result bounds and wall-clock
|
||||
worker timeout.
|
||||
|
||||
Each side retains:
|
||||
|
||||
- its syntax-provider request, acceptance, diagnostics and coverage;
|
||||
- the exact execution or replacement request sent to its worker;
|
||||
- a deterministic display key plus the complete authoritative request;
|
||||
- completion, timeout, cancellation or worker-failure status;
|
||||
- engine, runtime and adapter identity when the engine returns one;
|
||||
- user and effective flags;
|
||||
- native ranges and normalized editor UTF-16 ranges;
|
||||
- compile acceptance, matches, captures and replacement output.
|
||||
|
||||
The orchestrator owns two syntax supervisors and the multi-engine execution
|
||||
supervisor. The two engine calls run concurrently through independently
|
||||
killable flavour workers. A side timeout kills only that worker. The shared
|
||||
Cancel action terminates every active comparison worker. Comparison remains
|
||||
outside `EngineSupervisor`, whose responsibility is one engine operation and
|
||||
lifecycle boundary.
|
||||
|
||||
Match pairs are aligned by equal normalized UTF-16 ranges first. Unpaired
|
||||
matches use an explicitly labelled ordinal fallback. Captures are aligned by
|
||||
normalized range, then capture identity, then an explicitly labelled ordinal
|
||||
fallback. Original native offsets remain attached to both results.
|
||||
|
||||
No equivalence claim is made when syntax or compilation is rejected, a worker
|
||||
times out/fails/is cancelled, results or values are truncated, or replacement
|
||||
output is incomplete. A completed matching result is labelled only as
|
||||
agreement for the current subject; it is evidence, not a proof that two
|
||||
patterns are equivalent. Timeout is never interpreted as “no match.”
|
||||
|
||||
The retained DTO is bounded to 5,000 displayed difference records and 2,000
|
||||
match alignments while still counting all differences in the bounded engine
|
||||
result. The panel renders at most 250 differences and 100 alignments at once.
|
||||
|
||||
## PCRE2 C17 generator
|
||||
|
||||
The first reviewed code-generation target is the PCRE2 10.47 8-bit C API.
|
||||
Other languages remain unavailable until their generators have equivalent
|
||||
escaping, semantic and toolchain gates.
|
||||
|
||||
The generator consumes a validated PCRE2 snapshot and:
|
||||
|
||||
- requires the registered PCRE2 10.47 version and flag/option allowlists;
|
||||
- refuses unpaired UTF-16 surrogates rather than emitting lossy UTF-8;
|
||||
- emits pattern, subject and replacement as independent explicit UTF-8 byte
|
||||
arrays, so quotes, backslashes, newlines, NUL, Unicode, dollar signs and
|
||||
braces cannot escape into C source syntax;
|
||||
- enables mandatory `PCRE2_UTF` and `PCRE2_UCP`;
|
||||
- maps `i`, `m`, `s`, `x`, `U` and `J` to named PCRE2 compile constants;
|
||||
- implements application-level `g`/scan-all behavior explicitly;
|
||||
- handles the PCRE2 empty-match anchored retry before advancing one UTF-8
|
||||
character;
|
||||
- configures native match, depth and heap limits;
|
||||
- enforces match/capture row caps in the generated match loop;
|
||||
- bounds replacement output with a host buffer and withholds output when its
|
||||
post-run result-row bound is exceeded;
|
||||
- reports compile offsets and native API errors and cleans up all PCRE2 state;
|
||||
- rejects compilation against a PCRE2 header other than 10.47 and verifies the
|
||||
runtime version.
|
||||
|
||||
The generated program states limitations that cannot be mapped honestly:
|
||||
|
||||
- PCRE2 has no `g` compile flag;
|
||||
- match/capture rows and replacement-output size are host-application bounds,
|
||||
not PCRE2 matching options;
|
||||
- replacement result-row limits can be checked only after
|
||||
`pcre2_substitute()` has completed;
|
||||
- PCRE2 exposes no native wall-clock timeout, so hard elapsed-time
|
||||
cancellation belongs to the containing process;
|
||||
- C API offsets are UTF-8 bytes, not browser editor UTF-16 units.
|
||||
|
||||
Generated snippets are examples only and are never used to implement the
|
||||
browser runtime.
|
||||
|
||||
## Golden and toolchain gate
|
||||
|
||||
`scripts/pcre2-codegen-golden.test.ts` pins deterministic SHA-256 identities
|
||||
for match and replacement fixtures:
|
||||
|
||||
```text
|
||||
match 22b91d578a449b0ed5c30e6eadad30f19a2bcb26ac9db355c8ec7a69328675da
|
||||
replacement ab90a0d9831ea7b7145aa0072ac352b42baf9a795319d81f531b0f341b04a636
|
||||
```
|
||||
|
||||
The compile/execute gate was run against a host static build from official
|
||||
signed tag `pcre2-10.47`, commit
|
||||
`f454e231fe5006dd7ff8f4693fd2b8eb94333429`. Both generated fixtures compile
|
||||
under C17 with `-Wall -Wextra -Werror`; the match fixture verifies native UTF-8
|
||||
ranges and named captures, and the replacement fixture verifies exact Unicode
|
||||
output and substitution count.
|
||||
|
||||
To repeat the gate with an exact local PCRE2 installation:
|
||||
|
||||
```sh
|
||||
PCRE2_CODEGEN_PREFIX=/path/to/pcre2-10.47-prefix \
|
||||
npm test -- scripts/pcre2-codegen-golden.test.ts
|
||||
```
|
||||
|
||||
The prefix must provide `include/pcre2.h` and
|
||||
`lib64/libpcre2-8.a`. A different static-library location can be supplied
|
||||
through `PCRE2_CODEGEN_LIBRARY`.
|
||||
@@ -1,28 +1,74 @@
|
||||
# Engine builds
|
||||
|
||||
Version 0.1.0 ships no WebAssembly engine.
|
||||
The production application uses the browser's native `RegExp` for ECMAScript
|
||||
and bundles a verified PCRE2 10.47 WebAssembly pack. Normal application builds
|
||||
never download or compile an engine: the reviewed files under
|
||||
`public/engines/pcre2/` are copied into `dist/` and verified there.
|
||||
|
||||
`npm run engines:build` verifies that this release has no external pack to
|
||||
build. After the production build, `npm run engines:verify` rejects any
|
||||
undeclared engine asset and confirms the packaged ECMAScript-only notice.
|
||||
Neither command accesses the network.
|
||||
## Rebuilding PCRE2
|
||||
|
||||
## PCRE2 feasibility record
|
||||
The reproducible build accepts only:
|
||||
|
||||
The next flavour was spiked outside the application repository against official
|
||||
PCRE2 10.47:
|
||||
- PCRE2 tag `pcre2-10.47`, signed tag object
|
||||
`cd007b4466798f66d479d1442a407099e7c40050`, peeled commit
|
||||
`f454e231fe5006dd7ff8f4693fd2b8eb94333429` and tree
|
||||
`81a83a3552bd68d0ea7b7004f8bb6e7892f583ba`;
|
||||
- Emscripten 6.0.4, compiler revision
|
||||
`fe5be6afdff43ad58860d821fcc8572a23f92d19`, from emsdk commit
|
||||
`224ec5f9f2f72f09f9ce0e26d66bae7dbd8b692f`;
|
||||
- CMake 4.3.4 and Ninja 1.13.2;
|
||||
- 8-bit PCRE2 with Unicode enabled and JIT, threads and filesystem disabled.
|
||||
|
||||
- signed tag object `cd007b4466798f66d479d1442a407099e7c40050`;
|
||||
- peeled commit `f454e231fe5006dd7ff8f4693fd2b8eb94333429`;
|
||||
- licence `BSD-3-Clause WITH PCRE2-exception`;
|
||||
- Emscripten 6.0.4;
|
||||
- 8-bit library, Unicode enabled, JIT disabled.
|
||||
The source and compiler checkouts must already exist locally and be completely
|
||||
clean. No command clones, downloads or updates them:
|
||||
|
||||
Node and Chromium tests demonstrated version/config reporting, Unicode,
|
||||
named groups, global substitution, automatic and explicit callouts, and
|
||||
match/depth/heap errors. This was a technical spike only. No spike source,
|
||||
binary or PCRE2 source is included in v0.1.0.
|
||||
```sh
|
||||
npm run engines:pcre2:build -- \
|
||||
--source-dir /absolute/path/to/pcre2 \
|
||||
--emcc /absolute/path/to/emscripten/emcc
|
||||
npm run engines:pcre2:verify
|
||||
npm run engines:pcre2:install
|
||||
```
|
||||
|
||||
Production PCRE2 support remains gated on a reviewed bridge, copied DTOs,
|
||||
allocation cleanup, exact source build, UTF-8/UTF-16 mapping, trace/output caps,
|
||||
worker recovery, deterministic assets, licences and browser conformance.
|
||||
Use `--force` only to replace the exact ignored `.engine-build/pcre2` output.
|
||||
The explicit install command first re-verifies that pack and then atomically
|
||||
replaces only `public/engines/pcre2`.
|
||||
|
||||
The builder verifies Git objects and critical source/compiler hashes,
|
||||
sanitizes build flags, sets `SOURCE_DATE_EPOCH=1760997684`, builds serially and
|
||||
emits exactly:
|
||||
|
||||
- `pcre2.mjs` and `pcre2.wasm`;
|
||||
- deterministic `engine-metadata.json`;
|
||||
- the exact upstream `LICENSE.txt`;
|
||||
- `SHA256SUMS`.
|
||||
|
||||
The verifier checks the closed file set, metadata, source bridge hashes and all
|
||||
checksums. It validates and instantiates WebAssembly, checks ABI/engine/config
|
||||
identity, then executes native Unicode compile, bounded match and bounded
|
||||
substitution smoke cases plus normal and cap-stopped automatic-callout traces.
|
||||
|
||||
The immutable signed-tag object is pinned. OpenPGP trust validation remains a
|
||||
release-operator step because the offline builder does not install or trust a
|
||||
key.
|
||||
|
||||
## Runtime boundary
|
||||
|
||||
ABI version 3 has no retained native handles. Each call compiles, obtains
|
||||
capture names, matches/substitutes or traces and releases its code, match data
|
||||
and match context before returning. The caller supplies fixed-capacity
|
||||
result/name/output/event/mark buffers. Hard maxima are 1 MiB pattern, 16 MiB
|
||||
subject, 1,000 captures, 10,000 matches, 100,000 capture rows, 50,000 trace
|
||||
events, 10 MiB serialized trace data and 128 MiB PCRE2 heap limit; the
|
||||
workbench uses stricter pattern and replacement input limits where applicable.
|
||||
|
||||
PCRE2 always runs with UTF and UCP. Native UTF-8 byte offsets are retained in
|
||||
DTOs and converted only at valid code-point boundaries to half-open editor
|
||||
UTF-16 ranges. Lone UTF-16 surrogates are rejected before encoding.
|
||||
|
||||
Normal execution lives only in `pcre2.worker`. Instrumented
|
||||
`PCRE2_AUTO_CALLOUT` collection is a separate ABI operation loaded only by
|
||||
`pcre2-trace.worker`; normal match, replacement, tests, corpus and benchmarks
|
||||
never call it. Each supervisor terminates its worker on timeout, crash,
|
||||
cancellation or supersession and lazily creates a clean generation for the
|
||||
next request.
|
||||
|
||||
@@ -1,13 +1,39 @@
|
||||
# Execution traces
|
||||
|
||||
Version 0.1.0 exposes no execution trace.
|
||||
Browser ECMAScript APIs expose compilation, match, capture and replacement
|
||||
results, but no V8, SpiderMonkey or JavaScriptCore execution trace. Regex Tools
|
||||
does not relabel structural explanations, static findings or timing
|
||||
observations as an ECMAScript engine trace.
|
||||
|
||||
Browser ECMAScript APIs return compilation, match, capture and replacement
|
||||
results but not the runtime's internal backtracking events. Regex Tools does
|
||||
not label structural explanations, static hints or adjacent-result inference
|
||||
as an actual engine trace.
|
||||
PCRE2 10.47 has a separate actual automatic-callout vertical. ABI version 3
|
||||
compiles the requested pattern with `PCRE2_AUTO_CALLOUT`, registers an
|
||||
application callback, and copies only:
|
||||
|
||||
The next PCRE2 slice will use actual automatic/explicit callout events from an
|
||||
application-owned bridge. Event count, serialized bytes, match/depth/heap
|
||||
limits and wall-clock termination must all be enforced. Classifications such
|
||||
as “apparent backtrack” will remain visibly derived from adjacent events.
|
||||
- callout number;
|
||||
- pattern byte position and next-item byte length;
|
||||
- subject byte position;
|
||||
- capture-top and capture-last numbers;
|
||||
- bounded current mark text.
|
||||
|
||||
These fields have `reported` provenance. Pattern and subject byte boundaries
|
||||
are validated and normalized to exact editor UTF-16 positions while the native
|
||||
values remain visible.
|
||||
|
||||
Collection runs in `pcre2-trace.worker`, not the normal execution worker. It
|
||||
uses the selected PCRE2 match, depth and heap limits plus the supervisor wall
|
||||
clock. The callback retains only complete 32-byte records and mark slices under
|
||||
both a 50,000-event and 10 MiB serialized-data hard cap. At a cap it returns
|
||||
PCRE2’s reserved `PCRE2_ERROR_CALLOUT`, abandoning the native match; the result
|
||||
retains that native status, the last complete event and an explicit truncation
|
||||
flag. Timeout, cancellation or crash discards the complete worker.
|
||||
|
||||
The viewer derives “forward”, “same position” and “apparent backtrack” solely
|
||||
from the difference between adjacent reported subject positions. Each label
|
||||
has `derived` provenance. This is useful navigation, not a claim that the
|
||||
stream reports every internal action. PCRE2 start optimizations can also
|
||||
conclude some requests without any callout.
|
||||
|
||||
Tracing represents one exact `pcre2_match()` invocation. The application-level
|
||||
`g` iteration flag is not applied, and that fact is reported. Normal match,
|
||||
replacement, tests, corpus, comparisons and benchmarks never use the
|
||||
instrumented trace path.
|
||||
|
||||
@@ -1,9 +1,9 @@
|
||||
# Explanation model
|
||||
|
||||
The community provider converts regexpp nodes into a stable application AST
|
||||
inside the syntax worker. Nodes carry deterministic IDs, UTF-16 ranges, raw
|
||||
source, capture metadata, quantifier bounds, assertion kind, support status,
|
||||
provider identity and provenance.
|
||||
The ECMAScript community provider converts regexpp nodes into a stable
|
||||
application AST inside the syntax worker. Nodes carry deterministic IDs,
|
||||
UTF-16 ranges, raw source, capture metadata, quantifier bounds, assertion kind,
|
||||
support status, provider identity and provenance.
|
||||
|
||||
Explanations are original deterministic text selected by node type. Nullable
|
||||
and minimum/maximum consumed-length properties are derived recursively where
|
||||
@@ -15,6 +15,13 @@ regexpp 4.12.2 does not expose tolerant recovery. A malformed pattern therefore
|
||||
gets the provider's exact diagnostic plus an application-owned error node. No
|
||||
partial regexpp AST is claimed.
|
||||
|
||||
The PCRE2 provider currently emits a deliberately partial lexical tree for
|
||||
common capture forms. It does not infer branch-reset numbering or claim full
|
||||
nesting; authoritative compilation, capture counts, names and result ranges
|
||||
come from PCRE2 itself.
|
||||
|
||||
The explanation tree is structural. It is never called an actual execution
|
||||
trace. Selecting a node selects and scrolls its exact editor range; selecting
|
||||
pattern text chooses the smallest enclosing node.
|
||||
pattern text chooses the smallest enclosing node. PCRE2’s separate trace viewer
|
||||
contains actual bounded automatic callouts; its movement labels remain a
|
||||
distinct derived layer and are never copied into the structural explanation.
|
||||
|
||||
@@ -1,20 +1,55 @@
|
||||
# Flavour support
|
||||
|
||||
| Flavour | Syntax | Execution | Replacement | Captures | Trace | Native offsets |
|
||||
| ------------ | ------------------------------------------------------------------ | ----------------------------------------------- | ---------------------------------------------------- | -------------------------------------------------- | ----------- | ------------------- |
|
||||
| ECMAScript | regexpp 4.12.2 grammar; explanations partial/feature-matrix tested | Current browser `RegExp`; feature-matrix tested | Native matches; bounded ECMAScript `GetSubstitution` | Named/numbered; final repeated capture; no history | Unavailable | UTF-16 |
|
||||
| PCRE2 | Unavailable in release | Technical spike only; not shipped | Unavailable | Unavailable | Unavailable | Planned UTF-8 bytes |
|
||||
| PCRE | Unavailable | Unavailable | Unavailable | Unavailable | Unavailable | Unavailable |
|
||||
| Python `re` | Unavailable | Unavailable | Unavailable | Unavailable | Unavailable | Planned code points |
|
||||
| Go `regexp` | Unavailable | Unavailable | Unavailable | Unavailable | Unavailable | Planned UTF-8 bytes |
|
||||
| Rust `regex` | Unavailable | Unavailable | Unavailable | Unavailable | Unavailable | Planned UTF-8 bytes |
|
||||
| .NET | Unavailable | Unavailable | Unavailable | Planned capture history | Unavailable | Planned UTF-16 |
|
||||
| Java | Unavailable | Unavailable | Unavailable | Unavailable | Unavailable | Planned UTF-16 |
|
||||
| Flavour | Syntax | Execution | Replacement | Captures | Trace | Analysis | Generated cases | Native offsets |
|
||||
| ------------ | ------------------------------------------------------------ | ------------------------------------------------------ | ---------------------------------------- | -------------------------------------------------- | ------------------------------------------------------------ | ------------------------------------------------------ | ------------------------------------------------------- | ------------------- |
|
||||
| ECMAScript | regexpp 4.12.2; explanations partial/feature-matrix tested | Current browser `RegExp`; conformance tested | Bounded ECMAScript `GetSubstitution` | Named/numbered; final repeated capture; no history | Unavailable | Experimental static heuristics + bounded native probes | Bounded normalized-AST sampling; actual-engine verified | UTF-16 |
|
||||
| PCRE2 10.47 | Application lexical provider partial; compiler authoritative | Pinned 8-bit WASM; bounded/killable/conformance tested | Native bounded `pcre2_substitute()` loop | Native named/numbered records; no capture history | Bounded reported automatic callouts; derived movement labels | Unavailable; ECMAScript heuristics are never applied | Unavailable; structural syntax provider is partial | UTF-8 bytes |
|
||||
| PCRE | Unavailable | Unavailable | Unavailable | Unavailable | Unavailable | Unavailable | Unavailable | Unavailable |
|
||||
| Python `re` | Unavailable | Unavailable | Unavailable | Unavailable | Unavailable | Unavailable | Unavailable | Planned code points |
|
||||
| Go `regexp` | Unavailable | Unavailable | Unavailable | Unavailable | Unavailable | Unavailable | Unavailable | Planned UTF-8 bytes |
|
||||
| Rust `regex` | Unavailable | Unavailable | Unavailable | Unavailable | Unavailable | Unavailable | Unavailable | Planned UTF-8 bytes |
|
||||
| .NET | Unavailable | Unavailable | Unavailable | Planned capture history | Unavailable | Unavailable | Unavailable | Planned UTF-16 |
|
||||
| Java | Unavailable | Unavailable | Unavailable | Unavailable | Unavailable | Unavailable | Unavailable | Planned UTF-16 |
|
||||
|
||||
ECMAScript tests cover `g`, `y`, `u`, internal `d`, zero-length iteration,
|
||||
named groups, unmatched/empty/repeated groups, lookbehind, backreferences,
|
||||
Unicode properties, astral input, replacement and result truncation.
|
||||
ECMAScript tests cover its complete exposed flag matrix, internal indices,
|
||||
zero-length iteration, captures, lookbehind, backreferences, Unicode properties
|
||||
and bounded replacement.
|
||||
|
||||
Actual browser identity is displayed because ECMAScript behavior can evolve
|
||||
with the runtime. Syntax acceptance and engine compilation are separate;
|
||||
neither silently rewrites the pattern.
|
||||
PCRE2 tests cover `g`, `i`, `m`, `s`, `x`, `U`, `J`, mandatory UTF/UCP,
|
||||
branch-reset groups, recursion, `\K`, duplicate names, native substitution,
|
||||
astral byte/UTF-16 ranges, zero-length iteration, compile errors, match limits
|
||||
and result/output truncation. Trace tests cover reported automatic callouts,
|
||||
bounded mark copying, native stop at the complete-event cap, compile ranges,
|
||||
worker loading and browser rendering.
|
||||
|
||||
ECMAScript static findings and generated-input timing observations are
|
||||
advisory. They describe only the selected ECMAScript 2025 pattern, browser
|
||||
runtime and bounded inputs; they are not proofs of safety or general
|
||||
complexity.
|
||||
|
||||
ECMAScript generated-case version 1 uses only a complete accepted normalized
|
||||
AST, an explicit deterministic seed and bounded sampling strategies. Every
|
||||
retained positive or negative label is confirmed by the actual selected
|
||||
browser engine. PCRE2 generation is unavailable because its current lexical
|
||||
provider is intentionally partial; ECMAScript behavior is never relabelled as
|
||||
PCRE2.
|
||||
|
||||
The PCRE2 explanation provider deliberately does not invent full structural
|
||||
coverage. In particular, branch-reset numbering is omitted from the syntax tree
|
||||
while actual capture numbers and names still come from PCRE2 results.
|
||||
|
||||
ECMAScript and PCRE2 support exact-request comparison for syntax acceptance,
|
||||
engine compilation, matches, captures and engine-native replacement. Results
|
||||
align by normalized UTF-16 ranges while retaining each native offset model.
|
||||
Timeout, cancellation, rejection and truncation are never presented as
|
||||
equivalent results.
|
||||
|
||||
Pattern formatting is available only for an exact accepted ECMAScript regexpp
|
||||
snapshot. It canonicalizes literal slashes and raw control/line-separator
|
||||
representations, then requires paired actual-engine and applicable exact-test
|
||||
validation before apply. PCRE2 formatting is unavailable because the current
|
||||
provider is deliberately partial; its syntax is never treated as ECMAScript.
|
||||
|
||||
PCRE2 is the only current code-generation target. The reviewed C17 generator is
|
||||
pinned to the PCRE2 10.47 8-bit API; other language generators remain
|
||||
unavailable until they pass equivalent escaping and named-toolchain gates.
|
||||
|
||||
57
docs/FORMATTING.md
Normal file
57
docs/FORMATTING.md
Normal file
@@ -0,0 +1,57 @@
|
||||
# Pattern formatting
|
||||
|
||||
Regex Tools includes a deliberately narrow, local ECMAScript formatter. It
|
||||
canonicalizes representations that can be changed without inventing a visual
|
||||
layout:
|
||||
|
||||
- an unescaped literal `/` becomes `\/`, making the pattern safe to copy into
|
||||
an ECMAScript regex literal;
|
||||
- raw NUL and control code units become fixed-width `\xNN` or their canonical
|
||||
short escape (`\t`, `\n`, `\v`, `\f`, `\r`);
|
||||
- raw U+2028 and U+2029 line separators become `\u2028` and `\u2029`;
|
||||
- legacy escaped raw controls are replaced as one parser-reported literal
|
||||
token, so the formatter does not introduce an extra backslash.
|
||||
|
||||
Existing explicit escape spellings, groups, alternatives, quantifiers, inline
|
||||
flags and literal whitespace are otherwise preserved byte for byte. ECMAScript
|
||||
regex whitespace is normally significant, so the tool does not insert
|
||||
indentation, line breaks or comments and does not claim to be a structural
|
||||
pretty-printer. The transform is idempotent.
|
||||
|
||||
## Eligibility
|
||||
|
||||
Formatting requires the exact current, accepted, non-recovered syntax snapshot
|
||||
from the bundled `@eslint-community/regexpp` provider. Transformations are
|
||||
derived from the provider's literal-token ranges rather than from a heuristic
|
||||
text scan. A stale parse, a malformed pattern, a partial provider, PCRE2 or any
|
||||
other flavour is reported as unavailable and is never silently rewritten.
|
||||
|
||||
## Mandatory validation before apply
|
||||
|
||||
The preview cannot be applied directly. Regex Tools first:
|
||||
|
||||
1. reparses source and candidate independently and checks capture numbering,
|
||||
names, repetition and parent-capture shape;
|
||||
2. compiles and runs the source and candidate in separate actual ECMAScript
|
||||
engine workers against the current subject and replacement;
|
||||
3. compares exact normalized match, capture and replacement output, including
|
||||
the engine identity;
|
||||
4. reruns every enabled unit test whose flavour, version, pattern, flags and
|
||||
engine options exactly match the active source snapshot, and compares both
|
||||
semantic output and assertion outcome.
|
||||
|
||||
A compile rejection, one-sided timeout, crash, worker error, cancellation,
|
||||
truncated result, engine-identity change, incomplete test suite or semantic
|
||||
difference blocks application. Matching fixed timeouts are comparable only for
|
||||
an exact unit-test pair; two timeouts on the current interactive snapshot are
|
||||
inconclusive. If no unit test is applicable, the UI says so and still requires
|
||||
the current subject/replacement gate.
|
||||
|
||||
Validation is bounded to 1,000 tests, the normal worker/input limits and a
|
||||
60-second aggregate test wall budget. A test is not started unless its complete
|
||||
configured timeout still fits that budget. The user must explicitly confirm
|
||||
the exact validated candidate before the Apply button is enabled.
|
||||
|
||||
Passing these gates is strong, reproducible evidence for the current snapshot
|
||||
and exact tests. It is not a mathematical proof of equivalence for every
|
||||
possible subject.
|
||||
135
docs/GENERATED_CASES.md
Normal file
135
docs/GENERATED_CASES.md
Normal file
@@ -0,0 +1,135 @@
|
||||
# Generated test cases
|
||||
|
||||
Regex Tools generator version 1 creates deterministic candidate subjects from
|
||||
the application-owned normalized AST and then verifies every candidate through
|
||||
the selected actual engine adapter. Generation is local-only and bounded. It is
|
||||
sampling, not proof of language coverage, equivalence or safety.
|
||||
|
||||
## Supported scope
|
||||
|
||||
Version 1 is scoped to an accepted ECMAScript 2025 normalized AST from the
|
||||
bundled regexpp-based provider. It requests:
|
||||
|
||||
- a shortest structural candidate using the shortest visible alternative and
|
||||
each quantifier minimum;
|
||||
- each structurally visible alternative;
|
||||
- quantifier minimum, adjacent and bounded upper values;
|
||||
- optional-absent and optional-present values;
|
||||
- bounded character-class, dot and Unicode-property representatives;
|
||||
- newline and non-boundary context around anchors;
|
||||
- deletion, replacement and boundary mutations as likely near misses;
|
||||
- deterministic random alternative, quantifier and class choices.
|
||||
|
||||
The coverage report says `covered`, `partial`, `unsupported` or
|
||||
`not applicable` for each category. “Covered” means that the documented bounded
|
||||
sampling strategy was requested. It does not mean every string in the regular
|
||||
language was enumerated.
|
||||
|
||||
PCRE2 generation is unavailable in version 1. Its actual engine is complete for
|
||||
the advertised execution operations, but its current structural syntax
|
||||
provider is intentionally partial. Regex Tools does not feed PCRE2 syntax to
|
||||
the ECMAScript generator or relabel ECMAScript behavior.
|
||||
|
||||
## Unsupported and partial constructs
|
||||
|
||||
The report includes exact normalized source ranges and a reason for every
|
||||
recognized gap, capped at 250 rendered entries. Version 1 does not solve:
|
||||
|
||||
- capture-dependent backreferences;
|
||||
- lookaround or word-boundary constraints symbolically;
|
||||
- recursion, subroutine calls or conditional execution;
|
||||
- branch-reset and atomic grouping;
|
||||
- engine control verbs or callouts;
|
||||
- complex Unicode-set algebra and properties of strings;
|
||||
- recovered or provider-unsupported syntax.
|
||||
|
||||
Surrounding candidates may still happen to satisfy a lookaround or boundary.
|
||||
They are retained only when the actual selected engine confirms the requested
|
||||
match outcome. That confirmation does not upgrade the generator’s structural
|
||||
coverage claim.
|
||||
|
||||
## Determinism and seed
|
||||
|
||||
The seed is explicit text of 1–128 UTF-16 units. Generator version 1 hashes its
|
||||
UTF-16 code units with a documented stable 32-bit FNV-1a step and drives an
|
||||
application-owned 32-bit deterministic sequence. It never calls
|
||||
`Math.random()`.
|
||||
|
||||
The same generator version, normalized AST, flags, settings and seed produce
|
||||
the same candidate order and candidate identifiers. Verification timing is
|
||||
environment-dependent, and strict wall-time or engine-timeout bounds can
|
||||
therefore stop two runs at different points. The UI records both the original
|
||||
seed and its eight-digit hash. Generated unit tests retain:
|
||||
|
||||
- generator id and version;
|
||||
- the original seed;
|
||||
- candidate id;
|
||||
- intended positive or negative outcome.
|
||||
|
||||
That provenance survives project and test-suite JSON export when subjects are
|
||||
included. Editing a generated test turns it into a user-authored test and
|
||||
removes the generation provenance.
|
||||
|
||||
## Actual-engine verification
|
||||
|
||||
Candidate synthesis runs in a dedicated killable worker. The verifier then
|
||||
sends each exact candidate, pattern, flavour version, flags, engine options and
|
||||
scan mode through `EngineSupervisor`, which selects the registered real engine
|
||||
worker.
|
||||
|
||||
- An intended positive is labelled `should-match` only after the engine returns
|
||||
at least one match.
|
||||
- An intended negative is labelled `should-not-match` only after the engine
|
||||
returns no match.
|
||||
- A candidate with the opposite result is discarded and retained only as a
|
||||
bounded advanced diagnostic.
|
||||
- Engine compile rejection, timeout, worker crash, cancellation and ordinary
|
||||
execution error remain distinct outcomes.
|
||||
- Cancellation terminates both candidate-synthesis and active engine workers.
|
||||
- Configuration changes invalidate results immediately; old cases are never
|
||||
displayed against a new pattern or engine configuration.
|
||||
|
||||
The engine name, version, adapter, runtime and native offset unit shown with the
|
||||
result come from the actual execution result, not static marketing metadata.
|
||||
|
||||
## Bounds
|
||||
|
||||
Defaults:
|
||||
|
||||
| Bound | Default |
|
||||
| -------------------------------------- | ------------------------: |
|
||||
| Retained verified cases | 48 |
|
||||
| Candidate attempts | 192 |
|
||||
| Seeded random variants | 16 |
|
||||
| Per-case engine timeout | 500 ms |
|
||||
| Aggregate wall time | 15 s |
|
||||
| Bytes per candidate subject | 64 KiB |
|
||||
| Aggregate candidate subject bytes | 1 MiB |
|
||||
| Synthesized repetitions per quantifier | 32 |
|
||||
| Normalized AST traversal | 20,000 nodes / 512 levels |
|
||||
|
||||
Hard caps:
|
||||
|
||||
| Bound | Hard cap |
|
||||
| -------------------------------------- | -------: |
|
||||
| Retained verified cases | 1,000 |
|
||||
| Candidate attempts | 4,000 |
|
||||
| Per-case engine timeout | 10 s |
|
||||
| Aggregate wall time | 30 s |
|
||||
| Bytes per candidate subject | 16 MiB |
|
||||
| Aggregate candidate subject bytes | 16 MiB |
|
||||
| Synthesized repetitions per quantifier | 1,024 |
|
||||
| Advanced discard diagnostics | 250 |
|
||||
|
||||
UTF-8 sizes are checked before a candidate is retained for verification.
|
||||
Quantifier expansion is preflighted before `String.prototype.repeat` allocates
|
||||
the output. Reaching a case, attempt, byte, AST, repetition or wall-time bound
|
||||
is visible in the result; it is never reported as complete coverage.
|
||||
|
||||
## Privacy and persistence
|
||||
|
||||
No pattern, candidate, verified subject, diagnostic or seed is uploaded.
|
||||
Results and settings are ephemeral until the user explicitly adds cases to the
|
||||
unit-test suite or downloads the verified JSON artifact. Test-suite export
|
||||
omits subjects by default because generated subjects may still encode literal
|
||||
pattern content. Enabling subject export is an explicit user choice.
|
||||
71
docs/MINIMIZATION.md
Normal file
71
docs/MINIMIZATION.md
Normal file
@@ -0,0 +1,71 @@
|
||||
# Subject minimization
|
||||
|
||||
The **Minimize** workspace reduces one failing subject locally against the
|
||||
selected real ECMAScript or PCRE2 worker. It can preserve:
|
||||
|
||||
- the exact failure class of any supported saved unit-test assertion;
|
||||
- an identified capture's wrong participation, value or UTF-16 range;
|
||||
- an engine worker timeout at one fixed deadline;
|
||||
- an ECMAScript/PCRE2 semantic mismatch with the same mismatch-kind set; or
|
||||
- the same one-sided comparison timeout while the other engine completes with
|
||||
authoritative, untruncated output.
|
||||
|
||||
The fixed pattern, ordered flags, options, replacement and engine versions form
|
||||
the target identity. Syntax is parsed once to obtain capture metadata. Every
|
||||
accepted subject candidate is then verified through the selected execution
|
||||
worker; the reducer does not contain a substitute regex implementation.
|
||||
|
||||
## Deterministic transforms
|
||||
|
||||
Subjects must be well-formed Unicode. The reducer never splits a surrogate
|
||||
pair and rejects input containing a lone surrogate.
|
||||
|
||||
Reduction order is stable:
|
||||
|
||||
1. contiguous ddmin chunk deletion, left to right;
|
||||
2. single Unicode-scalar deletion, left to right; then
|
||||
3. lower-rank replacement by `a`, `0`, space, newline and `!`, in that order.
|
||||
|
||||
The last two passes restart after every accepted change and continue to a fixed
|
||||
point. A complete pass with no accepted candidate establishes only **local
|
||||
minimality under those transforms**. It is not a proof of the shortest,
|
||||
globally minimal or semantically simplest reproducer. Evaluation or wall-budget
|
||||
exhaustion, an inconclusive candidate, cancellation and worker failure always
|
||||
return a result without a minimality claim.
|
||||
|
||||
## Exact predicates
|
||||
|
||||
A baseline run must reproduce the requested failure before reduction starts.
|
||||
Unit-test failures retain their assertion class; a wrong capture value cannot
|
||||
turn into a missing capture, and a semantic comparison retains the exact sorted
|
||||
set of baseline mismatch kinds. Rejected patterns and truncated match,
|
||||
capture or replacement data are not accepted as semantic evidence.
|
||||
|
||||
Timeout is a worker deadline outcome, never a no-match. An explicit timeout
|
||||
predicate accepts only `WorkerRequestError("timeout")` at the configured fixed
|
||||
deadline. Comparison timeout reduction additionally requires the same flavour
|
||||
to time out and the peer flavour to complete with the same engine identity and
|
||||
complete data. Crash, unreadable worker response, ordinary worker error and
|
||||
cancellation have separate statuses. Crash minimization is unsupported and a
|
||||
crash never satisfies a timeout predicate.
|
||||
|
||||
For a `must-time-out` unit-test assertion, an ordinary completion is the test
|
||||
failure being minimized; a timeout makes that assertion pass. The separate
|
||||
**Exact engine timeout** target is used to minimize an observed timeout.
|
||||
|
||||
## Bounds and progress
|
||||
|
||||
- Subject: 1 MiB UTF-8, separate from the larger interactive editor cap.
|
||||
- Candidate evaluations: at most 2,000, including the baseline.
|
||||
- Aggregate syntax-setup and reduction wall time: at most 60 seconds.
|
||||
- Candidate worker deadline: fixed for the complete run, at most 10 seconds.
|
||||
- Retained accepted-transform history: 200 entries; totals remain exact.
|
||||
|
||||
A candidate starts only when its full fixed deadline fits inside the remaining
|
||||
aggregate wall budget. This prevents the aggregate deadline from being
|
||||
misreported as an engine timeout. Progress exposes phase, evaluation and wall
|
||||
budgets, accepted reductions, current scalar/byte sizes and the latest
|
||||
observation. Cancellation terminates syntax and execution workers.
|
||||
|
||||
The minimized subject can be copied or deliberately applied to the main editor.
|
||||
Inputs and results stay in memory and are neither uploaded nor persisted.
|
||||
@@ -7,10 +7,13 @@ community
|
||||
```
|
||||
|
||||
It uses only reviewed open-source dependencies and builds without private
|
||||
registry credentials. Version 0.1.0 parses ECMAScript with regexpp 4.12.2.
|
||||
registry credentials. ECMAScript uses regexpp 4.12.2. PCRE2 uses an
|
||||
application-owned lexical provider for partial explanations and the actual
|
||||
bundled PCRE2 compiler for authoritative acceptance.
|
||||
|
||||
The product deliberately does not implement or reference an `@r101/parser`
|
||||
profile. No commercial package alias, import, tarball, credential, licence key
|
||||
or private CI path exists. Additional flavour syntax will be implemented
|
||||
incrementally as open-source providers behind the same application-owned
|
||||
interface.
|
||||
interface. Partial coverage is exposed as metadata and never presented as a
|
||||
complete parser.
|
||||
|
||||
66
docs/PCRE2_NEXT_SLICES.md
Normal file
66
docs/PCRE2_NEXT_SLICES.md
Normal file
@@ -0,0 +1,66 @@
|
||||
# PCRE2 advanced slices
|
||||
|
||||
This document records the reviewed boundaries for advanced PCRE2 work. A
|
||||
feature is exposed only after its complete runtime and UI path passes the
|
||||
stated gates.
|
||||
|
||||
## Automatic-callout trace — implemented
|
||||
|
||||
ABI version 3 and the dedicated trace worker now provide:
|
||||
|
||||
- a new ABI operation separate from normal matching, compiled with
|
||||
`PCRE2_AUTO_CALLOUT`;
|
||||
- a fixed copied event record containing only reported callout number, pattern
|
||||
byte position, next-item length, subject byte position, capture top/last and
|
||||
bounded mark text;
|
||||
- caller-supplied event and text buffers capped by both
|
||||
`maximumTraceEvents` and `maximumTraceBytes`;
|
||||
- explicit `eventsTruncated`, native status and last-complete-event fields;
|
||||
- the existing match/depth/heap limits and supervisor timeout;
|
||||
- cleanup identical to normal ABI calls.
|
||||
|
||||
Tracing runs in its own worker and never replaces the normal result used by
|
||||
Match, Replace, Tests, Corpus or Analysis. “Forward”, “same position” and
|
||||
“apparent backtrack” are derived from adjacent reported positions only and
|
||||
carry derived provenance. The viewer explicitly states that a callout stream
|
||||
is not a complete account of every internal engine action.
|
||||
|
||||
## ECMAScript versus PCRE2 comparison — implemented
|
||||
|
||||
The comparison orchestrator issues two explicit requests with
|
||||
`Promise.allSettled` through independently killable flavour workers. It does
|
||||
not translate flags, patterns or replacement syntax.
|
||||
|
||||
The comparison DTO retains:
|
||||
|
||||
- each exact request identity, engine identity and independently complete or
|
||||
timed-out result;
|
||||
- match/capture differences aligned by editor UTF-16 ranges, while retaining
|
||||
each native offset model;
|
||||
- explicit “not comparable” states for syntax/compile failure, truncation,
|
||||
timeout, cancellation, worker failure or flavour-only constructs;
|
||||
- a shared wall-clock cancellation action that terminates both workers.
|
||||
|
||||
The orchestrator remains outside `EngineSupervisor`, preserving the
|
||||
single-engine lifecycle boundary. Shared patterns are sent unchanged and
|
||||
explicit per-flavour variants remain user-owned. Agreement is labelled only as
|
||||
evidence for the current subject.
|
||||
|
||||
## PCRE2 C code generation — implemented
|
||||
|
||||
The first reviewed target consumes a validated request snapshot and uses an
|
||||
application-owned C17 template. It:
|
||||
|
||||
- identifies exact PCRE2 10.47, 8-bit, UTF/UCP semantics and selected flags;
|
||||
- emits pattern, subject and replacement independently as exact UTF-8 byte
|
||||
arrays;
|
||||
- includes match/depth/heap limits and error handling in generated C examples;
|
||||
- states when a target binding cannot express an application-level `g` flag or
|
||||
a Regex Tools output/result cap directly;
|
||||
- ships deterministic golden match/replacement fixtures that compile and
|
||||
execute against official PCRE2 10.47 before the target is advertised.
|
||||
|
||||
No generated snippet may be used as the implementation of the browser runtime.
|
||||
Other target languages remain unavailable until their own escaping and
|
||||
toolchain gates pass. See
|
||||
[`COMPARISON_AND_CODEGEN.md`](COMPARISON_AND_CODEGEN.md).
|
||||
@@ -10,6 +10,18 @@ browser, engine version, pattern and subject. The UI therefore reports observed
|
||||
elapsed time and exact runtime identity but makes no universal throughput or
|
||||
safety claim.
|
||||
|
||||
PCRE2 cold execution includes loading and instantiating the self-hosted
|
||||
WebAssembly module inside its worker; warm requests reuse that instance.
|
||||
Every request still compiles and frees its pattern so no unbounded compiled-code
|
||||
cache can accumulate. Native match-step, depth and heap limits complement the
|
||||
supervisor wall clock, and cancellation discards the complete worker heap.
|
||||
|
||||
Automatic-callout tracing is deliberately excluded from every benchmark. It
|
||||
compiles an independently instrumented PCRE2 pattern in a separate worker and
|
||||
can materially change runtime work. The trace path retains at most 50,000
|
||||
complete 32-byte event records and 10 MiB total serialized event/mark bytes;
|
||||
the viewer renders at most 2,000 retained events.
|
||||
|
||||
The production bundle separates syntax and execution workers. Interactive
|
||||
syntax/extraction trees and editor decoration sets render at most 2,000 items
|
||||
each and report both the rendered and actual totals. These are presentation
|
||||
@@ -62,5 +74,57 @@ representation against a byte budget before materializing or writing it.
|
||||
IndexedDB schema version 2 indexes `updatedAt`; latest-project loading opens one
|
||||
descending cursor and validates only that record instead of calling `getAll()`.
|
||||
|
||||
Formal cold/warm benchmarks, p95 statistics and growth charts are deferred to
|
||||
the analysis milestone and will run in killable workers.
|
||||
Corpus runs accept at most 256 documents, 16 MiB per document and 256 MiB of
|
||||
aggregate UTF-8 input. Jobs run sequentially so only one engine request is
|
||||
active at a time. A batch retains at most 100,000 matches and 64 MiB of applied
|
||||
output, and stops after five minutes of aggregate wall time. Each document also
|
||||
uses the selected manual timeout. Progress is reported after every document;
|
||||
cancellation kills the current worker and labels all remaining documents.
|
||||
Result tables contain summaries, not match/capture objects. ZIP export is
|
||||
available only for an all-exact apply batch and is created asynchronously.
|
||||
|
||||
Independent-line mode preflights at most 100,000 logical lines and processes
|
||||
each as an isolated engine subject while preserving line separators for exact
|
||||
apply output. Per document, corpus extraction retains at most 32 group summaries
|
||||
(including the full match), three 160-UTF-16-unit samples per group and 100
|
||||
unique diagnostics. Output previews show 2,000 units. Content-free summary
|
||||
exports are capped at 4 MiB; the applied archive is capped at 68 MiB in addition
|
||||
to the 64 MiB uncompressed-output budget.
|
||||
|
||||
ECMAScript analysis reports cold compilation separately from warm native
|
||||
execution. The default benchmark uses three warm-up and 15 measured samples;
|
||||
user caps are 100 warm-ups and 1,000 samples. It reports median and
|
||||
nearest-rank p95 values plus separate timing and outcome/growth plots with the
|
||||
exact runtime identity. Each sample defaults to a 2-second killable-worker
|
||||
limit (10-second cap), and the aggregate analysis wall time is 30 seconds.
|
||||
|
||||
Growth probes retain at most 24 steps, generate at most 1 MiB by default under
|
||||
the central 16 MiB subject cap, and reject more than 10,000,000 repetitions
|
||||
before string allocation. Replacement timing is skipped when match collection
|
||||
is truncated or the estimated output exceeds 8 Mi UTF-16 units. Results are
|
||||
observations for the chosen runtime and generated inputs, never a general
|
||||
complexity proof. See [`ANALYSIS.md`](ANALYSIS.md).
|
||||
|
||||
Generated-case synthesis defaults to 48 retained cases from at most 192
|
||||
candidates, 16 seeded-random variants, 64 KiB per candidate, 1 MiB aggregate
|
||||
candidate bytes and 15 seconds of aggregate wall time. Every candidate then
|
||||
uses a separate selected-engine request with a 500 ms default timeout.
|
||||
Generation preflights UTF-8 size and repetition bounds, caps normalized-AST
|
||||
traversal at 20,000 nodes / 512 levels and retains only bounded diagnostics.
|
||||
Hard caps are documented in [`GENERATED_CASES.md`](GENERATED_CASES.md).
|
||||
|
||||
Subject minimization accepts at most 1 MiB UTF-8, performs at most 2,000 real
|
||||
engine candidate evaluations and stops after at most 60 seconds including
|
||||
syntax setup. One fixed worker timeout of at most 10 seconds applies to every
|
||||
candidate. A candidate starts only when that complete deadline fits in the
|
||||
remaining aggregate budget. Accepted-transform history retains 200 entries;
|
||||
evaluation and accepted totals remain exact. See
|
||||
[`MINIMIZATION.md`](MINIMIZATION.md).
|
||||
|
||||
Formatter validation uses two syntax supervisors and two actual-engine
|
||||
supervisors so source/candidate outcomes cannot share compiled state. The
|
||||
current subject/replacement snapshot always runs; at most 1,000 enabled tests
|
||||
with the exact active identity are replayed. Test validation has a 60-second
|
||||
aggregate wall budget, and a test starts only when its complete configured
|
||||
worker timeout still fits. Preview details retain at most 10,000 transforms and
|
||||
the UI renders the first 200. See [`FORMATTING.md`](FORMATTING.md).
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# Portal requirements
|
||||
|
||||
Regex Tools 0.1.0 is a static nested-path-safe application.
|
||||
Regex Tools is a static nested-path-safe application.
|
||||
|
||||
Required delivery behavior:
|
||||
|
||||
@@ -9,11 +9,17 @@ Required delivery behavior:
|
||||
revalidation/no-cache resources.
|
||||
- Hashed Vite assets may use immutable long-term caching.
|
||||
- CSP must allow `worker-src 'self' blob:`.
|
||||
- No WebAssembly MIME rule or `wasm-unsafe-eval` is required by v0.1.0.
|
||||
- `engines/pcre2/pcre2.wasm` must use `application/wasm`.
|
||||
- `script-src` must permit self-hosted WebAssembly compilation, normally with
|
||||
`'wasm-unsafe-eval'`; broad `'unsafe-eval'` is neither needed nor recommended.
|
||||
- `engines/pcre2/pcre2.mjs`, metadata, checksums, licence and WASM should use
|
||||
revalidation or a release-scoped immutable cache. They are not
|
||||
content-hashed filenames and must never drift under one release identity.
|
||||
|
||||
The Portal must consume the immutable release ZIP, verify its SHA-256, verify
|
||||
manifest ID `de.add-ideas.regex-tools` and version `0.1.0`, and mount it at a
|
||||
manifest ID `de.add-ideas.regex-tools` and version `0.2.0`, and mount it at a
|
||||
relative target such as `apps/regex/`.
|
||||
|
||||
When PCRE2 ships later, the smallest additional change will be
|
||||
`application/wasm` delivery and `script-src 'self' 'wasm-unsafe-eval'`.
|
||||
The package, manifest, tag, artifact URL and checksum must all use the v0.2.0
|
||||
identity. The historical v0.1.0 artifact coordinates remain immutable and must
|
||||
not be overwritten or reused.
|
||||
|
||||
@@ -8,7 +8,7 @@ Inspection date: 2026-07-24.
|
||||
| Toolbox SDK | `53c40a61ba1581246f65773fcbb1c1cfd31ac98e` / 0.2.2 | Apache-2.0 | Contract, shell and release conventions; package APIs used, no source adapted. |
|
||||
| Toolbox Portal | `a9c31c8986c40a0097966318e925083302e91e13` / 0.5.0 | AGPL-3.0-only | Assembly/lock conventions inspected; no source copied into the app. |
|
||||
| regexpp | 4.12.2; npm integrity `sha512-EriSTlt5OC9/7SXkRSCAhfSxxoSUgBm33OH+IkwbdpgoqsSsUg7y3uh+IICI/Qg4BBWr3U2i39RpmycbxMq4ew==` | MIT | Pinned ECMAScript syntax provider. Provider AST is normalized in a worker. |
|
||||
| PCRE2 | signed tag 10.47; commit `f454e231fe5006dd7ff8f4693fd2b8eb94333429` | BSD-3-Clause WITH PCRE2-exception | External technical spike only; not shipped or copied. |
|
||||
| PCRE2 | signed tag 10.47; commit `f454e231fe5006dd7ff8f4693fd2b8eb94333429` | BSD-3-Clause WITH PCRE2-exception | Pinned offline build; generated verified WebAssembly pack is shipped, source is not vendored. |
|
||||
| ECMAScript specification | <https://tc39.es/ecma262/>; living specification inspected 2026-07-24 | ECMA terms | Semantics reference; no prose copied. |
|
||||
| regex101 | <https://regex101.com/> and public parser documentation | Product/commercial terms | Product reference only. No source, branding, explanations, reference prose or visual design copied. |
|
||||
| `recheck` | 4.5.0 | MIT | Metadata inspected; not installed or shipped. |
|
||||
@@ -17,5 +17,5 @@ Inspection date: 2026-07-24.
|
||||
| RegexLib | Website inspected | Redistribution licence not established | No fixture copied. |
|
||||
| RGXP.RU | Website inspected | Redistribution licence not established | No fixture copied. |
|
||||
|
||||
All v0.1.0 conformance cases are project-authored. See
|
||||
All v0.1.0 and v0.2.0 conformance cases are project-authored. See
|
||||
`tests/fixtures/README.md`.
|
||||
|
||||
@@ -18,6 +18,9 @@ on publication failure, replaces an existing exact version only with
|
||||
`--force`, and produces:
|
||||
|
||||
```text
|
||||
release/regex-tools-0.1.0.zip
|
||||
release/regex-tools-0.1.0.zip.sha256
|
||||
release/regex-tools-0.2.0.zip
|
||||
release/regex-tools-0.2.0.zip.sha256
|
||||
```
|
||||
|
||||
The existing `release/regex-tools-0.1.0.zip` and matching sidecar are historical
|
||||
immutable artifacts. Creating v0.2.0 must not replace, rename or remove them.
|
||||
|
||||
@@ -9,6 +9,16 @@ untrusted.
|
||||
- A terminated worker is never reused.
|
||||
- Pattern, subject, replacement/list template, match, capture and output sizes
|
||||
are bounded.
|
||||
- PCRE2 adds native match-step, depth and heap caps. Its C ABI returns only
|
||||
caller-owned bounded records and releases code, match data and match context
|
||||
on every exit path.
|
||||
- PCRE2 automatic-callout tracing is a separate ABI and worker path. It copies
|
||||
only complete fixed records and bounded mark text, stops matching at the
|
||||
50,000-event or 10 MiB serialized trace cap, and is killed on timeout,
|
||||
cancellation or crash.
|
||||
- PCRE2 runs only in mandatory UTF/UCP mode. Exact UTF-8 byte boundaries are
|
||||
normalized to editor UTF-16 offsets; lone surrogates are rejected instead of
|
||||
being silently replaced by browser encoding.
|
||||
- Replacement templates are strings; executable callbacks are unavailable.
|
||||
- Replacement/list token decorations, rows, chips, diagnostics and list
|
||||
evaluation work have separate presentation/operation caps; incomplete list
|
||||
@@ -25,14 +35,51 @@ untrusted.
|
||||
- Unit-test assertion diagnostics never stringify complete large values and
|
||||
retain at most 2 KiB UTF-8 per case. Oversized replacement output is not
|
||||
copied into an exact current-result draft.
|
||||
- Corpus file input is local, recognized UTF-8 text only. It is bounded before
|
||||
decoding, rejects NUL/binary input, runs sequentially in a dedicated
|
||||
cancellable supervisor and never modifies source files.
|
||||
- Corpus JSON/CSV/NDJSON summaries never contain source, capture-sample or
|
||||
applied-output content. CSV cells are formula-injection protected. Local UI
|
||||
previews are explicitly bounded. Applied files require an explicit
|
||||
content-export acknowledgement; names are normalized against archive path
|
||||
traversal and collisions. Incomplete applied output cannot be downloaded or
|
||||
included in the all-document ZIP, whose bytes have a separate hard limit.
|
||||
- Corpus documents, contents, outputs and results are never placed in project
|
||||
JSON or IndexedDB. Schema-v1 imports that attempt to inject corpus payloads
|
||||
are rejected.
|
||||
- ECMAScript growth analysis preflights repetition count and UTF-8 bytes before
|
||||
allocating a generated string. The analysis worker is terminated on timeout,
|
||||
cancellation or crash; replacement timing also preflights its output bound.
|
||||
Analysis settings and results are neither persisted nor uploaded.
|
||||
- Generated-case synthesis runs in its own killable worker under AST, depth,
|
||||
attempt, repetition and UTF-8 byte caps. Labels come only from bounded
|
||||
requests to the selected real engine worker; a wrong outcome, timeout, crash,
|
||||
cancellation or compile rejection is never converted into a verified case.
|
||||
Settings, discarded candidates and unselected results are ephemeral and
|
||||
never uploaded.
|
||||
- Imported generated-test provenance is accepted only for the exact supported
|
||||
generator id/version and must agree with the test assertion outcome. Unknown
|
||||
provenance fields and future versions are rejected; editing a case removes
|
||||
its provenance.
|
||||
- ECMAScript formatting consumes only exact accepted regexpp literal-token
|
||||
ranges and never treats PCRE2 as ECMAScript. Source/candidate syntax and
|
||||
execution use separate killable workers; compile failure, truncation,
|
||||
timeout asymmetry, engine-identity drift, incomplete tests or any semantic
|
||||
difference fails closed. Preview/results are ephemeral and apply requires an
|
||||
explicit confirmation.
|
||||
- Subject minimization rejects lone surrogates, caps input at 1 MiB UTF-8,
|
||||
evaluation count at 2,000 and aggregate wall time at 60 seconds. Each
|
||||
predicate check runs in the selected killable engine worker. Timeout, crash,
|
||||
cancellation and worker error remain distinct; truncated results fail
|
||||
closed. Subjects and results are ephemeral and never uploaded or persisted.
|
||||
- No backend, uploads, telemetry, remote corpus URL, CDN, remote font, `eval`,
|
||||
`Function`, arbitrary command line or unsafe HTML rendering exists.
|
||||
|
||||
Version 0.1.0 needs a CSP equivalent to:
|
||||
The PCRE2-enabled build needs a CSP equivalent to:
|
||||
|
||||
```text
|
||||
default-src 'self';
|
||||
script-src 'self';
|
||||
script-src 'self' 'wasm-unsafe-eval';
|
||||
worker-src 'self' blob:;
|
||||
connect-src 'self';
|
||||
img-src 'self' data:;
|
||||
@@ -44,8 +91,9 @@ base-uri 'none';
|
||||
form-action 'none';
|
||||
```
|
||||
|
||||
The Toolbox shell currently requires inline styles. No `unsafe-eval` or
|
||||
`wasm-unsafe-eval` is required until an actual WebAssembly flavour ships.
|
||||
The Toolbox shell currently requires inline styles. Corpus ZIP generation and
|
||||
PCRE2 use self-hosted modules and local workers under the existing
|
||||
`worker-src` policy. Broad `unsafe-eval` is not required.
|
||||
|
||||
Report vulnerabilities privately to the repository owner before public issue
|
||||
details are posted.
|
||||
|
||||
Reference in New Issue
Block a user