regex-tools/docs/COMPARISON_AND_CODEGEN.md

ECMAScript/PCRE2 comparison and PCRE2 C generation

Comparison contract

The comparison panel sends two explicit requests: one to the native browser ECMAScript worker and one to the bundled PCRE2 10.47 worker. It does not translate patterns, flags, options or replacement templates.

This is an explicit two-engine feature, not a generic active-flavour comparison. Selecting PHP, Perl, Python, Ruby, Java, C++, Go, Rust, .NET or Scala compatibility for the main workbench does not insert it into either side or route it through ECMAScript/PCRE2.

Two pattern models are available:

• Shared pattern sends the same string unchanged to both engines.
• Per-flavour variants retains two explicit ports supplied by the user.

Flags are selected independently from each flavour's registered allowlist. PCRE2 match, depth and heap bounds are part of its exact request. Both sides share the selected subject, scan-all intent, host result bounds and wall-clock worker timeout.

Each side retains:

• its syntax-provider request, acceptance, diagnostics and coverage;
• the exact execution or replacement request sent to its worker;
• a deterministic display key plus the complete authoritative request;
• completion, timeout, cancellation or worker-failure status;
• engine, runtime and adapter identity when the engine returns one;
• user and effective flags;
• native ranges and normalized editor UTF-16 ranges;
• compile acceptance, matches, captures and replacement output.

The orchestrator owns two syntax supervisors and the multi-engine execution supervisor. The two engine calls run concurrently through independently killable flavour workers. A side timeout kills only that worker. The shared Cancel action terminates every active comparison worker. Comparison remains outside EngineSupervisor, whose responsibility is one engine operation and lifecycle boundary.

Match pairs are aligned by equal normalized UTF-16 ranges first. Unpaired matches use an explicitly labelled ordinal fallback. Captures are aligned by normalized range, then capture identity, then an explicitly labelled ordinal fallback. Original native offsets remain attached to both results.

No equivalence claim is made when syntax or compilation is rejected, a worker times out/fails/is cancelled, results or values are truncated, or replacement output is incomplete. A completed matching result is labelled only as agreement for the current subject; it is evidence, not a proof that two patterns are equivalent. Timeout is never interpreted as “no match.”

The retained DTO is bounded to 5,000 displayed difference records and 2,000 match alignments while still counting all differences in the bounded engine result. The panel renders at most 250 differences and 100 alignments at once.

PCRE2 C17 generator

The first reviewed code-generation target is the standalone PCRE2 10.47 8-bit C API. Every other target remains unavailable until its generator has equivalent escaping, semantic and named-runtime/toolchain gates. In particular, PHP's embedded PCRE2 10.44 is not routed through the standalone-PCRE2 generator; an executable browser profile does not by itself imply a code-generation target.

The generator consumes a validated PCRE2 snapshot and:

• requires the registered PCRE2 10.47 version and flag/option allowlists;
• refuses unpaired UTF-16 surrogates rather than emitting lossy UTF-8;
• emits pattern, subject and replacement as independent explicit UTF-8 byte arrays, so quotes, backslashes, newlines, NUL, Unicode, dollar signs and braces cannot escape into C source syntax;
• enables mandatory PCRE2_UTF and PCRE2_UCP;
• maps i, m, s, x, U and J to named PCRE2 compile constants;
• implements application-level g/scan-all behavior explicitly;
• handles the PCRE2 empty-match anchored retry before advancing one UTF-8 character;
• configures native match, depth and heap limits;
• enforces match/capture row caps in the generated match loop;
• bounds replacement output with a host buffer and withholds output when its post-run result-row bound is exceeded;
• reports compile offsets and native API errors and cleans up all PCRE2 state;
• rejects compilation against a PCRE2 header other than 10.47 and verifies the runtime version.

The generated program states limitations that cannot be mapped honestly:

• PCRE2 has no g compile flag;
• match/capture rows and replacement-output size are host-application bounds, not PCRE2 matching options;
• replacement result-row limits can be checked only after pcre2_substitute() has completed;
• PCRE2 exposes no native wall-clock timeout, so hard elapsed-time cancellation belongs to the containing process;
• C API offsets are UTF-8 bytes, not browser editor UTF-16 units.

Generated snippets are examples only and are never used to implement the browser runtime.

Golden and toolchain gate

scripts/pcre2-codegen-golden.test.ts pins deterministic SHA-256 identities for match and replacement fixtures:

match       22b91d578a449b0ed5c30e6eadad30f19a2bcb26ac9db355c8ec7a69328675da
replacement ab90a0d9831ea7b7145aa0072ac352b42baf9a795319d81f531b0f341b04a636

The compile/execute gate was run against a host static build from official signed tag pcre2-10.47, commit f454e231fe5006dd7ff8f4693fd2b8eb94333429. Both generated fixtures compile under C17 with -Wall -Wextra -Werror; the match fixture verifies native UTF-8 ranges and named captures, and the replacement fixture verifies exact Unicode output and substitution count.

To repeat the gate with an exact local PCRE2 installation:

PCRE2_CODEGEN_PREFIX=/path/to/pcre2-10.47-prefix \
  npm test -- scripts/pcre2-codegen-golden.test.ts

The prefix must provide include/pcre2.h and lib64/libpcre2-8.a. A different static-library location can be supplied through PCRE2_CODEGEN_LIBRARY.