166 lines
7.1 KiB
Markdown
166 lines
7.1 KiB
Markdown
# Engine builds
|
|
|
|
The production application uses the browser's native `RegExp` for ECMAScript
|
|
and bundles verified PCRE2, Python and Java runtime packs. Normal application
|
|
builds never download or compile an engine: reviewed files under
|
|
`public/engines/` are copied into `dist/` and verified there.
|
|
|
|
## Rebuilding PCRE2
|
|
|
|
The reproducible build accepts only:
|
|
|
|
- PCRE2 tag `pcre2-10.47`, signed tag object
|
|
`cd007b4466798f66d479d1442a407099e7c40050`, peeled commit
|
|
`f454e231fe5006dd7ff8f4693fd2b8eb94333429` and tree
|
|
`81a83a3552bd68d0ea7b7004f8bb6e7892f583ba`;
|
|
- Emscripten 6.0.4, compiler revision
|
|
`fe5be6afdff43ad58860d821fcc8572a23f92d19`, from emsdk commit
|
|
`224ec5f9f2f72f09f9ce0e26d66bae7dbd8b692f`;
|
|
- CMake 4.3.4 and Ninja 1.13.2;
|
|
- 8-bit PCRE2 with Unicode enabled and JIT, threads and filesystem disabled.
|
|
|
|
The source and compiler checkouts must already exist locally and be completely
|
|
clean. No command clones, downloads or updates them:
|
|
|
|
```sh
|
|
npm run engines:pcre2:build -- \
|
|
--source-dir /absolute/path/to/pcre2 \
|
|
--emcc /absolute/path/to/emscripten/emcc
|
|
npm run engines:pcre2:verify
|
|
npm run engines:pcre2:install
|
|
```
|
|
|
|
Use `--force` only to replace the exact ignored `.engine-build/pcre2` output.
|
|
The explicit install command first re-verifies that pack and then atomically
|
|
replaces only `public/engines/pcre2`.
|
|
|
|
The builder verifies Git objects and critical source/compiler hashes,
|
|
sanitizes build flags, sets `SOURCE_DATE_EPOCH=1760997684`, builds serially and
|
|
emits exactly:
|
|
|
|
- `pcre2.mjs` and `pcre2.wasm`;
|
|
- deterministic `engine-metadata.json`;
|
|
- the exact upstream `LICENSE.txt`;
|
|
- `SHA256SUMS`.
|
|
|
|
The verifier checks the closed file set, metadata, source bridge hashes and all
|
|
checksums. It validates and instantiates WebAssembly, checks ABI/engine/config
|
|
identity, then executes native Unicode compile, bounded match and bounded
|
|
substitution smoke cases plus normal and cap-stopped automatic-callout traces.
|
|
|
|
The immutable signed-tag object is pinned. OpenPGP trust validation remains a
|
|
release-operator step because the offline builder does not install or trust a
|
|
key.
|
|
|
|
## Installing the Python pack
|
|
|
|
The Python pack is derived from the exact `pyodide@314.0.3` npm package locked
|
|
with integrity:
|
|
|
|
```text
|
|
sha512-sK40My6m8tmBUYtYH9au9rXUeh9x0wfahtHdOlGmJxZDsKBGKtP6KznyFB2+u/klbQTdDionR0uaVd176zVQzQ==
|
|
```
|
|
|
|
That Pyodide release is tag/commit
|
|
`ac57031be7564f864d061cb37c5c152e59f83ad4`. Direct executable identity
|
|
verification reports CPython 3.14.2; the shipped `pyodide-lock.json`
|
|
independently reports `info.python` as 3.14.0, and the verifier records rather
|
|
than conflates those two upstream identities. The pack copies only the reviewed
|
|
loader, WebAssembly runtime, Python standard library and lock metadata required
|
|
for a local CPython `re` worker. It adds deterministic engine metadata, a
|
|
closed-file-set checksum manifest and exact Pyodide/CPython licence material.
|
|
|
|
The install operation reads from the already installed pinned npm package. It
|
|
does not contact a CDN or package index. It verifies source package identity,
|
|
file hashes, runtime metadata and licence inputs before atomically replacing
|
|
only `public/engines/python/`. The distribution verifier checks the same closed
|
|
set under `dist/`, validates WebAssembly magic and confirms that all runtime
|
|
references remain relative and self-hosted.
|
|
|
|
```sh
|
|
node scripts/python-engine-pack.mjs \
|
|
node_modules/pyodide \
|
|
.engine-build/python
|
|
node scripts/install-python-engine.mjs .engine-build/python
|
|
```
|
|
|
|
The installed closed set is:
|
|
|
|
- `pyodide.mjs`, `pyodide.asm.mjs` and `pyodide.asm.wasm`;
|
|
- `python_stdlib.zip` and `pyodide-lock.json`;
|
|
- `engine-metadata.json` and `SHA256SUMS`; and
|
|
- `LICENSE.pyodide.txt` and `LICENSE.cpython.txt`.
|
|
|
|
## Building the Java pack
|
|
|
|
The Java bridge under `engines/java/` wraps TeaVM 0.15.0
|
|
`java.util.regex.Pattern` and `Matcher`. TeaVM tag/commit
|
|
`ee91b03e616c4b45401cd11fb0cd7eb0daf6649b` is pinned through the Maven build.
|
|
The build produces a minified ES2015 module without source maps, source files,
|
|
remote imports or an OpenJDK runtime.
|
|
|
|
The pack gate verifies the Maven coordinates, bridge source identity, TeaVM
|
|
engine identity and self-test, generated module, deterministic metadata,
|
|
checksums, Apache-2.0 licence and TeaVM NOTICE. Only the verified pack is
|
|
installed under `public/engines/java/`.
|
|
|
|
```sh
|
|
node scripts/java-engine-pack.mjs
|
|
node scripts/install-java-engine.mjs
|
|
```
|
|
|
|
The pack command performs two clean offline Maven builds and requires identical
|
|
output. The installed closed set is `java-regex.mjs`,
|
|
`engine-metadata.json`, `SHA256SUMS`, `LICENSE.txt` and `NOTICE.txt`.
|
|
|
|
TeaVM supplies its own Apache Harmony-derived class-library implementation.
|
|
The resulting engine is therefore identified as
|
|
`TeaVM java.util.regex 0.15.0`, not OpenJDK. TeaVM 0.15.0 does not implement
|
|
OpenJDK's `Pattern.UNICODE_CHARACTER_CLASS`; the bridge treats the application
|
|
`U` flag as a compatibility request, implies Unicode case folding and reports
|
|
that predefined-character classes and word boundaries retain TeaVM behaviour.
|
|
|
|
## Runtime boundaries
|
|
|
|
### PCRE2
|
|
|
|
ABI version 3 has no retained native handles. Each call compiles, obtains
|
|
capture names, matches/substitutes or traces and releases its code, match data
|
|
and match context before returning. The caller supplies fixed-capacity
|
|
result/name/output/event/mark buffers. Hard maxima are 1 MiB pattern, 16 MiB
|
|
subject, 1,000 captures, 10,000 matches, 100,000 capture rows, 50,000 trace
|
|
events, 10 MiB serialized trace data and 128 MiB PCRE2 heap limit; the
|
|
workbench uses stricter pattern and replacement input limits where applicable.
|
|
|
|
PCRE2 always runs with UTF and UCP. Native UTF-8 byte offsets are retained in
|
|
DTOs and converted only at valid code-point boundaries to half-open editor
|
|
UTF-16 ranges. Lone UTF-16 surrogates are rejected before encoding.
|
|
|
|
Normal execution lives only in `pcre2.worker`. Instrumented
|
|
`PCRE2_AUTO_CALLOUT` collection is a separate ABI operation loaded only by
|
|
`pcre2-trace.worker`; normal match, replacement, tests, corpus and benchmarks
|
|
never call it. Each supervisor terminates its worker on timeout, crash,
|
|
cancellation or supersession and lazily creates a clean generation for the
|
|
next request.
|
|
|
|
### Python
|
|
|
|
The Python worker loads the self-hosted Pyodide pack and verifies Pyodide,
|
|
CPython and `re` identities before accepting work. The application bridge is
|
|
evaluated inside that interpreter and exposes bounded compile, match and native
|
|
replacement operations. It reports CPython's Unicode code-point positions;
|
|
the TypeScript adapter retains them and converts only valid boundaries to
|
|
half-open editor UTF-16 ranges. Runtime load has a separate startup deadline.
|
|
Execution timeout, cancellation or crash terminates the worker and its complete
|
|
Python/WebAssembly heap.
|
|
|
|
### Java
|
|
|
|
The Java worker imports the self-hosted TeaVM ES module and checks the bridge
|
|
ABI, declared engine identity and compiled self-test before accepting work.
|
|
Every request compiles a `Pattern`, obtains bounded `Matcher` results or native
|
|
replacement output, and returns native half-open UTF-16 ranges. No Java virtual
|
|
machine, OpenJDK classes, remote service or ECMAScript regex fallback is
|
|
involved. Runtime load has a separate startup deadline; termination discards
|
|
the complete worker module state.
|