Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
5e962ff0f3 |
@@ -0,0 +1,39 @@
|
||||
name: Verify
|
||||
|
||||
on:
|
||||
push:
|
||||
branches: [main]
|
||||
pull_request:
|
||||
workflow_dispatch:
|
||||
|
||||
concurrency:
|
||||
group: verify-${{ gitea.repository }}-${{ gitea.ref }}
|
||||
cancel-in-progress: true
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
jobs:
|
||||
verify:
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 45
|
||||
env:
|
||||
CI: "true"
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
- uses: actions/setup-node@v4
|
||||
with:
|
||||
node-version: "22"
|
||||
cache: npm
|
||||
- name: Select declared npm version
|
||||
run: npm install --global npm@11.17.0
|
||||
- name: Install dependencies
|
||||
run: npm ci
|
||||
- name: Audit runtime dependencies
|
||||
run: npm audit --omit=dev --audit-level=moderate
|
||||
- name: Check, test, and build
|
||||
run: npm run check
|
||||
- name: Install browser engines
|
||||
run: npx playwright install --with-deps chromium firefox webkit
|
||||
- name: Browser tests
|
||||
run: npm run test:browser
|
||||
@@ -1,5 +1,12 @@
|
||||
# Changelog
|
||||
|
||||
## 0.2.0 - 2026-09-02
|
||||
|
||||
- Added composable literal and line transforms with allocation preflights, plus BOM, UTF-8 validity, zero-byte and byte-newline evidence.
|
||||
- Added paired text/artifact-evidence downloads with source/output hashes and
|
||||
pipeline provenance; automatic Toolbox handoff remains intentionally disabled
|
||||
pending the coordinated Portal consumer rollout.
|
||||
|
||||
## 0.1.0 - 2026-09-01
|
||||
|
||||
- Added the initial local-first Text Tools workbench.
|
||||
|
||||
@@ -1,19 +1,25 @@
|
||||
# Text Tools
|
||||
|
||||
Transform and inspect plain text locally in the browser.
|
||||
Compose, inspect and evidence plain-text transformations locally in the browser.
|
||||
|
||||
Text Tools is a standalone local-first application in the [add·ideas Toolbox](https://git.add-ideas.de/lotobo/toolbox-portal). Inputs are processed in the browser and are not uploaded.
|
||||
|
||||
## Version 0.1 scope
|
||||
## Version 0.2 scope
|
||||
|
||||
- Ordered pipelines for line endings, trimming, whitespace, stable sorting and deduplication
|
||||
- Ordered pipelines for line endings, trimming, whitespace, stable sorting, deduplication, literal replacement, line prefix/suffix/filter/number/join and line-order reversal
|
||||
- Locale-aware case conversion, Unicode NFC/NFD/NFKC/NFKD and best-effort Latin transliteration
|
||||
- JSON/HTML/URL/Base64/hex escaping, wrapping and delimited-column selection
|
||||
- Explicit UTF-8, UTF-16LE and Latin-1 file decoding plus text/code-point/line inventory
|
||||
- Versioned JSON recipe import/export and per-step size/change reports
|
||||
- JSON/HTML/URL/Base64/hex escaping plus strict inverse decode/unescape stages; binary decoders accept canonical encodings and require valid UTF-8 text
|
||||
- Bounded RFC-style quoted CSV column selection across embedded delimiters and newlines, alongside the retained literal multi-character delimiter-per-line mode
|
||||
- Explicit UTF-8, UTF-16LE and Latin-1 file decoding plus BOM, malformed UTF-8 offset, zero-byte parity, byte-level newline and replacement-character evidence
|
||||
- Versioned JSON recipe import/export, per-step size/change reports, and a paired text-plus-JSON artifact download with source/output SHA-256 and transformation provenance
|
||||
|
||||
Pipelines are limited to 2,000,000 UTF-16 input units and an 8×/32 MiB output expansion. File input is limited to 16 MiB before decoding. Encoding is selected explicitly rather than guessed; compatibility normalisation, transliteration, narrow encodings and several transforms can be lossy, so source, result, and warnings remain visible. See [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) and [docs/PRIVACY-SECURITY.md](docs/PRIVACY-SECURITY.md).
|
||||
|
||||
The artifact evidence is a portable download contract, not an automatic Toolbox
|
||||
handoff. SDK 0.3.0 provides the shared transfer contract, but automatic Open With
|
||||
remains disabled in this release until the Portal consumer rollout is
|
||||
coordinated. Local downloads stay explicit and portable in the meantime.
|
||||
|
||||
## Development
|
||||
|
||||
Requires Node.js 22 and npm 11.
|
||||
@@ -26,7 +32,7 @@ npm run test:browser
|
||||
|
||||
## Release
|
||||
|
||||
`npm run release:artifact` creates a deterministic `release/text-tools-0.1.0.zip` and checksum sidecar.
|
||||
`npm run release:artifact` creates a deterministic `release/text-tools-0.2.0.zip` and checksum sidecar.
|
||||
|
||||
## Licence
|
||||
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
# Corresponding source
|
||||
|
||||
The corresponding source for Text Tools 0.1.0 is available at:
|
||||
The corresponding source for Text Tools 0.2.0 is available at:
|
||||
|
||||
https://git.add-ideas.de/lotobo/text-tools/src/tag/v0.1.0
|
||||
https://git.add-ideas.de/lotobo/text-tools/src/tag/v0.2.0
|
||||
|
||||
Build with Node.js 22, npm 11, `npm ci`, and `npm run release:artifact`.
|
||||
|
||||
@@ -1,12 +1,12 @@
|
||||
# Third-party notices
|
||||
|
||||
Text Tools 0.1.0 directly depends on these runtime packages:
|
||||
Text Tools 0.2.0 directly depends on these runtime packages:
|
||||
|
||||
| Package | Pinned version | Declared licence |
|
||||
| -------------------------------- | -------------: | ---------------- |
|
||||
| `@add-ideas/toolbox-contract` | 0.2.3 | Apache-2.0 |
|
||||
| `@add-ideas/toolbox-helpers` | 0.1.0 | GPL-3.0-or-later |
|
||||
| `@add-ideas/toolbox-shell-react` | 0.2.3 | Apache-2.0 |
|
||||
| `@add-ideas/toolbox-contract` | 0.3.0 | Apache-2.0 |
|
||||
| `@add-ideas/toolbox-helpers` | 0.2.0 | GPL-3.0-or-later |
|
||||
| `@add-ideas/toolbox-shell-react` | 0.3.0 | Apache-2.0 |
|
||||
| `react` | 19.2.8 | MIT |
|
||||
| `react-dom` | 19.2.8 | MIT |
|
||||
|
||||
|
||||
@@ -2,6 +2,8 @@
|
||||
|
||||
Text Tools is a static React/Vite application wrapped in the shared Toolbox shell. `text/pipeline.ts` defines an ordered, typed list of transformations. `applyPipeline` starts from the exact source, applies enabled steps one at a time, checks expansion after every step, and returns the last successful output with per-step before/after counts and warnings.
|
||||
|
||||
Text encoding/decoding, line-ending conversion and Unicode-aware case primitives come from `@add-ideas/toolbox-helpers`. File decoding uses an explicit encoding and fatal-or-replacement choice; it does not guess a charset. Recipe JSON carries a schema/app identity, accepts at most 100 supported steps, discards supplied IDs and assigns local monotonic IDs.
|
||||
Text encoding/decoding, line-ending conversion, bounded CSV parsing and Unicode-aware case primitives come from `@add-ideas/toolbox-helpers`. File decoding uses an explicit encoding and fatal-or-replacement choice; it does not guess a charset. Inverse JSON/HTML/URL/Base64/hex stages are strict: malformed syntax and non-UTF-8 byte output fail the step rather than being silently repaired. Column selection offers a quoted CSV mode that preserves embedded delimiters/newlines and a separate literal-delimiter-per-line mode. Recipe JSON carries a schema/app identity, accepts at most 100 supported steps, discards supplied IDs and assigns local monotonic IDs.
|
||||
|
||||
Version 0.1 operations are synchronous and bounded, so the app creates no worker and has no persistence or server API. Relative entry and asset URLs keep the build relocatable below a nested portal path.
|
||||
`text/evidence.ts` inspects original bytes before decoding, including BOM, exact UTF-8 validity/first invalid offset, zero-byte parity and CR/LF/CRLF counts. Artifact export emits the selected output bytes and a deterministic evidence document containing canonical-source and exact-output SHA-256 values, inventories, recipe and step reports. The document records that direct Toolbox handoff is unavailable in this release; the SDK 0.3.0 transfer contract is intentionally not activated until the Portal consumer rollout is coordinated.
|
||||
|
||||
Version 0.2 operations are synchronous and bounded, so the app creates no worker and has no persistence or server API. Relative entry and asset URLs keep the build relocatable below a nested portal path.
|
||||
|
||||
@@ -2,6 +2,8 @@
|
||||
|
||||
Source text, imported files, recipes and results stay in page memory. There is no telemetry, analytics, account, persistence or runtime network path. Imported text is rendered as text/preformatted content, not executable HTML; HTML escaping produces a string and does not preview it as markup.
|
||||
|
||||
Files are rejected above 16 MiB before reading. Decoded/pasted pipeline input is limited to 2,000,000 UTF-16 units, recipe text to 1,000,000 units/100 steps, and output to both 8× the original (with a small-input floor) and 32 MiB. These limits reduce accidental expansion but do not prove that output is safe for a downstream interpreter.
|
||||
Files are rejected above 16 MiB before reading. Decoded/pasted pipeline input is limited to 2,000,000 UTF-16 units, recipe text to 1,000,000 units/100 steps, and output to both 8× the original (with a small-input floor) and 32 MiB. Quoted CSV selection additionally limits rows, columns and field size. Strict inverse stages reject malformed escapes/encodings and invalid UTF-8; they do not attempt permissive repair. These limits reduce accidental expansion but do not prove that output is safe for a downstream interpreter.
|
||||
|
||||
Compatibility normalisation, case conversion, transliteration, whitespace/line transforms, column selection and narrow encodings can change or lose information. Locale sorting depends on the browser's `Intl.Collator`. The exact source, output, per-step counts and warnings remain visible so users can review those changes before copying or downloading.
|
||||
|
||||
Byte evidence does not guess an encoding or prove the selected decoder is correct. A valid UTF-8 byte sequence may represent text intended for another encoding, zero-byte parity is only a UTF-16 review signal, and replacement characters may already have existed in the source. Artifact evidence hashes canonical UTF-8 source text and exact exported bytes, clearly labels both, and includes filenames and transformation details that may themselves be sensitive.
|
||||
|
||||
Generated
+19
-19
@@ -1,22 +1,22 @@
|
||||
{
|
||||
"name": "text-tools",
|
||||
"version": "0.1.0",
|
||||
"version": "0.2.0",
|
||||
"lockfileVersion": 3,
|
||||
"requires": true,
|
||||
"packages": {
|
||||
"": {
|
||||
"name": "text-tools",
|
||||
"version": "0.1.0",
|
||||
"version": "0.2.0",
|
||||
"license": "GPL-3.0-or-later",
|
||||
"dependencies": {
|
||||
"@add-ideas/toolbox-contract": "0.2.3",
|
||||
"@add-ideas/toolbox-helpers": "0.1.0",
|
||||
"@add-ideas/toolbox-shell-react": "0.2.3",
|
||||
"@add-ideas/toolbox-contract": "0.3.0",
|
||||
"@add-ideas/toolbox-helpers": "0.2.0",
|
||||
"@add-ideas/toolbox-shell-react": "0.3.0",
|
||||
"react": "19.2.8",
|
||||
"react-dom": "19.2.8"
|
||||
},
|
||||
"devDependencies": {
|
||||
"@add-ideas/toolbox-testkit": "0.2.3",
|
||||
"@add-ideas/toolbox-testkit": "0.3.0",
|
||||
"@eslint/js": "10.0.1",
|
||||
"@playwright/test": "1.62.1",
|
||||
"@testing-library/jest-dom": "6.9.1",
|
||||
@@ -42,24 +42,25 @@
|
||||
}
|
||||
},
|
||||
"node_modules/@add-ideas/toolbox-contract": {
|
||||
"version": "0.2.3",
|
||||
"license": "Apache-2.0",
|
||||
"engines": {
|
||||
"node": ">=20"
|
||||
}
|
||||
"version": "0.3.0",
|
||||
"resolved": "https://git.add-ideas.de/api/packages/lotobo/npm/%40add-ideas%2Ftoolbox-contract/-/0.3.0/toolbox-contract-0.3.0.tgz",
|
||||
"integrity": "sha512-dKrK7BjOFwqJaBfJuhKxZKIld4sH0AKjEn6a0yLnbdMUFY+fFv4VSLGV2tNSBD016gumc2iNqOjUj/ld7x4rtA==",
|
||||
"license": "Apache-2.0"
|
||||
},
|
||||
"node_modules/@add-ideas/toolbox-helpers": {
|
||||
"version": "0.1.0",
|
||||
"version": "0.2.0",
|
||||
"license": "GPL-3.0-or-later",
|
||||
"engines": {
|
||||
"node": ">=22"
|
||||
}
|
||||
},
|
||||
"node_modules/@add-ideas/toolbox-shell-react": {
|
||||
"version": "0.2.3",
|
||||
"version": "0.3.0",
|
||||
"resolved": "https://git.add-ideas.de/api/packages/lotobo/npm/%40add-ideas%2Ftoolbox-shell-react/-/0.3.0/toolbox-shell-react-0.3.0.tgz",
|
||||
"integrity": "sha512-74p6JzAOG0YCAKdlc1hLofV4ZIko7vb448S75cIiM88PKm93EHl5VD7g8YVyfM56Ui97UY9dmy+Whiq4sGzpsg==",
|
||||
"license": "Apache-2.0",
|
||||
"dependencies": {
|
||||
"@add-ideas/toolbox-contract": "0.2.3"
|
||||
"@add-ideas/toolbox-contract": "0.3.0"
|
||||
},
|
||||
"peerDependencies": {
|
||||
"react": ">=18 <20",
|
||||
@@ -67,17 +68,16 @@
|
||||
}
|
||||
},
|
||||
"node_modules/@add-ideas/toolbox-testkit": {
|
||||
"version": "0.2.3",
|
||||
"version": "0.3.0",
|
||||
"resolved": "https://git.add-ideas.de/api/packages/lotobo/npm/%40add-ideas%2Ftoolbox-testkit/-/0.3.0/toolbox-testkit-0.3.0.tgz",
|
||||
"integrity": "sha512-4Fk+oSvZFspOMIXr8Xy040nhAaBsIQAzsGyXWSpjn3+k3yBKq7nB1r5zCHhsXzfdLzvPDAx2KcmSNOhM330D9w==",
|
||||
"dev": true,
|
||||
"license": "Apache-2.0",
|
||||
"dependencies": {
|
||||
"@add-ideas/toolbox-contract": "0.2.3"
|
||||
"@add-ideas/toolbox-contract": "0.3.0"
|
||||
},
|
||||
"bin": {
|
||||
"toolbox-check": "dist/cli.js"
|
||||
},
|
||||
"engines": {
|
||||
"node": ">=20"
|
||||
}
|
||||
},
|
||||
"node_modules/@adobe/css-tools": {
|
||||
|
||||
+5
-5
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "text-tools",
|
||||
"version": "0.1.0",
|
||||
"version": "0.2.0",
|
||||
"description": "Transform and inspect plain text locally in the browser.",
|
||||
"license": "GPL-3.0-or-later",
|
||||
"author": "Albrecht Degering",
|
||||
@@ -39,14 +39,14 @@
|
||||
"release:artifact": "npm run check && npm run test:browser && npm run package:release -- --force"
|
||||
},
|
||||
"dependencies": {
|
||||
"@add-ideas/toolbox-contract": "0.2.3",
|
||||
"@add-ideas/toolbox-helpers": "0.1.0",
|
||||
"@add-ideas/toolbox-shell-react": "0.2.3",
|
||||
"@add-ideas/toolbox-contract": "0.3.0",
|
||||
"@add-ideas/toolbox-helpers": "0.2.0",
|
||||
"@add-ideas/toolbox-shell-react": "0.3.0",
|
||||
"react": "19.2.8",
|
||||
"react-dom": "19.2.8"
|
||||
},
|
||||
"devDependencies": {
|
||||
"@add-ideas/toolbox-testkit": "0.2.3",
|
||||
"@add-ideas/toolbox-testkit": "0.3.0",
|
||||
"@eslint/js": "10.0.1",
|
||||
"@playwright/test": "1.62.1",
|
||||
"@testing-library/jest-dom": "6.9.1",
|
||||
|
||||
+20
-2
@@ -15,7 +15,25 @@ export default defineConfig({
|
||||
timeout: 180_000,
|
||||
},
|
||||
projects: [
|
||||
{ name: "chromium", use: { ...devices["Desktop Chrome"] } },
|
||||
{ name: "firefox", use: { ...devices["Desktop Firefox"] } },
|
||||
{
|
||||
name: "chromium",
|
||||
testIgnore: /responsive\.spec\.ts/,
|
||||
use: { ...devices["Desktop Chrome"] },
|
||||
},
|
||||
{
|
||||
name: "firefox",
|
||||
testIgnore: /responsive\.spec\.ts/,
|
||||
use: { ...devices["Desktop Firefox"] },
|
||||
},
|
||||
{
|
||||
name: "webkit",
|
||||
testIgnore: /responsive\.spec\.ts/,
|
||||
use: { ...devices["Desktop Safari"] },
|
||||
},
|
||||
{
|
||||
name: "mobile-chromium",
|
||||
testMatch: /responsive\.spec\.ts/,
|
||||
use: { ...devices["Pixel 5"] },
|
||||
},
|
||||
],
|
||||
});
|
||||
|
||||
@@ -1,5 +1,12 @@
|
||||
# Changelog
|
||||
|
||||
## 0.2.0 - 2026-09-02
|
||||
|
||||
- Added composable literal and line transforms with allocation preflights, plus BOM, UTF-8 validity, zero-byte and byte-newline evidence.
|
||||
- Added paired text/artifact-evidence downloads with source/output hashes and
|
||||
pipeline provenance; automatic Toolbox handoff remains intentionally disabled
|
||||
pending the coordinated Portal consumer rollout.
|
||||
|
||||
## 0.1.0 - 2026-09-01
|
||||
|
||||
- Added the initial local-first Text Tools workbench.
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
==============================================================================
|
||||
@add-ideas/toolbox-contract@0.2.3
|
||||
@add-ideas/toolbox-contract@0.3.0
|
||||
Declared licence: Apache-2.0
|
||||
==============================================================================
|
||||
--- LICENSE ---
|
||||
@@ -198,7 +198,7 @@ Declared licence: Apache-2.0
|
||||
|
||||
|
||||
==============================================================================
|
||||
@add-ideas/toolbox-helpers@0.1.0
|
||||
@add-ideas/toolbox-helpers@0.2.0
|
||||
Declared licence: GPL-3.0-or-later
|
||||
==============================================================================
|
||||
--- LICENSE ---
|
||||
@@ -879,7 +879,7 @@ Public License instead of this License. But first, please read
|
||||
|
||||
|
||||
==============================================================================
|
||||
@add-ideas/toolbox-shell-react@0.2.3
|
||||
@add-ideas/toolbox-shell-react@0.3.0
|
||||
Declared licence: Apache-2.0
|
||||
==============================================================================
|
||||
--- LICENSE ---
|
||||
|
||||
+13
-7
@@ -1,19 +1,25 @@
|
||||
# Text Tools
|
||||
|
||||
Transform and inspect plain text locally in the browser.
|
||||
Compose, inspect and evidence plain-text transformations locally in the browser.
|
||||
|
||||
Text Tools is a standalone local-first application in the [add·ideas Toolbox](https://git.add-ideas.de/lotobo/toolbox-portal). Inputs are processed in the browser and are not uploaded.
|
||||
|
||||
## Version 0.1 scope
|
||||
## Version 0.2 scope
|
||||
|
||||
- Ordered pipelines for line endings, trimming, whitespace, stable sorting and deduplication
|
||||
- Ordered pipelines for line endings, trimming, whitespace, stable sorting, deduplication, literal replacement, line prefix/suffix/filter/number/join and line-order reversal
|
||||
- Locale-aware case conversion, Unicode NFC/NFD/NFKC/NFKD and best-effort Latin transliteration
|
||||
- JSON/HTML/URL/Base64/hex escaping, wrapping and delimited-column selection
|
||||
- Explicit UTF-8, UTF-16LE and Latin-1 file decoding plus text/code-point/line inventory
|
||||
- Versioned JSON recipe import/export and per-step size/change reports
|
||||
- JSON/HTML/URL/Base64/hex escaping plus strict inverse decode/unescape stages; binary decoders accept canonical encodings and require valid UTF-8 text
|
||||
- Bounded RFC-style quoted CSV column selection across embedded delimiters and newlines, alongside the retained literal multi-character delimiter-per-line mode
|
||||
- Explicit UTF-8, UTF-16LE and Latin-1 file decoding plus BOM, malformed UTF-8 offset, zero-byte parity, byte-level newline and replacement-character evidence
|
||||
- Versioned JSON recipe import/export, per-step size/change reports, and a paired text-plus-JSON artifact download with source/output SHA-256 and transformation provenance
|
||||
|
||||
Pipelines are limited to 2,000,000 UTF-16 input units and an 8×/32 MiB output expansion. File input is limited to 16 MiB before decoding. Encoding is selected explicitly rather than guessed; compatibility normalisation, transliteration, narrow encodings and several transforms can be lossy, so source, result, and warnings remain visible. See [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) and [docs/PRIVACY-SECURITY.md](docs/PRIVACY-SECURITY.md).
|
||||
|
||||
The artifact evidence is a portable download contract, not an automatic Toolbox
|
||||
handoff. SDK 0.3.0 provides the shared transfer contract, but automatic Open With
|
||||
remains disabled in this release until the Portal consumer rollout is
|
||||
coordinated. Local downloads stay explicit and portable in the meantime.
|
||||
|
||||
## Development
|
||||
|
||||
Requires Node.js 22 and npm 11.
|
||||
@@ -26,7 +32,7 @@ npm run test:browser
|
||||
|
||||
## Release
|
||||
|
||||
`npm run release:artifact` creates a deterministic `release/text-tools-0.1.0.zip` and checksum sidecar.
|
||||
`npm run release:artifact` creates a deterministic `release/text-tools-0.2.0.zip` and checksum sidecar.
|
||||
|
||||
## Licence
|
||||
|
||||
|
||||
+2
-2
@@ -1,7 +1,7 @@
|
||||
# Corresponding source
|
||||
|
||||
The corresponding source for Text Tools 0.1.0 is available at:
|
||||
The corresponding source for Text Tools 0.2.0 is available at:
|
||||
|
||||
https://git.add-ideas.de/lotobo/text-tools/src/tag/v0.1.0
|
||||
https://git.add-ideas.de/lotobo/text-tools/src/tag/v0.2.0
|
||||
|
||||
Build with Node.js 22, npm 11, `npm ci`, and `npm run release:artifact`.
|
||||
|
||||
@@ -1,12 +1,12 @@
|
||||
# Third-party notices
|
||||
|
||||
Text Tools 0.1.0 directly depends on these runtime packages:
|
||||
Text Tools 0.2.0 directly depends on these runtime packages:
|
||||
|
||||
| Package | Pinned version | Declared licence |
|
||||
| -------------------------------- | -------------: | ---------------- |
|
||||
| `@add-ideas/toolbox-contract` | 0.2.3 | Apache-2.0 |
|
||||
| `@add-ideas/toolbox-helpers` | 0.1.0 | GPL-3.0-or-later |
|
||||
| `@add-ideas/toolbox-shell-react` | 0.2.3 | Apache-2.0 |
|
||||
| `@add-ideas/toolbox-contract` | 0.3.0 | Apache-2.0 |
|
||||
| `@add-ideas/toolbox-helpers` | 0.2.0 | GPL-3.0-or-later |
|
||||
| `@add-ideas/toolbox-shell-react` | 0.3.0 | Apache-2.0 |
|
||||
| `react` | 19.2.8 | MIT |
|
||||
| `react-dom` | 19.2.8 | MIT |
|
||||
|
||||
|
||||
@@ -2,6 +2,8 @@
|
||||
|
||||
Text Tools is a static React/Vite application wrapped in the shared Toolbox shell. `text/pipeline.ts` defines an ordered, typed list of transformations. `applyPipeline` starts from the exact source, applies enabled steps one at a time, checks expansion after every step, and returns the last successful output with per-step before/after counts and warnings.
|
||||
|
||||
Text encoding/decoding, line-ending conversion and Unicode-aware case primitives come from `@add-ideas/toolbox-helpers`. File decoding uses an explicit encoding and fatal-or-replacement choice; it does not guess a charset. Recipe JSON carries a schema/app identity, accepts at most 100 supported steps, discards supplied IDs and assigns local monotonic IDs.
|
||||
Text encoding/decoding, line-ending conversion, bounded CSV parsing and Unicode-aware case primitives come from `@add-ideas/toolbox-helpers`. File decoding uses an explicit encoding and fatal-or-replacement choice; it does not guess a charset. Inverse JSON/HTML/URL/Base64/hex stages are strict: malformed syntax and non-UTF-8 byte output fail the step rather than being silently repaired. Column selection offers a quoted CSV mode that preserves embedded delimiters/newlines and a separate literal-delimiter-per-line mode. Recipe JSON carries a schema/app identity, accepts at most 100 supported steps, discards supplied IDs and assigns local monotonic IDs.
|
||||
|
||||
Version 0.1 operations are synchronous and bounded, so the app creates no worker and has no persistence or server API. Relative entry and asset URLs keep the build relocatable below a nested portal path.
|
||||
`text/evidence.ts` inspects original bytes before decoding, including BOM, exact UTF-8 validity/first invalid offset, zero-byte parity and CR/LF/CRLF counts. Artifact export emits the selected output bytes and a deterministic evidence document containing canonical-source and exact-output SHA-256 values, inventories, recipe and step reports. The document records that direct Toolbox handoff is unavailable in this release; the SDK 0.3.0 transfer contract is intentionally not activated until the Portal consumer rollout is coordinated.
|
||||
|
||||
Version 0.2 operations are synchronous and bounded, so the app creates no worker and has no persistence or server API. Relative entry and asset URLs keep the build relocatable below a nested portal path.
|
||||
|
||||
@@ -2,6 +2,8 @@
|
||||
|
||||
Source text, imported files, recipes and results stay in page memory. There is no telemetry, analytics, account, persistence or runtime network path. Imported text is rendered as text/preformatted content, not executable HTML; HTML escaping produces a string and does not preview it as markup.
|
||||
|
||||
Files are rejected above 16 MiB before reading. Decoded/pasted pipeline input is limited to 2,000,000 UTF-16 units, recipe text to 1,000,000 units/100 steps, and output to both 8× the original (with a small-input floor) and 32 MiB. These limits reduce accidental expansion but do not prove that output is safe for a downstream interpreter.
|
||||
Files are rejected above 16 MiB before reading. Decoded/pasted pipeline input is limited to 2,000,000 UTF-16 units, recipe text to 1,000,000 units/100 steps, and output to both 8× the original (with a small-input floor) and 32 MiB. Quoted CSV selection additionally limits rows, columns and field size. Strict inverse stages reject malformed escapes/encodings and invalid UTF-8; they do not attempt permissive repair. These limits reduce accidental expansion but do not prove that output is safe for a downstream interpreter.
|
||||
|
||||
Compatibility normalisation, case conversion, transliteration, whitespace/line transforms, column selection and narrow encodings can change or lose information. Locale sorting depends on the browser's `Intl.Collator`. The exact source, output, per-step counts and warnings remain visible so users can review those changes before copying or downloading.
|
||||
|
||||
Byte evidence does not guess an encoding or prove the selected decoder is correct. A valid UTF-8 byte sequence may represent text intended for another encoding, zero-byte parity is only a UTF-16 review signal, and replacement characters may already have existed in the source. Artifact evidence hashes canonical UTF-8 source text and exact exported bytes, clearly labels both, and includes filenames and transformation details that may themselves be sensitive.
|
||||
|
||||
+1
-1
@@ -1,5 +1,5 @@
|
||||
const CACHE_PREFIX = "text-tools-shell-";
|
||||
const CACHE_NAME = CACHE_PREFIX + "0.1.0";
|
||||
const CACHE_NAME = CACHE_PREFIX + "0.2.0";
|
||||
const CORE = ["./", "./manifest.webmanifest", "./favicon.svg"];
|
||||
self.addEventListener("install", (event) => {
|
||||
event.waitUntil(
|
||||
|
||||
+34
-3
@@ -3,12 +3,21 @@
|
||||
"schemaVersion": 1,
|
||||
"id": "de.add-ideas.text-tools",
|
||||
"name": "Text Tools",
|
||||
"version": "0.1.0",
|
||||
"description": "Transform and inspect plain text locally in the browser.",
|
||||
"version": "0.2.0",
|
||||
"description": "Compose text transforms and export encoding evidence locally.",
|
||||
"entry": "./",
|
||||
"icon": "./favicon.svg",
|
||||
"categories": ["text", "developer", "productivity"],
|
||||
"tags": ["text", "unicode", "normalize", "sort", "escape", "encoding"],
|
||||
"tags": [
|
||||
"text",
|
||||
"unicode",
|
||||
"pipeline",
|
||||
"normalize",
|
||||
"sort",
|
||||
"escape",
|
||||
"encoding",
|
||||
"newline"
|
||||
],
|
||||
"integration": {
|
||||
"contextVersion": 1,
|
||||
"launchModes": ["navigate", "new-tab"],
|
||||
@@ -21,6 +30,28 @@
|
||||
"crossOriginIsolated": false,
|
||||
"topLevelContext": false
|
||||
},
|
||||
"io": {
|
||||
"accepts": [
|
||||
{
|
||||
"mediaType": "text/*",
|
||||
"extensions": [".txt", ".csv", ".md", ".log"],
|
||||
"label": "Bounded text files"
|
||||
}
|
||||
],
|
||||
"produces": [
|
||||
{
|
||||
"mediaType": "text/plain",
|
||||
"extensions": [".txt"],
|
||||
"label": "Transformed text"
|
||||
},
|
||||
{
|
||||
"mediaType": "application/json",
|
||||
"extensions": [".json"],
|
||||
"label": "Recipe and artifact evidence"
|
||||
}
|
||||
]
|
||||
},
|
||||
"capabilities": { "required": [], "optional": ["web-crypto"] },
|
||||
"privacy": {
|
||||
"processing": "local",
|
||||
"fileUploads": true,
|
||||
|
||||
+200
-15
@@ -1,8 +1,8 @@
|
||||
import { useState } from "react";
|
||||
import {
|
||||
decodeText,
|
||||
encodeText,
|
||||
triggerBlobDownload,
|
||||
triggerBlobDownloads,
|
||||
type TextEncoding,
|
||||
} from "@add-ideas/toolbox-helpers";
|
||||
import {
|
||||
@@ -13,6 +13,11 @@ import {
|
||||
type StepType,
|
||||
type TransformStep,
|
||||
} from "../text/pipeline";
|
||||
import {
|
||||
createTextArtifactEvidence,
|
||||
decodeTextWithEvidence,
|
||||
type TextByteEvidence,
|
||||
} from "../text/evidence";
|
||||
|
||||
const initial = " Crème brûlée \r\nAlpha\nalpha\r\n Cedar \n";
|
||||
const initialSteps: TransformStep[] = [
|
||||
@@ -31,8 +36,16 @@ const STEP_LABELS: Record<StepType, string> = {
|
||||
normalize: "Unicode normalization",
|
||||
transliterate: "Best-effort transliteration",
|
||||
escape: "Escape / encode",
|
||||
unescape: "Decode / unescape strictly",
|
||||
wrap: "Wrap text",
|
||||
columns: "Select/reorder columns",
|
||||
"replace-literal": "Replace literal text",
|
||||
"prefix-lines": "Prefix lines",
|
||||
"suffix-lines": "Suffix lines",
|
||||
"filter-lines": "Keep matching lines",
|
||||
"number-lines": "Number lines",
|
||||
"join-lines": "Join lines",
|
||||
"reverse-lines": "Reverse line order",
|
||||
};
|
||||
|
||||
function Option({
|
||||
@@ -43,9 +56,13 @@ function Option({
|
||||
change: (option: string) => void;
|
||||
}) {
|
||||
if (
|
||||
["trim-lines", "trim-document", "dedupe-lines", "transliterate"].includes(
|
||||
step.type,
|
||||
)
|
||||
[
|
||||
"trim-lines",
|
||||
"trim-document",
|
||||
"dedupe-lines",
|
||||
"transliterate",
|
||||
"reverse-lines",
|
||||
].includes(step.type)
|
||||
)
|
||||
return <span className="muted">No options</span>;
|
||||
if (step.type === "line-endings")
|
||||
@@ -104,10 +121,10 @@ function Option({
|
||||
))}
|
||||
</select>
|
||||
);
|
||||
if (step.type === "escape")
|
||||
if (step.type === "escape" || step.type === "unescape")
|
||||
return (
|
||||
<select
|
||||
aria-label="Escape target"
|
||||
aria-label={step.type === "escape" ? "Escape target" : "Decode source"}
|
||||
value={step.option}
|
||||
onChange={(event) => change(event.target.value)}
|
||||
>
|
||||
@@ -136,14 +153,90 @@ function Option({
|
||||
onChange={(event) => change(event.target.value)}
|
||||
/>
|
||||
);
|
||||
if (step.type === "replace-literal") {
|
||||
let values: [string, string] = ["", ""];
|
||||
try {
|
||||
const parsed = JSON.parse(step.option) as unknown;
|
||||
if (
|
||||
Array.isArray(parsed) &&
|
||||
parsed.length === 2 &&
|
||||
parsed.every((value) => typeof value === "string")
|
||||
)
|
||||
values = parsed as [string, string];
|
||||
} catch {
|
||||
/* Keep editable empty values; apply will report invalid stored JSON. */
|
||||
}
|
||||
return (
|
||||
<div className="column-options">
|
||||
<input
|
||||
aria-label="Literal search text"
|
||||
value={values[0]}
|
||||
onChange={(event) =>
|
||||
change(JSON.stringify([event.target.value, values[1]]))
|
||||
}
|
||||
/>
|
||||
<input
|
||||
aria-label="Literal replacement text"
|
||||
value={values[1]}
|
||||
onChange={(event) =>
|
||||
change(JSON.stringify([values[0], event.target.value]))
|
||||
}
|
||||
/>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
if (
|
||||
[
|
||||
"prefix-lines",
|
||||
"suffix-lines",
|
||||
"filter-lines",
|
||||
"number-lines",
|
||||
"join-lines",
|
||||
].includes(step.type)
|
||||
)
|
||||
return (
|
||||
<input
|
||||
aria-label="Column settings"
|
||||
aria-label={`${STEP_LABELS[step.type]} option`}
|
||||
value={step.option}
|
||||
onChange={(event) => change(event.target.value)}
|
||||
placeholder=",|3,1,2"
|
||||
placeholder={
|
||||
step.type === "join-lines"
|
||||
? "Delimiter; \\n and \\t supported"
|
||||
: "Text"
|
||||
}
|
||||
/>
|
||||
);
|
||||
const pieces = step.option.split("|");
|
||||
const mode = pieces.length >= 3 ? pieces[0]! : "literal";
|
||||
const delimiter = pieces.length >= 3 ? pieces[1]! : pieces[0] || ",";
|
||||
const order =
|
||||
pieces.length >= 3 ? pieces.slice(2).join("|") : pieces[1] || "1";
|
||||
const encode = (nextMode: string, nextDelimiter: string, nextOrder: string) =>
|
||||
change(`${nextMode}|${nextDelimiter}|${nextOrder}`);
|
||||
return (
|
||||
<div className="column-options">
|
||||
<select
|
||||
aria-label="Column parsing mode"
|
||||
value={mode}
|
||||
onChange={(event) => encode(event.target.value, delimiter, order)}
|
||||
>
|
||||
<option value="csv">Quoted CSV records</option>
|
||||
<option value="literal">Literal delimiter per line</option>
|
||||
</select>
|
||||
<input
|
||||
aria-label="Column delimiter"
|
||||
value={delimiter}
|
||||
onChange={(event) => encode(mode, event.target.value, order)}
|
||||
placeholder=", or \\t"
|
||||
/>
|
||||
<input
|
||||
aria-label="Column order"
|
||||
value={order}
|
||||
onChange={(event) => encode(mode, delimiter, event.target.value)}
|
||||
placeholder="3,1,2"
|
||||
/>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
|
||||
function Inventory({ value }: { value: string }) {
|
||||
@@ -188,6 +281,8 @@ export function Workbench() {
|
||||
const [fatalDecode, setFatalDecode] = useState(true);
|
||||
const [error, setError] = useState("");
|
||||
const [recipe, setRecipe] = useState("");
|
||||
const [sourceEvidence, setSourceEvidence] = useState<TextByteEvidence>();
|
||||
const [sourceName, setSourceName] = useState("pasted-text.txt");
|
||||
const outputLoss =
|
||||
outputEncoding === "latin1" &&
|
||||
[...result.output].some((character) => character.codePointAt(0)! > 255);
|
||||
@@ -218,16 +313,18 @@ export function Workbench() {
|
||||
return;
|
||||
}
|
||||
try {
|
||||
const decoded = decodeText(
|
||||
const decoded = decodeTextWithEvidence(
|
||||
new Uint8Array(await file.arrayBuffer()),
|
||||
inputEncoding,
|
||||
fatalDecode,
|
||||
);
|
||||
if (decoded.length > 2_000_000)
|
||||
if (decoded.text.length > 2_000_000)
|
||||
throw new Error(
|
||||
"Decoded text exceeds the 2,000,000 UTF-16-unit pipeline limit.",
|
||||
);
|
||||
setSource(decoded);
|
||||
setSource(decoded.text);
|
||||
setSourceEvidence(decoded.evidence);
|
||||
setSourceName(file.name);
|
||||
setError("");
|
||||
} catch (reason) {
|
||||
setError(
|
||||
@@ -253,7 +350,7 @@ export function Workbench() {
|
||||
};
|
||||
const exportRecipe = () => {
|
||||
const value = JSON.stringify(
|
||||
{ schemaVersion: 1, app: "text-tools", version: "0.1.0", steps },
|
||||
{ schemaVersion: 1, app: "text-tools", version: "0.2.0", steps },
|
||||
null,
|
||||
2,
|
||||
);
|
||||
@@ -263,6 +360,41 @@ export function Workbench() {
|
||||
"text-tools-recipe.json",
|
||||
);
|
||||
};
|
||||
const exportArtifact = async () => {
|
||||
try {
|
||||
const bytes = encodeText(result.output, outputEncoding);
|
||||
const evidence = await createTextArtifactEvidence({
|
||||
sourceName,
|
||||
sourceEvidence,
|
||||
sourceText: source,
|
||||
outputName: `transformed-${outputEncoding}.txt`,
|
||||
outputEncoding,
|
||||
outputBytes: bytes,
|
||||
pipeline: result,
|
||||
steps,
|
||||
});
|
||||
triggerBlobDownloads(
|
||||
[
|
||||
{
|
||||
blob: new Blob([bytes as BlobPart], { type: "text/plain" }),
|
||||
filename: evidence.output.name,
|
||||
},
|
||||
{
|
||||
blob: new Blob([JSON.stringify(evidence, null, 2) + "\n"], {
|
||||
type: "application/json",
|
||||
}),
|
||||
filename: "text-tools-artifact-evidence.json",
|
||||
},
|
||||
],
|
||||
{ maximumFiles: 2, order: "input", revokeDelayMs: 1_000 },
|
||||
);
|
||||
setError("");
|
||||
} catch (reason) {
|
||||
setError(
|
||||
reason instanceof Error ? reason.message : "Artifact export failed.",
|
||||
);
|
||||
}
|
||||
};
|
||||
const importRecipe = () => {
|
||||
try {
|
||||
if (recipe.length > 1_000_000)
|
||||
@@ -315,7 +447,8 @@ export function Workbench() {
|
||||
<h1>Text Tools</h1>
|
||||
<p>
|
||||
Build an ordered, visible transformation pipeline for normalization,
|
||||
lines, casing, escaping, wrapping, and columns.
|
||||
lines, casing, strict escaping/decoding, wrapping, and quoted or
|
||||
literal-delimited columns.
|
||||
</p>
|
||||
</div>
|
||||
<span className="privacy-pill">Browser-local</span>
|
||||
@@ -362,11 +495,54 @@ export function Workbench() {
|
||||
</div>
|
||||
<textarea
|
||||
value={source}
|
||||
onChange={(event) => setSource(event.target.value)}
|
||||
onChange={(event) => {
|
||||
setSource(event.target.value);
|
||||
setSourceEvidence(undefined);
|
||||
setSourceName("pasted-text.txt");
|
||||
}}
|
||||
spellCheck={false}
|
||||
aria-label="Text source"
|
||||
/>
|
||||
<Inventory value={source} />
|
||||
{sourceEvidence && (
|
||||
<details>
|
||||
<summary>Byte decoding evidence</summary>
|
||||
<dl className="inventory">
|
||||
<div>
|
||||
<dt>Selected / BOM</dt>
|
||||
<dd>
|
||||
{sourceEvidence.selectedEncoding} /{" "}
|
||||
{sourceEvidence.bom?.encoding ?? "none"}
|
||||
</dd>
|
||||
</div>
|
||||
<div>
|
||||
<dt>UTF-8 validation</dt>
|
||||
<dd>
|
||||
{sourceEvidence.utf8.valid
|
||||
? "valid"
|
||||
: `invalid at byte ${sourceEvidence.utf8.firstInvalidOffset}`}
|
||||
</dd>
|
||||
</div>
|
||||
<div>
|
||||
<dt>Replacement characters</dt>
|
||||
<dd>{sourceEvidence.replacementCharacters}</dd>
|
||||
</div>
|
||||
<div>
|
||||
<dt>Byte newlines CRLF / LF / CR</dt>
|
||||
<dd>
|
||||
{sourceEvidence.byteNewlines.crlf} /{" "}
|
||||
{sourceEvidence.byteNewlines.bareLf} /{" "}
|
||||
{sourceEvidence.byteNewlines.bareCr}
|
||||
</dd>
|
||||
</div>
|
||||
</dl>
|
||||
{sourceEvidence.warnings.map((warning) => (
|
||||
<p className="warning" key={warning}>
|
||||
{warning}
|
||||
</p>
|
||||
))}
|
||||
</details>
|
||||
)}
|
||||
</section>
|
||||
<section className="panel workspace" aria-labelledby="output-heading">
|
||||
<div className="panel-heading">
|
||||
@@ -386,6 +562,9 @@ export function Workbench() {
|
||||
<button type="button" onClick={download}>
|
||||
Download
|
||||
</button>
|
||||
<button type="button" onClick={() => void exportArtifact()}>
|
||||
Export artifact + evidence
|
||||
</button>
|
||||
</div>
|
||||
</div>
|
||||
<textarea
|
||||
@@ -533,6 +712,7 @@ export function Workbench() {
|
||||
<summary>Versioned recipe import/export</summary>
|
||||
<div className="recipe">
|
||||
<textarea
|
||||
aria-label="Text Tools recipe JSON"
|
||||
value={recipe}
|
||||
onChange={(event) => setRecipe(event.target.value)}
|
||||
placeholder="Paste a Text Tools recipe JSON here."
|
||||
@@ -551,7 +731,12 @@ export function Workbench() {
|
||||
Encoding detection is limited to an explicit choice; no arbitrary
|
||||
charset guess is made. Transliteration, compatibility normalization,
|
||||
escaping, column omission, and narrow encodings can be lossy, so the
|
||||
exact source and output remain visible.
|
||||
exact source and output remain visible. Portable artifact evidence
|
||||
includes exact hashes, selected encodings, newline facts and the
|
||||
ordered recipe. SDK 0.3.0 provides the shared transfer contract, but
|
||||
one-click Open With remains disabled until the Portal consumer rollout
|
||||
is coordinated. This version provides explicit portable downloads in
|
||||
the meantime.
|
||||
</p>
|
||||
</section>
|
||||
</main>
|
||||
|
||||
@@ -298,6 +298,14 @@ textarea {
|
||||
display: flex;
|
||||
gap: 0.25rem;
|
||||
}
|
||||
.column-options {
|
||||
display: grid;
|
||||
grid-template-columns: minmax(8rem, 1fr) minmax(4rem, 0.55fr) minmax(
|
||||
7rem,
|
||||
1fr
|
||||
);
|
||||
gap: 0.4rem;
|
||||
}
|
||||
.step-actions button {
|
||||
min-width: 2.55rem;
|
||||
padding: 0.4rem;
|
||||
@@ -361,6 +369,9 @@ summary {
|
||||
.steps li {
|
||||
grid-template-columns: 1fr;
|
||||
}
|
||||
.column-options {
|
||||
grid-template-columns: 1fr;
|
||||
}
|
||||
}
|
||||
@media (max-width: 42rem) {
|
||||
.hero {
|
||||
@@ -369,4 +380,7 @@ summary {
|
||||
.privacy-pill {
|
||||
order: -1;
|
||||
}
|
||||
.encoding-row {
|
||||
grid-template-columns: minmax(0, 1fr);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -0,0 +1,251 @@
|
||||
import {
|
||||
decodeText,
|
||||
digestHex,
|
||||
encodeText,
|
||||
type TextEncoding,
|
||||
} from "@add-ideas/toolbox-helpers";
|
||||
|
||||
import {
|
||||
textInventory,
|
||||
type PipelineResult,
|
||||
type StepReport,
|
||||
type TransformStep,
|
||||
} from "./pipeline";
|
||||
|
||||
export interface TextByteEvidence {
|
||||
readonly schemaVersion: 1;
|
||||
readonly byteLength: number;
|
||||
readonly selectedEncoding: TextEncoding;
|
||||
readonly fatalDecode: boolean;
|
||||
readonly bom?: { encoding: Exclude<TextEncoding, "latin1">; bytes: number };
|
||||
readonly utf8: { valid: boolean; firstInvalidOffset?: number };
|
||||
readonly zeroBytes: {
|
||||
total: number;
|
||||
evenOffsets: number;
|
||||
oddOffsets: number;
|
||||
};
|
||||
readonly byteNewlines: { crlf: number; bareLf: number; bareCr: number };
|
||||
readonly replacementCharacters: number;
|
||||
readonly warnings: readonly string[];
|
||||
}
|
||||
|
||||
export interface TextArtifactEvidence {
|
||||
readonly schemaVersion: 1;
|
||||
readonly artifactType: "de.add-ideas.toolbox.text/v1";
|
||||
readonly createdBy: { app: "text-tools"; version: "0.2.0" };
|
||||
readonly source: {
|
||||
name: string;
|
||||
canonicalUtf8Sha256: string;
|
||||
inventory: ReturnType<typeof textInventory>;
|
||||
byteEvidence?: TextByteEvidence;
|
||||
};
|
||||
readonly output: {
|
||||
name: string;
|
||||
mediaType: "text/plain";
|
||||
encoding: TextEncoding;
|
||||
bytes: number;
|
||||
sha256: string;
|
||||
inventory: ReturnType<typeof textInventory>;
|
||||
};
|
||||
readonly pipeline: {
|
||||
steps: readonly TransformStep[];
|
||||
reports: readonly StepReport[];
|
||||
warnings: readonly string[];
|
||||
};
|
||||
readonly handoff: {
|
||||
supportedByThisBuild: false;
|
||||
note: string;
|
||||
};
|
||||
}
|
||||
|
||||
function bom(bytes: Uint8Array): TextByteEvidence["bom"] {
|
||||
if (bytes[0] === 0xef && bytes[1] === 0xbb && bytes[2] === 0xbf)
|
||||
return { encoding: "utf-8", bytes: 3 };
|
||||
if (bytes[0] === 0xff && bytes[1] === 0xfe)
|
||||
return { encoding: "utf-16le", bytes: 2 };
|
||||
if (bytes[0] === 0xfe && bytes[1] === 0xff)
|
||||
return { encoding: "utf-16be", bytes: 2 };
|
||||
return undefined;
|
||||
}
|
||||
|
||||
function utf8InvalidOffset(bytes: Uint8Array): number | undefined {
|
||||
const continuation = (index: number) =>
|
||||
index < bytes.length && (bytes[index]! & 0xc0) === 0x80;
|
||||
for (let index = 0; index < bytes.length; index += 1) {
|
||||
const first = bytes[index]!;
|
||||
if (first <= 0x7f) continue;
|
||||
if (first >= 0xc2 && first <= 0xdf) {
|
||||
if (!continuation(index + 1)) return index;
|
||||
index += 1;
|
||||
continue;
|
||||
}
|
||||
if (first >= 0xe0 && first <= 0xef) {
|
||||
const second = bytes[index + 1];
|
||||
if (
|
||||
second === undefined ||
|
||||
(first === 0xe0 && (second < 0xa0 || second > 0xbf)) ||
|
||||
(first === 0xed && (second < 0x80 || second > 0x9f)) ||
|
||||
(first !== 0xe0 &&
|
||||
first !== 0xed &&
|
||||
(second < 0x80 || second > 0xbf)) ||
|
||||
!continuation(index + 2)
|
||||
)
|
||||
return index;
|
||||
index += 2;
|
||||
continue;
|
||||
}
|
||||
if (first >= 0xf0 && first <= 0xf4) {
|
||||
const second = bytes[index + 1];
|
||||
if (
|
||||
second === undefined ||
|
||||
(first === 0xf0 && (second < 0x90 || second > 0xbf)) ||
|
||||
(first === 0xf4 && (second < 0x80 || second > 0x8f)) ||
|
||||
(first !== 0xf0 &&
|
||||
first !== 0xf4 &&
|
||||
(second < 0x80 || second > 0xbf)) ||
|
||||
!continuation(index + 2) ||
|
||||
!continuation(index + 3)
|
||||
)
|
||||
return index;
|
||||
index += 3;
|
||||
continue;
|
||||
}
|
||||
return index;
|
||||
}
|
||||
return undefined;
|
||||
}
|
||||
|
||||
function inspectByteNewlines(bytes: Uint8Array) {
|
||||
let crlf = 0;
|
||||
let bareLf = 0;
|
||||
let bareCr = 0;
|
||||
for (let index = 0; index < bytes.length; index += 1) {
|
||||
if (bytes[index] === 0x0d && bytes[index + 1] === 0x0a) {
|
||||
crlf += 1;
|
||||
index += 1;
|
||||
} else if (bytes[index] === 0x0a) bareLf += 1;
|
||||
else if (bytes[index] === 0x0d) bareCr += 1;
|
||||
}
|
||||
return { crlf, bareLf, bareCr };
|
||||
}
|
||||
|
||||
export function decodeTextWithEvidence(
|
||||
bytes: Uint8Array,
|
||||
selectedEncoding: TextEncoding,
|
||||
fatalDecode: boolean,
|
||||
): { text: string; evidence: TextByteEvidence } {
|
||||
if (bytes.byteLength > 16 * 1024 * 1024)
|
||||
throw new RangeError("Text byte evidence is limited to 16 MiB.");
|
||||
const detectedBom = bom(bytes);
|
||||
const invalidOffset = utf8InvalidOffset(bytes);
|
||||
let zeroTotal = 0;
|
||||
let zeroEven = 0;
|
||||
let zeroOdd = 0;
|
||||
bytes.forEach((value, index) => {
|
||||
if (value !== 0) return;
|
||||
zeroTotal += 1;
|
||||
if (index % 2) zeroOdd += 1;
|
||||
else zeroEven += 1;
|
||||
});
|
||||
const text = decodeText(bytes, selectedEncoding, fatalDecode);
|
||||
const warnings: string[] = [];
|
||||
if (detectedBom && detectedBom.encoding !== selectedEncoding)
|
||||
warnings.push(
|
||||
`The ${detectedBom.encoding.toUpperCase()} BOM conflicts with the selected ${selectedEncoding.toUpperCase()} decoder.`,
|
||||
);
|
||||
if (!detectedBom)
|
||||
warnings.push(
|
||||
"No byte-order mark is present; the selected encoding is an explicit user choice, not a detection claim.",
|
||||
);
|
||||
if (selectedEncoding.startsWith("utf-16") && bytes.length % 2)
|
||||
warnings.push("UTF-16 input has an odd trailing byte.");
|
||||
if (selectedEncoding !== "utf-8" && invalidOffset === undefined)
|
||||
warnings.push(
|
||||
"The same bytes are also well-formed UTF-8; encoding intent cannot be inferred from validity alone.",
|
||||
);
|
||||
const replacementCharacters = [...text].filter(
|
||||
(character) => character === "\uFFFD",
|
||||
).length;
|
||||
if (replacementCharacters)
|
||||
warnings.push(
|
||||
`${replacementCharacters} replacement character(s) appear in decoded text; they may be source data or decoder substitutions.`,
|
||||
);
|
||||
return {
|
||||
text,
|
||||
evidence: Object.freeze({
|
||||
schemaVersion: 1,
|
||||
byteLength: bytes.byteLength,
|
||||
selectedEncoding,
|
||||
fatalDecode,
|
||||
...(detectedBom ? { bom: detectedBom } : {}),
|
||||
utf8: {
|
||||
valid: invalidOffset === undefined,
|
||||
...(invalidOffset === undefined
|
||||
? {}
|
||||
: { firstInvalidOffset: invalidOffset }),
|
||||
},
|
||||
zeroBytes: {
|
||||
total: zeroTotal,
|
||||
evenOffsets: zeroEven,
|
||||
oddOffsets: zeroOdd,
|
||||
},
|
||||
byteNewlines: inspectByteNewlines(bytes),
|
||||
replacementCharacters,
|
||||
warnings: Object.freeze(warnings),
|
||||
}),
|
||||
};
|
||||
}
|
||||
|
||||
export async function createTextArtifactEvidence(input: {
|
||||
sourceName: string;
|
||||
sourceEvidence?: TextByteEvidence;
|
||||
sourceText: string;
|
||||
outputName: string;
|
||||
outputEncoding: TextEncoding;
|
||||
outputBytes: Uint8Array;
|
||||
pipeline: PipelineResult;
|
||||
steps: readonly TransformStep[];
|
||||
}): Promise<TextArtifactEvidence> {
|
||||
if (input.outputBytes.byteLength > 32 * 1024 * 1024)
|
||||
throw new RangeError("Artifact output exceeds the 32 MiB evidence limit.");
|
||||
const canonicalSource = encodeText(input.sourceText, "utf-8");
|
||||
return Object.freeze({
|
||||
schemaVersion: 1,
|
||||
artifactType: "de.add-ideas.toolbox.text/v1",
|
||||
createdBy: {
|
||||
app: "text-tools" as const,
|
||||
version: "0.2.0" as const,
|
||||
},
|
||||
source: {
|
||||
name: input.sourceName,
|
||||
canonicalUtf8Sha256: await digestHex(
|
||||
canonicalSource,
|
||||
"SHA-256",
|
||||
32 * 1024 * 1024,
|
||||
),
|
||||
inventory: textInventory(input.sourceText),
|
||||
...(input.sourceEvidence ? { byteEvidence: input.sourceEvidence } : {}),
|
||||
},
|
||||
output: {
|
||||
name: input.outputName,
|
||||
mediaType: "text/plain" as const,
|
||||
encoding: input.outputEncoding,
|
||||
bytes: input.outputBytes.byteLength,
|
||||
sha256: await digestHex(input.outputBytes, "SHA-256", 32 * 1024 * 1024),
|
||||
inventory: textInventory(input.pipeline.output),
|
||||
},
|
||||
pipeline: {
|
||||
steps: Object.freeze(
|
||||
input.steps.map((step) => Object.freeze({ ...step })),
|
||||
),
|
||||
reports: Object.freeze(
|
||||
input.pipeline.steps.map((report) => Object.freeze({ ...report })),
|
||||
),
|
||||
warnings: Object.freeze([...input.pipeline.warnings]),
|
||||
},
|
||||
handoff: {
|
||||
supportedByThisBuild: false as const,
|
||||
note: "The portable files are ready for explicit local transfer. SDK 0.3.0 provides the shared contract, but Open With remains disabled until the coordinated Portal consumer rollout.",
|
||||
},
|
||||
});
|
||||
}
|
||||
+243
-8
@@ -1,9 +1,14 @@
|
||||
import {
|
||||
base64ToBytes,
|
||||
bytesToBase64,
|
||||
bytesToHex,
|
||||
convertLineEndings,
|
||||
decodeText,
|
||||
encodeText,
|
||||
hexToBytes,
|
||||
normalizeUnicode,
|
||||
parseCsv,
|
||||
stringifyCsv,
|
||||
transformCase,
|
||||
type CaseTransform,
|
||||
type LineEnding,
|
||||
@@ -20,8 +25,16 @@ export type StepType =
|
||||
| "normalize"
|
||||
| "transliterate"
|
||||
| "escape"
|
||||
| "unescape"
|
||||
| "wrap"
|
||||
| "columns";
|
||||
| "columns"
|
||||
| "replace-literal"
|
||||
| "prefix-lines"
|
||||
| "suffix-lines"
|
||||
| "filter-lines"
|
||||
| "number-lines"
|
||||
| "join-lines"
|
||||
| "reverse-lines";
|
||||
export interface TransformStep {
|
||||
id: string;
|
||||
type: StepType;
|
||||
@@ -86,11 +99,27 @@ function wrapText(value: string, width: number): string {
|
||||
.join("\n");
|
||||
}
|
||||
|
||||
function transformColumns(value: string, option: string): string {
|
||||
const [delimiter = ",", order = "1"] = option.split("|");
|
||||
function transformColumns(
|
||||
value: string,
|
||||
option: string,
|
||||
): {
|
||||
value: string;
|
||||
warning?: string;
|
||||
} {
|
||||
const pieces = option.split("|");
|
||||
const mode = pieces.length >= 3 ? pieces[0] : "literal";
|
||||
const delimiterInput = pieces.length >= 3 ? pieces[1] : pieces[0];
|
||||
const order = pieces.length >= 3 ? pieces.slice(2).join("|") : pieces[1];
|
||||
const delimiter = delimiterInput === "\\t" ? "\t" : delimiterInput || ",";
|
||||
if (!delimiter || delimiter.length > 8)
|
||||
throw new Error("Column delimiter must contain 1–8 characters.");
|
||||
const indices = order.split(",").map((entry) => Number(entry.trim()) - 1);
|
||||
if (mode !== "literal" && mode !== "csv")
|
||||
throw new Error("Column parsing mode must be csv or literal.");
|
||||
if (mode === "csv" && delimiter.length !== 1)
|
||||
throw new Error("Quoted CSV mode requires a one-character delimiter.");
|
||||
const indices = (order || "1")
|
||||
.split(",")
|
||||
.map((entry) => Number(entry.trim()) - 1);
|
||||
if (
|
||||
!indices.length ||
|
||||
indices.some(
|
||||
@@ -98,12 +127,95 @@ function transformColumns(value: string, option: string): string {
|
||||
)
|
||||
)
|
||||
throw new Error("Column order uses 1-based indices such as 3,1,2.");
|
||||
return lines(value)
|
||||
if (mode === "literal")
|
||||
return {
|
||||
value: lines(value)
|
||||
.map((line) => {
|
||||
const cells = line.split(delimiter);
|
||||
return indices.map((index) => cells[index] ?? "").join(delimiter);
|
||||
})
|
||||
.join("\n");
|
||||
.join("\n"),
|
||||
};
|
||||
const rows = parseCsv(value, {
|
||||
delimiter,
|
||||
maxRows: 200_000,
|
||||
maxColumns: 1_000,
|
||||
maxFieldChars: 2_000_000,
|
||||
});
|
||||
return {
|
||||
value: stringifyCsv(
|
||||
rows.map((row) => indices.map((index) => row[index] ?? "")),
|
||||
{
|
||||
delimiter,
|
||||
maxRows: 200_000,
|
||||
maxColumns: 1_000,
|
||||
maxFieldChars: 2_000_000,
|
||||
},
|
||||
),
|
||||
warning:
|
||||
"Quoted CSV rows were parsed across embedded delimiters/newlines and serialized canonically with CRLF row endings.",
|
||||
};
|
||||
}
|
||||
|
||||
function strictHtmlUnescape(value: string): string {
|
||||
const entities: Record<string, string> = {
|
||||
"&": "&",
|
||||
"<": "<",
|
||||
">": ">",
|
||||
""": '"',
|
||||
"'": "'",
|
||||
};
|
||||
let output = "";
|
||||
for (let index = 0; index < value.length; index += 1) {
|
||||
if (value[index] !== "&") {
|
||||
output += value[index];
|
||||
continue;
|
||||
}
|
||||
const end = value.indexOf(";", index + 1);
|
||||
if (end < 0 || end - index > 6)
|
||||
throw new SyntaxError(
|
||||
`HTML entity at UTF-16 index ${index} is malformed.`,
|
||||
);
|
||||
const entity = value.slice(index, end + 1);
|
||||
const decoded = entities[entity];
|
||||
if (decoded === undefined)
|
||||
throw new SyntaxError(
|
||||
`HTML entity ${entity} is not emitted by the matching escape stage.`,
|
||||
);
|
||||
output += decoded;
|
||||
index = end;
|
||||
}
|
||||
return output;
|
||||
}
|
||||
|
||||
function strictJsonUnescape(value: string): string {
|
||||
const parsed = JSON.parse(`"${value}"`) as unknown;
|
||||
if (typeof parsed !== "string")
|
||||
throw new SyntaxError("JSON escape input is invalid.");
|
||||
return parsed;
|
||||
}
|
||||
|
||||
function unescapeValue(value: string, option: string): string {
|
||||
if (option === "json") return strictJsonUnescape(value);
|
||||
if (option === "html") return strictHtmlUnescape(value);
|
||||
if (option === "url") return decodeURIComponent(value);
|
||||
if (option === "base64")
|
||||
return decodeText(
|
||||
base64ToBytes(value, { maxOutputBytes: MAX_OUTPUT }),
|
||||
"utf-8",
|
||||
true,
|
||||
);
|
||||
if (option === "hex")
|
||||
return decodeText(
|
||||
hexToBytes(value, {
|
||||
maxOutputBytes: MAX_OUTPUT,
|
||||
allowWhitespace: false,
|
||||
allowPrefix: false,
|
||||
}),
|
||||
"utf-8",
|
||||
true,
|
||||
);
|
||||
throw new Error("Unsupported unescape source.");
|
||||
}
|
||||
|
||||
function transliterate(value: string): string {
|
||||
@@ -119,6 +231,74 @@ function transliterate(value: string): string {
|
||||
.replaceAll("ł", "l");
|
||||
}
|
||||
|
||||
function parseReplacement(option: string): readonly [string, string] {
|
||||
let parsed: unknown;
|
||||
try {
|
||||
parsed = JSON.parse(option);
|
||||
} catch {
|
||||
throw new SyntaxError("Literal replacement options are invalid.");
|
||||
}
|
||||
if (
|
||||
!Array.isArray(parsed) ||
|
||||
parsed.length !== 2 ||
|
||||
parsed.some((value) => typeof value !== "string")
|
||||
)
|
||||
throw new SyntaxError(
|
||||
"Literal replacement requires [search, replacement].",
|
||||
);
|
||||
const [search, replacement] = parsed as [string, string];
|
||||
if (!search) throw new SyntaxError("Literal search text must not be empty.");
|
||||
if (search.length > 100_000 || replacement.length > 1_000_000)
|
||||
throw new RangeError(
|
||||
"Literal replacement option exceeds its safety limit.",
|
||||
);
|
||||
return [search, replacement];
|
||||
}
|
||||
|
||||
function decodedDelimiter(option: string): string {
|
||||
if (option.length > 1_000)
|
||||
throw new RangeError("Line join delimiter exceeds 1,000 characters.");
|
||||
return option.replaceAll("\\n", "\n").replaceAll("\\t", "\t");
|
||||
}
|
||||
|
||||
function assertProjectedOutput(units: number): void {
|
||||
if (!Number.isSafeInteger(units) || units > MAX_OUTPUT)
|
||||
throw new RangeError(
|
||||
"Transformation would exceed the 32 MiB output limit.",
|
||||
);
|
||||
}
|
||||
|
||||
function replaceLiteral(
|
||||
value: string,
|
||||
search: string,
|
||||
replacement: string,
|
||||
): string {
|
||||
let matches = 0;
|
||||
let offset = 0;
|
||||
while ((offset = value.indexOf(search, offset)) >= 0) {
|
||||
matches += 1;
|
||||
offset += search.length;
|
||||
assertProjectedOutput(
|
||||
value.length + matches * (replacement.length - search.length),
|
||||
);
|
||||
}
|
||||
return value.replaceAll(search, replacement);
|
||||
}
|
||||
|
||||
function decorateLines(
|
||||
value: string,
|
||||
addition: string,
|
||||
side: "prefix" | "suffix",
|
||||
): string {
|
||||
if (addition.length > 100_000)
|
||||
throw new RangeError("Line decoration exceeds 100,000 UTF-16 units.");
|
||||
const values = lines(value);
|
||||
assertProjectedOutput(value.length + values.length * addition.length);
|
||||
return values
|
||||
.map((line) => (side === "prefix" ? addition + line : line + addition))
|
||||
.join("\n");
|
||||
}
|
||||
|
||||
function applyStep(
|
||||
value: string,
|
||||
step: TransformStep,
|
||||
@@ -188,10 +368,57 @@ function applyStep(
|
||||
return { value: bytesToHex(encodeText(value)) };
|
||||
throw new Error("Unsupported escape target.");
|
||||
}
|
||||
case "unescape":
|
||||
return {
|
||||
value: unescapeValue(value, step.option),
|
||||
warning:
|
||||
step.option === "base64" || step.option === "hex"
|
||||
? "Decoded bytes are required to be well-formed UTF-8 text; arbitrary binary is rejected."
|
||||
: undefined,
|
||||
};
|
||||
case "wrap":
|
||||
return { value: wrapText(value, Number(step.option)) };
|
||||
case "columns":
|
||||
return { value: transformColumns(value, step.option) };
|
||||
return transformColumns(value, step.option);
|
||||
case "replace-literal": {
|
||||
const [search, replacement] = parseReplacement(step.option);
|
||||
return { value: replaceLiteral(value, search, replacement) };
|
||||
}
|
||||
case "prefix-lines":
|
||||
return { value: decorateLines(value, step.option, "prefix") };
|
||||
case "suffix-lines":
|
||||
return { value: decorateLines(value, step.option, "suffix") };
|
||||
case "filter-lines": {
|
||||
if (!step.option || step.option.length > 100_000)
|
||||
throw new SyntaxError("Line filter text must not be empty.");
|
||||
return {
|
||||
value: lines(value)
|
||||
.filter((line) => line.includes(step.option))
|
||||
.join("\n"),
|
||||
warning: "Line filtering removes every line without the literal text.",
|
||||
};
|
||||
}
|
||||
case "number-lines": {
|
||||
const start = Number(step.option || "1");
|
||||
if (!Number.isSafeInteger(start) || Math.abs(start) > 1_000_000_000)
|
||||
throw new RangeError("Line-number start must be a bounded integer.");
|
||||
return {
|
||||
value: lines(value)
|
||||
.map((line, index) => `${start + index}. ${line}`)
|
||||
.join("\n"),
|
||||
};
|
||||
}
|
||||
case "join-lines": {
|
||||
const values = lines(value);
|
||||
const delimiter = decodedDelimiter(step.option);
|
||||
assertProjectedOutput(
|
||||
values.reduce((total, line) => total + line.length, 0) +
|
||||
Math.max(0, values.length - 1) * delimiter.length,
|
||||
);
|
||||
return { value: values.join(delimiter) };
|
||||
}
|
||||
case "reverse-lines":
|
||||
return { value: lines(value).reverse().join("\n") };
|
||||
}
|
||||
}
|
||||
|
||||
@@ -243,8 +470,16 @@ export function createStep(type: StepType): TransformStep {
|
||||
normalize: "NFC",
|
||||
transliterate: "",
|
||||
escape: "json",
|
||||
unescape: "json",
|
||||
wrap: "80",
|
||||
columns: ",|1",
|
||||
columns: "csv|,|1",
|
||||
"replace-literal": JSON.stringify(["old", "new"]),
|
||||
"prefix-lines": "> ",
|
||||
"suffix-lines": "",
|
||||
"filter-lines": "text",
|
||||
"number-lines": "1",
|
||||
"join-lines": " ",
|
||||
"reverse-lines": "",
|
||||
};
|
||||
return {
|
||||
id: `step-${++nextStepId}`,
|
||||
|
||||
@@ -3,12 +3,21 @@
|
||||
"schemaVersion": 1,
|
||||
"id": "de.add-ideas.text-tools",
|
||||
"name": "Text Tools",
|
||||
"version": "0.1.0",
|
||||
"description": "Transform and inspect plain text locally in the browser.",
|
||||
"version": "0.2.0",
|
||||
"description": "Compose text transforms and export encoding evidence locally.",
|
||||
"entry": "./",
|
||||
"icon": "./favicon.svg",
|
||||
"categories": ["text", "developer", "productivity"],
|
||||
"tags": ["text", "unicode", "normalize", "sort", "escape", "encoding"],
|
||||
"tags": [
|
||||
"text",
|
||||
"unicode",
|
||||
"pipeline",
|
||||
"normalize",
|
||||
"sort",
|
||||
"escape",
|
||||
"encoding",
|
||||
"newline"
|
||||
],
|
||||
"integration": {
|
||||
"contextVersion": 1,
|
||||
"launchModes": ["navigate", "new-tab"],
|
||||
@@ -21,6 +30,28 @@
|
||||
"crossOriginIsolated": false,
|
||||
"topLevelContext": false
|
||||
},
|
||||
"io": {
|
||||
"accepts": [
|
||||
{
|
||||
"mediaType": "text/*",
|
||||
"extensions": [".txt", ".csv", ".md", ".log"],
|
||||
"label": "Bounded text files"
|
||||
}
|
||||
],
|
||||
"produces": [
|
||||
{
|
||||
"mediaType": "text/plain",
|
||||
"extensions": [".txt"],
|
||||
"label": "Transformed text"
|
||||
},
|
||||
{
|
||||
"mediaType": "application/json",
|
||||
"extensions": [".json"],
|
||||
"label": "Recipe and artifact evidence"
|
||||
}
|
||||
]
|
||||
},
|
||||
"capabilities": { "required": [], "optional": ["web-crypto"] },
|
||||
"privacy": {
|
||||
"processing": "local",
|
||||
"fileUploads": true,
|
||||
|
||||
+1
-1
@@ -1 +1 @@
|
||||
export const APP_VERSION = "0.1.0";
|
||||
export const APP_VERSION = "0.2.0";
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
import { expect, test, type Page } from "@playwright/test";
|
||||
import { Buffer } from "node:buffer";
|
||||
|
||||
const ORIGIN = "http://127.0.0.1:4173";
|
||||
async function localOnly(page: Page) {
|
||||
@@ -43,6 +44,35 @@ test("applies the visible text pipeline and preserves its last output", async ({
|
||||
expect(external).toEqual([]);
|
||||
});
|
||||
|
||||
test("shows byte evidence and composes an added literal transform", async ({
|
||||
page,
|
||||
}) => {
|
||||
await page.goto("/deep/nested/text/");
|
||||
await page.getByLabel("Decode as").selectOption("utf-8");
|
||||
await page.locator('input[type="file"]').setInputFiles({
|
||||
name: "bom.txt",
|
||||
mimeType: "text/plain",
|
||||
buffer: Buffer.from([0xef, 0xbb, 0xbf, ...Buffer.from("old\r\nold")]),
|
||||
});
|
||||
await page.getByText("Byte decoding evidence").click();
|
||||
await expect(page.getByText("utf-8 / utf-8")).toBeVisible();
|
||||
await expect(
|
||||
page
|
||||
.locator("details")
|
||||
.filter({ hasText: "Byte decoding evidence" })
|
||||
.getByText("1 / 0 / 0", { exact: true }),
|
||||
).toBeVisible();
|
||||
await page.getByLabel("New transformation").selectOption("replace-literal");
|
||||
await page.getByRole("button", { name: "Add step" }).click();
|
||||
await page.getByLabel("Literal search text").fill("old");
|
||||
await page.getByLabel("Literal replacement text").fill("new");
|
||||
await page.getByRole("button", { name: "Apply pipeline" }).click();
|
||||
await expect(page.getByLabel("Transformed output")).toHaveValue("new");
|
||||
await expect(
|
||||
page.getByRole("button", { name: "Export artifact + evidence" }),
|
||||
).toBeVisible();
|
||||
});
|
||||
|
||||
test("serves the release identity and hardened headers", async ({
|
||||
request,
|
||||
}) => {
|
||||
@@ -55,7 +85,7 @@ test("serves the release identity and hardened headers", async ({
|
||||
const manifest = await request.get("/deep/nested/text/toolbox-app.json");
|
||||
await expect(manifest.json()).resolves.toMatchObject({
|
||||
id: "de.add-ideas.text-tools",
|
||||
version: "0.1.0",
|
||||
version: "0.2.0",
|
||||
entry: "./",
|
||||
});
|
||||
});
|
||||
|
||||
@@ -0,0 +1,18 @@
|
||||
import { expect, test } from "@playwright/test";
|
||||
|
||||
test("keeps the primary workspace inside a narrow viewport", async ({
|
||||
page,
|
||||
}) => {
|
||||
await page.goto("/deep/nested/text/");
|
||||
await expect(page.locator("main").first()).toBeVisible();
|
||||
await expect(
|
||||
page.locator("main .loading, main .workbench-loading"),
|
||||
).toHaveCount(0);
|
||||
|
||||
const widths = await page.evaluate(() => ({
|
||||
content: document.documentElement.scrollWidth,
|
||||
viewport: document.documentElement.clientWidth,
|
||||
}));
|
||||
expect(widths.viewport).toBeLessThanOrEqual(430);
|
||||
expect(widths.content).toBeLessThanOrEqual(widths.viewport + 1);
|
||||
});
|
||||
@@ -4,6 +4,10 @@ import {
|
||||
textInventory,
|
||||
type TransformStep,
|
||||
} from "../../src/text/pipeline";
|
||||
import {
|
||||
createTextArtifactEvidence,
|
||||
decodeTextWithEvidence,
|
||||
} from "../../src/text/evidence";
|
||||
|
||||
const step = (type: TransformStep["type"], option = ""): TransformStep => ({
|
||||
id: type,
|
||||
@@ -28,6 +32,36 @@ describe("text pipelines", () => {
|
||||
expect(
|
||||
applyPipeline("<img src=x onerror=1>", [step("escape", "html")]).output,
|
||||
).toBe("<img src=x onerror=1>"));
|
||||
it("strictly reverses each escape/encoding stage", () => {
|
||||
const source = 'café <x> & "quoted"\n';
|
||||
for (const option of ["json", "html", "url", "base64", "hex"])
|
||||
expect(
|
||||
applyPipeline(applyPipeline(source, [step("escape", option)]).output, [
|
||||
step("unescape", option),
|
||||
]).output,
|
||||
).toBe(source);
|
||||
expect(() => applyPipeline("©", [step("unescape", "html")])).toThrow(
|
||||
/not emitted/u,
|
||||
);
|
||||
expect(() => applyPipeline("w6k", [step("unescape", "base64")])).toThrow(
|
||||
/canonical|padding|UTF-8/u,
|
||||
);
|
||||
expect(() => applyPipeline("%ZZ", [step("unescape", "url")])).toThrow();
|
||||
});
|
||||
it("selects columns from quoted CSV records including embedded newlines", () => {
|
||||
const result = applyPipeline(
|
||||
'name,note,id\r\nAlice,"hello, world",1\r\nBob,"two\nlines",2\r\n',
|
||||
[step("columns", "csv|,|3,1,2")],
|
||||
);
|
||||
expect(result.output).toContain('1,Alice,"hello, world"');
|
||||
expect(result.output).toContain('2,Bob,"two\nlines"');
|
||||
expect(result.warnings[0]).toMatch(/Quoted CSV/u);
|
||||
});
|
||||
it("retains multi-character literal-delimiter mode", () =>
|
||||
expect(
|
||||
applyPipeline("left::right\na::b", [step("columns", "literal|::|2,1")])
|
||||
.output,
|
||||
).toBe("right::left\nb::a"));
|
||||
it("reports lossy transliteration", () =>
|
||||
expect(applyPipeline("Crème", [step("transliterate")]).warnings[0]).toMatch(
|
||||
/lossy/u,
|
||||
@@ -39,4 +73,75 @@ describe("text pipelines", () => {
|
||||
bareCr: 1,
|
||||
finalNewline: true,
|
||||
}));
|
||||
it("composes literal replacement, filtering, numbering and line decoration", () => {
|
||||
const result = applyPipeline("old apple\nold pear\nbanana", [
|
||||
step("replace-literal", JSON.stringify(["old", "fresh"])),
|
||||
step("filter-lines", "fresh"),
|
||||
step("prefix-lines", "- "),
|
||||
step("number-lines", "10"),
|
||||
step("suffix-lines", "!"),
|
||||
step("join-lines", " | "),
|
||||
]);
|
||||
expect(result.output).toBe("10. - fresh apple! | 11. - fresh pear!");
|
||||
expect(result.warnings).toEqual(
|
||||
expect.arrayContaining([expect.stringMatching(/filtering removes/u)]),
|
||||
);
|
||||
});
|
||||
|
||||
it("reverses lines without reversing grapheme contents", () =>
|
||||
expect(applyPipeline("😀a\nb", [step("reverse-lines")]).output).toBe(
|
||||
"b\n😀a",
|
||||
));
|
||||
|
||||
it("preflights expanding transforms before allocating their output", () => {
|
||||
expect(() =>
|
||||
applyPipeline("x".repeat(40), [
|
||||
step("replace-literal", JSON.stringify(["x", "y".repeat(1_000_000)])),
|
||||
]),
|
||||
).toThrow(/32 MiB/u);
|
||||
expect(() =>
|
||||
applyPipeline("a\nb", [step("prefix-lines", "x".repeat(100_001))]),
|
||||
).toThrow(/100,000/u);
|
||||
});
|
||||
});
|
||||
|
||||
describe("byte and portable artifact evidence", () => {
|
||||
it("records BOM conflicts, UTF-8 validity and byte newline forms", () => {
|
||||
const bytes = Uint8Array.of(0xef, 0xbb, 0xbf, 0x61, 0x0d, 0x0a, 0x62);
|
||||
const decoded = decodeTextWithEvidence(bytes, "latin1", true);
|
||||
expect(decoded.evidence).toMatchObject({
|
||||
byteLength: 7,
|
||||
bom: { encoding: "utf-8", bytes: 3 },
|
||||
utf8: { valid: true },
|
||||
byteNewlines: { crlf: 1, bareLf: 0, bareCr: 0 },
|
||||
});
|
||||
expect(decoded.evidence.warnings.join(" ")).toMatch(/conflicts/u);
|
||||
const malformed = decodeTextWithEvidence(
|
||||
Uint8Array.of(0xc3, 0x28),
|
||||
"latin1",
|
||||
true,
|
||||
);
|
||||
expect(malformed.evidence.utf8).toEqual({
|
||||
valid: false,
|
||||
firstInvalidOffset: 0,
|
||||
});
|
||||
});
|
||||
|
||||
it("emits exact output hashes and an honest handoff marker", async () => {
|
||||
const steps = [step("case", "upper")];
|
||||
const pipeline = applyPipeline("hello", steps);
|
||||
const bytes = new TextEncoder().encode(pipeline.output);
|
||||
const evidence = await createTextArtifactEvidence({
|
||||
sourceName: "input.txt",
|
||||
sourceText: "hello",
|
||||
outputName: "output.txt",
|
||||
outputEncoding: "utf-8",
|
||||
outputBytes: bytes,
|
||||
pipeline,
|
||||
steps,
|
||||
});
|
||||
expect(evidence.output.sha256).toMatch(/^[a-f0-9]{64}$/u);
|
||||
expect(evidence.handoff.supportedByThisBuild).toBe(false);
|
||||
expect(evidence.pipeline.steps[0]).toMatchObject({ type: "case" });
|
||||
});
|
||||
});
|
||||
|
||||
+1
-1
@@ -4,7 +4,7 @@
|
||||
"target": "ES2023",
|
||||
"lib": ["ES2023", "DOM", "DOM.Iterable"],
|
||||
"module": "ESNext",
|
||||
"types": ["vite/client"],
|
||||
"types": ["vite/client", "node"],
|
||||
"allowArbitraryExtensions": true,
|
||||
"skipLibCheck": true,
|
||||
"moduleResolution": "Bundler",
|
||||
|
||||
Reference in New Issue
Block a user