Release Text Tools 0.2.0
Verify / verify (push) Canceled after 0s

This commit is contained in:
2026-09-02 07:24:52 +02:00
parent c69bd5af12
commit 5e962ff0f3
29 changed files with 1093 additions and 102 deletions
+39
View File
@@ -0,0 +1,39 @@
name: Verify
on:
push:
branches: [main]
pull_request:
workflow_dispatch:
concurrency:
group: verify-${{ gitea.repository }}-${{ gitea.ref }}
cancel-in-progress: true
permissions:
contents: read
jobs:
verify:
runs-on: ubuntu-latest
timeout-minutes: 45
env:
CI: "true"
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: "22"
cache: npm
- name: Select declared npm version
run: npm install --global npm@11.17.0
- name: Install dependencies
run: npm ci
- name: Audit runtime dependencies
run: npm audit --omit=dev --audit-level=moderate
- name: Check, test, and build
run: npm run check
- name: Install browser engines
run: npx playwright install --with-deps chromium firefox webkit
- name: Browser tests
run: npm run test:browser
+7
View File
@@ -1,5 +1,12 @@
# Changelog # Changelog
## 0.2.0 - 2026-09-02
- Added composable literal and line transforms with allocation preflights, plus BOM, UTF-8 validity, zero-byte and byte-newline evidence.
- Added paired text/artifact-evidence downloads with source/output hashes and
pipeline provenance; automatic Toolbox handoff remains intentionally disabled
pending the coordinated Portal consumer rollout.
## 0.1.0 - 2026-09-01 ## 0.1.0 - 2026-09-01
- Added the initial local-first Text Tools workbench. - Added the initial local-first Text Tools workbench.
+13 -7
View File
@@ -1,19 +1,25 @@
# Text Tools # Text Tools
Transform and inspect plain text locally in the browser. Compose, inspect and evidence plain-text transformations locally in the browser.
Text Tools is a standalone local-first application in the [add·ideas Toolbox](https://git.add-ideas.de/lotobo/toolbox-portal). Inputs are processed in the browser and are not uploaded. Text Tools is a standalone local-first application in the [add·ideas Toolbox](https://git.add-ideas.de/lotobo/toolbox-portal). Inputs are processed in the browser and are not uploaded.
## Version 0.1 scope ## Version 0.2 scope
- Ordered pipelines for line endings, trimming, whitespace, stable sorting and deduplication - Ordered pipelines for line endings, trimming, whitespace, stable sorting, deduplication, literal replacement, line prefix/suffix/filter/number/join and line-order reversal
- Locale-aware case conversion, Unicode NFC/NFD/NFKC/NFKD and best-effort Latin transliteration - Locale-aware case conversion, Unicode NFC/NFD/NFKC/NFKD and best-effort Latin transliteration
- JSON/HTML/URL/Base64/hex escaping, wrapping and delimited-column selection - JSON/HTML/URL/Base64/hex escaping plus strict inverse decode/unescape stages; binary decoders accept canonical encodings and require valid UTF-8 text
- Explicit UTF-8, UTF-16LE and Latin-1 file decoding plus text/code-point/line inventory - Bounded RFC-style quoted CSV column selection across embedded delimiters and newlines, alongside the retained literal multi-character delimiter-per-line mode
- Versioned JSON recipe import/export and per-step size/change reports - Explicit UTF-8, UTF-16LE and Latin-1 file decoding plus BOM, malformed UTF-8 offset, zero-byte parity, byte-level newline and replacement-character evidence
- Versioned JSON recipe import/export, per-step size/change reports, and a paired text-plus-JSON artifact download with source/output SHA-256 and transformation provenance
Pipelines are limited to 2,000,000 UTF-16 input units and an 8×/32 MiB output expansion. File input is limited to 16 MiB before decoding. Encoding is selected explicitly rather than guessed; compatibility normalisation, transliteration, narrow encodings and several transforms can be lossy, so source, result, and warnings remain visible. See [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) and [docs/PRIVACY-SECURITY.md](docs/PRIVACY-SECURITY.md). Pipelines are limited to 2,000,000 UTF-16 input units and an 8×/32 MiB output expansion. File input is limited to 16 MiB before decoding. Encoding is selected explicitly rather than guessed; compatibility normalisation, transliteration, narrow encodings and several transforms can be lossy, so source, result, and warnings remain visible. See [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) and [docs/PRIVACY-SECURITY.md](docs/PRIVACY-SECURITY.md).
The artifact evidence is a portable download contract, not an automatic Toolbox
handoff. SDK 0.3.0 provides the shared transfer contract, but automatic Open With
remains disabled in this release until the Portal consumer rollout is
coordinated. Local downloads stay explicit and portable in the meantime.
## Development ## Development
Requires Node.js 22 and npm 11. Requires Node.js 22 and npm 11.
@@ -26,7 +32,7 @@ npm run test:browser
## Release ## Release
`npm run release:artifact` creates a deterministic `release/text-tools-0.1.0.zip` and checksum sidecar. `npm run release:artifact` creates a deterministic `release/text-tools-0.2.0.zip` and checksum sidecar.
## Licence ## Licence
+2 -2
View File
@@ -1,7 +1,7 @@
# Corresponding source # Corresponding source
The corresponding source for Text Tools 0.1.0 is available at: The corresponding source for Text Tools 0.2.0 is available at:
https://git.add-ideas.de/lotobo/text-tools/src/tag/v0.1.0 https://git.add-ideas.de/lotobo/text-tools/src/tag/v0.2.0
Build with Node.js 22, npm 11, `npm ci`, and `npm run release:artifact`. Build with Node.js 22, npm 11, `npm ci`, and `npm run release:artifact`.
+4 -4
View File
@@ -1,12 +1,12 @@
# Third-party notices # Third-party notices
Text Tools 0.1.0 directly depends on these runtime packages: Text Tools 0.2.0 directly depends on these runtime packages:
| Package | Pinned version | Declared licence | | Package | Pinned version | Declared licence |
| -------------------------------- | -------------: | ---------------- | | -------------------------------- | -------------: | ---------------- |
| `@add-ideas/toolbox-contract` | 0.2.3 | Apache-2.0 | | `@add-ideas/toolbox-contract` | 0.3.0 | Apache-2.0 |
| `@add-ideas/toolbox-helpers` | 0.1.0 | GPL-3.0-or-later | | `@add-ideas/toolbox-helpers` | 0.2.0 | GPL-3.0-or-later |
| `@add-ideas/toolbox-shell-react` | 0.2.3 | Apache-2.0 | | `@add-ideas/toolbox-shell-react` | 0.3.0 | Apache-2.0 |
| `react` | 19.2.8 | MIT | | `react` | 19.2.8 | MIT |
| `react-dom` | 19.2.8 | MIT | | `react-dom` | 19.2.8 | MIT |
+4 -2
View File
@@ -2,6 +2,8 @@
Text Tools is a static React/Vite application wrapped in the shared Toolbox shell. `text/pipeline.ts` defines an ordered, typed list of transformations. `applyPipeline` starts from the exact source, applies enabled steps one at a time, checks expansion after every step, and returns the last successful output with per-step before/after counts and warnings. Text Tools is a static React/Vite application wrapped in the shared Toolbox shell. `text/pipeline.ts` defines an ordered, typed list of transformations. `applyPipeline` starts from the exact source, applies enabled steps one at a time, checks expansion after every step, and returns the last successful output with per-step before/after counts and warnings.
Text encoding/decoding, line-ending conversion and Unicode-aware case primitives come from `@add-ideas/toolbox-helpers`. File decoding uses an explicit encoding and fatal-or-replacement choice; it does not guess a charset. Recipe JSON carries a schema/app identity, accepts at most 100 supported steps, discards supplied IDs and assigns local monotonic IDs. Text encoding/decoding, line-ending conversion, bounded CSV parsing and Unicode-aware case primitives come from `@add-ideas/toolbox-helpers`. File decoding uses an explicit encoding and fatal-or-replacement choice; it does not guess a charset. Inverse JSON/HTML/URL/Base64/hex stages are strict: malformed syntax and non-UTF-8 byte output fail the step rather than being silently repaired. Column selection offers a quoted CSV mode that preserves embedded delimiters/newlines and a separate literal-delimiter-per-line mode. Recipe JSON carries a schema/app identity, accepts at most 100 supported steps, discards supplied IDs and assigns local monotonic IDs.
Version 0.1 operations are synchronous and bounded, so the app creates no worker and has no persistence or server API. Relative entry and asset URLs keep the build relocatable below a nested portal path. `text/evidence.ts` inspects original bytes before decoding, including BOM, exact UTF-8 validity/first invalid offset, zero-byte parity and CR/LF/CRLF counts. Artifact export emits the selected output bytes and a deterministic evidence document containing canonical-source and exact-output SHA-256 values, inventories, recipe and step reports. The document records that direct Toolbox handoff is unavailable in this release; the SDK 0.3.0 transfer contract is intentionally not activated until the Portal consumer rollout is coordinated.
Version 0.2 operations are synchronous and bounded, so the app creates no worker and has no persistence or server API. Relative entry and asset URLs keep the build relocatable below a nested portal path.
+3 -1
View File
@@ -2,6 +2,8 @@
Source text, imported files, recipes and results stay in page memory. There is no telemetry, analytics, account, persistence or runtime network path. Imported text is rendered as text/preformatted content, not executable HTML; HTML escaping produces a string and does not preview it as markup. Source text, imported files, recipes and results stay in page memory. There is no telemetry, analytics, account, persistence or runtime network path. Imported text is rendered as text/preformatted content, not executable HTML; HTML escaping produces a string and does not preview it as markup.
Files are rejected above 16 MiB before reading. Decoded/pasted pipeline input is limited to 2,000,000 UTF-16 units, recipe text to 1,000,000 units/100 steps, and output to both 8× the original (with a small-input floor) and 32 MiB. These limits reduce accidental expansion but do not prove that output is safe for a downstream interpreter. Files are rejected above 16 MiB before reading. Decoded/pasted pipeline input is limited to 2,000,000 UTF-16 units, recipe text to 1,000,000 units/100 steps, and output to both 8× the original (with a small-input floor) and 32 MiB. Quoted CSV selection additionally limits rows, columns and field size. Strict inverse stages reject malformed escapes/encodings and invalid UTF-8; they do not attempt permissive repair. These limits reduce accidental expansion but do not prove that output is safe for a downstream interpreter.
Compatibility normalisation, case conversion, transliteration, whitespace/line transforms, column selection and narrow encodings can change or lose information. Locale sorting depends on the browser's `Intl.Collator`. The exact source, output, per-step counts and warnings remain visible so users can review those changes before copying or downloading. Compatibility normalisation, case conversion, transliteration, whitespace/line transforms, column selection and narrow encodings can change or lose information. Locale sorting depends on the browser's `Intl.Collator`. The exact source, output, per-step counts and warnings remain visible so users can review those changes before copying or downloading.
Byte evidence does not guess an encoding or prove the selected decoder is correct. A valid UTF-8 byte sequence may represent text intended for another encoding, zero-byte parity is only a UTF-16 review signal, and replacement characters may already have existed in the source. Artifact evidence hashes canonical UTF-8 source text and exact exported bytes, clearly labels both, and includes filenames and transformation details that may themselves be sensitive.
+19 -19
View File
@@ -1,22 +1,22 @@
{ {
"name": "text-tools", "name": "text-tools",
"version": "0.1.0", "version": "0.2.0",
"lockfileVersion": 3, "lockfileVersion": 3,
"requires": true, "requires": true,
"packages": { "packages": {
"": { "": {
"name": "text-tools", "name": "text-tools",
"version": "0.1.0", "version": "0.2.0",
"license": "GPL-3.0-or-later", "license": "GPL-3.0-or-later",
"dependencies": { "dependencies": {
"@add-ideas/toolbox-contract": "0.2.3", "@add-ideas/toolbox-contract": "0.3.0",
"@add-ideas/toolbox-helpers": "0.1.0", "@add-ideas/toolbox-helpers": "0.2.0",
"@add-ideas/toolbox-shell-react": "0.2.3", "@add-ideas/toolbox-shell-react": "0.3.0",
"react": "19.2.8", "react": "19.2.8",
"react-dom": "19.2.8" "react-dom": "19.2.8"
}, },
"devDependencies": { "devDependencies": {
"@add-ideas/toolbox-testkit": "0.2.3", "@add-ideas/toolbox-testkit": "0.3.0",
"@eslint/js": "10.0.1", "@eslint/js": "10.0.1",
"@playwright/test": "1.62.1", "@playwright/test": "1.62.1",
"@testing-library/jest-dom": "6.9.1", "@testing-library/jest-dom": "6.9.1",
@@ -42,24 +42,25 @@
} }
}, },
"node_modules/@add-ideas/toolbox-contract": { "node_modules/@add-ideas/toolbox-contract": {
"version": "0.2.3", "version": "0.3.0",
"license": "Apache-2.0", "resolved": "https://git.add-ideas.de/api/packages/lotobo/npm/%40add-ideas%2Ftoolbox-contract/-/0.3.0/toolbox-contract-0.3.0.tgz",
"engines": { "integrity": "sha512-dKrK7BjOFwqJaBfJuhKxZKIld4sH0AKjEn6a0yLnbdMUFY+fFv4VSLGV2tNSBD016gumc2iNqOjUj/ld7x4rtA==",
"node": ">=20" "license": "Apache-2.0"
}
}, },
"node_modules/@add-ideas/toolbox-helpers": { "node_modules/@add-ideas/toolbox-helpers": {
"version": "0.1.0", "version": "0.2.0",
"license": "GPL-3.0-or-later", "license": "GPL-3.0-or-later",
"engines": { "engines": {
"node": ">=22" "node": ">=22"
} }
}, },
"node_modules/@add-ideas/toolbox-shell-react": { "node_modules/@add-ideas/toolbox-shell-react": {
"version": "0.2.3", "version": "0.3.0",
"resolved": "https://git.add-ideas.de/api/packages/lotobo/npm/%40add-ideas%2Ftoolbox-shell-react/-/0.3.0/toolbox-shell-react-0.3.0.tgz",
"integrity": "sha512-74p6JzAOG0YCAKdlc1hLofV4ZIko7vb448S75cIiM88PKm93EHl5VD7g8YVyfM56Ui97UY9dmy+Whiq4sGzpsg==",
"license": "Apache-2.0", "license": "Apache-2.0",
"dependencies": { "dependencies": {
"@add-ideas/toolbox-contract": "0.2.3" "@add-ideas/toolbox-contract": "0.3.0"
}, },
"peerDependencies": { "peerDependencies": {
"react": ">=18 <20", "react": ">=18 <20",
@@ -67,17 +68,16 @@
} }
}, },
"node_modules/@add-ideas/toolbox-testkit": { "node_modules/@add-ideas/toolbox-testkit": {
"version": "0.2.3", "version": "0.3.0",
"resolved": "https://git.add-ideas.de/api/packages/lotobo/npm/%40add-ideas%2Ftoolbox-testkit/-/0.3.0/toolbox-testkit-0.3.0.tgz",
"integrity": "sha512-4Fk+oSvZFspOMIXr8Xy040nhAaBsIQAzsGyXWSpjn3+k3yBKq7nB1r5zCHhsXzfdLzvPDAx2KcmSNOhM330D9w==",
"dev": true, "dev": true,
"license": "Apache-2.0", "license": "Apache-2.0",
"dependencies": { "dependencies": {
"@add-ideas/toolbox-contract": "0.2.3" "@add-ideas/toolbox-contract": "0.3.0"
}, },
"bin": { "bin": {
"toolbox-check": "dist/cli.js" "toolbox-check": "dist/cli.js"
},
"engines": {
"node": ">=20"
} }
}, },
"node_modules/@adobe/css-tools": { "node_modules/@adobe/css-tools": {
+5 -5
View File
@@ -1,6 +1,6 @@
{ {
"name": "text-tools", "name": "text-tools",
"version": "0.1.0", "version": "0.2.0",
"description": "Transform and inspect plain text locally in the browser.", "description": "Transform and inspect plain text locally in the browser.",
"license": "GPL-3.0-or-later", "license": "GPL-3.0-or-later",
"author": "Albrecht Degering", "author": "Albrecht Degering",
@@ -39,14 +39,14 @@
"release:artifact": "npm run check && npm run test:browser && npm run package:release -- --force" "release:artifact": "npm run check && npm run test:browser && npm run package:release -- --force"
}, },
"dependencies": { "dependencies": {
"@add-ideas/toolbox-contract": "0.2.3", "@add-ideas/toolbox-contract": "0.3.0",
"@add-ideas/toolbox-helpers": "0.1.0", "@add-ideas/toolbox-helpers": "0.2.0",
"@add-ideas/toolbox-shell-react": "0.2.3", "@add-ideas/toolbox-shell-react": "0.3.0",
"react": "19.2.8", "react": "19.2.8",
"react-dom": "19.2.8" "react-dom": "19.2.8"
}, },
"devDependencies": { "devDependencies": {
"@add-ideas/toolbox-testkit": "0.2.3", "@add-ideas/toolbox-testkit": "0.3.0",
"@eslint/js": "10.0.1", "@eslint/js": "10.0.1",
"@playwright/test": "1.62.1", "@playwright/test": "1.62.1",
"@testing-library/jest-dom": "6.9.1", "@testing-library/jest-dom": "6.9.1",
+20 -2
View File
@@ -15,7 +15,25 @@ export default defineConfig({
timeout: 180_000, timeout: 180_000,
}, },
projects: [ projects: [
{ name: "chromium", use: { ...devices["Desktop Chrome"] } }, {
{ name: "firefox", use: { ...devices["Desktop Firefox"] } }, name: "chromium",
testIgnore: /responsive\.spec\.ts/,
use: { ...devices["Desktop Chrome"] },
},
{
name: "firefox",
testIgnore: /responsive\.spec\.ts/,
use: { ...devices["Desktop Firefox"] },
},
{
name: "webkit",
testIgnore: /responsive\.spec\.ts/,
use: { ...devices["Desktop Safari"] },
},
{
name: "mobile-chromium",
testMatch: /responsive\.spec\.ts/,
use: { ...devices["Pixel 5"] },
},
], ],
}); });
+7
View File
@@ -1,5 +1,12 @@
# Changelog # Changelog
## 0.2.0 - 2026-09-02
- Added composable literal and line transforms with allocation preflights, plus BOM, UTF-8 validity, zero-byte and byte-newline evidence.
- Added paired text/artifact-evidence downloads with source/output hashes and
pipeline provenance; automatic Toolbox handoff remains intentionally disabled
pending the coordinated Portal consumer rollout.
## 0.1.0 - 2026-09-01 ## 0.1.0 - 2026-09-01
- Added the initial local-first Text Tools workbench. - Added the initial local-first Text Tools workbench.
+3 -3
View File
@@ -1,5 +1,5 @@
============================================================================== ==============================================================================
@add-ideas/toolbox-contract@0.2.3 @add-ideas/toolbox-contract@0.3.0
Declared licence: Apache-2.0 Declared licence: Apache-2.0
============================================================================== ==============================================================================
--- LICENSE --- --- LICENSE ---
@@ -198,7 +198,7 @@ Declared licence: Apache-2.0
============================================================================== ==============================================================================
@add-ideas/toolbox-helpers@0.1.0 @add-ideas/toolbox-helpers@0.2.0
Declared licence: GPL-3.0-or-later Declared licence: GPL-3.0-or-later
============================================================================== ==============================================================================
--- LICENSE --- --- LICENSE ---
@@ -879,7 +879,7 @@ Public License instead of this License. But first, please read
============================================================================== ==============================================================================
@add-ideas/toolbox-shell-react@0.2.3 @add-ideas/toolbox-shell-react@0.3.0
Declared licence: Apache-2.0 Declared licence: Apache-2.0
============================================================================== ==============================================================================
--- LICENSE --- --- LICENSE ---
+13 -7
View File
@@ -1,19 +1,25 @@
# Text Tools # Text Tools
Transform and inspect plain text locally in the browser. Compose, inspect and evidence plain-text transformations locally in the browser.
Text Tools is a standalone local-first application in the [add·ideas Toolbox](https://git.add-ideas.de/lotobo/toolbox-portal). Inputs are processed in the browser and are not uploaded. Text Tools is a standalone local-first application in the [add·ideas Toolbox](https://git.add-ideas.de/lotobo/toolbox-portal). Inputs are processed in the browser and are not uploaded.
## Version 0.1 scope ## Version 0.2 scope
- Ordered pipelines for line endings, trimming, whitespace, stable sorting and deduplication - Ordered pipelines for line endings, trimming, whitespace, stable sorting, deduplication, literal replacement, line prefix/suffix/filter/number/join and line-order reversal
- Locale-aware case conversion, Unicode NFC/NFD/NFKC/NFKD and best-effort Latin transliteration - Locale-aware case conversion, Unicode NFC/NFD/NFKC/NFKD and best-effort Latin transliteration
- JSON/HTML/URL/Base64/hex escaping, wrapping and delimited-column selection - JSON/HTML/URL/Base64/hex escaping plus strict inverse decode/unescape stages; binary decoders accept canonical encodings and require valid UTF-8 text
- Explicit UTF-8, UTF-16LE and Latin-1 file decoding plus text/code-point/line inventory - Bounded RFC-style quoted CSV column selection across embedded delimiters and newlines, alongside the retained literal multi-character delimiter-per-line mode
- Versioned JSON recipe import/export and per-step size/change reports - Explicit UTF-8, UTF-16LE and Latin-1 file decoding plus BOM, malformed UTF-8 offset, zero-byte parity, byte-level newline and replacement-character evidence
- Versioned JSON recipe import/export, per-step size/change reports, and a paired text-plus-JSON artifact download with source/output SHA-256 and transformation provenance
Pipelines are limited to 2,000,000 UTF-16 input units and an 8×/32 MiB output expansion. File input is limited to 16 MiB before decoding. Encoding is selected explicitly rather than guessed; compatibility normalisation, transliteration, narrow encodings and several transforms can be lossy, so source, result, and warnings remain visible. See [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) and [docs/PRIVACY-SECURITY.md](docs/PRIVACY-SECURITY.md). Pipelines are limited to 2,000,000 UTF-16 input units and an 8×/32 MiB output expansion. File input is limited to 16 MiB before decoding. Encoding is selected explicitly rather than guessed; compatibility normalisation, transliteration, narrow encodings and several transforms can be lossy, so source, result, and warnings remain visible. See [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) and [docs/PRIVACY-SECURITY.md](docs/PRIVACY-SECURITY.md).
The artifact evidence is a portable download contract, not an automatic Toolbox
handoff. SDK 0.3.0 provides the shared transfer contract, but automatic Open With
remains disabled in this release until the Portal consumer rollout is
coordinated. Local downloads stay explicit and portable in the meantime.
## Development ## Development
Requires Node.js 22 and npm 11. Requires Node.js 22 and npm 11.
@@ -26,7 +32,7 @@ npm run test:browser
## Release ## Release
`npm run release:artifact` creates a deterministic `release/text-tools-0.1.0.zip` and checksum sidecar. `npm run release:artifact` creates a deterministic `release/text-tools-0.2.0.zip` and checksum sidecar.
## Licence ## Licence
+2 -2
View File
@@ -1,7 +1,7 @@
# Corresponding source # Corresponding source
The corresponding source for Text Tools 0.1.0 is available at: The corresponding source for Text Tools 0.2.0 is available at:
https://git.add-ideas.de/lotobo/text-tools/src/tag/v0.1.0 https://git.add-ideas.de/lotobo/text-tools/src/tag/v0.2.0
Build with Node.js 22, npm 11, `npm ci`, and `npm run release:artifact`. Build with Node.js 22, npm 11, `npm ci`, and `npm run release:artifact`.
+4 -4
View File
@@ -1,12 +1,12 @@
# Third-party notices # Third-party notices
Text Tools 0.1.0 directly depends on these runtime packages: Text Tools 0.2.0 directly depends on these runtime packages:
| Package | Pinned version | Declared licence | | Package | Pinned version | Declared licence |
| -------------------------------- | -------------: | ---------------- | | -------------------------------- | -------------: | ---------------- |
| `@add-ideas/toolbox-contract` | 0.2.3 | Apache-2.0 | | `@add-ideas/toolbox-contract` | 0.3.0 | Apache-2.0 |
| `@add-ideas/toolbox-helpers` | 0.1.0 | GPL-3.0-or-later | | `@add-ideas/toolbox-helpers` | 0.2.0 | GPL-3.0-or-later |
| `@add-ideas/toolbox-shell-react` | 0.2.3 | Apache-2.0 | | `@add-ideas/toolbox-shell-react` | 0.3.0 | Apache-2.0 |
| `react` | 19.2.8 | MIT | | `react` | 19.2.8 | MIT |
| `react-dom` | 19.2.8 | MIT | | `react-dom` | 19.2.8 | MIT |
+4 -2
View File
@@ -2,6 +2,8 @@
Text Tools is a static React/Vite application wrapped in the shared Toolbox shell. `text/pipeline.ts` defines an ordered, typed list of transformations. `applyPipeline` starts from the exact source, applies enabled steps one at a time, checks expansion after every step, and returns the last successful output with per-step before/after counts and warnings. Text Tools is a static React/Vite application wrapped in the shared Toolbox shell. `text/pipeline.ts` defines an ordered, typed list of transformations. `applyPipeline` starts from the exact source, applies enabled steps one at a time, checks expansion after every step, and returns the last successful output with per-step before/after counts and warnings.
Text encoding/decoding, line-ending conversion and Unicode-aware case primitives come from `@add-ideas/toolbox-helpers`. File decoding uses an explicit encoding and fatal-or-replacement choice; it does not guess a charset. Recipe JSON carries a schema/app identity, accepts at most 100 supported steps, discards supplied IDs and assigns local monotonic IDs. Text encoding/decoding, line-ending conversion, bounded CSV parsing and Unicode-aware case primitives come from `@add-ideas/toolbox-helpers`. File decoding uses an explicit encoding and fatal-or-replacement choice; it does not guess a charset. Inverse JSON/HTML/URL/Base64/hex stages are strict: malformed syntax and non-UTF-8 byte output fail the step rather than being silently repaired. Column selection offers a quoted CSV mode that preserves embedded delimiters/newlines and a separate literal-delimiter-per-line mode. Recipe JSON carries a schema/app identity, accepts at most 100 supported steps, discards supplied IDs and assigns local monotonic IDs.
Version 0.1 operations are synchronous and bounded, so the app creates no worker and has no persistence or server API. Relative entry and asset URLs keep the build relocatable below a nested portal path. `text/evidence.ts` inspects original bytes before decoding, including BOM, exact UTF-8 validity/first invalid offset, zero-byte parity and CR/LF/CRLF counts. Artifact export emits the selected output bytes and a deterministic evidence document containing canonical-source and exact-output SHA-256 values, inventories, recipe and step reports. The document records that direct Toolbox handoff is unavailable in this release; the SDK 0.3.0 transfer contract is intentionally not activated until the Portal consumer rollout is coordinated.
Version 0.2 operations are synchronous and bounded, so the app creates no worker and has no persistence or server API. Relative entry and asset URLs keep the build relocatable below a nested portal path.
+3 -1
View File
@@ -2,6 +2,8 @@
Source text, imported files, recipes and results stay in page memory. There is no telemetry, analytics, account, persistence or runtime network path. Imported text is rendered as text/preformatted content, not executable HTML; HTML escaping produces a string and does not preview it as markup. Source text, imported files, recipes and results stay in page memory. There is no telemetry, analytics, account, persistence or runtime network path. Imported text is rendered as text/preformatted content, not executable HTML; HTML escaping produces a string and does not preview it as markup.
Files are rejected above 16 MiB before reading. Decoded/pasted pipeline input is limited to 2,000,000 UTF-16 units, recipe text to 1,000,000 units/100 steps, and output to both 8× the original (with a small-input floor) and 32 MiB. These limits reduce accidental expansion but do not prove that output is safe for a downstream interpreter. Files are rejected above 16 MiB before reading. Decoded/pasted pipeline input is limited to 2,000,000 UTF-16 units, recipe text to 1,000,000 units/100 steps, and output to both 8× the original (with a small-input floor) and 32 MiB. Quoted CSV selection additionally limits rows, columns and field size. Strict inverse stages reject malformed escapes/encodings and invalid UTF-8; they do not attempt permissive repair. These limits reduce accidental expansion but do not prove that output is safe for a downstream interpreter.
Compatibility normalisation, case conversion, transliteration, whitespace/line transforms, column selection and narrow encodings can change or lose information. Locale sorting depends on the browser's `Intl.Collator`. The exact source, output, per-step counts and warnings remain visible so users can review those changes before copying or downloading. Compatibility normalisation, case conversion, transliteration, whitespace/line transforms, column selection and narrow encodings can change or lose information. Locale sorting depends on the browser's `Intl.Collator`. The exact source, output, per-step counts and warnings remain visible so users can review those changes before copying or downloading.
Byte evidence does not guess an encoding or prove the selected decoder is correct. A valid UTF-8 byte sequence may represent text intended for another encoding, zero-byte parity is only a UTF-16 review signal, and replacement characters may already have existed in the source. Artifact evidence hashes canonical UTF-8 source text and exact exported bytes, clearly labels both, and includes filenames and transformation details that may themselves be sensitive.
+1 -1
View File
@@ -1,5 +1,5 @@
const CACHE_PREFIX = "text-tools-shell-"; const CACHE_PREFIX = "text-tools-shell-";
const CACHE_NAME = CACHE_PREFIX + "0.1.0"; const CACHE_NAME = CACHE_PREFIX + "0.2.0";
const CORE = ["./", "./manifest.webmanifest", "./favicon.svg"]; const CORE = ["./", "./manifest.webmanifest", "./favicon.svg"];
self.addEventListener("install", (event) => { self.addEventListener("install", (event) => {
event.waitUntil( event.waitUntil(
+34 -3
View File
@@ -3,12 +3,21 @@
"schemaVersion": 1, "schemaVersion": 1,
"id": "de.add-ideas.text-tools", "id": "de.add-ideas.text-tools",
"name": "Text Tools", "name": "Text Tools",
"version": "0.1.0", "version": "0.2.0",
"description": "Transform and inspect plain text locally in the browser.", "description": "Compose text transforms and export encoding evidence locally.",
"entry": "./", "entry": "./",
"icon": "./favicon.svg", "icon": "./favicon.svg",
"categories": ["text", "developer", "productivity"], "categories": ["text", "developer", "productivity"],
"tags": ["text", "unicode", "normalize", "sort", "escape", "encoding"], "tags": [
"text",
"unicode",
"pipeline",
"normalize",
"sort",
"escape",
"encoding",
"newline"
],
"integration": { "integration": {
"contextVersion": 1, "contextVersion": 1,
"launchModes": ["navigate", "new-tab"], "launchModes": ["navigate", "new-tab"],
@@ -21,6 +30,28 @@
"crossOriginIsolated": false, "crossOriginIsolated": false,
"topLevelContext": false "topLevelContext": false
}, },
"io": {
"accepts": [
{
"mediaType": "text/*",
"extensions": [".txt", ".csv", ".md", ".log"],
"label": "Bounded text files"
}
],
"produces": [
{
"mediaType": "text/plain",
"extensions": [".txt"],
"label": "Transformed text"
},
{
"mediaType": "application/json",
"extensions": [".json"],
"label": "Recipe and artifact evidence"
}
]
},
"capabilities": { "required": [], "optional": ["web-crypto"] },
"privacy": { "privacy": {
"processing": "local", "processing": "local",
"fileUploads": true, "fileUploads": true,
+204 -19
View File
@@ -1,8 +1,8 @@
import { useState } from "react"; import { useState } from "react";
import { import {
decodeText,
encodeText, encodeText,
triggerBlobDownload, triggerBlobDownload,
triggerBlobDownloads,
type TextEncoding, type TextEncoding,
} from "@add-ideas/toolbox-helpers"; } from "@add-ideas/toolbox-helpers";
import { import {
@@ -13,6 +13,11 @@ import {
type StepType, type StepType,
type TransformStep, type TransformStep,
} from "../text/pipeline"; } from "../text/pipeline";
import {
createTextArtifactEvidence,
decodeTextWithEvidence,
type TextByteEvidence,
} from "../text/evidence";
const initial = " Crème brûlée \r\nAlpha\nalpha\r\n Cedar \n"; const initial = " Crème brûlée \r\nAlpha\nalpha\r\n Cedar \n";
const initialSteps: TransformStep[] = [ const initialSteps: TransformStep[] = [
@@ -31,8 +36,16 @@ const STEP_LABELS: Record<StepType, string> = {
normalize: "Unicode normalization", normalize: "Unicode normalization",
transliterate: "Best-effort transliteration", transliterate: "Best-effort transliteration",
escape: "Escape / encode", escape: "Escape / encode",
unescape: "Decode / unescape strictly",
wrap: "Wrap text", wrap: "Wrap text",
columns: "Select/reorder columns", columns: "Select/reorder columns",
"replace-literal": "Replace literal text",
"prefix-lines": "Prefix lines",
"suffix-lines": "Suffix lines",
"filter-lines": "Keep matching lines",
"number-lines": "Number lines",
"join-lines": "Join lines",
"reverse-lines": "Reverse line order",
}; };
function Option({ function Option({
@@ -43,9 +56,13 @@ function Option({
change: (option: string) => void; change: (option: string) => void;
}) { }) {
if ( if (
["trim-lines", "trim-document", "dedupe-lines", "transliterate"].includes( [
step.type, "trim-lines",
) "trim-document",
"dedupe-lines",
"transliterate",
"reverse-lines",
].includes(step.type)
) )
return <span className="muted">No options</span>; return <span className="muted">No options</span>;
if (step.type === "line-endings") if (step.type === "line-endings")
@@ -104,10 +121,10 @@ function Option({
))} ))}
</select> </select>
); );
if (step.type === "escape") if (step.type === "escape" || step.type === "unescape")
return ( return (
<select <select
aria-label="Escape target" aria-label={step.type === "escape" ? "Escape target" : "Decode source"}
value={step.option} value={step.option}
onChange={(event) => change(event.target.value)} onChange={(event) => change(event.target.value)}
> >
@@ -136,13 +153,89 @@ function Option({
onChange={(event) => change(event.target.value)} onChange={(event) => change(event.target.value)}
/> />
); );
if (step.type === "replace-literal") {
let values: [string, string] = ["", ""];
try {
const parsed = JSON.parse(step.option) as unknown;
if (
Array.isArray(parsed) &&
parsed.length === 2 &&
parsed.every((value) => typeof value === "string")
)
values = parsed as [string, string];
} catch {
/* Keep editable empty values; apply will report invalid stored JSON. */
}
return (
<div className="column-options">
<input
aria-label="Literal search text"
value={values[0]}
onChange={(event) =>
change(JSON.stringify([event.target.value, values[1]]))
}
/>
<input
aria-label="Literal replacement text"
value={values[1]}
onChange={(event) =>
change(JSON.stringify([values[0], event.target.value]))
}
/>
</div>
);
}
if (
[
"prefix-lines",
"suffix-lines",
"filter-lines",
"number-lines",
"join-lines",
].includes(step.type)
)
return (
<input
aria-label={`${STEP_LABELS[step.type]} option`}
value={step.option}
onChange={(event) => change(event.target.value)}
placeholder={
step.type === "join-lines"
? "Delimiter; \\n and \\t supported"
: "Text"
}
/>
);
const pieces = step.option.split("|");
const mode = pieces.length >= 3 ? pieces[0]! : "literal";
const delimiter = pieces.length >= 3 ? pieces[1]! : pieces[0] || ",";
const order =
pieces.length >= 3 ? pieces.slice(2).join("|") : pieces[1] || "1";
const encode = (nextMode: string, nextDelimiter: string, nextOrder: string) =>
change(`${nextMode}|${nextDelimiter}|${nextOrder}`);
return ( return (
<input <div className="column-options">
aria-label="Column settings" <select
value={step.option} aria-label="Column parsing mode"
onChange={(event) => change(event.target.value)} value={mode}
placeholder=",|3,1,2" onChange={(event) => encode(event.target.value, delimiter, order)}
/> >
<option value="csv">Quoted CSV records</option>
<option value="literal">Literal delimiter per line</option>
</select>
<input
aria-label="Column delimiter"
value={delimiter}
onChange={(event) => encode(mode, event.target.value, order)}
placeholder=", or \\t"
/>
<input
aria-label="Column order"
value={order}
onChange={(event) => encode(mode, delimiter, event.target.value)}
placeholder="3,1,2"
/>
</div>
); );
} }
@@ -188,6 +281,8 @@ export function Workbench() {
const [fatalDecode, setFatalDecode] = useState(true); const [fatalDecode, setFatalDecode] = useState(true);
const [error, setError] = useState(""); const [error, setError] = useState("");
const [recipe, setRecipe] = useState(""); const [recipe, setRecipe] = useState("");
const [sourceEvidence, setSourceEvidence] = useState<TextByteEvidence>();
const [sourceName, setSourceName] = useState("pasted-text.txt");
const outputLoss = const outputLoss =
outputEncoding === "latin1" && outputEncoding === "latin1" &&
[...result.output].some((character) => character.codePointAt(0)! > 255); [...result.output].some((character) => character.codePointAt(0)! > 255);
@@ -218,16 +313,18 @@ export function Workbench() {
return; return;
} }
try { try {
const decoded = decodeText( const decoded = decodeTextWithEvidence(
new Uint8Array(await file.arrayBuffer()), new Uint8Array(await file.arrayBuffer()),
inputEncoding, inputEncoding,
fatalDecode, fatalDecode,
); );
if (decoded.length > 2_000_000) if (decoded.text.length > 2_000_000)
throw new Error( throw new Error(
"Decoded text exceeds the 2,000,000 UTF-16-unit pipeline limit.", "Decoded text exceeds the 2,000,000 UTF-16-unit pipeline limit.",
); );
setSource(decoded); setSource(decoded.text);
setSourceEvidence(decoded.evidence);
setSourceName(file.name);
setError(""); setError("");
} catch (reason) { } catch (reason) {
setError( setError(
@@ -253,7 +350,7 @@ export function Workbench() {
}; };
const exportRecipe = () => { const exportRecipe = () => {
const value = JSON.stringify( const value = JSON.stringify(
{ schemaVersion: 1, app: "text-tools", version: "0.1.0", steps }, { schemaVersion: 1, app: "text-tools", version: "0.2.0", steps },
null, null,
2, 2,
); );
@@ -263,6 +360,41 @@ export function Workbench() {
"text-tools-recipe.json", "text-tools-recipe.json",
); );
}; };
const exportArtifact = async () => {
try {
const bytes = encodeText(result.output, outputEncoding);
const evidence = await createTextArtifactEvidence({
sourceName,
sourceEvidence,
sourceText: source,
outputName: `transformed-${outputEncoding}.txt`,
outputEncoding,
outputBytes: bytes,
pipeline: result,
steps,
});
triggerBlobDownloads(
[
{
blob: new Blob([bytes as BlobPart], { type: "text/plain" }),
filename: evidence.output.name,
},
{
blob: new Blob([JSON.stringify(evidence, null, 2) + "\n"], {
type: "application/json",
}),
filename: "text-tools-artifact-evidence.json",
},
],
{ maximumFiles: 2, order: "input", revokeDelayMs: 1_000 },
);
setError("");
} catch (reason) {
setError(
reason instanceof Error ? reason.message : "Artifact export failed.",
);
}
};
const importRecipe = () => { const importRecipe = () => {
try { try {
if (recipe.length > 1_000_000) if (recipe.length > 1_000_000)
@@ -315,7 +447,8 @@ export function Workbench() {
<h1>Text Tools</h1> <h1>Text Tools</h1>
<p> <p>
Build an ordered, visible transformation pipeline for normalization, Build an ordered, visible transformation pipeline for normalization,
lines, casing, escaping, wrapping, and columns. lines, casing, strict escaping/decoding, wrapping, and quoted or
literal-delimited columns.
</p> </p>
</div> </div>
<span className="privacy-pill">Browser-local</span> <span className="privacy-pill">Browser-local</span>
@@ -362,11 +495,54 @@ export function Workbench() {
</div> </div>
<textarea <textarea
value={source} value={source}
onChange={(event) => setSource(event.target.value)} onChange={(event) => {
setSource(event.target.value);
setSourceEvidence(undefined);
setSourceName("pasted-text.txt");
}}
spellCheck={false} spellCheck={false}
aria-label="Text source" aria-label="Text source"
/> />
<Inventory value={source} /> <Inventory value={source} />
{sourceEvidence && (
<details>
<summary>Byte decoding evidence</summary>
<dl className="inventory">
<div>
<dt>Selected / BOM</dt>
<dd>
{sourceEvidence.selectedEncoding} /{" "}
{sourceEvidence.bom?.encoding ?? "none"}
</dd>
</div>
<div>
<dt>UTF-8 validation</dt>
<dd>
{sourceEvidence.utf8.valid
? "valid"
: `invalid at byte ${sourceEvidence.utf8.firstInvalidOffset}`}
</dd>
</div>
<div>
<dt>Replacement characters</dt>
<dd>{sourceEvidence.replacementCharacters}</dd>
</div>
<div>
<dt>Byte newlines CRLF / LF / CR</dt>
<dd>
{sourceEvidence.byteNewlines.crlf} /{" "}
{sourceEvidence.byteNewlines.bareLf} /{" "}
{sourceEvidence.byteNewlines.bareCr}
</dd>
</div>
</dl>
{sourceEvidence.warnings.map((warning) => (
<p className="warning" key={warning}>
{warning}
</p>
))}
</details>
)}
</section> </section>
<section className="panel workspace" aria-labelledby="output-heading"> <section className="panel workspace" aria-labelledby="output-heading">
<div className="panel-heading"> <div className="panel-heading">
@@ -386,6 +562,9 @@ export function Workbench() {
<button type="button" onClick={download}> <button type="button" onClick={download}>
Download Download
</button> </button>
<button type="button" onClick={() => void exportArtifact()}>
Export artifact + evidence
</button>
</div> </div>
</div> </div>
<textarea <textarea
@@ -533,6 +712,7 @@ export function Workbench() {
<summary>Versioned recipe import/export</summary> <summary>Versioned recipe import/export</summary>
<div className="recipe"> <div className="recipe">
<textarea <textarea
aria-label="Text Tools recipe JSON"
value={recipe} value={recipe}
onChange={(event) => setRecipe(event.target.value)} onChange={(event) => setRecipe(event.target.value)}
placeholder="Paste a Text Tools recipe JSON here." placeholder="Paste a Text Tools recipe JSON here."
@@ -551,7 +731,12 @@ export function Workbench() {
Encoding detection is limited to an explicit choice; no arbitrary Encoding detection is limited to an explicit choice; no arbitrary
charset guess is made. Transliteration, compatibility normalization, charset guess is made. Transliteration, compatibility normalization,
escaping, column omission, and narrow encodings can be lossy, so the escaping, column omission, and narrow encodings can be lossy, so the
exact source and output remain visible. exact source and output remain visible. Portable artifact evidence
includes exact hashes, selected encodings, newline facts and the
ordered recipe. SDK 0.3.0 provides the shared transfer contract, but
one-click Open With remains disabled until the Portal consumer rollout
is coordinated. This version provides explicit portable downloads in
the meantime.
</p> </p>
</section> </section>
</main> </main>
+14
View File
@@ -298,6 +298,14 @@ textarea {
display: flex; display: flex;
gap: 0.25rem; gap: 0.25rem;
} }
.column-options {
display: grid;
grid-template-columns: minmax(8rem, 1fr) minmax(4rem, 0.55fr) minmax(
7rem,
1fr
);
gap: 0.4rem;
}
.step-actions button { .step-actions button {
min-width: 2.55rem; min-width: 2.55rem;
padding: 0.4rem; padding: 0.4rem;
@@ -361,6 +369,9 @@ summary {
.steps li { .steps li {
grid-template-columns: 1fr; grid-template-columns: 1fr;
} }
.column-options {
grid-template-columns: 1fr;
}
} }
@media (max-width: 42rem) { @media (max-width: 42rem) {
.hero { .hero {
@@ -369,4 +380,7 @@ summary {
.privacy-pill { .privacy-pill {
order: -1; order: -1;
} }
.encoding-row {
grid-template-columns: minmax(0, 1fr);
}
} }
+251
View File
@@ -0,0 +1,251 @@
import {
decodeText,
digestHex,
encodeText,
type TextEncoding,
} from "@add-ideas/toolbox-helpers";
import {
textInventory,
type PipelineResult,
type StepReport,
type TransformStep,
} from "./pipeline";
export interface TextByteEvidence {
readonly schemaVersion: 1;
readonly byteLength: number;
readonly selectedEncoding: TextEncoding;
readonly fatalDecode: boolean;
readonly bom?: { encoding: Exclude<TextEncoding, "latin1">; bytes: number };
readonly utf8: { valid: boolean; firstInvalidOffset?: number };
readonly zeroBytes: {
total: number;
evenOffsets: number;
oddOffsets: number;
};
readonly byteNewlines: { crlf: number; bareLf: number; bareCr: number };
readonly replacementCharacters: number;
readonly warnings: readonly string[];
}
export interface TextArtifactEvidence {
readonly schemaVersion: 1;
readonly artifactType: "de.add-ideas.toolbox.text/v1";
readonly createdBy: { app: "text-tools"; version: "0.2.0" };
readonly source: {
name: string;
canonicalUtf8Sha256: string;
inventory: ReturnType<typeof textInventory>;
byteEvidence?: TextByteEvidence;
};
readonly output: {
name: string;
mediaType: "text/plain";
encoding: TextEncoding;
bytes: number;
sha256: string;
inventory: ReturnType<typeof textInventory>;
};
readonly pipeline: {
steps: readonly TransformStep[];
reports: readonly StepReport[];
warnings: readonly string[];
};
readonly handoff: {
supportedByThisBuild: false;
note: string;
};
}
function bom(bytes: Uint8Array): TextByteEvidence["bom"] {
if (bytes[0] === 0xef && bytes[1] === 0xbb && bytes[2] === 0xbf)
return { encoding: "utf-8", bytes: 3 };
if (bytes[0] === 0xff && bytes[1] === 0xfe)
return { encoding: "utf-16le", bytes: 2 };
if (bytes[0] === 0xfe && bytes[1] === 0xff)
return { encoding: "utf-16be", bytes: 2 };
return undefined;
}
function utf8InvalidOffset(bytes: Uint8Array): number | undefined {
const continuation = (index: number) =>
index < bytes.length && (bytes[index]! & 0xc0) === 0x80;
for (let index = 0; index < bytes.length; index += 1) {
const first = bytes[index]!;
if (first <= 0x7f) continue;
if (first >= 0xc2 && first <= 0xdf) {
if (!continuation(index + 1)) return index;
index += 1;
continue;
}
if (first >= 0xe0 && first <= 0xef) {
const second = bytes[index + 1];
if (
second === undefined ||
(first === 0xe0 && (second < 0xa0 || second > 0xbf)) ||
(first === 0xed && (second < 0x80 || second > 0x9f)) ||
(first !== 0xe0 &&
first !== 0xed &&
(second < 0x80 || second > 0xbf)) ||
!continuation(index + 2)
)
return index;
index += 2;
continue;
}
if (first >= 0xf0 && first <= 0xf4) {
const second = bytes[index + 1];
if (
second === undefined ||
(first === 0xf0 && (second < 0x90 || second > 0xbf)) ||
(first === 0xf4 && (second < 0x80 || second > 0x8f)) ||
(first !== 0xf0 &&
first !== 0xf4 &&
(second < 0x80 || second > 0xbf)) ||
!continuation(index + 2) ||
!continuation(index + 3)
)
return index;
index += 3;
continue;
}
return index;
}
return undefined;
}
function inspectByteNewlines(bytes: Uint8Array) {
let crlf = 0;
let bareLf = 0;
let bareCr = 0;
for (let index = 0; index < bytes.length; index += 1) {
if (bytes[index] === 0x0d && bytes[index + 1] === 0x0a) {
crlf += 1;
index += 1;
} else if (bytes[index] === 0x0a) bareLf += 1;
else if (bytes[index] === 0x0d) bareCr += 1;
}
return { crlf, bareLf, bareCr };
}
export function decodeTextWithEvidence(
bytes: Uint8Array,
selectedEncoding: TextEncoding,
fatalDecode: boolean,
): { text: string; evidence: TextByteEvidence } {
if (bytes.byteLength > 16 * 1024 * 1024)
throw new RangeError("Text byte evidence is limited to 16 MiB.");
const detectedBom = bom(bytes);
const invalidOffset = utf8InvalidOffset(bytes);
let zeroTotal = 0;
let zeroEven = 0;
let zeroOdd = 0;
bytes.forEach((value, index) => {
if (value !== 0) return;
zeroTotal += 1;
if (index % 2) zeroOdd += 1;
else zeroEven += 1;
});
const text = decodeText(bytes, selectedEncoding, fatalDecode);
const warnings: string[] = [];
if (detectedBom && detectedBom.encoding !== selectedEncoding)
warnings.push(
`The ${detectedBom.encoding.toUpperCase()} BOM conflicts with the selected ${selectedEncoding.toUpperCase()} decoder.`,
);
if (!detectedBom)
warnings.push(
"No byte-order mark is present; the selected encoding is an explicit user choice, not a detection claim.",
);
if (selectedEncoding.startsWith("utf-16") && bytes.length % 2)
warnings.push("UTF-16 input has an odd trailing byte.");
if (selectedEncoding !== "utf-8" && invalidOffset === undefined)
warnings.push(
"The same bytes are also well-formed UTF-8; encoding intent cannot be inferred from validity alone.",
);
const replacementCharacters = [...text].filter(
(character) => character === "\uFFFD",
).length;
if (replacementCharacters)
warnings.push(
`${replacementCharacters} replacement character(s) appear in decoded text; they may be source data or decoder substitutions.`,
);
return {
text,
evidence: Object.freeze({
schemaVersion: 1,
byteLength: bytes.byteLength,
selectedEncoding,
fatalDecode,
...(detectedBom ? { bom: detectedBom } : {}),
utf8: {
valid: invalidOffset === undefined,
...(invalidOffset === undefined
? {}
: { firstInvalidOffset: invalidOffset }),
},
zeroBytes: {
total: zeroTotal,
evenOffsets: zeroEven,
oddOffsets: zeroOdd,
},
byteNewlines: inspectByteNewlines(bytes),
replacementCharacters,
warnings: Object.freeze(warnings),
}),
};
}
export async function createTextArtifactEvidence(input: {
sourceName: string;
sourceEvidence?: TextByteEvidence;
sourceText: string;
outputName: string;
outputEncoding: TextEncoding;
outputBytes: Uint8Array;
pipeline: PipelineResult;
steps: readonly TransformStep[];
}): Promise<TextArtifactEvidence> {
if (input.outputBytes.byteLength > 32 * 1024 * 1024)
throw new RangeError("Artifact output exceeds the 32 MiB evidence limit.");
const canonicalSource = encodeText(input.sourceText, "utf-8");
return Object.freeze({
schemaVersion: 1,
artifactType: "de.add-ideas.toolbox.text/v1",
createdBy: {
app: "text-tools" as const,
version: "0.2.0" as const,
},
source: {
name: input.sourceName,
canonicalUtf8Sha256: await digestHex(
canonicalSource,
"SHA-256",
32 * 1024 * 1024,
),
inventory: textInventory(input.sourceText),
...(input.sourceEvidence ? { byteEvidence: input.sourceEvidence } : {}),
},
output: {
name: input.outputName,
mediaType: "text/plain" as const,
encoding: input.outputEncoding,
bytes: input.outputBytes.byteLength,
sha256: await digestHex(input.outputBytes, "SHA-256", 32 * 1024 * 1024),
inventory: textInventory(input.pipeline.output),
},
pipeline: {
steps: Object.freeze(
input.steps.map((step) => Object.freeze({ ...step })),
),
reports: Object.freeze(
input.pipeline.steps.map((report) => Object.freeze({ ...report })),
),
warnings: Object.freeze([...input.pipeline.warnings]),
},
handoff: {
supportedByThisBuild: false as const,
note: "The portable files are ready for explicit local transfer. SDK 0.3.0 provides the shared contract, but Open With remains disabled until the coordinated Portal consumer rollout.",
},
});
}
+247 -12
View File
@@ -1,9 +1,14 @@
import { import {
base64ToBytes,
bytesToBase64, bytesToBase64,
bytesToHex, bytesToHex,
convertLineEndings, convertLineEndings,
decodeText,
encodeText, encodeText,
hexToBytes,
normalizeUnicode, normalizeUnicode,
parseCsv,
stringifyCsv,
transformCase, transformCase,
type CaseTransform, type CaseTransform,
type LineEnding, type LineEnding,
@@ -20,8 +25,16 @@ export type StepType =
| "normalize" | "normalize"
| "transliterate" | "transliterate"
| "escape" | "escape"
| "unescape"
| "wrap" | "wrap"
| "columns"; | "columns"
| "replace-literal"
| "prefix-lines"
| "suffix-lines"
| "filter-lines"
| "number-lines"
| "join-lines"
| "reverse-lines";
export interface TransformStep { export interface TransformStep {
id: string; id: string;
type: StepType; type: StepType;
@@ -86,11 +99,27 @@ function wrapText(value: string, width: number): string {
.join("\n"); .join("\n");
} }
function transformColumns(value: string, option: string): string { function transformColumns(
const [delimiter = ",", order = "1"] = option.split("|"); value: string,
option: string,
): {
value: string;
warning?: string;
} {
const pieces = option.split("|");
const mode = pieces.length >= 3 ? pieces[0] : "literal";
const delimiterInput = pieces.length >= 3 ? pieces[1] : pieces[0];
const order = pieces.length >= 3 ? pieces.slice(2).join("|") : pieces[1];
const delimiter = delimiterInput === "\\t" ? "\t" : delimiterInput || ",";
if (!delimiter || delimiter.length > 8) if (!delimiter || delimiter.length > 8)
throw new Error("Column delimiter must contain 18 characters."); throw new Error("Column delimiter must contain 18 characters.");
const indices = order.split(",").map((entry) => Number(entry.trim()) - 1); if (mode !== "literal" && mode !== "csv")
throw new Error("Column parsing mode must be csv or literal.");
if (mode === "csv" && delimiter.length !== 1)
throw new Error("Quoted CSV mode requires a one-character delimiter.");
const indices = (order || "1")
.split(",")
.map((entry) => Number(entry.trim()) - 1);
if ( if (
!indices.length || !indices.length ||
indices.some( indices.some(
@@ -98,12 +127,95 @@ function transformColumns(value: string, option: string): string {
) )
) )
throw new Error("Column order uses 1-based indices such as 3,1,2."); throw new Error("Column order uses 1-based indices such as 3,1,2.");
return lines(value) if (mode === "literal")
.map((line) => { return {
const cells = line.split(delimiter); value: lines(value)
return indices.map((index) => cells[index] ?? "").join(delimiter); .map((line) => {
}) const cells = line.split(delimiter);
.join("\n"); return indices.map((index) => cells[index] ?? "").join(delimiter);
})
.join("\n"),
};
const rows = parseCsv(value, {
delimiter,
maxRows: 200_000,
maxColumns: 1_000,
maxFieldChars: 2_000_000,
});
return {
value: stringifyCsv(
rows.map((row) => indices.map((index) => row[index] ?? "")),
{
delimiter,
maxRows: 200_000,
maxColumns: 1_000,
maxFieldChars: 2_000_000,
},
),
warning:
"Quoted CSV rows were parsed across embedded delimiters/newlines and serialized canonically with CRLF row endings.",
};
}
function strictHtmlUnescape(value: string): string {
const entities: Record<string, string> = {
"&amp;": "&",
"&lt;": "<",
"&gt;": ">",
"&quot;": '"',
"&#39;": "'",
};
let output = "";
for (let index = 0; index < value.length; index += 1) {
if (value[index] !== "&") {
output += value[index];
continue;
}
const end = value.indexOf(";", index + 1);
if (end < 0 || end - index > 6)
throw new SyntaxError(
`HTML entity at UTF-16 index ${index} is malformed.`,
);
const entity = value.slice(index, end + 1);
const decoded = entities[entity];
if (decoded === undefined)
throw new SyntaxError(
`HTML entity ${entity} is not emitted by the matching escape stage.`,
);
output += decoded;
index = end;
}
return output;
}
function strictJsonUnescape(value: string): string {
const parsed = JSON.parse(`"${value}"`) as unknown;
if (typeof parsed !== "string")
throw new SyntaxError("JSON escape input is invalid.");
return parsed;
}
function unescapeValue(value: string, option: string): string {
if (option === "json") return strictJsonUnescape(value);
if (option === "html") return strictHtmlUnescape(value);
if (option === "url") return decodeURIComponent(value);
if (option === "base64")
return decodeText(
base64ToBytes(value, { maxOutputBytes: MAX_OUTPUT }),
"utf-8",
true,
);
if (option === "hex")
return decodeText(
hexToBytes(value, {
maxOutputBytes: MAX_OUTPUT,
allowWhitespace: false,
allowPrefix: false,
}),
"utf-8",
true,
);
throw new Error("Unsupported unescape source.");
} }
function transliterate(value: string): string { function transliterate(value: string): string {
@@ -119,6 +231,74 @@ function transliterate(value: string): string {
.replaceAll("ł", "l"); .replaceAll("ł", "l");
} }
function parseReplacement(option: string): readonly [string, string] {
let parsed: unknown;
try {
parsed = JSON.parse(option);
} catch {
throw new SyntaxError("Literal replacement options are invalid.");
}
if (
!Array.isArray(parsed) ||
parsed.length !== 2 ||
parsed.some((value) => typeof value !== "string")
)
throw new SyntaxError(
"Literal replacement requires [search, replacement].",
);
const [search, replacement] = parsed as [string, string];
if (!search) throw new SyntaxError("Literal search text must not be empty.");
if (search.length > 100_000 || replacement.length > 1_000_000)
throw new RangeError(
"Literal replacement option exceeds its safety limit.",
);
return [search, replacement];
}
function decodedDelimiter(option: string): string {
if (option.length > 1_000)
throw new RangeError("Line join delimiter exceeds 1,000 characters.");
return option.replaceAll("\\n", "\n").replaceAll("\\t", "\t");
}
function assertProjectedOutput(units: number): void {
if (!Number.isSafeInteger(units) || units > MAX_OUTPUT)
throw new RangeError(
"Transformation would exceed the 32 MiB output limit.",
);
}
function replaceLiteral(
value: string,
search: string,
replacement: string,
): string {
let matches = 0;
let offset = 0;
while ((offset = value.indexOf(search, offset)) >= 0) {
matches += 1;
offset += search.length;
assertProjectedOutput(
value.length + matches * (replacement.length - search.length),
);
}
return value.replaceAll(search, replacement);
}
function decorateLines(
value: string,
addition: string,
side: "prefix" | "suffix",
): string {
if (addition.length > 100_000)
throw new RangeError("Line decoration exceeds 100,000 UTF-16 units.");
const values = lines(value);
assertProjectedOutput(value.length + values.length * addition.length);
return values
.map((line) => (side === "prefix" ? addition + line : line + addition))
.join("\n");
}
function applyStep( function applyStep(
value: string, value: string,
step: TransformStep, step: TransformStep,
@@ -188,10 +368,57 @@ function applyStep(
return { value: bytesToHex(encodeText(value)) }; return { value: bytesToHex(encodeText(value)) };
throw new Error("Unsupported escape target."); throw new Error("Unsupported escape target.");
} }
case "unescape":
return {
value: unescapeValue(value, step.option),
warning:
step.option === "base64" || step.option === "hex"
? "Decoded bytes are required to be well-formed UTF-8 text; arbitrary binary is rejected."
: undefined,
};
case "wrap": case "wrap":
return { value: wrapText(value, Number(step.option)) }; return { value: wrapText(value, Number(step.option)) };
case "columns": case "columns":
return { value: transformColumns(value, step.option) }; return transformColumns(value, step.option);
case "replace-literal": {
const [search, replacement] = parseReplacement(step.option);
return { value: replaceLiteral(value, search, replacement) };
}
case "prefix-lines":
return { value: decorateLines(value, step.option, "prefix") };
case "suffix-lines":
return { value: decorateLines(value, step.option, "suffix") };
case "filter-lines": {
if (!step.option || step.option.length > 100_000)
throw new SyntaxError("Line filter text must not be empty.");
return {
value: lines(value)
.filter((line) => line.includes(step.option))
.join("\n"),
warning: "Line filtering removes every line without the literal text.",
};
}
case "number-lines": {
const start = Number(step.option || "1");
if (!Number.isSafeInteger(start) || Math.abs(start) > 1_000_000_000)
throw new RangeError("Line-number start must be a bounded integer.");
return {
value: lines(value)
.map((line, index) => `${start + index}. ${line}`)
.join("\n"),
};
}
case "join-lines": {
const values = lines(value);
const delimiter = decodedDelimiter(step.option);
assertProjectedOutput(
values.reduce((total, line) => total + line.length, 0) +
Math.max(0, values.length - 1) * delimiter.length,
);
return { value: values.join(delimiter) };
}
case "reverse-lines":
return { value: lines(value).reverse().join("\n") };
} }
} }
@@ -243,8 +470,16 @@ export function createStep(type: StepType): TransformStep {
normalize: "NFC", normalize: "NFC",
transliterate: "", transliterate: "",
escape: "json", escape: "json",
unescape: "json",
wrap: "80", wrap: "80",
columns: ",|1", columns: "csv|,|1",
"replace-literal": JSON.stringify(["old", "new"]),
"prefix-lines": "> ",
"suffix-lines": "",
"filter-lines": "text",
"number-lines": "1",
"join-lines": " ",
"reverse-lines": "",
}; };
return { return {
id: `step-${++nextStepId}`, id: `step-${++nextStepId}`,
+34 -3
View File
@@ -3,12 +3,21 @@
"schemaVersion": 1, "schemaVersion": 1,
"id": "de.add-ideas.text-tools", "id": "de.add-ideas.text-tools",
"name": "Text Tools", "name": "Text Tools",
"version": "0.1.0", "version": "0.2.0",
"description": "Transform and inspect plain text locally in the browser.", "description": "Compose text transforms and export encoding evidence locally.",
"entry": "./", "entry": "./",
"icon": "./favicon.svg", "icon": "./favicon.svg",
"categories": ["text", "developer", "productivity"], "categories": ["text", "developer", "productivity"],
"tags": ["text", "unicode", "normalize", "sort", "escape", "encoding"], "tags": [
"text",
"unicode",
"pipeline",
"normalize",
"sort",
"escape",
"encoding",
"newline"
],
"integration": { "integration": {
"contextVersion": 1, "contextVersion": 1,
"launchModes": ["navigate", "new-tab"], "launchModes": ["navigate", "new-tab"],
@@ -21,6 +30,28 @@
"crossOriginIsolated": false, "crossOriginIsolated": false,
"topLevelContext": false "topLevelContext": false
}, },
"io": {
"accepts": [
{
"mediaType": "text/*",
"extensions": [".txt", ".csv", ".md", ".log"],
"label": "Bounded text files"
}
],
"produces": [
{
"mediaType": "text/plain",
"extensions": [".txt"],
"label": "Transformed text"
},
{
"mediaType": "application/json",
"extensions": [".json"],
"label": "Recipe and artifact evidence"
}
]
},
"capabilities": { "required": [], "optional": ["web-crypto"] },
"privacy": { "privacy": {
"processing": "local", "processing": "local",
"fileUploads": true, "fileUploads": true,
+1 -1
View File
@@ -1 +1 @@
export const APP_VERSION = "0.1.0"; export const APP_VERSION = "0.2.0";
+31 -1
View File
@@ -1,4 +1,5 @@
import { expect, test, type Page } from "@playwright/test"; import { expect, test, type Page } from "@playwright/test";
import { Buffer } from "node:buffer";
const ORIGIN = "http://127.0.0.1:4173"; const ORIGIN = "http://127.0.0.1:4173";
async function localOnly(page: Page) { async function localOnly(page: Page) {
@@ -43,6 +44,35 @@ test("applies the visible text pipeline and preserves its last output", async ({
expect(external).toEqual([]); expect(external).toEqual([]);
}); });
test("shows byte evidence and composes an added literal transform", async ({
page,
}) => {
await page.goto("/deep/nested/text/");
await page.getByLabel("Decode as").selectOption("utf-8");
await page.locator('input[type="file"]').setInputFiles({
name: "bom.txt",
mimeType: "text/plain",
buffer: Buffer.from([0xef, 0xbb, 0xbf, ...Buffer.from("old\r\nold")]),
});
await page.getByText("Byte decoding evidence").click();
await expect(page.getByText("utf-8 / utf-8")).toBeVisible();
await expect(
page
.locator("details")
.filter({ hasText: "Byte decoding evidence" })
.getByText("1 / 0 / 0", { exact: true }),
).toBeVisible();
await page.getByLabel("New transformation").selectOption("replace-literal");
await page.getByRole("button", { name: "Add step" }).click();
await page.getByLabel("Literal search text").fill("old");
await page.getByLabel("Literal replacement text").fill("new");
await page.getByRole("button", { name: "Apply pipeline" }).click();
await expect(page.getByLabel("Transformed output")).toHaveValue("new");
await expect(
page.getByRole("button", { name: "Export artifact + evidence" }),
).toBeVisible();
});
test("serves the release identity and hardened headers", async ({ test("serves the release identity and hardened headers", async ({
request, request,
}) => { }) => {
@@ -55,7 +85,7 @@ test("serves the release identity and hardened headers", async ({
const manifest = await request.get("/deep/nested/text/toolbox-app.json"); const manifest = await request.get("/deep/nested/text/toolbox-app.json");
await expect(manifest.json()).resolves.toMatchObject({ await expect(manifest.json()).resolves.toMatchObject({
id: "de.add-ideas.text-tools", id: "de.add-ideas.text-tools",
version: "0.1.0", version: "0.2.0",
entry: "./", entry: "./",
}); });
}); });
+18
View File
@@ -0,0 +1,18 @@
import { expect, test } from "@playwright/test";
test("keeps the primary workspace inside a narrow viewport", async ({
page,
}) => {
await page.goto("/deep/nested/text/");
await expect(page.locator("main").first()).toBeVisible();
await expect(
page.locator("main .loading, main .workbench-loading"),
).toHaveCount(0);
const widths = await page.evaluate(() => ({
content: document.documentElement.scrollWidth,
viewport: document.documentElement.clientWidth,
}));
expect(widths.viewport).toBeLessThanOrEqual(430);
expect(widths.content).toBeLessThanOrEqual(widths.viewport + 1);
});
+105
View File
@@ -4,6 +4,10 @@ import {
textInventory, textInventory,
type TransformStep, type TransformStep,
} from "../../src/text/pipeline"; } from "../../src/text/pipeline";
import {
createTextArtifactEvidence,
decodeTextWithEvidence,
} from "../../src/text/evidence";
const step = (type: TransformStep["type"], option = ""): TransformStep => ({ const step = (type: TransformStep["type"], option = ""): TransformStep => ({
id: type, id: type,
@@ -28,6 +32,36 @@ describe("text pipelines", () => {
expect( expect(
applyPipeline("<img src=x onerror=1>", [step("escape", "html")]).output, applyPipeline("<img src=x onerror=1>", [step("escape", "html")]).output,
).toBe("&lt;img src=x onerror=1&gt;")); ).toBe("&lt;img src=x onerror=1&gt;"));
it("strictly reverses each escape/encoding stage", () => {
const source = 'café <x> & "quoted"\n';
for (const option of ["json", "html", "url", "base64", "hex"])
expect(
applyPipeline(applyPipeline(source, [step("escape", option)]).output, [
step("unescape", option),
]).output,
).toBe(source);
expect(() => applyPipeline("&copy;", [step("unescape", "html")])).toThrow(
/not emitted/u,
);
expect(() => applyPipeline("w6k", [step("unescape", "base64")])).toThrow(
/canonical|padding|UTF-8/u,
);
expect(() => applyPipeline("%ZZ", [step("unescape", "url")])).toThrow();
});
it("selects columns from quoted CSV records including embedded newlines", () => {
const result = applyPipeline(
'name,note,id\r\nAlice,"hello, world",1\r\nBob,"two\nlines",2\r\n',
[step("columns", "csv|,|3,1,2")],
);
expect(result.output).toContain('1,Alice,"hello, world"');
expect(result.output).toContain('2,Bob,"two\nlines"');
expect(result.warnings[0]).toMatch(/Quoted CSV/u);
});
it("retains multi-character literal-delimiter mode", () =>
expect(
applyPipeline("left::right\na::b", [step("columns", "literal|::|2,1")])
.output,
).toBe("right::left\nb::a"));
it("reports lossy transliteration", () => it("reports lossy transliteration", () =>
expect(applyPipeline("Crème", [step("transliterate")]).warnings[0]).toMatch( expect(applyPipeline("Crème", [step("transliterate")]).warnings[0]).toMatch(
/lossy/u, /lossy/u,
@@ -39,4 +73,75 @@ describe("text pipelines", () => {
bareCr: 1, bareCr: 1,
finalNewline: true, finalNewline: true,
})); }));
it("composes literal replacement, filtering, numbering and line decoration", () => {
const result = applyPipeline("old apple\nold pear\nbanana", [
step("replace-literal", JSON.stringify(["old", "fresh"])),
step("filter-lines", "fresh"),
step("prefix-lines", "- "),
step("number-lines", "10"),
step("suffix-lines", "!"),
step("join-lines", " | "),
]);
expect(result.output).toBe("10. - fresh apple! | 11. - fresh pear!");
expect(result.warnings).toEqual(
expect.arrayContaining([expect.stringMatching(/filtering removes/u)]),
);
});
it("reverses lines without reversing grapheme contents", () =>
expect(applyPipeline("😀a\nb", [step("reverse-lines")]).output).toBe(
"b\n😀a",
));
it("preflights expanding transforms before allocating their output", () => {
expect(() =>
applyPipeline("x".repeat(40), [
step("replace-literal", JSON.stringify(["x", "y".repeat(1_000_000)])),
]),
).toThrow(/32 MiB/u);
expect(() =>
applyPipeline("a\nb", [step("prefix-lines", "x".repeat(100_001))]),
).toThrow(/100,000/u);
});
});
describe("byte and portable artifact evidence", () => {
it("records BOM conflicts, UTF-8 validity and byte newline forms", () => {
const bytes = Uint8Array.of(0xef, 0xbb, 0xbf, 0x61, 0x0d, 0x0a, 0x62);
const decoded = decodeTextWithEvidence(bytes, "latin1", true);
expect(decoded.evidence).toMatchObject({
byteLength: 7,
bom: { encoding: "utf-8", bytes: 3 },
utf8: { valid: true },
byteNewlines: { crlf: 1, bareLf: 0, bareCr: 0 },
});
expect(decoded.evidence.warnings.join(" ")).toMatch(/conflicts/u);
const malformed = decodeTextWithEvidence(
Uint8Array.of(0xc3, 0x28),
"latin1",
true,
);
expect(malformed.evidence.utf8).toEqual({
valid: false,
firstInvalidOffset: 0,
});
});
it("emits exact output hashes and an honest handoff marker", async () => {
const steps = [step("case", "upper")];
const pipeline = applyPipeline("hello", steps);
const bytes = new TextEncoder().encode(pipeline.output);
const evidence = await createTextArtifactEvidence({
sourceName: "input.txt",
sourceText: "hello",
outputName: "output.txt",
outputEncoding: "utf-8",
outputBytes: bytes,
pipeline,
steps,
});
expect(evidence.output.sha256).toMatch(/^[a-f0-9]{64}$/u);
expect(evidence.handoff.supportedByThisBuild).toBe(false);
expect(evidence.pipeline.steps[0]).toMatchObject({ type: "case" });
});
}); });
+1 -1
View File
@@ -4,7 +4,7 @@
"target": "ES2023", "target": "ES2023",
"lib": ["ES2023", "DOM", "DOM.Iterable"], "lib": ["ES2023", "DOM", "DOM.Iterable"],
"module": "ESNext", "module": "ESNext",
"types": ["vite/client"], "types": ["vite/client", "node"],
"allowArbitraryExtensions": true, "allowArbitraryExtensions": true,
"skipLibCheck": true, "skipLibCheck": true,
"moduleResolution": "Bundler", "moduleResolution": "Bundler",