feat(translator-tools): add locale review and glossary workflow

This commit is contained in:
2026-09-02 21:39:02 +02:00
parent 5e962ff0f3
commit 655797c4fe
32 changed files with 2084 additions and 2192 deletions
+6 -8
View File
@@ -1,13 +1,11 @@
# Changelog
## 0.2.0 - 2026-09-02
## Unreleased
- Added composable literal and line transforms with allocation preflights, plus BOM, UTF-8 validity, zero-byte and byte-newline evidence.
- Added paired text/artifact-evidence downloads with source/output hashes and
pipeline provenance; automatic Toolbox handoff remains intentionally disabled
pending the coordinated Portal consumer rollout.
- Added local translator workspace for i18next-like and single-locale JSON review.
- Added language pair selection, glossary storage and fuzzy in-context suggestions.
- Added local-only import/export pipeline with portal handoff integration.
## 0.1.0 - 2026-09-01
## 0.1.0 - 2026-09-02
- Added the initial local-first Text Tools workbench.
- Added Toolbox integration, bounded input handling, offline shell and deterministic release packaging.
- Initial local-first translator review implementation.
+18 -17
View File
@@ -1,24 +1,25 @@
# Text Tools
# Translator Tools
Compose, inspect and evidence plain-text transformations locally in the browser.
Translator Tools is a standalone local-first application in the
[add·ideas Toolbox](https://git.add-ideas.de/lotobo/toolbox-portal). Inputs are
parsed and edited in the browser only; no locale strings are uploaded.
Text Tools is a standalone local-first application in the [add·ideas Toolbox](https://git.add-ideas.de/lotobo/toolbox-portal). Inputs are processed in the browser and are not uploaded.
## Quick scope
## Version 0.2 scope
- Import locale files (`.json`, `.js`, `.txt`) in i18next-like or single-locale forms.
- Review authoritative language against one target language at a time.
- Edit source and target strings and copy-source quickly.
- Add translation glossary entries per language pair and receive fuzzy suggestions.
- Add new languages and export a reviewer-updated translation bundle locally.
- Send exports through the same-tool transfer channel to the Portal.
- Ordered pipelines for line endings, trimming, whitespace, stable sorting, deduplication, literal replacement, line prefix/suffix/filter/number/join and line-order reversal
- Locale-aware case conversion, Unicode NFC/NFD/NFKC/NFKD and best-effort Latin transliteration
- JSON/HTML/URL/Base64/hex escaping plus strict inverse decode/unescape stages; binary decoders accept canonical encodings and require valid UTF-8 text
- Bounded RFC-style quoted CSV column selection across embedded delimiters and newlines, alongside the retained literal multi-character delimiter-per-line mode
- Explicit UTF-8, UTF-16LE and Latin-1 file decoding plus BOM, malformed UTF-8 offset, zero-byte parity, byte-level newline and replacement-character evidence
- Versioned JSON recipe import/export, per-step size/change reports, and a paired text-plus-JSON artifact download with source/output SHA-256 and transformation provenance
## Notes
Pipelines are limited to 2,000,000 UTF-16 input units and an 8×/32 MiB output expansion. File input is limited to 16 MiB before decoding. Encoding is selected explicitly rather than guessed; compatibility normalisation, transliteration, narrow encodings and several transforms can be lossy, so source, result, and warnings remain visible. See [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) and [docs/PRIVACY-SECURITY.md](docs/PRIVACY-SECURITY.md).
The artifact evidence is a portable download contract, not an automatic Toolbox
handoff. SDK 0.3.0 provides the shared transfer contract, but automatic Open With
remains disabled in this release until the Portal consumer rollout is
coordinated. Local downloads stay explicit and portable in the meantime.
- The parser accepts JavaScript-style locale assignment forms such as
`const de = { ... }` and tolerant spacing/commas.
- Parsing and transfers are bounded to 5 MiB per imported/exported transfer file.
- Language detection relies on file name hints (`de_*.json`) and explicit user
selection.
## Development
@@ -32,7 +33,7 @@ npm run test:browser
## Release
`npm run release:artifact` creates a deterministic `release/text-tools-0.2.0.zip` and checksum sidecar.
`npm run release:artifact` creates a deterministic `release/translator-tools-0.1.0.zip` and checksum sidecar.
## Licence
+1 -1
View File
@@ -1,6 +1,6 @@
# Third-party notices
Text Tools 0.2.0 directly depends on these runtime packages:
Translator Tools 0.1.0 directly depends on these runtime packages:
| Package | Pinned version | Declared licence |
| -------------------------------- | -------------: | ---------------- |
+11 -4
View File
@@ -1,9 +1,16 @@
# Architecture
Text Tools is a static React/Vite application wrapped in the shared Toolbox shell. `text/pipeline.ts` defines an ordered, typed list of transformations. `applyPipeline` starts from the exact source, applies enabled steps one at a time, checks expansion after every step, and returns the last successful output with per-step before/after counts and warnings.
Translator Tools is a static React/Vite application wrapped in the shared Toolbox shell.
Text encoding/decoding, line-ending conversion, bounded CSV parsing and Unicode-aware case primitives come from `@add-ideas/toolbox-helpers`. File decoding uses an explicit encoding and fatal-or-replacement choice; it does not guess a charset. Inverse JSON/HTML/URL/Base64/hex stages are strict: malformed syntax and non-UTF-8 byte output fail the step rather than being silently repaired. Column selection offers a quoted CSV mode that preserves embedded delimiters/newlines and a separate literal-delimiter-per-line mode. Recipe JSON carries a schema/app identity, accepts at most 100 supported steps, discards supplied IDs and assigns local monotonic IDs.
The workbench parses a single imported locale file in-browser, normalizes either:
`text/evidence.ts` inspects original bytes before decoding, including BOM, exact UTF-8 validity/first invalid offset, zero-byte parity and CR/LF/CRLF counts. Artifact export emits the selected output bytes and a deterministic evidence document containing canonical-source and exact-output SHA-256 values, inventories, recipe and step reports. The document records that direct Toolbox handoff is unavailable in this release; the SDK 0.3.0 transfer contract is intentionally not activated until the Portal consumer rollout is coordinated.
- i18next-like language maps (`{ "en": {...}, "de": {...} }`)
- single-locale objects (`{ "key": "value" }`, or JS-style assignment forms like `const de = {...}`).
Version 0.2 operations are synchronous and bounded, so the app creates no worker and has no persistence or server API. Relative entry and asset URLs keep the build relocatable below a nested portal path.
Strings are flattened into stable dot paths for review/editing and can be serialized
back with an inverse unflattening step. Suggestions come from local glossary entries
and are matched by language pair and fuzzy string distance.
No background workers are required today. Persistent state is limited to a local
Storage glossary cache; everything else remains in process memory until download or
transfer.
+11 -4
View File
@@ -1,9 +1,16 @@
# Privacy and security
Source text, imported files, recipes and results stay in page memory. There is no telemetry, analytics, account, persistence or runtime network path. Imported text is rendered as text/preformatted content, not executable HTML; HTML escaping produces a string and does not preview it as markup.
Locale files and generated outputs stay in-browser and are not uploaded.
Files are rejected above 16 MiB before reading. Decoded/pasted pipeline input is limited to 2,000,000 UTF-16 units, recipe text to 1,000,000 units/100 steps, and output to both 8× the original (with a small-input floor) and 32 MiB. Quoted CSV selection additionally limits rows, columns and field size. Strict inverse stages reject malformed escapes/encodings and invalid UTF-8; they do not attempt permissive repair. These limits reduce accidental expansion but do not prove that output is safe for a downstream interpreter.
No accounts, telemetry or runtime network requests are used for the core workflow.
The app stores glossary terms in local browser storage so they are available on the same
device for repeated reviews.
Compatibility normalisation, case conversion, transliteration, whitespace/line transforms, column selection and narrow encodings can change or lose information. Locale sorting depends on the browser's `Intl.Collator`. The exact source, output, per-step counts and warnings remain visible so users can review those changes before copying or downloading.
Parser/serializer limits are bounded to prevent accidental expansion:
Byte evidence does not guess an encoding or prove the selected decoder is correct. A valid UTF-8 byte sequence may represent text intended for another encoding, zero-byte parity is only a UTF-16 review signal, and replacement characters may already have existed in the source. Artifact evidence hashes canonical UTF-8 source text and exact exported bytes, clearly labels both, and includes filenames and transformation details that may themselves be sensitive.
- Imports and exports are capped at 5 MiB each.
- Large objects are flattened and rendered with per-session bounds.
Suggested translations are purely local matching; they do not call external services.
Transfers to the Portal are signed local handoffs only and never stored by a remote
processor by default.
+486 -71
View File
File diff suppressed because it is too large Load Diff
+7 -6
View File
@@ -1,16 +1,16 @@
{
"name": "text-tools",
"version": "0.2.0",
"description": "Transform and inspect plain text locally in the browser.",
"name": "translator-tools",
"version": "0.1.0",
"description": "Review and translate i18next-like JSON language files locally.",
"license": "GPL-3.0-or-later",
"author": "Albrecht Degering",
"repository": {
"type": "git",
"url": "git+https://git.add-ideas.de/lotobo/text-tools.git"
"url": "git+https://git.add-ideas.de/lotobo/translator-tools.git"
},
"homepage": "https://git.add-ideas.de/lotobo/text-tools",
"homepage": "https://git.add-ideas.de/lotobo/translator-tools",
"bugs": {
"url": "https://git.add-ideas.de/lotobo/text-tools/issues"
"url": "https://git.add-ideas.de/lotobo/translator-tools/issues"
},
"private": true,
"type": "module",
@@ -42,6 +42,7 @@
"@add-ideas/toolbox-contract": "0.3.0",
"@add-ideas/toolbox-helpers": "0.2.0",
"@add-ideas/toolbox-shell-react": "0.3.0",
"json5": "^2.2.3",
"react": "19.2.8",
"react-dom": "19.2.8"
},
+6 -8
View File
@@ -1,13 +1,11 @@
# Changelog
## 0.2.0 - 2026-09-02
## Unreleased
- Added composable literal and line transforms with allocation preflights, plus BOM, UTF-8 validity, zero-byte and byte-newline evidence.
- Added paired text/artifact-evidence downloads with source/output hashes and
pipeline provenance; automatic Toolbox handoff remains intentionally disabled
pending the coordinated Portal consumer rollout.
- Added local translator workspace for i18next-like and single-locale JSON review.
- Added language pair selection, glossary storage and fuzzy in-context suggestions.
- Added local-only import/export pipeline with portal handoff integration.
## 0.1.0 - 2026-09-01
## 0.1.0 - 2026-09-02
- Added the initial local-first Text Tools workbench.
- Added Toolbox integration, bounded input handling, offline shell and deterministic release packaging.
- Initial local-first translator review implementation.
+30
View File
@@ -1077,6 +1077,36 @@ Declared licence: Apache-2.0
limitations under the License.
==============================================================================
json5@2.2.3
Declared licence: MIT
==============================================================================
--- LICENSE.md ---
MIT License
Copyright (c) 2012-2018 Aseem Kishore, and [others].
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
[others]: https://github.com/json5/json5/contributors
==============================================================================
react@19.2.8
Declared licence: MIT
+18 -17
View File
@@ -1,24 +1,25 @@
# Text Tools
# Translator Tools
Compose, inspect and evidence plain-text transformations locally in the browser.
Translator Tools is a standalone local-first application in the
[add·ideas Toolbox](https://git.add-ideas.de/lotobo/toolbox-portal). Inputs are
parsed and edited in the browser only; no locale strings are uploaded.
Text Tools is a standalone local-first application in the [add·ideas Toolbox](https://git.add-ideas.de/lotobo/toolbox-portal). Inputs are processed in the browser and are not uploaded.
## Quick scope
## Version 0.2 scope
- Import locale files (`.json`, `.js`, `.txt`) in i18next-like or single-locale forms.
- Review authoritative language against one target language at a time.
- Edit source and target strings and copy-source quickly.
- Add translation glossary entries per language pair and receive fuzzy suggestions.
- Add new languages and export a reviewer-updated translation bundle locally.
- Send exports through the same-tool transfer channel to the Portal.
- Ordered pipelines for line endings, trimming, whitespace, stable sorting, deduplication, literal replacement, line prefix/suffix/filter/number/join and line-order reversal
- Locale-aware case conversion, Unicode NFC/NFD/NFKC/NFKD and best-effort Latin transliteration
- JSON/HTML/URL/Base64/hex escaping plus strict inverse decode/unescape stages; binary decoders accept canonical encodings and require valid UTF-8 text
- Bounded RFC-style quoted CSV column selection across embedded delimiters and newlines, alongside the retained literal multi-character delimiter-per-line mode
- Explicit UTF-8, UTF-16LE and Latin-1 file decoding plus BOM, malformed UTF-8 offset, zero-byte parity, byte-level newline and replacement-character evidence
- Versioned JSON recipe import/export, per-step size/change reports, and a paired text-plus-JSON artifact download with source/output SHA-256 and transformation provenance
## Notes
Pipelines are limited to 2,000,000 UTF-16 input units and an 8×/32 MiB output expansion. File input is limited to 16 MiB before decoding. Encoding is selected explicitly rather than guessed; compatibility normalisation, transliteration, narrow encodings and several transforms can be lossy, so source, result, and warnings remain visible. See [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) and [docs/PRIVACY-SECURITY.md](docs/PRIVACY-SECURITY.md).
The artifact evidence is a portable download contract, not an automatic Toolbox
handoff. SDK 0.3.0 provides the shared transfer contract, but automatic Open With
remains disabled in this release until the Portal consumer rollout is
coordinated. Local downloads stay explicit and portable in the meantime.
- The parser accepts JavaScript-style locale assignment forms such as
`const de = { ... }` and tolerant spacing/commas.
- Parsing and transfers are bounded to 5 MiB per imported/exported transfer file.
- Language detection relies on file name hints (`de_*.json`) and explicit user
selection.
## Development
@@ -32,7 +33,7 @@ npm run test:browser
## Release
`npm run release:artifact` creates a deterministic `release/text-tools-0.2.0.zip` and checksum sidecar.
`npm run release:artifact` creates a deterministic `release/translator-tools-0.1.0.zip` and checksum sidecar.
## Licence
+1 -1
View File
@@ -1,6 +1,6 @@
# Third-party notices
Text Tools 0.2.0 directly depends on these runtime packages:
Translator Tools 0.1.0 directly depends on these runtime packages:
| Package | Pinned version | Declared licence |
| -------------------------------- | -------------: | ---------------- |
+11 -4
View File
@@ -1,9 +1,16 @@
# Architecture
Text Tools is a static React/Vite application wrapped in the shared Toolbox shell. `text/pipeline.ts` defines an ordered, typed list of transformations. `applyPipeline` starts from the exact source, applies enabled steps one at a time, checks expansion after every step, and returns the last successful output with per-step before/after counts and warnings.
Translator Tools is a static React/Vite application wrapped in the shared Toolbox shell.
Text encoding/decoding, line-ending conversion, bounded CSV parsing and Unicode-aware case primitives come from `@add-ideas/toolbox-helpers`. File decoding uses an explicit encoding and fatal-or-replacement choice; it does not guess a charset. Inverse JSON/HTML/URL/Base64/hex stages are strict: malformed syntax and non-UTF-8 byte output fail the step rather than being silently repaired. Column selection offers a quoted CSV mode that preserves embedded delimiters/newlines and a separate literal-delimiter-per-line mode. Recipe JSON carries a schema/app identity, accepts at most 100 supported steps, discards supplied IDs and assigns local monotonic IDs.
The workbench parses a single imported locale file in-browser, normalizes either:
`text/evidence.ts` inspects original bytes before decoding, including BOM, exact UTF-8 validity/first invalid offset, zero-byte parity and CR/LF/CRLF counts. Artifact export emits the selected output bytes and a deterministic evidence document containing canonical-source and exact-output SHA-256 values, inventories, recipe and step reports. The document records that direct Toolbox handoff is unavailable in this release; the SDK 0.3.0 transfer contract is intentionally not activated until the Portal consumer rollout is coordinated.
- i18next-like language maps (`{ "en": {...}, "de": {...} }`)
- single-locale objects (`{ "key": "value" }`, or JS-style assignment forms like `const de = {...}`).
Version 0.2 operations are synchronous and bounded, so the app creates no worker and has no persistence or server API. Relative entry and asset URLs keep the build relocatable below a nested portal path.
Strings are flattened into stable dot paths for review/editing and can be serialized
back with an inverse unflattening step. Suggestions come from local glossary entries
and are matched by language pair and fuzzy string distance.
No background workers are required today. Persistent state is limited to a local
Storage glossary cache; everything else remains in process memory until download or
transfer.
+11 -4
View File
@@ -1,9 +1,16 @@
# Privacy and security
Source text, imported files, recipes and results stay in page memory. There is no telemetry, analytics, account, persistence or runtime network path. Imported text is rendered as text/preformatted content, not executable HTML; HTML escaping produces a string and does not preview it as markup.
Locale files and generated outputs stay in-browser and are not uploaded.
Files are rejected above 16 MiB before reading. Decoded/pasted pipeline input is limited to 2,000,000 UTF-16 units, recipe text to 1,000,000 units/100 steps, and output to both 8× the original (with a small-input floor) and 32 MiB. Quoted CSV selection additionally limits rows, columns and field size. Strict inverse stages reject malformed escapes/encodings and invalid UTF-8; they do not attempt permissive repair. These limits reduce accidental expansion but do not prove that output is safe for a downstream interpreter.
No accounts, telemetry or runtime network requests are used for the core workflow.
The app stores glossary terms in local browser storage so they are available on the same
device for repeated reviews.
Compatibility normalisation, case conversion, transliteration, whitespace/line transforms, column selection and narrow encodings can change or lose information. Locale sorting depends on the browser's `Intl.Collator`. The exact source, output, per-step counts and warnings remain visible so users can review those changes before copying or downloading.
Parser/serializer limits are bounded to prevent accidental expansion:
Byte evidence does not guess an encoding or prove the selected decoder is correct. A valid UTF-8 byte sequence may represent text intended for another encoding, zero-byte parity is only a UTF-16 review signal, and replacement characters may already have existed in the source. Artifact evidence hashes canonical UTF-8 source text and exact exported bytes, clearly labels both, and includes filenames and transformation details that may themselves be sensitive.
- Imports and exports are capped at 5 MiB each.
- Large objects are flattened and rendered with per-session bounds.
Suggested translations are purely local matching; they do not call external services.
Transfers to the Portal are signed local handoffs only and never stored by a remote
processor by default.
+1 -1
View File
@@ -1 +1 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 64 64" role="img" aria-label="Text Tools"><path fill="#5b4ec4" d="M11 4h30l12 12v44H11z"/><path fill="#fff" opacity=".28" d="M41 4v14h12"/><text x="32" y="43" fill="#fff" font-family="system-ui,sans-serif" font-size="18" font-weight="800" text-anchor="middle">TE</text></svg>
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 64 64" role="img" aria-label="Translator Tools"><path fill="#5b4ec4" d="M11 4h30l12 12v44H11z"/><path fill="#fff" opacity=".28" d="M41 4v14h12"/><text x="32" y="43" fill="#fff" font-family="system-ui,sans-serif" font-size="18" font-weight="800" text-anchor="middle">TR</text></svg>

Before

Width:  |  Height:  |  Size: 329 B

After

Width:  |  Height:  |  Size: 335 B

+3 -3
View File
@@ -1,7 +1,7 @@
{
"name": "Text Tools",
"short_name": "Text",
"description": "Transform and inspect plain text locally in the browser.",
"name": "Translator Tools",
"short_name": "Translator",
"description": "Review and edit locale strings locally in the browser.",
"start_url": "./",
"scope": "./",
"display": "standalone",
+2 -2
View File
@@ -1,5 +1,5 @@
const CACHE_PREFIX = "text-tools-shell-";
const CACHE_NAME = CACHE_PREFIX + "0.2.0";
const CACHE_PREFIX = "translator-tools-shell-";
const CACHE_NAME = CACHE_PREFIX + "0.1.0";
const CORE = ["./", "./manifest.webmanifest", "./favicon.svg"];
self.addEventListener("install", (event) => {
event.waitUntil(
+23 -26
View File
@@ -1,22 +1,21 @@
{
"$schema": "https://git.add-ideas.de/lotobo/toolbox-sdk/raw/branch/main/schemas/toolbox-app.v1.schema.json",
"schemaVersion": 1,
"id": "de.add-ideas.text-tools",
"name": "Text Tools",
"version": "0.2.0",
"description": "Compose text transforms and export encoding evidence locally.",
"id": "de.add-ideas.translator-tools",
"name": "Translator Tools",
"version": "0.1.0",
"description": "Review and edit translation bundles in the browser with glossary suggestions.",
"entry": "./",
"icon": "./favicon.svg",
"categories": ["text", "developer", "productivity"],
"categories": ["text", "developer", "productivity", "i18n"],
"tags": [
"text",
"unicode",
"pipeline",
"normalize",
"sort",
"escape",
"encoding",
"newline"
"translation",
"i18n",
"locale",
"glossary",
"review",
"json",
"localization"
],
"integration": {
"contextVersion": 1,
@@ -26,32 +25,30 @@
"requirements": {
"secureContext": false,
"workers": false,
"indexedDb": false,
"indexedDb": true,
"crossOriginIsolated": false,
"topLevelContext": false
},
"io": {
"accepts": [
{
"mediaType": "text/*",
"extensions": [".txt", ".csv", ".md", ".log"],
"label": "Bounded text files"
"mediaType": "application/json",
"extensions": [".json", ".js"],
"label": "Locale JSON or i18next-like translation files"
}
],
"produces": [
{
"mediaType": "text/plain",
"extensions": [".txt"],
"label": "Transformed text"
},
{
"mediaType": "application/json",
"extensions": [".json"],
"label": "Recipe and artifact evidence"
"label": "Reviewed translation bundle"
}
]
},
"capabilities": { "required": [], "optional": ["web-crypto"] },
"capabilities": {
"required": [],
"optional": ["indexeddb", "local-artifact-handoff"]
},
"privacy": {
"processing": "local",
"fileUploads": true,
@@ -59,14 +56,14 @@
"label": "Inputs stay in this browser; nothing is uploaded."
},
"source": {
"repository": "https://git.add-ideas.de/lotobo/text-tools",
"repository": "https://git.add-ideas.de/lotobo/translator-tools",
"license": "GPL-3.0-or-later"
},
"actions": [
{
"id": "source",
"label": "Source",
"url": "https://git.add-ideas.de/lotobo/text-tools"
"url": "https://git.add-ideas.de/lotobo/translator-tools"
}
]
}
+24 -3
View File
@@ -1,10 +1,14 @@
import { lazy, Suspense, useState } from "react";
import { lazy, Suspense, useEffect, useMemo, useRef, useState } from "react";
import { AppShell } from "@add-ideas/toolbox-shell-react";
import "@add-ideas/toolbox-shell-react/styles.css";
import "./styles.css";
import { ErrorBoundary } from "./components/ErrorBoundary";
import { HelpDialog } from "./components/HelpDialog";
import { manifest } from "./toolbox/manifest";
import {
receiveTranslatorTransfer,
type IncomingTranslatorTransfer,
} from "./toolbox/transfer";
const Workbench = lazy(async () => ({
default: (await import("./components/Workbench")).Workbench,
@@ -12,6 +16,23 @@ const Workbench = lazy(async () => ({
export function App() {
const [helpOpen, setHelpOpen] = useState(false);
const [incoming, setIncoming] = useState<IncomingTranslatorTransfer>();
const incomingStarted = useRef(false);
const activeSourceLabel = useMemo(() => "Translator Tools", []);
useEffect(() => {
if (incomingStarted.current) return;
incomingStarted.current = true;
void receiveTranslatorTransfer()
.then((value) => setIncoming(value))
.catch((reason: unknown) =>
setIncoming({
status: "missing",
error: reason instanceof Error ? reason.message : String(reason),
}),
);
}, []);
return (
<ErrorBoundary>
<AppShell
@@ -22,11 +43,11 @@ export function App() {
<Suspense
fallback={
<p className="loading" role="status">
Preparing Text Tools
Preparing {activeSourceLabel}
</p>
}
>
<Workbench />
<Workbench incoming={incoming} />
</Suspense>
</AppShell>
<HelpDialog open={helpOpen} onClose={() => setHelpOpen(false)} />
+1 -1
View File
@@ -15,7 +15,7 @@ export class ErrorBoundary extends Component<
if (this.state.error)
return (
<main className="fatal">
<h1>Text Tools could not continue</h1>
<h1>Translator Tools could not continue</h1>
<p>{this.state.error.message}</p>
<button type="button" onClick={() => location.reload()}>
Reload
+8 -4
View File
@@ -25,16 +25,20 @@ export function HelpDialog({
<div className="dialog-heading">
<div>
<p className="eyebrow">Local-first help</p>
<h2 id="help-title">About Text Tools</h2>
<h2 id="help-title">About Translator Tools</h2>
</div>
<button type="button" onClick={onClose} aria-label="Close help">
×
</button>
</div>
<p>Transform and inspect plain text locally in the browser.</p>
<p>
All processing is performed in this browser. Imported data is treated as
untrusted and bounded before parsing.
Review and translate i18next-style locale JSON files directly in your
browser. No network upload is required.
</p>
<p>
Imported data is parsed with bounded input checks and can include
JavaScript-style objects (single quotes, trailing commas). Glossary
terms are saved to local storage in this browser only.
</p>
</dialog>
);
File diff suppressed because it is too large Load Diff
+247 -246
View File
@@ -1,26 +1,28 @@
:root {
--toolbox-background: #f6f7fb;
--toolbox-background: #f4f7fc;
--toolbox-surface: #fff;
--toolbox-surface-soft: #eff1f7;
--toolbox-text: #202332;
--toolbox-muted: #656b7d;
--toolbox-border: #d9dce7;
--toolbox-accent: #5b4ec4;
--toolbox-accent-hover: #493caf;
--toolbox-accent-soft: #ece9ff;
--toolbox-text: #1f2433;
--toolbox-muted: #62697c;
--toolbox-border: #d9deea;
--toolbox-accent: #4f46e5;
--toolbox-accent-hover: #4437d6;
--toolbox-accent-soft: #eceaff;
--toolbox-accent-contrast: #fff;
--toolbox-focus: #137d75;
--toolbox-focus: #107d74;
--toolbox-danger: #b42342;
}
* {
box-sizing: border-box;
}
html {
min-width: 20rem;
min-height: 100%;
background: var(--toolbox-background);
scrollbar-gutter: stable;
}
body {
min-width: 20rem;
min-height: 100vh;
@@ -29,15 +31,17 @@ body {
color: var(--toolbox-text);
font-family: Inter, ui-sans-serif, system-ui, sans-serif;
}
button,
input,
select,
textarea {
font: inherit;
}
button,
.button {
min-height: 2.55rem;
min-height: 2.45rem;
display: inline-flex;
align-items: center;
justify-content: center;
@@ -50,40 +54,70 @@ button,
font-weight: 720;
cursor: pointer;
}
button:hover:not(:disabled),
.button:hover {
border-color: var(--toolbox-accent);
background: var(--toolbox-surface-soft);
}
button:disabled {
opacity: 0.62;
cursor: not-allowed;
}
.secondary {
background: #f7f7ff;
}
button.primary {
border-color: var(--toolbox-accent);
background: var(--toolbox-accent);
color: var(--toolbox-accent-contrast);
}
button.primary:hover {
background: var(--toolbox-accent-hover);
}
:where(button, input, select, textarea, a):focus-visible {
outline: 3px solid color-mix(in srgb, var(--toolbox-focus) 42%, transparent);
outline-offset: 2px;
}
input,
select,
textarea {
width: 100%;
min-height: 2.55rem;
min-height: 2.45rem;
padding: 0.58rem 0.7rem;
border: 1px solid var(--toolbox-border);
border-radius: 0.62rem;
background: var(--toolbox-surface);
color: var(--toolbox-text);
}
textarea {
min-height: 10rem;
min-height: 9rem;
resize: vertical;
font-family: ui-monospace, SFMono-Regular, Consolas, monospace;
line-height: 1.48;
line-height: 1.4;
}
a {
color: inherit;
}
.toolbox-shell__main {
width: min(100%, 90rem);
padding: clamp(0.75rem, 1.8vw, 1.5rem);
}
.workbench {
display: grid;
gap: 1rem;
}
.hero,
.panel {
border: 1px solid var(--toolbox-border);
@@ -91,296 +125,263 @@ textarea {
background: var(--toolbox-surface);
box-shadow: 0 8px 28px rgb(30 36 70 / 4%);
}
.hero {
display: flex;
justify-content: space-between;
gap: 1rem;
align-items: flex-start;
padding: clamp(1.1rem, 3vw, 2rem);
padding: clamp(1rem, 3vw, 1.5rem);
}
.hero h1,
.panel h2,
.panel h3,
.help-dialog h2,
.fatal h1 {
.help-dialog h2 {
margin: 0;
letter-spacing: -0.025em;
}
.hero p:not(.eyebrow) {
max-width: 52rem;
margin: 0.55rem 0 0;
color: var(--toolbox-muted);
line-height: 1.55;
}
.eyebrow {
margin: 0 0 0.3rem;
color: var(--toolbox-accent);
font-size: 0.69rem;
font-weight: 820;
letter-spacing: 0.115em;
text-transform: uppercase;
}
.privacy-pill {
flex: 0 0 auto;
padding: 0.38rem 0.62rem;
padding: 0.4rem 0.7rem;
border-radius: 999px;
font-size: 0.84rem;
border: 1px solid var(--toolbox-border);
background: var(--toolbox-accent-soft);
color: var(--toolbox-accent);
font-size: 0.75rem;
font-weight: 760;
color: #2f2a6d;
align-self: flex-start;
}
.panel {
padding: 1rem;
}
.panel-heading {
display: flex;
justify-content: space-between;
gap: 1rem;
align-items: end;
margin-bottom: 0.9rem;
align-items: flex-start;
margin-bottom: 0.85rem;
}
.capability-grid {
display: grid;
grid-template-columns: repeat(auto-fit, minmax(min(100%, 13rem), 1fr));
gap: 0.75rem;
.panel-heading h2 {
margin: 0.16rem 0 0;
}
.capability-grid article {
padding: 0.9rem;
border: 1px solid var(--toolbox-border);
border-radius: 0.72rem;
background: var(--toolbox-surface-soft);
.eyebrow {
margin: 0;
color: var(--toolbox-accent);
font-size: 0.67rem;
font-weight: 820;
letter-spacing: 0.115em;
}
.capability-grid p {
margin: 0.4rem 0 0;
color: var(--toolbox-muted);
line-height: 1.48;
}
.workspace-tabs {
.actions {
display: flex;
gap: 0.4rem;
overflow-x: auto;
padding-bottom: 0.2rem;
}
.workspace-tabs button[aria-selected="true"] {
border-color: var(--toolbox-accent);
background: var(--toolbox-accent);
color: var(--toolbox-accent-contrast);
}
.grid {
display: grid;
grid-template-columns: repeat(auto-fit, minmax(min(100%, 20rem), 1fr));
gap: 0.85rem;
flex-wrap: wrap;
gap: 0.5rem;
}
.field {
display: grid;
gap: 0.35rem;
}
.field > span {
font-size: 0.76rem;
font-weight: 750;
}
.muted {
color: var(--toolbox-muted);
}
.result {
padding: 0.8rem;
border: 1px solid var(--toolbox-border);
border-radius: 0.68rem;
background: var(--toolbox-surface-soft);
overflow-wrap: anywhere;
}
.loading,
.fatal {
width: min(100% - 2rem, 60rem);
margin: 2rem auto;
padding: 1rem;
}
.help-dialog {
width: min(36rem, calc(100% - 2rem));
border: 1px solid var(--toolbox-border);
border-radius: 0.9rem;
background: var(--toolbox-surface);
color: var(--toolbox-text);
}
.help-dialog::backdrop {
background: rgb(20 24 45 / 55%);
}
.dialog-heading {
display: flex;
justify-content: space-between;
gap: 1rem;
align-items: start;
}
.workspace {
display: grid;
gap: 0.85rem;
}
.editor-grid {
display: grid;
grid-template-columns: repeat(2, minmax(0, 1fr));
gap: 1rem;
}
.editor-grid textarea {
min-height: 22rem;
}
.encoding-row {
display: grid;
grid-template-columns: minmax(10rem, 16rem) minmax(12rem, 1fr);
gap: 0.7rem;
align-items: end;
}
.actions {
display: flex;
gap: 0.5rem;
flex-wrap: wrap;
}
.primary {
border-color: var(--toolbox-accent);
background: var(--toolbox-accent);
color: var(--toolbox-accent-contrast);
}
.file-button {
position: relative;
overflow: hidden;
}
.file-button input {
position: absolute;
inset: 0;
opacity: 0;
cursor: pointer;
}
.check {
display: flex;
gap: 0.5rem;
.field-inline {
display: inline-flex;
align-items: center;
min-height: 2.55rem;
}
.check input {
width: auto;
min-height: auto;
}
.inventory {
display: grid;
grid-template-columns: repeat(auto-fit, minmax(7rem, 1fr));
gap: 0.45rem;
margin: 0;
}
.inventory div {
padding: 0.55rem;
border-radius: 0.55rem;
background: var(--toolbox-surface-soft);
}
.inventory dt {
color: var(--toolbox-muted);
font-size: 0.66rem;
font-weight: 750;
}
.inventory dd {
margin: 0.15rem 0 0;
}
.steps {
display: grid;
gap: 0.55rem;
margin: 0;
padding: 0;
list-style: none;
}
.steps li {
display: grid;
grid-template-columns: minmax(12rem, 0.8fr) minmax(12rem, 1.4fr) auto;
gap: 0.6rem;
align-items: center;
padding: 0.6rem;
border: 1px solid var(--toolbox-border);
border-radius: 0.65rem;
}
.step-actions {
display: flex;
gap: 0.25rem;
}
.column-options {
display: grid;
grid-template-columns: minmax(8rem, 1fr) minmax(4rem, 0.55fr) minmax(
7rem,
1fr
);
gap: 0.4rem;
}
.step-actions button {
min-width: 2.55rem;
padding: 0.4rem;
.field > span {
font-size: 0.92rem;
color: var(--toolbox-muted);
}
.warning,
.error,
.notice {
margin: 0;
padding: 0.7rem;
.form-grid {
display: grid;
gap: 0.8rem;
grid-template-columns: repeat(auto-fit, minmax(12rem, 1fr));
}
.field-group {
display: flex;
gap: 0.5rem;
}
.field-group input {
flex: 1;
}
.table-wrap {
overflow-x: auto;
border: 1px solid var(--toolbox-border);
border-radius: 0.65rem;
line-height: 1.5;
border-radius: 0.8rem;
}
.warning {
border-color: #d9a72e;
background: #fff8df;
color: #725000;
}
.error {
border-color: var(--toolbox-danger);
color: var(--toolbox-danger);
}
.notice {
background: var(--toolbox-surface-soft);
}
table {
.review-grid {
width: 100%;
border-collapse: collapse;
}
th,
td {
padding: 0.55rem;
.review-grid th,
.review-grid td {
border-bottom: 1px solid var(--toolbox-border);
text-align: left;
border-right: 1px solid var(--toolbox-border);
padding: 0.55rem;
vertical-align: top;
}
th {
.review-grid th:last-child,
.review-grid td:last-child {
border-right: none;
}
.review-grid thead {
background: #f9fbff;
}
.review-grid textarea {
min-height: 6rem;
}
.key-cell {
width: 24ch;
max-width: 24ch;
white-space: nowrap;
overflow: hidden;
text-overflow: ellipsis;
}
.count {
display: inline-flex;
align-items: center;
color: var(--toolbox-muted);
font-size: 0.7rem;
text-transform: uppercase;
}
details {
padding: 0.4rem 0.5rem;
border-radius: 0.5rem;
background: #f8f9fc;
border: 1px solid var(--toolbox-border);
border-radius: 0.65rem;
}
summary {
padding: 0.65rem;
cursor: pointer;
font-weight: 750;
.muted {
color: var(--toolbox-muted);
}
.recipe {
.row-actions {
margin-top: 0.35rem;
display: flex;
flex-wrap: wrap;
gap: 0.35rem;
}
.row-actions .chip,
.chip {
border-radius: 999px;
min-height: auto;
padding: 0.35rem 0.56rem;
}
.chip-list {
display: flex;
flex-wrap: wrap;
gap: 0.35rem;
}
.notice {
margin: 0.75rem 0 0;
color: #1f5a75;
}
.error {
margin: 0.75rem 0 0;
color: var(--toolbox-danger);
font-weight: 700;
}
.help-dialog {
margin: auto;
width: min(50rem, 95vw);
border: 1px solid var(--toolbox-border);
border-radius: 1rem;
padding: 0;
}
.help-dialog .dialog-heading {
display: flex;
justify-content: space-between;
align-items: center;
padding: 1rem;
border-bottom: 1px solid var(--toolbox-border);
}
.help-dialog p {
margin: 0.7rem 1rem;
}
.glossary,
.language-list,
.term-list,
.workspace ul {
list-style: none;
margin: 0;
padding: 0;
}
.glossary-form {
display: grid;
gap: 0.6rem;
padding: 0 0.65rem 0.65rem;
gap: 0.55rem;
margin: 0.5rem 0 0;
grid-template-columns: repeat(auto-fit, minmax(12rem, 1fr));
}
@media (max-width: 62rem) {
.editor-grid {
grid-template-columns: 1fr;
}
.term-list,
.language-list {
margin-top: 0.65rem;
display: grid;
gap: 0.45rem;
}
@media (max-width: 48rem) {
.steps li {
grid-template-columns: 1fr;
}
.column-options {
grid-template-columns: 1fr;
}
.term-list li,
.language-list li {
border: 1px solid var(--toolbox-border);
border-radius: 0.6rem;
background: #fbfcff;
display: flex;
justify-content: space-between;
gap: 0.75rem;
align-items: center;
padding: 0.55rem;
}
@media (max-width: 42rem) {
.hero {
.term-list p {
margin: 0.15rem 0 0;
}
.sr-only {
position: absolute;
width: 1px;
height: 1px;
padding: 0;
margin: -1px;
overflow: hidden;
clip: rect(0, 0, 0, 0);
white-space: nowrap;
border-width: 0;
}
@media (max-width: 720px) {
.panel-heading {
flex-direction: column;
align-items: stretch;
}
.privacy-pill {
order: -1;
}
.encoding-row {
grid-template-columns: minmax(0, 1fr);
.form-grid {
grid-template-columns: 1fr;
}
}
-251
View File
@@ -1,251 +0,0 @@
import {
decodeText,
digestHex,
encodeText,
type TextEncoding,
} from "@add-ideas/toolbox-helpers";
import {
textInventory,
type PipelineResult,
type StepReport,
type TransformStep,
} from "./pipeline";
export interface TextByteEvidence {
readonly schemaVersion: 1;
readonly byteLength: number;
readonly selectedEncoding: TextEncoding;
readonly fatalDecode: boolean;
readonly bom?: { encoding: Exclude<TextEncoding, "latin1">; bytes: number };
readonly utf8: { valid: boolean; firstInvalidOffset?: number };
readonly zeroBytes: {
total: number;
evenOffsets: number;
oddOffsets: number;
};
readonly byteNewlines: { crlf: number; bareLf: number; bareCr: number };
readonly replacementCharacters: number;
readonly warnings: readonly string[];
}
export interface TextArtifactEvidence {
readonly schemaVersion: 1;
readonly artifactType: "de.add-ideas.toolbox.text/v1";
readonly createdBy: { app: "text-tools"; version: "0.2.0" };
readonly source: {
name: string;
canonicalUtf8Sha256: string;
inventory: ReturnType<typeof textInventory>;
byteEvidence?: TextByteEvidence;
};
readonly output: {
name: string;
mediaType: "text/plain";
encoding: TextEncoding;
bytes: number;
sha256: string;
inventory: ReturnType<typeof textInventory>;
};
readonly pipeline: {
steps: readonly TransformStep[];
reports: readonly StepReport[];
warnings: readonly string[];
};
readonly handoff: {
supportedByThisBuild: false;
note: string;
};
}
function bom(bytes: Uint8Array): TextByteEvidence["bom"] {
if (bytes[0] === 0xef && bytes[1] === 0xbb && bytes[2] === 0xbf)
return { encoding: "utf-8", bytes: 3 };
if (bytes[0] === 0xff && bytes[1] === 0xfe)
return { encoding: "utf-16le", bytes: 2 };
if (bytes[0] === 0xfe && bytes[1] === 0xff)
return { encoding: "utf-16be", bytes: 2 };
return undefined;
}
function utf8InvalidOffset(bytes: Uint8Array): number | undefined {
const continuation = (index: number) =>
index < bytes.length && (bytes[index]! & 0xc0) === 0x80;
for (let index = 0; index < bytes.length; index += 1) {
const first = bytes[index]!;
if (first <= 0x7f) continue;
if (first >= 0xc2 && first <= 0xdf) {
if (!continuation(index + 1)) return index;
index += 1;
continue;
}
if (first >= 0xe0 && first <= 0xef) {
const second = bytes[index + 1];
if (
second === undefined ||
(first === 0xe0 && (second < 0xa0 || second > 0xbf)) ||
(first === 0xed && (second < 0x80 || second > 0x9f)) ||
(first !== 0xe0 &&
first !== 0xed &&
(second < 0x80 || second > 0xbf)) ||
!continuation(index + 2)
)
return index;
index += 2;
continue;
}
if (first >= 0xf0 && first <= 0xf4) {
const second = bytes[index + 1];
if (
second === undefined ||
(first === 0xf0 && (second < 0x90 || second > 0xbf)) ||
(first === 0xf4 && (second < 0x80 || second > 0x8f)) ||
(first !== 0xf0 &&
first !== 0xf4 &&
(second < 0x80 || second > 0xbf)) ||
!continuation(index + 2) ||
!continuation(index + 3)
)
return index;
index += 3;
continue;
}
return index;
}
return undefined;
}
function inspectByteNewlines(bytes: Uint8Array) {
let crlf = 0;
let bareLf = 0;
let bareCr = 0;
for (let index = 0; index < bytes.length; index += 1) {
if (bytes[index] === 0x0d && bytes[index + 1] === 0x0a) {
crlf += 1;
index += 1;
} else if (bytes[index] === 0x0a) bareLf += 1;
else if (bytes[index] === 0x0d) bareCr += 1;
}
return { crlf, bareLf, bareCr };
}
export function decodeTextWithEvidence(
bytes: Uint8Array,
selectedEncoding: TextEncoding,
fatalDecode: boolean,
): { text: string; evidence: TextByteEvidence } {
if (bytes.byteLength > 16 * 1024 * 1024)
throw new RangeError("Text byte evidence is limited to 16 MiB.");
const detectedBom = bom(bytes);
const invalidOffset = utf8InvalidOffset(bytes);
let zeroTotal = 0;
let zeroEven = 0;
let zeroOdd = 0;
bytes.forEach((value, index) => {
if (value !== 0) return;
zeroTotal += 1;
if (index % 2) zeroOdd += 1;
else zeroEven += 1;
});
const text = decodeText(bytes, selectedEncoding, fatalDecode);
const warnings: string[] = [];
if (detectedBom && detectedBom.encoding !== selectedEncoding)
warnings.push(
`The ${detectedBom.encoding.toUpperCase()} BOM conflicts with the selected ${selectedEncoding.toUpperCase()} decoder.`,
);
if (!detectedBom)
warnings.push(
"No byte-order mark is present; the selected encoding is an explicit user choice, not a detection claim.",
);
if (selectedEncoding.startsWith("utf-16") && bytes.length % 2)
warnings.push("UTF-16 input has an odd trailing byte.");
if (selectedEncoding !== "utf-8" && invalidOffset === undefined)
warnings.push(
"The same bytes are also well-formed UTF-8; encoding intent cannot be inferred from validity alone.",
);
const replacementCharacters = [...text].filter(
(character) => character === "\uFFFD",
).length;
if (replacementCharacters)
warnings.push(
`${replacementCharacters} replacement character(s) appear in decoded text; they may be source data or decoder substitutions.`,
);
return {
text,
evidence: Object.freeze({
schemaVersion: 1,
byteLength: bytes.byteLength,
selectedEncoding,
fatalDecode,
...(detectedBom ? { bom: detectedBom } : {}),
utf8: {
valid: invalidOffset === undefined,
...(invalidOffset === undefined
? {}
: { firstInvalidOffset: invalidOffset }),
},
zeroBytes: {
total: zeroTotal,
evenOffsets: zeroEven,
oddOffsets: zeroOdd,
},
byteNewlines: inspectByteNewlines(bytes),
replacementCharacters,
warnings: Object.freeze(warnings),
}),
};
}
export async function createTextArtifactEvidence(input: {
sourceName: string;
sourceEvidence?: TextByteEvidence;
sourceText: string;
outputName: string;
outputEncoding: TextEncoding;
outputBytes: Uint8Array;
pipeline: PipelineResult;
steps: readonly TransformStep[];
}): Promise<TextArtifactEvidence> {
if (input.outputBytes.byteLength > 32 * 1024 * 1024)
throw new RangeError("Artifact output exceeds the 32 MiB evidence limit.");
const canonicalSource = encodeText(input.sourceText, "utf-8");
return Object.freeze({
schemaVersion: 1,
artifactType: "de.add-ideas.toolbox.text/v1",
createdBy: {
app: "text-tools" as const,
version: "0.2.0" as const,
},
source: {
name: input.sourceName,
canonicalUtf8Sha256: await digestHex(
canonicalSource,
"SHA-256",
32 * 1024 * 1024,
),
inventory: textInventory(input.sourceText),
...(input.sourceEvidence ? { byteEvidence: input.sourceEvidence } : {}),
},
output: {
name: input.outputName,
mediaType: "text/plain" as const,
encoding: input.outputEncoding,
bytes: input.outputBytes.byteLength,
sha256: await digestHex(input.outputBytes, "SHA-256", 32 * 1024 * 1024),
inventory: textInventory(input.pipeline.output),
},
pipeline: {
steps: Object.freeze(
input.steps.map((step) => Object.freeze({ ...step })),
),
reports: Object.freeze(
input.pipeline.steps.map((report) => Object.freeze({ ...report })),
),
warnings: Object.freeze([...input.pipeline.warnings]),
},
handoff: {
supportedByThisBuild: false as const,
note: "The portable files are ready for explicit local transfer. SDK 0.3.0 provides the shared contract, but Open With remains disabled until the coordinated Portal consumer rollout.",
},
});
}
-515
View File
@@ -1,515 +0,0 @@
import {
base64ToBytes,
bytesToBase64,
bytesToHex,
convertLineEndings,
decodeText,
encodeText,
hexToBytes,
normalizeUnicode,
parseCsv,
stringifyCsv,
transformCase,
type CaseTransform,
type LineEnding,
} from "@add-ideas/toolbox-helpers";
export type StepType =
| "line-endings"
| "trim-lines"
| "trim-document"
| "collapse-whitespace"
| "sort-lines"
| "dedupe-lines"
| "case"
| "normalize"
| "transliterate"
| "escape"
| "unescape"
| "wrap"
| "columns"
| "replace-literal"
| "prefix-lines"
| "suffix-lines"
| "filter-lines"
| "number-lines"
| "join-lines"
| "reverse-lines";
export interface TransformStep {
id: string;
type: StepType;
enabled: boolean;
option: string;
}
export interface StepReport {
id: string;
type: StepType;
beforeUnits: number;
afterUnits: number;
changed: boolean;
warning?: string;
}
export interface PipelineResult {
output: string;
steps: StepReport[];
warnings: string[];
}
const MAX_INPUT = 2_000_000;
const MAX_OUTPUT = 32 * 1024 * 1024;
let nextStepId = 0;
function lines(value: string): string[] {
return value.replaceAll("\r\n", "\n").replaceAll("\r", "\n").split("\n");
}
function htmlEscape(value: string): string {
return value
.replaceAll("&", "&amp;")
.replaceAll("<", "&lt;")
.replaceAll(">", "&gt;")
.replaceAll('"', "&quot;")
.replaceAll("'", "&#39;");
}
function stableUnique(values: string[]): string[] {
const seen = new Set<string>();
return values.filter((value) => !seen.has(value) && !!seen.add(value));
}
function wrapText(value: string, width: number): string {
if (!Number.isSafeInteger(width) || width < 1 || width > 10_000)
throw new Error("Wrap width must be 110,000 code points.");
return lines(value)
.flatMap((line) => {
const words = line.split(/\s+/u);
const output: string[] = [];
let current = "";
for (const word of words) {
if (!word) continue;
if (!current) current = word;
else if ([...current, " ", ...word].length <= width)
current += ` ${word}`;
else {
output.push(current);
current = word;
}
}
output.push(current);
return output;
})
.join("\n");
}
function transformColumns(
value: string,
option: string,
): {
value: string;
warning?: string;
} {
const pieces = option.split("|");
const mode = pieces.length >= 3 ? pieces[0] : "literal";
const delimiterInput = pieces.length >= 3 ? pieces[1] : pieces[0];
const order = pieces.length >= 3 ? pieces.slice(2).join("|") : pieces[1];
const delimiter = delimiterInput === "\\t" ? "\t" : delimiterInput || ",";
if (!delimiter || delimiter.length > 8)
throw new Error("Column delimiter must contain 18 characters.");
if (mode !== "literal" && mode !== "csv")
throw new Error("Column parsing mode must be csv or literal.");
if (mode === "csv" && delimiter.length !== 1)
throw new Error("Quoted CSV mode requires a one-character delimiter.");
const indices = (order || "1")
.split(",")
.map((entry) => Number(entry.trim()) - 1);
if (
!indices.length ||
indices.some(
(index) => !Number.isSafeInteger(index) || index < 0 || index > 999,
)
)
throw new Error("Column order uses 1-based indices such as 3,1,2.");
if (mode === "literal")
return {
value: lines(value)
.map((line) => {
const cells = line.split(delimiter);
return indices.map((index) => cells[index] ?? "").join(delimiter);
})
.join("\n"),
};
const rows = parseCsv(value, {
delimiter,
maxRows: 200_000,
maxColumns: 1_000,
maxFieldChars: 2_000_000,
});
return {
value: stringifyCsv(
rows.map((row) => indices.map((index) => row[index] ?? "")),
{
delimiter,
maxRows: 200_000,
maxColumns: 1_000,
maxFieldChars: 2_000_000,
},
),
warning:
"Quoted CSV rows were parsed across embedded delimiters/newlines and serialized canonically with CRLF row endings.",
};
}
function strictHtmlUnescape(value: string): string {
const entities: Record<string, string> = {
"&amp;": "&",
"&lt;": "<",
"&gt;": ">",
"&quot;": '"',
"&#39;": "'",
};
let output = "";
for (let index = 0; index < value.length; index += 1) {
if (value[index] !== "&") {
output += value[index];
continue;
}
const end = value.indexOf(";", index + 1);
if (end < 0 || end - index > 6)
throw new SyntaxError(
`HTML entity at UTF-16 index ${index} is malformed.`,
);
const entity = value.slice(index, end + 1);
const decoded = entities[entity];
if (decoded === undefined)
throw new SyntaxError(
`HTML entity ${entity} is not emitted by the matching escape stage.`,
);
output += decoded;
index = end;
}
return output;
}
function strictJsonUnescape(value: string): string {
const parsed = JSON.parse(`"${value}"`) as unknown;
if (typeof parsed !== "string")
throw new SyntaxError("JSON escape input is invalid.");
return parsed;
}
function unescapeValue(value: string, option: string): string {
if (option === "json") return strictJsonUnescape(value);
if (option === "html") return strictHtmlUnescape(value);
if (option === "url") return decodeURIComponent(value);
if (option === "base64")
return decodeText(
base64ToBytes(value, { maxOutputBytes: MAX_OUTPUT }),
"utf-8",
true,
);
if (option === "hex")
return decodeText(
hexToBytes(value, {
maxOutputBytes: MAX_OUTPUT,
allowWhitespace: false,
allowPrefix: false,
}),
"utf-8",
true,
);
throw new Error("Unsupported unescape source.");
}
function transliterate(value: string): string {
return value
.normalize("NFKD")
.replaceAll(/\p{Mark}+/gu, "")
.replaceAll("ß", "ss")
.replaceAll("Æ", "AE")
.replaceAll("æ", "ae")
.replaceAll("Ø", "O")
.replaceAll("ø", "o")
.replaceAll("Ł", "L")
.replaceAll("ł", "l");
}
function parseReplacement(option: string): readonly [string, string] {
let parsed: unknown;
try {
parsed = JSON.parse(option);
} catch {
throw new SyntaxError("Literal replacement options are invalid.");
}
if (
!Array.isArray(parsed) ||
parsed.length !== 2 ||
parsed.some((value) => typeof value !== "string")
)
throw new SyntaxError(
"Literal replacement requires [search, replacement].",
);
const [search, replacement] = parsed as [string, string];
if (!search) throw new SyntaxError("Literal search text must not be empty.");
if (search.length > 100_000 || replacement.length > 1_000_000)
throw new RangeError(
"Literal replacement option exceeds its safety limit.",
);
return [search, replacement];
}
function decodedDelimiter(option: string): string {
if (option.length > 1_000)
throw new RangeError("Line join delimiter exceeds 1,000 characters.");
return option.replaceAll("\\n", "\n").replaceAll("\\t", "\t");
}
function assertProjectedOutput(units: number): void {
if (!Number.isSafeInteger(units) || units > MAX_OUTPUT)
throw new RangeError(
"Transformation would exceed the 32 MiB output limit.",
);
}
function replaceLiteral(
value: string,
search: string,
replacement: string,
): string {
let matches = 0;
let offset = 0;
while ((offset = value.indexOf(search, offset)) >= 0) {
matches += 1;
offset += search.length;
assertProjectedOutput(
value.length + matches * (replacement.length - search.length),
);
}
return value.replaceAll(search, replacement);
}
function decorateLines(
value: string,
addition: string,
side: "prefix" | "suffix",
): string {
if (addition.length > 100_000)
throw new RangeError("Line decoration exceeds 100,000 UTF-16 units.");
const values = lines(value);
assertProjectedOutput(value.length + values.length * addition.length);
return values
.map((line) => (side === "prefix" ? addition + line : line + addition))
.join("\n");
}
function applyStep(
value: string,
step: TransformStep,
): { value: string; warning?: string } {
switch (step.type) {
case "line-endings":
return { value: convertLineEndings(value, step.option as LineEnding) };
case "trim-lines":
return {
value: lines(value)
.map((line) => line.trim())
.join("\n"),
};
case "trim-document":
return { value: value.trim() };
case "collapse-whitespace":
return {
value:
step.option === "all"
? value.replaceAll(/\s+/gu, " ")
: lines(value)
.map((line) => line.replaceAll(/[\t ]+/gu, " "))
.join("\n"),
};
case "sort-lines": {
const locale = step.option || "en";
return {
value: lines(value)
.map((line, index) => ({ line, index }))
.sort(
(left, right) =>
left.line.localeCompare(right.line, locale, {
numeric: true,
sensitivity: "variant",
}) || left.index - right.index,
)
.map(({ line }) => line)
.join("\n"),
warning: `Sort order uses the host Intl.Collator for locale ${locale}.`,
};
}
case "dedupe-lines":
return { value: stableUnique(lines(value)).join("\n") };
case "case":
return { value: transformCase(value, step.option as CaseTransform) };
case "normalize":
return {
value: normalizeUnicode(
value,
step.option as "NFC" | "NFD" | "NFKC" | "NFKD",
),
};
case "transliterate":
return {
value: transliterate(value),
warning:
"Best-effort Latin transliteration is lossy and incomplete; it is not language-aware.",
};
case "escape": {
if (step.option === "json")
return { value: JSON.stringify(value).slice(1, -1) };
if (step.option === "html") return { value: htmlEscape(value) };
if (step.option === "url") return { value: encodeURIComponent(value) };
if (step.option === "base64")
return { value: bytesToBase64(encodeText(value)) };
if (step.option === "hex")
return { value: bytesToHex(encodeText(value)) };
throw new Error("Unsupported escape target.");
}
case "unescape":
return {
value: unescapeValue(value, step.option),
warning:
step.option === "base64" || step.option === "hex"
? "Decoded bytes are required to be well-formed UTF-8 text; arbitrary binary is rejected."
: undefined,
};
case "wrap":
return { value: wrapText(value, Number(step.option)) };
case "columns":
return transformColumns(value, step.option);
case "replace-literal": {
const [search, replacement] = parseReplacement(step.option);
return { value: replaceLiteral(value, search, replacement) };
}
case "prefix-lines":
return { value: decorateLines(value, step.option, "prefix") };
case "suffix-lines":
return { value: decorateLines(value, step.option, "suffix") };
case "filter-lines": {
if (!step.option || step.option.length > 100_000)
throw new SyntaxError("Line filter text must not be empty.");
return {
value: lines(value)
.filter((line) => line.includes(step.option))
.join("\n"),
warning: "Line filtering removes every line without the literal text.",
};
}
case "number-lines": {
const start = Number(step.option || "1");
if (!Number.isSafeInteger(start) || Math.abs(start) > 1_000_000_000)
throw new RangeError("Line-number start must be a bounded integer.");
return {
value: lines(value)
.map((line, index) => `${start + index}. ${line}`)
.join("\n"),
};
}
case "join-lines": {
const values = lines(value);
const delimiter = decodedDelimiter(step.option);
assertProjectedOutput(
values.reduce((total, line) => total + line.length, 0) +
Math.max(0, values.length - 1) * delimiter.length,
);
return { value: values.join(delimiter) };
}
case "reverse-lines":
return { value: lines(value).reverse().join("\n") };
}
}
export function applyPipeline(
input: string,
steps: readonly TransformStep[],
): PipelineResult {
if (input.length > MAX_INPUT)
throw new Error(
`Input exceeds ${MAX_INPUT.toLocaleString()} UTF-16 units.`,
);
let output = input;
const reports: StepReport[] = [];
const warnings: string[] = [];
for (const step of steps) {
if (!step.enabled) continue;
const before = output;
const result = applyStep(before, step);
output = result.value;
if (
output.length > MAX_OUTPUT ||
output.length > Math.max(1024, input.length * 8)
)
throw new Error(
"Pipeline output exceeded its 8× / 32 MiB expansion limit.",
);
reports.push({
id: step.id,
type: step.type,
beforeUnits: before.length,
afterUnits: output.length,
changed: before !== output,
warning: result.warning,
});
if (result.warning) warnings.push(result.warning);
}
return { output, steps: reports, warnings: [...new Set(warnings)] };
}
export function createStep(type: StepType): TransformStep {
const defaults: Record<StepType, string> = {
"line-endings": "lf",
"trim-lines": "",
"trim-document": "",
"collapse-whitespace": "line",
"sort-lines": "en",
"dedupe-lines": "",
case: "lower",
normalize: "NFC",
transliterate: "",
escape: "json",
unescape: "json",
wrap: "80",
columns: "csv|,|1",
"replace-literal": JSON.stringify(["old", "new"]),
"prefix-lines": "> ",
"suffix-lines": "",
"filter-lines": "text",
"number-lines": "1",
"join-lines": " ",
"reverse-lines": "",
};
return {
id: `step-${++nextStepId}`,
type,
enabled: true,
option: defaults[type],
};
}
export function textInventory(value: string) {
let codePoints = 0;
for (let index = 0; index < value.length; index += 1) {
const unit = value.charCodeAt(index);
if (
unit >= 0xd800 &&
unit <= 0xdbff &&
index + 1 < value.length &&
value.charCodeAt(index + 1) >= 0xdc00 &&
value.charCodeAt(index + 1) <= 0xdfff
)
index += 1;
codePoints += 1;
}
return {
utf16Units: value.length,
codePoints,
lines: value ? value.split(/\r\n|\r|\n/u).length : 0,
crlf: (value.match(/\r\n/gu) ?? []).length,
bareLf: (value.match(/(?<!\r)\n/gu) ?? []).length,
bareCr: (value.match(/\r(?!\n)/gu) ?? []).length,
finalNewline: /(?:\r\n|\r|\n)$/u.test(value),
};
}
+23 -26
View File
@@ -1,22 +1,21 @@
{
"$schema": "https://git.add-ideas.de/lotobo/toolbox-sdk/raw/branch/main/schemas/toolbox-app.v1.schema.json",
"schemaVersion": 1,
"id": "de.add-ideas.text-tools",
"name": "Text Tools",
"version": "0.2.0",
"description": "Compose text transforms and export encoding evidence locally.",
"id": "de.add-ideas.translator-tools",
"name": "Translator Tools",
"version": "0.1.0",
"description": "Review and edit translation bundles in the browser with glossary suggestions.",
"entry": "./",
"icon": "./favicon.svg",
"categories": ["text", "developer", "productivity"],
"categories": ["text", "developer", "productivity", "i18n"],
"tags": [
"text",
"unicode",
"pipeline",
"normalize",
"sort",
"escape",
"encoding",
"newline"
"translation",
"i18n",
"locale",
"glossary",
"review",
"json",
"localization"
],
"integration": {
"contextVersion": 1,
@@ -26,32 +25,30 @@
"requirements": {
"secureContext": false,
"workers": false,
"indexedDb": false,
"indexedDb": true,
"crossOriginIsolated": false,
"topLevelContext": false
},
"io": {
"accepts": [
{
"mediaType": "text/*",
"extensions": [".txt", ".csv", ".md", ".log"],
"label": "Bounded text files"
"mediaType": "application/json",
"extensions": [".json", ".js"],
"label": "Locale JSON or i18next-like translation files"
}
],
"produces": [
{
"mediaType": "text/plain",
"extensions": [".txt"],
"label": "Transformed text"
},
{
"mediaType": "application/json",
"extensions": [".json"],
"label": "Recipe and artifact evidence"
"label": "Reviewed translation bundle"
}
]
},
"capabilities": { "required": [], "optional": ["web-crypto"] },
"capabilities": {
"required": [],
"optional": ["indexeddb", "local-artifact-handoff"]
},
"privacy": {
"processing": "local",
"fileUploads": true,
@@ -59,14 +56,14 @@
"label": "Inputs stay in this browser; nothing is uploaded."
},
"source": {
"repository": "https://git.add-ideas.de/lotobo/text-tools",
"repository": "https://git.add-ideas.de/lotobo/translator-tools",
"license": "GPL-3.0-or-later"
},
"actions": [
{
"id": "source",
"label": "Source",
"url": "https://git.add-ideas.de/lotobo/text-tools"
"url": "https://git.add-ideas.de/lotobo/translator-tools"
}
]
}
+146
View File
@@ -0,0 +1,146 @@
import {
TOOLBOX_TRANSFER_QUERY_PARAMETER,
consumeToolboxTransfer,
createToolboxTransfer,
createToolboxTransferUrl,
loadToolboxContext,
readToolboxTransferToken,
type ToolboxTransfer,
} from "@add-ideas/toolbox-contract";
import { APP_VERSION } from "../version";
export const TRANS_TOOL_ID = "de.add-ideas.translator-tools";
export const PORTAL_APP_ID = "de.add-ideas.toolbox-portal";
export const MAX_TRANSLATOR_TRANSFER_BYTES = 5 * 1024 * 1024;
interface ReceiveDependencies {
consume: typeof consumeToolboxTransfer;
readToken: typeof readToolboxTransferToken;
replaceUrl: (url: string) => void;
}
const receiveDefaults: ReceiveDependencies = {
consume: consumeToolboxTransfer,
readToken: readToolboxTransferToken,
replaceUrl: (url) =>
window.history.replaceState(window.history.state, "", url),
};
export type IncomingTranslatorTransfer =
| { status: "none" | "missing"; error?: string }
| { status: "ready"; file: File; transfer: ToolboxTransfer };
export async function receiveTranslatorTransfer(
href: string | URL = window.location.href,
dependencies: ReceiveDependencies = receiveDefaults,
): Promise<IncomingTranslatorTransfer> {
const url = new URL(href);
if (!url.searchParams.has(TOOLBOX_TRANSFER_QUERY_PARAMETER))
return { status: "none" };
let token: string | undefined;
try {
token = dependencies.readToken(url);
} finally {
url.searchParams.delete(TOOLBOX_TRANSFER_QUERY_PARAMETER);
dependencies.replaceUrl(`${url.pathname}${url.search}${url.hash}`);
}
if (!token) return { status: "missing" };
const transfer = await dependencies.consume(token, TRANS_TOOL_ID);
if (!transfer) return { status: "missing" };
if (transfer.files.length !== 1)
throw new RangeError(
"Translator Tools accepts exactly one file per handoff.",
);
const source = transfer.files[0];
if (!source) throw new RangeError("The handoff did not contain a file.");
if (!source.size || source.size > MAX_TRANSLATOR_TRANSFER_BYTES)
throw new RangeError(
"Transferred translation files must be between 1 byte and 5 MiB.",
);
if (source.size !== source.blob.size)
throw new RangeError("Transferred size evidence does not match the file.");
const mediaType = source.mediaType.split(";", 1)[0]!.toLowerCase();
if (
!mediaType.startsWith("application/json") &&
!/\.(?:json|js|txt)$/iu.test(source.name)
) {
throw new TypeError(
"The transferred file is not a supported locale/translation document.",
);
}
return {
status: "ready",
file: new File([source.blob], source.name, {
type: source.mediaType || source.blob.type,
}),
transfer,
};
}
interface SendDependencies {
createTransfer: typeof createToolboxTransfer;
loadContext: typeof loadToolboxContext;
}
const sendDefaults: SendDependencies = {
createTransfer: createToolboxTransfer,
loadContext: loadToolboxContext,
};
export async function prepareTranslatorPortalHandoff(
blob: Blob,
name: string,
dependencies: SendDependencies = sendDefaults,
): Promise<URL> {
if (!blob.size || blob.size > MAX_TRANSLATOR_TRANSFER_BYTES)
throw new RangeError(
"Translator exports must be between 1 byte and 5 MiB.",
);
const context = await dependencies.loadContext();
if (context.status !== "ready")
throw new Error(
"Open Translator Tools from the Toolbox Portal to send this output locally.",
);
const transfer = await dependencies.createTransfer({
source: { appId: TRANS_TOOL_ID, appVersion: APP_VERSION },
targetAppId: PORTAL_APP_ID,
files: [
{
blob,
name,
mediaType: blob.type || "application/json",
size: blob.size,
},
],
evidence: {
formatVersion: "1",
operation: "translator-workspace-output",
settings: {
preferredDestinationAppIds: [
"de.add-ideas.regex-tools",
"de.add-ideas.data-tools",
],
},
warnings: [
"Translator output uses your current local edits; review authoritative and target pairs before downstream use.",
],
},
});
return createToolboxTransferUrl(
context.context.catalog.homeUrl,
transfer.token,
{ location: context.context.catalog.homeUrl },
);
}
+1 -1
View File
@@ -1 +1 @@
export const APP_VERSION = "0.2.0";
export const APP_VERSION = "0.1.0";
-91
View File
@@ -1,91 +0,0 @@
import { expect, test, type Page } from "@playwright/test";
import { Buffer } from "node:buffer";
const ORIGIN = "http://127.0.0.1:4173";
async function localOnly(page: Page) {
const external: string[] = [];
await page.route("**/*", async (route) => {
const url = new URL(route.request().url());
if (url.origin !== ORIGIN) {
external.push(url.href);
await route.abort();
} else await route.continue();
});
return external;
}
test("runs from a nested path without external requests", async ({ page }) => {
const errors: string[] = [];
page.on("pageerror", (error) => errors.push(error.message));
page.on("console", (message) => {
if (message.type() === "error") errors.push(message.text());
});
const external = await localOnly(page);
await page.goto("/deep/nested/text/");
await expect(page.getByRole("heading", { name: "Text Tools" })).toBeVisible();
expect(external).toEqual([]);
expect(errors).toEqual([]);
});
test("applies the visible text pipeline and preserves its last output", async ({
page,
}) => {
const external = await localOnly(page);
await page.goto("/deep/nested/text/");
await page.getByLabel("Text source").fill(" Alpha \r\nAlpha\n");
await page.getByRole("button", { name: "Apply pipeline" }).click();
await expect(page.getByLabel("Transformed output")).toHaveValue("Alpha\n");
await expect(
page.getByRole("cell", { name: "Trim every line" }),
).toBeVisible();
await expect(
page.getByRole("cell", { name: "Deduplicate lines" }),
).toBeVisible();
expect(external).toEqual([]);
});
test("shows byte evidence and composes an added literal transform", async ({
page,
}) => {
await page.goto("/deep/nested/text/");
await page.getByLabel("Decode as").selectOption("utf-8");
await page.locator('input[type="file"]').setInputFiles({
name: "bom.txt",
mimeType: "text/plain",
buffer: Buffer.from([0xef, 0xbb, 0xbf, ...Buffer.from("old\r\nold")]),
});
await page.getByText("Byte decoding evidence").click();
await expect(page.getByText("utf-8 / utf-8")).toBeVisible();
await expect(
page
.locator("details")
.filter({ hasText: "Byte decoding evidence" })
.getByText("1 / 0 / 0", { exact: true }),
).toBeVisible();
await page.getByLabel("New transformation").selectOption("replace-literal");
await page.getByRole("button", { name: "Add step" }).click();
await page.getByLabel("Literal search text").fill("old");
await page.getByLabel("Literal replacement text").fill("new");
await page.getByRole("button", { name: "Apply pipeline" }).click();
await expect(page.getByLabel("Transformed output")).toHaveValue("new");
await expect(
page.getByRole("button", { name: "Export artifact + evidence" }),
).toBeVisible();
});
test("serves the release identity and hardened headers", async ({
request,
}) => {
const index = await request.get("/deep/nested/text/");
expect(index.ok()).toBe(true);
expect(index.headers()["content-security-policy"]).toContain(
"default-src 'self'",
);
expect(await index.text()).not.toMatch(/\b(?:src|href)=["']\//u);
const manifest = await request.get("/deep/nested/text/toolbox-app.json");
await expect(manifest.json()).resolves.toMatchObject({
id: "de.add-ideas.text-tools",
version: "0.2.0",
entry: "./",
});
});
-18
View File
@@ -1,18 +0,0 @@
import { expect, test } from "@playwright/test";
test("keeps the primary workspace inside a narrow viewport", async ({
page,
}) => {
await page.goto("/deep/nested/text/");
await expect(page.locator("main").first()).toBeVisible();
await expect(
page.locator("main .loading, main .workbench-loading"),
).toHaveCount(0);
const widths = await page.evaluate(() => ({
content: document.documentElement.scrollWidth,
viewport: document.documentElement.clientWidth,
}));
expect(widths.viewport).toBeLessThanOrEqual(430);
expect(widths.content).toBeLessThanOrEqual(widths.viewport + 1);
});
-17
View File
@@ -1,17 +0,0 @@
import { render, screen } from "@testing-library/react";
import { describe, expect, it, vi } from "vitest";
import { App } from "../../src/App";
describe("Text Tools", () => {
it("renders the local workbench and standard shell", async () => {
vi.stubGlobal(
"fetch",
vi.fn(async () => new Response("Not found", { status: 404 })),
);
render(<App />);
expect(
await screen.findByRole("heading", { name: "Text Tools" }),
).toBeVisible();
expect(await screen.findByText("Browser-local")).toBeVisible();
});
});
+13
View File
@@ -0,0 +1,13 @@
import { describe, expect, it } from "vitest";
import { manifest } from "../src/toolbox/manifest";
import { APP_VERSION } from "../src/version";
describe("translator-tools smoke", () => {
it("exposes a coherent toolbox manifest", () => {
expect(manifest.id).toBe("de.add-ideas.translator-tools");
expect(manifest.name).toBe("Translator Tools");
expect(manifest.version).toBe(APP_VERSION);
expect(manifest.categories).toContain("text");
expect(manifest.entry).toBe("./");
});
});
-147
View File
@@ -1,147 +0,0 @@
import { describe, expect, it } from "vitest";
import {
applyPipeline,
textInventory,
type TransformStep,
} from "../../src/text/pipeline";
import {
createTextArtifactEvidence,
decodeTextWithEvidence,
} from "../../src/text/evidence";
const step = (type: TransformStep["type"], option = ""): TransformStep => ({
id: type,
type,
option,
enabled: true,
});
describe("text pipelines", () => {
it("applies transformations in the declared order", () =>
expect(
applyPipeline(" b \nA\na", [
step("trim-lines"),
step("dedupe-lines"),
step("sort-lines", "en"),
]).output,
).toBe("a\nA\nb"));
it("keeps disabled steps out of the report", () =>
expect(
applyPipeline("A", [{ ...step("case", "lower"), enabled: false }]),
).toEqual({ output: "A", steps: [], warnings: [] }));
it("escapes HTML as data", () =>
expect(
applyPipeline("<img src=x onerror=1>", [step("escape", "html")]).output,
).toBe("&lt;img src=x onerror=1&gt;"));
it("strictly reverses each escape/encoding stage", () => {
const source = 'café <x> & "quoted"\n';
for (const option of ["json", "html", "url", "base64", "hex"])
expect(
applyPipeline(applyPipeline(source, [step("escape", option)]).output, [
step("unescape", option),
]).output,
).toBe(source);
expect(() => applyPipeline("&copy;", [step("unescape", "html")])).toThrow(
/not emitted/u,
);
expect(() => applyPipeline("w6k", [step("unescape", "base64")])).toThrow(
/canonical|padding|UTF-8/u,
);
expect(() => applyPipeline("%ZZ", [step("unescape", "url")])).toThrow();
});
it("selects columns from quoted CSV records including embedded newlines", () => {
const result = applyPipeline(
'name,note,id\r\nAlice,"hello, world",1\r\nBob,"two\nlines",2\r\n',
[step("columns", "csv|,|3,1,2")],
);
expect(result.output).toContain('1,Alice,"hello, world"');
expect(result.output).toContain('2,Bob,"two\nlines"');
expect(result.warnings[0]).toMatch(/Quoted CSV/u);
});
it("retains multi-character literal-delimiter mode", () =>
expect(
applyPipeline("left::right\na::b", [step("columns", "literal|::|2,1")])
.output,
).toBe("right::left\nb::a"));
it("reports lossy transliteration", () =>
expect(applyPipeline("Crème", [step("transliterate")]).warnings[0]).toMatch(
/lossy/u,
));
it("tracks exact line endings", () =>
expect(textInventory("a\r\nb\nc\r")).toMatchObject({
crlf: 1,
bareLf: 1,
bareCr: 1,
finalNewline: true,
}));
it("composes literal replacement, filtering, numbering and line decoration", () => {
const result = applyPipeline("old apple\nold pear\nbanana", [
step("replace-literal", JSON.stringify(["old", "fresh"])),
step("filter-lines", "fresh"),
step("prefix-lines", "- "),
step("number-lines", "10"),
step("suffix-lines", "!"),
step("join-lines", " | "),
]);
expect(result.output).toBe("10. - fresh apple! | 11. - fresh pear!");
expect(result.warnings).toEqual(
expect.arrayContaining([expect.stringMatching(/filtering removes/u)]),
);
});
it("reverses lines without reversing grapheme contents", () =>
expect(applyPipeline("😀a\nb", [step("reverse-lines")]).output).toBe(
"b\n😀a",
));
it("preflights expanding transforms before allocating their output", () => {
expect(() =>
applyPipeline("x".repeat(40), [
step("replace-literal", JSON.stringify(["x", "y".repeat(1_000_000)])),
]),
).toThrow(/32 MiB/u);
expect(() =>
applyPipeline("a\nb", [step("prefix-lines", "x".repeat(100_001))]),
).toThrow(/100,000/u);
});
});
describe("byte and portable artifact evidence", () => {
it("records BOM conflicts, UTF-8 validity and byte newline forms", () => {
const bytes = Uint8Array.of(0xef, 0xbb, 0xbf, 0x61, 0x0d, 0x0a, 0x62);
const decoded = decodeTextWithEvidence(bytes, "latin1", true);
expect(decoded.evidence).toMatchObject({
byteLength: 7,
bom: { encoding: "utf-8", bytes: 3 },
utf8: { valid: true },
byteNewlines: { crlf: 1, bareLf: 0, bareCr: 0 },
});
expect(decoded.evidence.warnings.join(" ")).toMatch(/conflicts/u);
const malformed = decodeTextWithEvidence(
Uint8Array.of(0xc3, 0x28),
"latin1",
true,
);
expect(malformed.evidence.utf8).toEqual({
valid: false,
firstInvalidOffset: 0,
});
});
it("emits exact output hashes and an honest handoff marker", async () => {
const steps = [step("case", "upper")];
const pipeline = applyPipeline("hello", steps);
const bytes = new TextEncoder().encode(pipeline.output);
const evidence = await createTextArtifactEvidence({
sourceName: "input.txt",
sourceText: "hello",
outputName: "output.txt",
outputEncoding: "utf-8",
outputBytes: bytes,
pipeline,
steps,
});
expect(evidence.output.sha256).toMatch(/^[a-f0-9]{64}$/u);
expect(evidence.handoff.supportedByThisBuild).toBe(false);
expect(evidence.pipeline.steps[0]).toMatchObject({ type: "case" });
});
});