feat(translator-tools): add locale review and glossary workflow
This commit is contained in:
+6
-8
@@ -1,13 +1,11 @@
|
|||||||
# Changelog
|
# Changelog
|
||||||
|
|
||||||
## 0.2.0 - 2026-09-02
|
## Unreleased
|
||||||
|
|
||||||
- Added composable literal and line transforms with allocation preflights, plus BOM, UTF-8 validity, zero-byte and byte-newline evidence.
|
- Added local translator workspace for i18next-like and single-locale JSON review.
|
||||||
- Added paired text/artifact-evidence downloads with source/output hashes and
|
- Added language pair selection, glossary storage and fuzzy in-context suggestions.
|
||||||
pipeline provenance; automatic Toolbox handoff remains intentionally disabled
|
- Added local-only import/export pipeline with portal handoff integration.
|
||||||
pending the coordinated Portal consumer rollout.
|
|
||||||
|
|
||||||
## 0.1.0 - 2026-09-01
|
## 0.1.0 - 2026-09-02
|
||||||
|
|
||||||
- Added the initial local-first Text Tools workbench.
|
- Initial local-first translator review implementation.
|
||||||
- Added Toolbox integration, bounded input handling, offline shell and deterministic release packaging.
|
|
||||||
|
|||||||
@@ -1,24 +1,25 @@
|
|||||||
# Text Tools
|
# Translator Tools
|
||||||
|
|
||||||
Compose, inspect and evidence plain-text transformations locally in the browser.
|
Translator Tools is a standalone local-first application in the
|
||||||
|
[add·ideas Toolbox](https://git.add-ideas.de/lotobo/toolbox-portal). Inputs are
|
||||||
|
parsed and edited in the browser only; no locale strings are uploaded.
|
||||||
|
|
||||||
Text Tools is a standalone local-first application in the [add·ideas Toolbox](https://git.add-ideas.de/lotobo/toolbox-portal). Inputs are processed in the browser and are not uploaded.
|
## Quick scope
|
||||||
|
|
||||||
## Version 0.2 scope
|
- Import locale files (`.json`, `.js`, `.txt`) in i18next-like or single-locale forms.
|
||||||
|
- Review authoritative language against one target language at a time.
|
||||||
|
- Edit source and target strings and copy-source quickly.
|
||||||
|
- Add translation glossary entries per language pair and receive fuzzy suggestions.
|
||||||
|
- Add new languages and export a reviewer-updated translation bundle locally.
|
||||||
|
- Send exports through the same-tool transfer channel to the Portal.
|
||||||
|
|
||||||
- Ordered pipelines for line endings, trimming, whitespace, stable sorting, deduplication, literal replacement, line prefix/suffix/filter/number/join and line-order reversal
|
## Notes
|
||||||
- Locale-aware case conversion, Unicode NFC/NFD/NFKC/NFKD and best-effort Latin transliteration
|
|
||||||
- JSON/HTML/URL/Base64/hex escaping plus strict inverse decode/unescape stages; binary decoders accept canonical encodings and require valid UTF-8 text
|
|
||||||
- Bounded RFC-style quoted CSV column selection across embedded delimiters and newlines, alongside the retained literal multi-character delimiter-per-line mode
|
|
||||||
- Explicit UTF-8, UTF-16LE and Latin-1 file decoding plus BOM, malformed UTF-8 offset, zero-byte parity, byte-level newline and replacement-character evidence
|
|
||||||
- Versioned JSON recipe import/export, per-step size/change reports, and a paired text-plus-JSON artifact download with source/output SHA-256 and transformation provenance
|
|
||||||
|
|
||||||
Pipelines are limited to 2,000,000 UTF-16 input units and an 8×/32 MiB output expansion. File input is limited to 16 MiB before decoding. Encoding is selected explicitly rather than guessed; compatibility normalisation, transliteration, narrow encodings and several transforms can be lossy, so source, result, and warnings remain visible. See [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) and [docs/PRIVACY-SECURITY.md](docs/PRIVACY-SECURITY.md).
|
- The parser accepts JavaScript-style locale assignment forms such as
|
||||||
|
`const de = { ... }` and tolerant spacing/commas.
|
||||||
The artifact evidence is a portable download contract, not an automatic Toolbox
|
- Parsing and transfers are bounded to 5 MiB per imported/exported transfer file.
|
||||||
handoff. SDK 0.3.0 provides the shared transfer contract, but automatic Open With
|
- Language detection relies on file name hints (`de_*.json`) and explicit user
|
||||||
remains disabled in this release until the Portal consumer rollout is
|
selection.
|
||||||
coordinated. Local downloads stay explicit and portable in the meantime.
|
|
||||||
|
|
||||||
## Development
|
## Development
|
||||||
|
|
||||||
@@ -32,7 +33,7 @@ npm run test:browser
|
|||||||
|
|
||||||
## Release
|
## Release
|
||||||
|
|
||||||
`npm run release:artifact` creates a deterministic `release/text-tools-0.2.0.zip` and checksum sidecar.
|
`npm run release:artifact` creates a deterministic `release/translator-tools-0.1.0.zip` and checksum sidecar.
|
||||||
|
|
||||||
## Licence
|
## Licence
|
||||||
|
|
||||||
|
|||||||
@@ -1,6 +1,6 @@
|
|||||||
# Third-party notices
|
# Third-party notices
|
||||||
|
|
||||||
Text Tools 0.2.0 directly depends on these runtime packages:
|
Translator Tools 0.1.0 directly depends on these runtime packages:
|
||||||
|
|
||||||
| Package | Pinned version | Declared licence |
|
| Package | Pinned version | Declared licence |
|
||||||
| -------------------------------- | -------------: | ---------------- |
|
| -------------------------------- | -------------: | ---------------- |
|
||||||
|
|||||||
+11
-4
@@ -1,9 +1,16 @@
|
|||||||
# Architecture
|
# Architecture
|
||||||
|
|
||||||
Text Tools is a static React/Vite application wrapped in the shared Toolbox shell. `text/pipeline.ts` defines an ordered, typed list of transformations. `applyPipeline` starts from the exact source, applies enabled steps one at a time, checks expansion after every step, and returns the last successful output with per-step before/after counts and warnings.
|
Translator Tools is a static React/Vite application wrapped in the shared Toolbox shell.
|
||||||
|
|
||||||
Text encoding/decoding, line-ending conversion, bounded CSV parsing and Unicode-aware case primitives come from `@add-ideas/toolbox-helpers`. File decoding uses an explicit encoding and fatal-or-replacement choice; it does not guess a charset. Inverse JSON/HTML/URL/Base64/hex stages are strict: malformed syntax and non-UTF-8 byte output fail the step rather than being silently repaired. Column selection offers a quoted CSV mode that preserves embedded delimiters/newlines and a separate literal-delimiter-per-line mode. Recipe JSON carries a schema/app identity, accepts at most 100 supported steps, discards supplied IDs and assigns local monotonic IDs.
|
The workbench parses a single imported locale file in-browser, normalizes either:
|
||||||
|
|
||||||
`text/evidence.ts` inspects original bytes before decoding, including BOM, exact UTF-8 validity/first invalid offset, zero-byte parity and CR/LF/CRLF counts. Artifact export emits the selected output bytes and a deterministic evidence document containing canonical-source and exact-output SHA-256 values, inventories, recipe and step reports. The document records that direct Toolbox handoff is unavailable in this release; the SDK 0.3.0 transfer contract is intentionally not activated until the Portal consumer rollout is coordinated.
|
- i18next-like language maps (`{ "en": {...}, "de": {...} }`)
|
||||||
|
- single-locale objects (`{ "key": "value" }`, or JS-style assignment forms like `const de = {...}`).
|
||||||
|
|
||||||
Version 0.2 operations are synchronous and bounded, so the app creates no worker and has no persistence or server API. Relative entry and asset URLs keep the build relocatable below a nested portal path.
|
Strings are flattened into stable dot paths for review/editing and can be serialized
|
||||||
|
back with an inverse unflattening step. Suggestions come from local glossary entries
|
||||||
|
and are matched by language pair and fuzzy string distance.
|
||||||
|
|
||||||
|
No background workers are required today. Persistent state is limited to a local
|
||||||
|
Storage glossary cache; everything else remains in process memory until download or
|
||||||
|
transfer.
|
||||||
|
|||||||
@@ -1,9 +1,16 @@
|
|||||||
# Privacy and security
|
# Privacy and security
|
||||||
|
|
||||||
Source text, imported files, recipes and results stay in page memory. There is no telemetry, analytics, account, persistence or runtime network path. Imported text is rendered as text/preformatted content, not executable HTML; HTML escaping produces a string and does not preview it as markup.
|
Locale files and generated outputs stay in-browser and are not uploaded.
|
||||||
|
|
||||||
Files are rejected above 16 MiB before reading. Decoded/pasted pipeline input is limited to 2,000,000 UTF-16 units, recipe text to 1,000,000 units/100 steps, and output to both 8× the original (with a small-input floor) and 32 MiB. Quoted CSV selection additionally limits rows, columns and field size. Strict inverse stages reject malformed escapes/encodings and invalid UTF-8; they do not attempt permissive repair. These limits reduce accidental expansion but do not prove that output is safe for a downstream interpreter.
|
No accounts, telemetry or runtime network requests are used for the core workflow.
|
||||||
|
The app stores glossary terms in local browser storage so they are available on the same
|
||||||
|
device for repeated reviews.
|
||||||
|
|
||||||
Compatibility normalisation, case conversion, transliteration, whitespace/line transforms, column selection and narrow encodings can change or lose information. Locale sorting depends on the browser's `Intl.Collator`. The exact source, output, per-step counts and warnings remain visible so users can review those changes before copying or downloading.
|
Parser/serializer limits are bounded to prevent accidental expansion:
|
||||||
|
|
||||||
Byte evidence does not guess an encoding or prove the selected decoder is correct. A valid UTF-8 byte sequence may represent text intended for another encoding, zero-byte parity is only a UTF-16 review signal, and replacement characters may already have existed in the source. Artifact evidence hashes canonical UTF-8 source text and exact exported bytes, clearly labels both, and includes filenames and transformation details that may themselves be sensitive.
|
- Imports and exports are capped at 5 MiB each.
|
||||||
|
- Large objects are flattened and rendered with per-session bounds.
|
||||||
|
|
||||||
|
Suggested translations are purely local matching; they do not call external services.
|
||||||
|
Transfers to the Portal are signed local handoffs only and never stored by a remote
|
||||||
|
processor by default.
|
||||||
|
|||||||
Generated
+486
-71
File diff suppressed because it is too large
Load Diff
+7
-6
@@ -1,16 +1,16 @@
|
|||||||
{
|
{
|
||||||
"name": "text-tools",
|
"name": "translator-tools",
|
||||||
"version": "0.2.0",
|
"version": "0.1.0",
|
||||||
"description": "Transform and inspect plain text locally in the browser.",
|
"description": "Review and translate i18next-like JSON language files locally.",
|
||||||
"license": "GPL-3.0-or-later",
|
"license": "GPL-3.0-or-later",
|
||||||
"author": "Albrecht Degering",
|
"author": "Albrecht Degering",
|
||||||
"repository": {
|
"repository": {
|
||||||
"type": "git",
|
"type": "git",
|
||||||
"url": "git+https://git.add-ideas.de/lotobo/text-tools.git"
|
"url": "git+https://git.add-ideas.de/lotobo/translator-tools.git"
|
||||||
},
|
},
|
||||||
"homepage": "https://git.add-ideas.de/lotobo/text-tools",
|
"homepage": "https://git.add-ideas.de/lotobo/translator-tools",
|
||||||
"bugs": {
|
"bugs": {
|
||||||
"url": "https://git.add-ideas.de/lotobo/text-tools/issues"
|
"url": "https://git.add-ideas.de/lotobo/translator-tools/issues"
|
||||||
},
|
},
|
||||||
"private": true,
|
"private": true,
|
||||||
"type": "module",
|
"type": "module",
|
||||||
@@ -42,6 +42,7 @@
|
|||||||
"@add-ideas/toolbox-contract": "0.3.0",
|
"@add-ideas/toolbox-contract": "0.3.0",
|
||||||
"@add-ideas/toolbox-helpers": "0.2.0",
|
"@add-ideas/toolbox-helpers": "0.2.0",
|
||||||
"@add-ideas/toolbox-shell-react": "0.3.0",
|
"@add-ideas/toolbox-shell-react": "0.3.0",
|
||||||
|
"json5": "^2.2.3",
|
||||||
"react": "19.2.8",
|
"react": "19.2.8",
|
||||||
"react-dom": "19.2.8"
|
"react-dom": "19.2.8"
|
||||||
},
|
},
|
||||||
|
|||||||
+6
-8
@@ -1,13 +1,11 @@
|
|||||||
# Changelog
|
# Changelog
|
||||||
|
|
||||||
## 0.2.0 - 2026-09-02
|
## Unreleased
|
||||||
|
|
||||||
- Added composable literal and line transforms with allocation preflights, plus BOM, UTF-8 validity, zero-byte and byte-newline evidence.
|
- Added local translator workspace for i18next-like and single-locale JSON review.
|
||||||
- Added paired text/artifact-evidence downloads with source/output hashes and
|
- Added language pair selection, glossary storage and fuzzy in-context suggestions.
|
||||||
pipeline provenance; automatic Toolbox handoff remains intentionally disabled
|
- Added local-only import/export pipeline with portal handoff integration.
|
||||||
pending the coordinated Portal consumer rollout.
|
|
||||||
|
|
||||||
## 0.1.0 - 2026-09-01
|
## 0.1.0 - 2026-09-02
|
||||||
|
|
||||||
- Added the initial local-first Text Tools workbench.
|
- Initial local-first translator review implementation.
|
||||||
- Added Toolbox integration, bounded input handling, offline shell and deterministic release packaging.
|
|
||||||
|
|||||||
@@ -1077,6 +1077,36 @@ Declared licence: Apache-2.0
|
|||||||
limitations under the License.
|
limitations under the License.
|
||||||
|
|
||||||
|
|
||||||
|
==============================================================================
|
||||||
|
json5@2.2.3
|
||||||
|
Declared licence: MIT
|
||||||
|
==============================================================================
|
||||||
|
--- LICENSE.md ---
|
||||||
|
MIT License
|
||||||
|
|
||||||
|
Copyright (c) 2012-2018 Aseem Kishore, and [others].
|
||||||
|
|
||||||
|
Permission is hereby granted, free of charge, to any person obtaining a copy
|
||||||
|
of this software and associated documentation files (the "Software"), to deal
|
||||||
|
in the Software without restriction, including without limitation the rights
|
||||||
|
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
||||||
|
copies of the Software, and to permit persons to whom the Software is
|
||||||
|
furnished to do so, subject to the following conditions:
|
||||||
|
|
||||||
|
The above copyright notice and this permission notice shall be included in all
|
||||||
|
copies or substantial portions of the Software.
|
||||||
|
|
||||||
|
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||||
|
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
||||||
|
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
||||||
|
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
||||||
|
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
||||||
|
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
||||||
|
SOFTWARE.
|
||||||
|
|
||||||
|
[others]: https://github.com/json5/json5/contributors
|
||||||
|
|
||||||
|
|
||||||
==============================================================================
|
==============================================================================
|
||||||
react@19.2.8
|
react@19.2.8
|
||||||
Declared licence: MIT
|
Declared licence: MIT
|
||||||
|
|||||||
+18
-17
@@ -1,24 +1,25 @@
|
|||||||
# Text Tools
|
# Translator Tools
|
||||||
|
|
||||||
Compose, inspect and evidence plain-text transformations locally in the browser.
|
Translator Tools is a standalone local-first application in the
|
||||||
|
[add·ideas Toolbox](https://git.add-ideas.de/lotobo/toolbox-portal). Inputs are
|
||||||
|
parsed and edited in the browser only; no locale strings are uploaded.
|
||||||
|
|
||||||
Text Tools is a standalone local-first application in the [add·ideas Toolbox](https://git.add-ideas.de/lotobo/toolbox-portal). Inputs are processed in the browser and are not uploaded.
|
## Quick scope
|
||||||
|
|
||||||
## Version 0.2 scope
|
- Import locale files (`.json`, `.js`, `.txt`) in i18next-like or single-locale forms.
|
||||||
|
- Review authoritative language against one target language at a time.
|
||||||
|
- Edit source and target strings and copy-source quickly.
|
||||||
|
- Add translation glossary entries per language pair and receive fuzzy suggestions.
|
||||||
|
- Add new languages and export a reviewer-updated translation bundle locally.
|
||||||
|
- Send exports through the same-tool transfer channel to the Portal.
|
||||||
|
|
||||||
- Ordered pipelines for line endings, trimming, whitespace, stable sorting, deduplication, literal replacement, line prefix/suffix/filter/number/join and line-order reversal
|
## Notes
|
||||||
- Locale-aware case conversion, Unicode NFC/NFD/NFKC/NFKD and best-effort Latin transliteration
|
|
||||||
- JSON/HTML/URL/Base64/hex escaping plus strict inverse decode/unescape stages; binary decoders accept canonical encodings and require valid UTF-8 text
|
|
||||||
- Bounded RFC-style quoted CSV column selection across embedded delimiters and newlines, alongside the retained literal multi-character delimiter-per-line mode
|
|
||||||
- Explicit UTF-8, UTF-16LE and Latin-1 file decoding plus BOM, malformed UTF-8 offset, zero-byte parity, byte-level newline and replacement-character evidence
|
|
||||||
- Versioned JSON recipe import/export, per-step size/change reports, and a paired text-plus-JSON artifact download with source/output SHA-256 and transformation provenance
|
|
||||||
|
|
||||||
Pipelines are limited to 2,000,000 UTF-16 input units and an 8×/32 MiB output expansion. File input is limited to 16 MiB before decoding. Encoding is selected explicitly rather than guessed; compatibility normalisation, transliteration, narrow encodings and several transforms can be lossy, so source, result, and warnings remain visible. See [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) and [docs/PRIVACY-SECURITY.md](docs/PRIVACY-SECURITY.md).
|
- The parser accepts JavaScript-style locale assignment forms such as
|
||||||
|
`const de = { ... }` and tolerant spacing/commas.
|
||||||
The artifact evidence is a portable download contract, not an automatic Toolbox
|
- Parsing and transfers are bounded to 5 MiB per imported/exported transfer file.
|
||||||
handoff. SDK 0.3.0 provides the shared transfer contract, but automatic Open With
|
- Language detection relies on file name hints (`de_*.json`) and explicit user
|
||||||
remains disabled in this release until the Portal consumer rollout is
|
selection.
|
||||||
coordinated. Local downloads stay explicit and portable in the meantime.
|
|
||||||
|
|
||||||
## Development
|
## Development
|
||||||
|
|
||||||
@@ -32,7 +33,7 @@ npm run test:browser
|
|||||||
|
|
||||||
## Release
|
## Release
|
||||||
|
|
||||||
`npm run release:artifact` creates a deterministic `release/text-tools-0.2.0.zip` and checksum sidecar.
|
`npm run release:artifact` creates a deterministic `release/translator-tools-0.1.0.zip` and checksum sidecar.
|
||||||
|
|
||||||
## Licence
|
## Licence
|
||||||
|
|
||||||
|
|||||||
@@ -1,6 +1,6 @@
|
|||||||
# Third-party notices
|
# Third-party notices
|
||||||
|
|
||||||
Text Tools 0.2.0 directly depends on these runtime packages:
|
Translator Tools 0.1.0 directly depends on these runtime packages:
|
||||||
|
|
||||||
| Package | Pinned version | Declared licence |
|
| Package | Pinned version | Declared licence |
|
||||||
| -------------------------------- | -------------: | ---------------- |
|
| -------------------------------- | -------------: | ---------------- |
|
||||||
|
|||||||
@@ -1,9 +1,16 @@
|
|||||||
# Architecture
|
# Architecture
|
||||||
|
|
||||||
Text Tools is a static React/Vite application wrapped in the shared Toolbox shell. `text/pipeline.ts` defines an ordered, typed list of transformations. `applyPipeline` starts from the exact source, applies enabled steps one at a time, checks expansion after every step, and returns the last successful output with per-step before/after counts and warnings.
|
Translator Tools is a static React/Vite application wrapped in the shared Toolbox shell.
|
||||||
|
|
||||||
Text encoding/decoding, line-ending conversion, bounded CSV parsing and Unicode-aware case primitives come from `@add-ideas/toolbox-helpers`. File decoding uses an explicit encoding and fatal-or-replacement choice; it does not guess a charset. Inverse JSON/HTML/URL/Base64/hex stages are strict: malformed syntax and non-UTF-8 byte output fail the step rather than being silently repaired. Column selection offers a quoted CSV mode that preserves embedded delimiters/newlines and a separate literal-delimiter-per-line mode. Recipe JSON carries a schema/app identity, accepts at most 100 supported steps, discards supplied IDs and assigns local monotonic IDs.
|
The workbench parses a single imported locale file in-browser, normalizes either:
|
||||||
|
|
||||||
`text/evidence.ts` inspects original bytes before decoding, including BOM, exact UTF-8 validity/first invalid offset, zero-byte parity and CR/LF/CRLF counts. Artifact export emits the selected output bytes and a deterministic evidence document containing canonical-source and exact-output SHA-256 values, inventories, recipe and step reports. The document records that direct Toolbox handoff is unavailable in this release; the SDK 0.3.0 transfer contract is intentionally not activated until the Portal consumer rollout is coordinated.
|
- i18next-like language maps (`{ "en": {...}, "de": {...} }`)
|
||||||
|
- single-locale objects (`{ "key": "value" }`, or JS-style assignment forms like `const de = {...}`).
|
||||||
|
|
||||||
Version 0.2 operations are synchronous and bounded, so the app creates no worker and has no persistence or server API. Relative entry and asset URLs keep the build relocatable below a nested portal path.
|
Strings are flattened into stable dot paths for review/editing and can be serialized
|
||||||
|
back with an inverse unflattening step. Suggestions come from local glossary entries
|
||||||
|
and are matched by language pair and fuzzy string distance.
|
||||||
|
|
||||||
|
No background workers are required today. Persistent state is limited to a local
|
||||||
|
Storage glossary cache; everything else remains in process memory until download or
|
||||||
|
transfer.
|
||||||
|
|||||||
@@ -1,9 +1,16 @@
|
|||||||
# Privacy and security
|
# Privacy and security
|
||||||
|
|
||||||
Source text, imported files, recipes and results stay in page memory. There is no telemetry, analytics, account, persistence or runtime network path. Imported text is rendered as text/preformatted content, not executable HTML; HTML escaping produces a string and does not preview it as markup.
|
Locale files and generated outputs stay in-browser and are not uploaded.
|
||||||
|
|
||||||
Files are rejected above 16 MiB before reading. Decoded/pasted pipeline input is limited to 2,000,000 UTF-16 units, recipe text to 1,000,000 units/100 steps, and output to both 8× the original (with a small-input floor) and 32 MiB. Quoted CSV selection additionally limits rows, columns and field size. Strict inverse stages reject malformed escapes/encodings and invalid UTF-8; they do not attempt permissive repair. These limits reduce accidental expansion but do not prove that output is safe for a downstream interpreter.
|
No accounts, telemetry or runtime network requests are used for the core workflow.
|
||||||
|
The app stores glossary terms in local browser storage so they are available on the same
|
||||||
|
device for repeated reviews.
|
||||||
|
|
||||||
Compatibility normalisation, case conversion, transliteration, whitespace/line transforms, column selection and narrow encodings can change or lose information. Locale sorting depends on the browser's `Intl.Collator`. The exact source, output, per-step counts and warnings remain visible so users can review those changes before copying or downloading.
|
Parser/serializer limits are bounded to prevent accidental expansion:
|
||||||
|
|
||||||
Byte evidence does not guess an encoding or prove the selected decoder is correct. A valid UTF-8 byte sequence may represent text intended for another encoding, zero-byte parity is only a UTF-16 review signal, and replacement characters may already have existed in the source. Artifact evidence hashes canonical UTF-8 source text and exact exported bytes, clearly labels both, and includes filenames and transformation details that may themselves be sensitive.
|
- Imports and exports are capped at 5 MiB each.
|
||||||
|
- Large objects are flattened and rendered with per-session bounds.
|
||||||
|
|
||||||
|
Suggested translations are purely local matching; they do not call external services.
|
||||||
|
Transfers to the Portal are signed local handoffs only and never stored by a remote
|
||||||
|
processor by default.
|
||||||
|
|||||||
+1
-1
@@ -1 +1 @@
|
|||||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 64 64" role="img" aria-label="Text Tools"><path fill="#5b4ec4" d="M11 4h30l12 12v44H11z"/><path fill="#fff" opacity=".28" d="M41 4v14h12"/><text x="32" y="43" fill="#fff" font-family="system-ui,sans-serif" font-size="18" font-weight="800" text-anchor="middle">TE</text></svg>
|
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 64 64" role="img" aria-label="Translator Tools"><path fill="#5b4ec4" d="M11 4h30l12 12v44H11z"/><path fill="#fff" opacity=".28" d="M41 4v14h12"/><text x="32" y="43" fill="#fff" font-family="system-ui,sans-serif" font-size="18" font-weight="800" text-anchor="middle">TR</text></svg>
|
||||||
|
|||||||
|
Before Width: | Height: | Size: 329 B After Width: | Height: | Size: 335 B |
@@ -1,7 +1,7 @@
|
|||||||
{
|
{
|
||||||
"name": "Text Tools",
|
"name": "Translator Tools",
|
||||||
"short_name": "Text",
|
"short_name": "Translator",
|
||||||
"description": "Transform and inspect plain text locally in the browser.",
|
"description": "Review and edit locale strings locally in the browser.",
|
||||||
"start_url": "./",
|
"start_url": "./",
|
||||||
"scope": "./",
|
"scope": "./",
|
||||||
"display": "standalone",
|
"display": "standalone",
|
||||||
|
|||||||
+2
-2
@@ -1,5 +1,5 @@
|
|||||||
const CACHE_PREFIX = "text-tools-shell-";
|
const CACHE_PREFIX = "translator-tools-shell-";
|
||||||
const CACHE_NAME = CACHE_PREFIX + "0.2.0";
|
const CACHE_NAME = CACHE_PREFIX + "0.1.0";
|
||||||
const CORE = ["./", "./manifest.webmanifest", "./favicon.svg"];
|
const CORE = ["./", "./manifest.webmanifest", "./favicon.svg"];
|
||||||
self.addEventListener("install", (event) => {
|
self.addEventListener("install", (event) => {
|
||||||
event.waitUntil(
|
event.waitUntil(
|
||||||
|
|||||||
+23
-26
@@ -1,22 +1,21 @@
|
|||||||
{
|
{
|
||||||
"$schema": "https://git.add-ideas.de/lotobo/toolbox-sdk/raw/branch/main/schemas/toolbox-app.v1.schema.json",
|
"$schema": "https://git.add-ideas.de/lotobo/toolbox-sdk/raw/branch/main/schemas/toolbox-app.v1.schema.json",
|
||||||
"schemaVersion": 1,
|
"schemaVersion": 1,
|
||||||
"id": "de.add-ideas.text-tools",
|
"id": "de.add-ideas.translator-tools",
|
||||||
"name": "Text Tools",
|
"name": "Translator Tools",
|
||||||
"version": "0.2.0",
|
"version": "0.1.0",
|
||||||
"description": "Compose text transforms and export encoding evidence locally.",
|
"description": "Review and edit translation bundles in the browser with glossary suggestions.",
|
||||||
"entry": "./",
|
"entry": "./",
|
||||||
"icon": "./favicon.svg",
|
"icon": "./favicon.svg",
|
||||||
"categories": ["text", "developer", "productivity"],
|
"categories": ["text", "developer", "productivity", "i18n"],
|
||||||
"tags": [
|
"tags": [
|
||||||
"text",
|
"translation",
|
||||||
"unicode",
|
"i18n",
|
||||||
"pipeline",
|
"locale",
|
||||||
"normalize",
|
"glossary",
|
||||||
"sort",
|
"review",
|
||||||
"escape",
|
"json",
|
||||||
"encoding",
|
"localization"
|
||||||
"newline"
|
|
||||||
],
|
],
|
||||||
"integration": {
|
"integration": {
|
||||||
"contextVersion": 1,
|
"contextVersion": 1,
|
||||||
@@ -26,32 +25,30 @@
|
|||||||
"requirements": {
|
"requirements": {
|
||||||
"secureContext": false,
|
"secureContext": false,
|
||||||
"workers": false,
|
"workers": false,
|
||||||
"indexedDb": false,
|
"indexedDb": true,
|
||||||
"crossOriginIsolated": false,
|
"crossOriginIsolated": false,
|
||||||
"topLevelContext": false
|
"topLevelContext": false
|
||||||
},
|
},
|
||||||
"io": {
|
"io": {
|
||||||
"accepts": [
|
"accepts": [
|
||||||
{
|
{
|
||||||
"mediaType": "text/*",
|
"mediaType": "application/json",
|
||||||
"extensions": [".txt", ".csv", ".md", ".log"],
|
"extensions": [".json", ".js"],
|
||||||
"label": "Bounded text files"
|
"label": "Locale JSON or i18next-like translation files"
|
||||||
}
|
}
|
||||||
],
|
],
|
||||||
"produces": [
|
"produces": [
|
||||||
{
|
|
||||||
"mediaType": "text/plain",
|
|
||||||
"extensions": [".txt"],
|
|
||||||
"label": "Transformed text"
|
|
||||||
},
|
|
||||||
{
|
{
|
||||||
"mediaType": "application/json",
|
"mediaType": "application/json",
|
||||||
"extensions": [".json"],
|
"extensions": [".json"],
|
||||||
"label": "Recipe and artifact evidence"
|
"label": "Reviewed translation bundle"
|
||||||
}
|
}
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
"capabilities": { "required": [], "optional": ["web-crypto"] },
|
"capabilities": {
|
||||||
|
"required": [],
|
||||||
|
"optional": ["indexeddb", "local-artifact-handoff"]
|
||||||
|
},
|
||||||
"privacy": {
|
"privacy": {
|
||||||
"processing": "local",
|
"processing": "local",
|
||||||
"fileUploads": true,
|
"fileUploads": true,
|
||||||
@@ -59,14 +56,14 @@
|
|||||||
"label": "Inputs stay in this browser; nothing is uploaded."
|
"label": "Inputs stay in this browser; nothing is uploaded."
|
||||||
},
|
},
|
||||||
"source": {
|
"source": {
|
||||||
"repository": "https://git.add-ideas.de/lotobo/text-tools",
|
"repository": "https://git.add-ideas.de/lotobo/translator-tools",
|
||||||
"license": "GPL-3.0-or-later"
|
"license": "GPL-3.0-or-later"
|
||||||
},
|
},
|
||||||
"actions": [
|
"actions": [
|
||||||
{
|
{
|
||||||
"id": "source",
|
"id": "source",
|
||||||
"label": "Source",
|
"label": "Source",
|
||||||
"url": "https://git.add-ideas.de/lotobo/text-tools"
|
"url": "https://git.add-ideas.de/lotobo/translator-tools"
|
||||||
}
|
}
|
||||||
]
|
]
|
||||||
}
|
}
|
||||||
|
|||||||
+24
-3
@@ -1,10 +1,14 @@
|
|||||||
import { lazy, Suspense, useState } from "react";
|
import { lazy, Suspense, useEffect, useMemo, useRef, useState } from "react";
|
||||||
import { AppShell } from "@add-ideas/toolbox-shell-react";
|
import { AppShell } from "@add-ideas/toolbox-shell-react";
|
||||||
import "@add-ideas/toolbox-shell-react/styles.css";
|
import "@add-ideas/toolbox-shell-react/styles.css";
|
||||||
import "./styles.css";
|
import "./styles.css";
|
||||||
import { ErrorBoundary } from "./components/ErrorBoundary";
|
import { ErrorBoundary } from "./components/ErrorBoundary";
|
||||||
import { HelpDialog } from "./components/HelpDialog";
|
import { HelpDialog } from "./components/HelpDialog";
|
||||||
import { manifest } from "./toolbox/manifest";
|
import { manifest } from "./toolbox/manifest";
|
||||||
|
import {
|
||||||
|
receiveTranslatorTransfer,
|
||||||
|
type IncomingTranslatorTransfer,
|
||||||
|
} from "./toolbox/transfer";
|
||||||
|
|
||||||
const Workbench = lazy(async () => ({
|
const Workbench = lazy(async () => ({
|
||||||
default: (await import("./components/Workbench")).Workbench,
|
default: (await import("./components/Workbench")).Workbench,
|
||||||
@@ -12,6 +16,23 @@ const Workbench = lazy(async () => ({
|
|||||||
|
|
||||||
export function App() {
|
export function App() {
|
||||||
const [helpOpen, setHelpOpen] = useState(false);
|
const [helpOpen, setHelpOpen] = useState(false);
|
||||||
|
const [incoming, setIncoming] = useState<IncomingTranslatorTransfer>();
|
||||||
|
const incomingStarted = useRef(false);
|
||||||
|
const activeSourceLabel = useMemo(() => "Translator Tools", []);
|
||||||
|
|
||||||
|
useEffect(() => {
|
||||||
|
if (incomingStarted.current) return;
|
||||||
|
incomingStarted.current = true;
|
||||||
|
void receiveTranslatorTransfer()
|
||||||
|
.then((value) => setIncoming(value))
|
||||||
|
.catch((reason: unknown) =>
|
||||||
|
setIncoming({
|
||||||
|
status: "missing",
|
||||||
|
error: reason instanceof Error ? reason.message : String(reason),
|
||||||
|
}),
|
||||||
|
);
|
||||||
|
}, []);
|
||||||
|
|
||||||
return (
|
return (
|
||||||
<ErrorBoundary>
|
<ErrorBoundary>
|
||||||
<AppShell
|
<AppShell
|
||||||
@@ -22,11 +43,11 @@ export function App() {
|
|||||||
<Suspense
|
<Suspense
|
||||||
fallback={
|
fallback={
|
||||||
<p className="loading" role="status">
|
<p className="loading" role="status">
|
||||||
Preparing Text Tools…
|
Preparing {activeSourceLabel}…
|
||||||
</p>
|
</p>
|
||||||
}
|
}
|
||||||
>
|
>
|
||||||
<Workbench />
|
<Workbench incoming={incoming} />
|
||||||
</Suspense>
|
</Suspense>
|
||||||
</AppShell>
|
</AppShell>
|
||||||
<HelpDialog open={helpOpen} onClose={() => setHelpOpen(false)} />
|
<HelpDialog open={helpOpen} onClose={() => setHelpOpen(false)} />
|
||||||
|
|||||||
@@ -15,7 +15,7 @@ export class ErrorBoundary extends Component<
|
|||||||
if (this.state.error)
|
if (this.state.error)
|
||||||
return (
|
return (
|
||||||
<main className="fatal">
|
<main className="fatal">
|
||||||
<h1>Text Tools could not continue</h1>
|
<h1>Translator Tools could not continue</h1>
|
||||||
<p>{this.state.error.message}</p>
|
<p>{this.state.error.message}</p>
|
||||||
<button type="button" onClick={() => location.reload()}>
|
<button type="button" onClick={() => location.reload()}>
|
||||||
Reload
|
Reload
|
||||||
|
|||||||
@@ -25,16 +25,20 @@ export function HelpDialog({
|
|||||||
<div className="dialog-heading">
|
<div className="dialog-heading">
|
||||||
<div>
|
<div>
|
||||||
<p className="eyebrow">Local-first help</p>
|
<p className="eyebrow">Local-first help</p>
|
||||||
<h2 id="help-title">About Text Tools</h2>
|
<h2 id="help-title">About Translator Tools</h2>
|
||||||
</div>
|
</div>
|
||||||
<button type="button" onClick={onClose} aria-label="Close help">
|
<button type="button" onClick={onClose} aria-label="Close help">
|
||||||
×
|
×
|
||||||
</button>
|
</button>
|
||||||
</div>
|
</div>
|
||||||
<p>Transform and inspect plain text locally in the browser.</p>
|
|
||||||
<p>
|
<p>
|
||||||
All processing is performed in this browser. Imported data is treated as
|
Review and translate i18next-style locale JSON files directly in your
|
||||||
untrusted and bounded before parsing.
|
browser. No network upload is required.
|
||||||
|
</p>
|
||||||
|
<p>
|
||||||
|
Imported data is parsed with bounded input checks and can include
|
||||||
|
JavaScript-style objects (single quotes, trailing commas). Glossary
|
||||||
|
terms are saved to local storage in this browser only.
|
||||||
</p>
|
</p>
|
||||||
</dialog>
|
</dialog>
|
||||||
);
|
);
|
||||||
|
|||||||
+950
-670
File diff suppressed because it is too large
Load Diff
+245
-244
@@ -1,26 +1,28 @@
|
|||||||
:root {
|
:root {
|
||||||
--toolbox-background: #f6f7fb;
|
--toolbox-background: #f4f7fc;
|
||||||
--toolbox-surface: #fff;
|
--toolbox-surface: #fff;
|
||||||
--toolbox-surface-soft: #eff1f7;
|
--toolbox-surface-soft: #eff1f7;
|
||||||
--toolbox-text: #202332;
|
--toolbox-text: #1f2433;
|
||||||
--toolbox-muted: #656b7d;
|
--toolbox-muted: #62697c;
|
||||||
--toolbox-border: #d9dce7;
|
--toolbox-border: #d9deea;
|
||||||
--toolbox-accent: #5b4ec4;
|
--toolbox-accent: #4f46e5;
|
||||||
--toolbox-accent-hover: #493caf;
|
--toolbox-accent-hover: #4437d6;
|
||||||
--toolbox-accent-soft: #ece9ff;
|
--toolbox-accent-soft: #eceaff;
|
||||||
--toolbox-accent-contrast: #fff;
|
--toolbox-accent-contrast: #fff;
|
||||||
--toolbox-focus: #137d75;
|
--toolbox-focus: #107d74;
|
||||||
--toolbox-danger: #b42342;
|
--toolbox-danger: #b42342;
|
||||||
}
|
}
|
||||||
|
|
||||||
* {
|
* {
|
||||||
box-sizing: border-box;
|
box-sizing: border-box;
|
||||||
}
|
}
|
||||||
|
|
||||||
html {
|
html {
|
||||||
min-width: 20rem;
|
min-width: 20rem;
|
||||||
min-height: 100%;
|
min-height: 100%;
|
||||||
background: var(--toolbox-background);
|
background: var(--toolbox-background);
|
||||||
scrollbar-gutter: stable;
|
|
||||||
}
|
}
|
||||||
|
|
||||||
body {
|
body {
|
||||||
min-width: 20rem;
|
min-width: 20rem;
|
||||||
min-height: 100vh;
|
min-height: 100vh;
|
||||||
@@ -29,15 +31,17 @@ body {
|
|||||||
color: var(--toolbox-text);
|
color: var(--toolbox-text);
|
||||||
font-family: Inter, ui-sans-serif, system-ui, sans-serif;
|
font-family: Inter, ui-sans-serif, system-ui, sans-serif;
|
||||||
}
|
}
|
||||||
|
|
||||||
button,
|
button,
|
||||||
input,
|
input,
|
||||||
select,
|
select,
|
||||||
textarea {
|
textarea {
|
||||||
font: inherit;
|
font: inherit;
|
||||||
}
|
}
|
||||||
|
|
||||||
button,
|
button,
|
||||||
.button {
|
.button {
|
||||||
min-height: 2.55rem;
|
min-height: 2.45rem;
|
||||||
display: inline-flex;
|
display: inline-flex;
|
||||||
align-items: center;
|
align-items: center;
|
||||||
justify-content: center;
|
justify-content: center;
|
||||||
@@ -50,40 +54,70 @@ button,
|
|||||||
font-weight: 720;
|
font-weight: 720;
|
||||||
cursor: pointer;
|
cursor: pointer;
|
||||||
}
|
}
|
||||||
|
|
||||||
button:hover:not(:disabled),
|
button:hover:not(:disabled),
|
||||||
.button:hover {
|
.button:hover {
|
||||||
border-color: var(--toolbox-accent);
|
border-color: var(--toolbox-accent);
|
||||||
background: var(--toolbox-surface-soft);
|
background: var(--toolbox-surface-soft);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
button:disabled {
|
||||||
|
opacity: 0.62;
|
||||||
|
cursor: not-allowed;
|
||||||
|
}
|
||||||
|
|
||||||
|
.secondary {
|
||||||
|
background: #f7f7ff;
|
||||||
|
}
|
||||||
|
|
||||||
|
button.primary {
|
||||||
|
border-color: var(--toolbox-accent);
|
||||||
|
background: var(--toolbox-accent);
|
||||||
|
color: var(--toolbox-accent-contrast);
|
||||||
|
}
|
||||||
|
|
||||||
|
button.primary:hover {
|
||||||
|
background: var(--toolbox-accent-hover);
|
||||||
|
}
|
||||||
|
|
||||||
:where(button, input, select, textarea, a):focus-visible {
|
:where(button, input, select, textarea, a):focus-visible {
|
||||||
outline: 3px solid color-mix(in srgb, var(--toolbox-focus) 42%, transparent);
|
outline: 3px solid color-mix(in srgb, var(--toolbox-focus) 42%, transparent);
|
||||||
outline-offset: 2px;
|
outline-offset: 2px;
|
||||||
}
|
}
|
||||||
|
|
||||||
input,
|
input,
|
||||||
select,
|
select,
|
||||||
textarea {
|
textarea {
|
||||||
width: 100%;
|
width: 100%;
|
||||||
min-height: 2.55rem;
|
min-height: 2.45rem;
|
||||||
padding: 0.58rem 0.7rem;
|
padding: 0.58rem 0.7rem;
|
||||||
border: 1px solid var(--toolbox-border);
|
border: 1px solid var(--toolbox-border);
|
||||||
border-radius: 0.62rem;
|
border-radius: 0.62rem;
|
||||||
background: var(--toolbox-surface);
|
background: var(--toolbox-surface);
|
||||||
color: var(--toolbox-text);
|
color: var(--toolbox-text);
|
||||||
}
|
}
|
||||||
|
|
||||||
textarea {
|
textarea {
|
||||||
min-height: 10rem;
|
min-height: 9rem;
|
||||||
resize: vertical;
|
resize: vertical;
|
||||||
font-family: ui-monospace, SFMono-Regular, Consolas, monospace;
|
font-family: ui-monospace, SFMono-Regular, Consolas, monospace;
|
||||||
line-height: 1.48;
|
line-height: 1.4;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
a {
|
||||||
|
color: inherit;
|
||||||
|
}
|
||||||
|
|
||||||
.toolbox-shell__main {
|
.toolbox-shell__main {
|
||||||
width: min(100%, 90rem);
|
width: min(100%, 90rem);
|
||||||
padding: clamp(0.75rem, 1.8vw, 1.5rem);
|
padding: clamp(0.75rem, 1.8vw, 1.5rem);
|
||||||
}
|
}
|
||||||
|
|
||||||
.workbench {
|
.workbench {
|
||||||
display: grid;
|
display: grid;
|
||||||
gap: 1rem;
|
gap: 1rem;
|
||||||
}
|
}
|
||||||
|
|
||||||
.hero,
|
.hero,
|
||||||
.panel {
|
.panel {
|
||||||
border: 1px solid var(--toolbox-border);
|
border: 1px solid var(--toolbox-border);
|
||||||
@@ -91,296 +125,263 @@ textarea {
|
|||||||
background: var(--toolbox-surface);
|
background: var(--toolbox-surface);
|
||||||
box-shadow: 0 8px 28px rgb(30 36 70 / 4%);
|
box-shadow: 0 8px 28px rgb(30 36 70 / 4%);
|
||||||
}
|
}
|
||||||
|
|
||||||
.hero {
|
.hero {
|
||||||
display: flex;
|
display: flex;
|
||||||
justify-content: space-between;
|
justify-content: space-between;
|
||||||
gap: 1rem;
|
gap: 1rem;
|
||||||
align-items: flex-start;
|
align-items: flex-start;
|
||||||
padding: clamp(1.1rem, 3vw, 2rem);
|
padding: clamp(1rem, 3vw, 1.5rem);
|
||||||
}
|
}
|
||||||
|
|
||||||
.hero h1,
|
.hero h1,
|
||||||
.panel h2,
|
.panel h2,
|
||||||
.panel h3,
|
.help-dialog h2 {
|
||||||
.help-dialog h2,
|
|
||||||
.fatal h1 {
|
|
||||||
margin: 0;
|
margin: 0;
|
||||||
letter-spacing: -0.025em;
|
letter-spacing: -0.025em;
|
||||||
}
|
}
|
||||||
|
|
||||||
.hero p:not(.eyebrow) {
|
.hero p:not(.eyebrow) {
|
||||||
max-width: 52rem;
|
max-width: 52rem;
|
||||||
margin: 0.55rem 0 0;
|
margin: 0.55rem 0 0;
|
||||||
color: var(--toolbox-muted);
|
color: var(--toolbox-muted);
|
||||||
line-height: 1.55;
|
line-height: 1.55;
|
||||||
}
|
}
|
||||||
.eyebrow {
|
|
||||||
margin: 0 0 0.3rem;
|
|
||||||
color: var(--toolbox-accent);
|
|
||||||
font-size: 0.69rem;
|
|
||||||
font-weight: 820;
|
|
||||||
letter-spacing: 0.115em;
|
|
||||||
text-transform: uppercase;
|
|
||||||
}
|
|
||||||
.privacy-pill {
|
.privacy-pill {
|
||||||
flex: 0 0 auto;
|
padding: 0.4rem 0.7rem;
|
||||||
padding: 0.38rem 0.62rem;
|
|
||||||
border-radius: 999px;
|
border-radius: 999px;
|
||||||
|
font-size: 0.84rem;
|
||||||
|
border: 1px solid var(--toolbox-border);
|
||||||
background: var(--toolbox-accent-soft);
|
background: var(--toolbox-accent-soft);
|
||||||
color: var(--toolbox-accent);
|
color: #2f2a6d;
|
||||||
font-size: 0.75rem;
|
align-self: flex-start;
|
||||||
font-weight: 760;
|
|
||||||
}
|
}
|
||||||
|
|
||||||
.panel {
|
.panel {
|
||||||
padding: 1rem;
|
padding: 1rem;
|
||||||
}
|
}
|
||||||
|
|
||||||
.panel-heading {
|
.panel-heading {
|
||||||
display: flex;
|
display: flex;
|
||||||
justify-content: space-between;
|
justify-content: space-between;
|
||||||
gap: 1rem;
|
gap: 1rem;
|
||||||
align-items: end;
|
align-items: flex-start;
|
||||||
margin-bottom: 0.9rem;
|
margin-bottom: 0.85rem;
|
||||||
}
|
}
|
||||||
.capability-grid {
|
|
||||||
display: grid;
|
.panel-heading h2 {
|
||||||
grid-template-columns: repeat(auto-fit, minmax(min(100%, 13rem), 1fr));
|
margin: 0.16rem 0 0;
|
||||||
gap: 0.75rem;
|
|
||||||
}
|
}
|
||||||
.capability-grid article {
|
|
||||||
padding: 0.9rem;
|
.eyebrow {
|
||||||
border: 1px solid var(--toolbox-border);
|
margin: 0;
|
||||||
border-radius: 0.72rem;
|
color: var(--toolbox-accent);
|
||||||
background: var(--toolbox-surface-soft);
|
font-size: 0.67rem;
|
||||||
|
font-weight: 820;
|
||||||
|
letter-spacing: 0.115em;
|
||||||
}
|
}
|
||||||
.capability-grid p {
|
|
||||||
margin: 0.4rem 0 0;
|
.actions {
|
||||||
color: var(--toolbox-muted);
|
|
||||||
line-height: 1.48;
|
|
||||||
}
|
|
||||||
.workspace-tabs {
|
|
||||||
display: flex;
|
display: flex;
|
||||||
gap: 0.4rem;
|
flex-wrap: wrap;
|
||||||
overflow-x: auto;
|
gap: 0.5rem;
|
||||||
padding-bottom: 0.2rem;
|
|
||||||
}
|
|
||||||
.workspace-tabs button[aria-selected="true"] {
|
|
||||||
border-color: var(--toolbox-accent);
|
|
||||||
background: var(--toolbox-accent);
|
|
||||||
color: var(--toolbox-accent-contrast);
|
|
||||||
}
|
|
||||||
.grid {
|
|
||||||
display: grid;
|
|
||||||
grid-template-columns: repeat(auto-fit, minmax(min(100%, 20rem), 1fr));
|
|
||||||
gap: 0.85rem;
|
|
||||||
}
|
}
|
||||||
|
|
||||||
.field {
|
.field {
|
||||||
display: grid;
|
display: grid;
|
||||||
gap: 0.35rem;
|
gap: 0.35rem;
|
||||||
}
|
}
|
||||||
.field > span {
|
|
||||||
font-size: 0.76rem;
|
.field-inline {
|
||||||
font-weight: 750;
|
display: inline-flex;
|
||||||
}
|
|
||||||
.muted {
|
|
||||||
color: var(--toolbox-muted);
|
|
||||||
}
|
|
||||||
.result {
|
|
||||||
padding: 0.8rem;
|
|
||||||
border: 1px solid var(--toolbox-border);
|
|
||||||
border-radius: 0.68rem;
|
|
||||||
background: var(--toolbox-surface-soft);
|
|
||||||
overflow-wrap: anywhere;
|
|
||||||
}
|
|
||||||
.loading,
|
|
||||||
.fatal {
|
|
||||||
width: min(100% - 2rem, 60rem);
|
|
||||||
margin: 2rem auto;
|
|
||||||
padding: 1rem;
|
|
||||||
}
|
|
||||||
.help-dialog {
|
|
||||||
width: min(36rem, calc(100% - 2rem));
|
|
||||||
border: 1px solid var(--toolbox-border);
|
|
||||||
border-radius: 0.9rem;
|
|
||||||
background: var(--toolbox-surface);
|
|
||||||
color: var(--toolbox-text);
|
|
||||||
}
|
|
||||||
.help-dialog::backdrop {
|
|
||||||
background: rgb(20 24 45 / 55%);
|
|
||||||
}
|
|
||||||
.dialog-heading {
|
|
||||||
display: flex;
|
|
||||||
justify-content: space-between;
|
|
||||||
gap: 1rem;
|
|
||||||
align-items: start;
|
|
||||||
}
|
|
||||||
.workspace {
|
|
||||||
display: grid;
|
|
||||||
gap: 0.85rem;
|
|
||||||
}
|
|
||||||
.editor-grid {
|
|
||||||
display: grid;
|
|
||||||
grid-template-columns: repeat(2, minmax(0, 1fr));
|
|
||||||
gap: 1rem;
|
|
||||||
}
|
|
||||||
.editor-grid textarea {
|
|
||||||
min-height: 22rem;
|
|
||||||
}
|
|
||||||
.encoding-row {
|
|
||||||
display: grid;
|
|
||||||
grid-template-columns: minmax(10rem, 16rem) minmax(12rem, 1fr);
|
|
||||||
gap: 0.7rem;
|
|
||||||
align-items: end;
|
|
||||||
}
|
|
||||||
.actions {
|
|
||||||
display: flex;
|
|
||||||
gap: 0.5rem;
|
|
||||||
flex-wrap: wrap;
|
|
||||||
}
|
|
||||||
.primary {
|
|
||||||
border-color: var(--toolbox-accent);
|
|
||||||
background: var(--toolbox-accent);
|
|
||||||
color: var(--toolbox-accent-contrast);
|
|
||||||
}
|
|
||||||
.file-button {
|
|
||||||
position: relative;
|
|
||||||
overflow: hidden;
|
|
||||||
}
|
|
||||||
.file-button input {
|
|
||||||
position: absolute;
|
|
||||||
inset: 0;
|
|
||||||
opacity: 0;
|
|
||||||
cursor: pointer;
|
|
||||||
}
|
|
||||||
.check {
|
|
||||||
display: flex;
|
|
||||||
gap: 0.5rem;
|
|
||||||
align-items: center;
|
align-items: center;
|
||||||
min-height: 2.55rem;
|
|
||||||
}
|
|
||||||
.check input {
|
|
||||||
width: auto;
|
|
||||||
min-height: auto;
|
|
||||||
}
|
|
||||||
.inventory {
|
|
||||||
display: grid;
|
|
||||||
grid-template-columns: repeat(auto-fit, minmax(7rem, 1fr));
|
|
||||||
gap: 0.45rem;
|
|
||||||
margin: 0;
|
|
||||||
}
|
|
||||||
.inventory div {
|
|
||||||
padding: 0.55rem;
|
|
||||||
border-radius: 0.55rem;
|
|
||||||
background: var(--toolbox-surface-soft);
|
|
||||||
}
|
|
||||||
.inventory dt {
|
|
||||||
color: var(--toolbox-muted);
|
|
||||||
font-size: 0.66rem;
|
|
||||||
font-weight: 750;
|
|
||||||
}
|
|
||||||
.inventory dd {
|
|
||||||
margin: 0.15rem 0 0;
|
|
||||||
}
|
|
||||||
.steps {
|
|
||||||
display: grid;
|
|
||||||
gap: 0.55rem;
|
|
||||||
margin: 0;
|
|
||||||
padding: 0;
|
|
||||||
list-style: none;
|
|
||||||
}
|
|
||||||
.steps li {
|
|
||||||
display: grid;
|
|
||||||
grid-template-columns: minmax(12rem, 0.8fr) minmax(12rem, 1.4fr) auto;
|
|
||||||
gap: 0.6rem;
|
|
||||||
align-items: center;
|
|
||||||
padding: 0.6rem;
|
|
||||||
border: 1px solid var(--toolbox-border);
|
|
||||||
border-radius: 0.65rem;
|
|
||||||
}
|
|
||||||
.step-actions {
|
|
||||||
display: flex;
|
|
||||||
gap: 0.25rem;
|
|
||||||
}
|
|
||||||
.column-options {
|
|
||||||
display: grid;
|
|
||||||
grid-template-columns: minmax(8rem, 1fr) minmax(4rem, 0.55fr) minmax(
|
|
||||||
7rem,
|
|
||||||
1fr
|
|
||||||
);
|
|
||||||
gap: 0.4rem;
|
gap: 0.4rem;
|
||||||
}
|
}
|
||||||
.step-actions button {
|
|
||||||
min-width: 2.55rem;
|
.field > span {
|
||||||
padding: 0.4rem;
|
font-size: 0.92rem;
|
||||||
|
color: var(--toolbox-muted);
|
||||||
}
|
}
|
||||||
.warning,
|
|
||||||
.error,
|
.form-grid {
|
||||||
.notice {
|
display: grid;
|
||||||
margin: 0;
|
gap: 0.8rem;
|
||||||
padding: 0.7rem;
|
grid-template-columns: repeat(auto-fit, minmax(12rem, 1fr));
|
||||||
|
}
|
||||||
|
|
||||||
|
.field-group {
|
||||||
|
display: flex;
|
||||||
|
gap: 0.5rem;
|
||||||
|
}
|
||||||
|
|
||||||
|
.field-group input {
|
||||||
|
flex: 1;
|
||||||
|
}
|
||||||
|
|
||||||
|
.table-wrap {
|
||||||
|
overflow-x: auto;
|
||||||
border: 1px solid var(--toolbox-border);
|
border: 1px solid var(--toolbox-border);
|
||||||
border-radius: 0.65rem;
|
border-radius: 0.8rem;
|
||||||
line-height: 1.5;
|
|
||||||
}
|
}
|
||||||
.warning {
|
|
||||||
border-color: #d9a72e;
|
.review-grid {
|
||||||
background: #fff8df;
|
|
||||||
color: #725000;
|
|
||||||
}
|
|
||||||
.error {
|
|
||||||
border-color: var(--toolbox-danger);
|
|
||||||
color: var(--toolbox-danger);
|
|
||||||
}
|
|
||||||
.notice {
|
|
||||||
background: var(--toolbox-surface-soft);
|
|
||||||
}
|
|
||||||
table {
|
|
||||||
width: 100%;
|
width: 100%;
|
||||||
border-collapse: collapse;
|
border-collapse: collapse;
|
||||||
}
|
}
|
||||||
th,
|
|
||||||
td {
|
.review-grid th,
|
||||||
padding: 0.55rem;
|
.review-grid td {
|
||||||
border-bottom: 1px solid var(--toolbox-border);
|
border-bottom: 1px solid var(--toolbox-border);
|
||||||
text-align: left;
|
border-right: 1px solid var(--toolbox-border);
|
||||||
|
padding: 0.55rem;
|
||||||
|
vertical-align: top;
|
||||||
}
|
}
|
||||||
th {
|
|
||||||
|
.review-grid th:last-child,
|
||||||
|
.review-grid td:last-child {
|
||||||
|
border-right: none;
|
||||||
|
}
|
||||||
|
|
||||||
|
.review-grid thead {
|
||||||
|
background: #f9fbff;
|
||||||
|
}
|
||||||
|
|
||||||
|
.review-grid textarea {
|
||||||
|
min-height: 6rem;
|
||||||
|
}
|
||||||
|
|
||||||
|
.key-cell {
|
||||||
|
width: 24ch;
|
||||||
|
max-width: 24ch;
|
||||||
|
white-space: nowrap;
|
||||||
|
overflow: hidden;
|
||||||
|
text-overflow: ellipsis;
|
||||||
|
}
|
||||||
|
|
||||||
|
.count {
|
||||||
|
display: inline-flex;
|
||||||
|
align-items: center;
|
||||||
color: var(--toolbox-muted);
|
color: var(--toolbox-muted);
|
||||||
font-size: 0.7rem;
|
padding: 0.4rem 0.5rem;
|
||||||
text-transform: uppercase;
|
border-radius: 0.5rem;
|
||||||
}
|
background: #f8f9fc;
|
||||||
details {
|
|
||||||
border: 1px solid var(--toolbox-border);
|
border: 1px solid var(--toolbox-border);
|
||||||
border-radius: 0.65rem;
|
|
||||||
}
|
}
|
||||||
summary {
|
|
||||||
padding: 0.65rem;
|
.muted {
|
||||||
cursor: pointer;
|
color: var(--toolbox-muted);
|
||||||
font-weight: 750;
|
|
||||||
}
|
}
|
||||||
.recipe {
|
|
||||||
|
.row-actions {
|
||||||
|
margin-top: 0.35rem;
|
||||||
|
display: flex;
|
||||||
|
flex-wrap: wrap;
|
||||||
|
gap: 0.35rem;
|
||||||
|
}
|
||||||
|
|
||||||
|
.row-actions .chip,
|
||||||
|
.chip {
|
||||||
|
border-radius: 999px;
|
||||||
|
min-height: auto;
|
||||||
|
padding: 0.35rem 0.56rem;
|
||||||
|
}
|
||||||
|
|
||||||
|
.chip-list {
|
||||||
|
display: flex;
|
||||||
|
flex-wrap: wrap;
|
||||||
|
gap: 0.35rem;
|
||||||
|
}
|
||||||
|
|
||||||
|
.notice {
|
||||||
|
margin: 0.75rem 0 0;
|
||||||
|
color: #1f5a75;
|
||||||
|
}
|
||||||
|
|
||||||
|
.error {
|
||||||
|
margin: 0.75rem 0 0;
|
||||||
|
color: var(--toolbox-danger);
|
||||||
|
font-weight: 700;
|
||||||
|
}
|
||||||
|
|
||||||
|
.help-dialog {
|
||||||
|
margin: auto;
|
||||||
|
width: min(50rem, 95vw);
|
||||||
|
border: 1px solid var(--toolbox-border);
|
||||||
|
border-radius: 1rem;
|
||||||
|
padding: 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
.help-dialog .dialog-heading {
|
||||||
|
display: flex;
|
||||||
|
justify-content: space-between;
|
||||||
|
align-items: center;
|
||||||
|
padding: 1rem;
|
||||||
|
border-bottom: 1px solid var(--toolbox-border);
|
||||||
|
}
|
||||||
|
|
||||||
|
.help-dialog p {
|
||||||
|
margin: 0.7rem 1rem;
|
||||||
|
}
|
||||||
|
|
||||||
|
.glossary,
|
||||||
|
.language-list,
|
||||||
|
.term-list,
|
||||||
|
.workspace ul {
|
||||||
|
list-style: none;
|
||||||
|
margin: 0;
|
||||||
|
padding: 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
.glossary-form {
|
||||||
display: grid;
|
display: grid;
|
||||||
gap: 0.6rem;
|
gap: 0.55rem;
|
||||||
padding: 0 0.65rem 0.65rem;
|
margin: 0.5rem 0 0;
|
||||||
|
grid-template-columns: repeat(auto-fit, minmax(12rem, 1fr));
|
||||||
}
|
}
|
||||||
@media (max-width: 62rem) {
|
|
||||||
.editor-grid {
|
.term-list,
|
||||||
grid-template-columns: 1fr;
|
.language-list {
|
||||||
|
margin-top: 0.65rem;
|
||||||
|
display: grid;
|
||||||
|
gap: 0.45rem;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
.term-list li,
|
||||||
|
.language-list li {
|
||||||
|
border: 1px solid var(--toolbox-border);
|
||||||
|
border-radius: 0.6rem;
|
||||||
|
background: #fbfcff;
|
||||||
|
display: flex;
|
||||||
|
justify-content: space-between;
|
||||||
|
gap: 0.75rem;
|
||||||
|
align-items: center;
|
||||||
|
padding: 0.55rem;
|
||||||
}
|
}
|
||||||
@media (max-width: 48rem) {
|
|
||||||
.steps li {
|
.term-list p {
|
||||||
grid-template-columns: 1fr;
|
margin: 0.15rem 0 0;
|
||||||
}
|
}
|
||||||
.column-options {
|
|
||||||
grid-template-columns: 1fr;
|
.sr-only {
|
||||||
|
position: absolute;
|
||||||
|
width: 1px;
|
||||||
|
height: 1px;
|
||||||
|
padding: 0;
|
||||||
|
margin: -1px;
|
||||||
|
overflow: hidden;
|
||||||
|
clip: rect(0, 0, 0, 0);
|
||||||
|
white-space: nowrap;
|
||||||
|
border-width: 0;
|
||||||
}
|
}
|
||||||
}
|
|
||||||
@media (max-width: 42rem) {
|
@media (max-width: 720px) {
|
||||||
.hero {
|
.panel-heading {
|
||||||
flex-direction: column;
|
flex-direction: column;
|
||||||
|
align-items: stretch;
|
||||||
}
|
}
|
||||||
.privacy-pill {
|
|
||||||
order: -1;
|
.form-grid {
|
||||||
}
|
grid-template-columns: 1fr;
|
||||||
.encoding-row {
|
|
||||||
grid-template-columns: minmax(0, 1fr);
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -1,251 +0,0 @@
|
|||||||
import {
|
|
||||||
decodeText,
|
|
||||||
digestHex,
|
|
||||||
encodeText,
|
|
||||||
type TextEncoding,
|
|
||||||
} from "@add-ideas/toolbox-helpers";
|
|
||||||
|
|
||||||
import {
|
|
||||||
textInventory,
|
|
||||||
type PipelineResult,
|
|
||||||
type StepReport,
|
|
||||||
type TransformStep,
|
|
||||||
} from "./pipeline";
|
|
||||||
|
|
||||||
export interface TextByteEvidence {
|
|
||||||
readonly schemaVersion: 1;
|
|
||||||
readonly byteLength: number;
|
|
||||||
readonly selectedEncoding: TextEncoding;
|
|
||||||
readonly fatalDecode: boolean;
|
|
||||||
readonly bom?: { encoding: Exclude<TextEncoding, "latin1">; bytes: number };
|
|
||||||
readonly utf8: { valid: boolean; firstInvalidOffset?: number };
|
|
||||||
readonly zeroBytes: {
|
|
||||||
total: number;
|
|
||||||
evenOffsets: number;
|
|
||||||
oddOffsets: number;
|
|
||||||
};
|
|
||||||
readonly byteNewlines: { crlf: number; bareLf: number; bareCr: number };
|
|
||||||
readonly replacementCharacters: number;
|
|
||||||
readonly warnings: readonly string[];
|
|
||||||
}
|
|
||||||
|
|
||||||
export interface TextArtifactEvidence {
|
|
||||||
readonly schemaVersion: 1;
|
|
||||||
readonly artifactType: "de.add-ideas.toolbox.text/v1";
|
|
||||||
readonly createdBy: { app: "text-tools"; version: "0.2.0" };
|
|
||||||
readonly source: {
|
|
||||||
name: string;
|
|
||||||
canonicalUtf8Sha256: string;
|
|
||||||
inventory: ReturnType<typeof textInventory>;
|
|
||||||
byteEvidence?: TextByteEvidence;
|
|
||||||
};
|
|
||||||
readonly output: {
|
|
||||||
name: string;
|
|
||||||
mediaType: "text/plain";
|
|
||||||
encoding: TextEncoding;
|
|
||||||
bytes: number;
|
|
||||||
sha256: string;
|
|
||||||
inventory: ReturnType<typeof textInventory>;
|
|
||||||
};
|
|
||||||
readonly pipeline: {
|
|
||||||
steps: readonly TransformStep[];
|
|
||||||
reports: readonly StepReport[];
|
|
||||||
warnings: readonly string[];
|
|
||||||
};
|
|
||||||
readonly handoff: {
|
|
||||||
supportedByThisBuild: false;
|
|
||||||
note: string;
|
|
||||||
};
|
|
||||||
}
|
|
||||||
|
|
||||||
function bom(bytes: Uint8Array): TextByteEvidence["bom"] {
|
|
||||||
if (bytes[0] === 0xef && bytes[1] === 0xbb && bytes[2] === 0xbf)
|
|
||||||
return { encoding: "utf-8", bytes: 3 };
|
|
||||||
if (bytes[0] === 0xff && bytes[1] === 0xfe)
|
|
||||||
return { encoding: "utf-16le", bytes: 2 };
|
|
||||||
if (bytes[0] === 0xfe && bytes[1] === 0xff)
|
|
||||||
return { encoding: "utf-16be", bytes: 2 };
|
|
||||||
return undefined;
|
|
||||||
}
|
|
||||||
|
|
||||||
function utf8InvalidOffset(bytes: Uint8Array): number | undefined {
|
|
||||||
const continuation = (index: number) =>
|
|
||||||
index < bytes.length && (bytes[index]! & 0xc0) === 0x80;
|
|
||||||
for (let index = 0; index < bytes.length; index += 1) {
|
|
||||||
const first = bytes[index]!;
|
|
||||||
if (first <= 0x7f) continue;
|
|
||||||
if (first >= 0xc2 && first <= 0xdf) {
|
|
||||||
if (!continuation(index + 1)) return index;
|
|
||||||
index += 1;
|
|
||||||
continue;
|
|
||||||
}
|
|
||||||
if (first >= 0xe0 && first <= 0xef) {
|
|
||||||
const second = bytes[index + 1];
|
|
||||||
if (
|
|
||||||
second === undefined ||
|
|
||||||
(first === 0xe0 && (second < 0xa0 || second > 0xbf)) ||
|
|
||||||
(first === 0xed && (second < 0x80 || second > 0x9f)) ||
|
|
||||||
(first !== 0xe0 &&
|
|
||||||
first !== 0xed &&
|
|
||||||
(second < 0x80 || second > 0xbf)) ||
|
|
||||||
!continuation(index + 2)
|
|
||||||
)
|
|
||||||
return index;
|
|
||||||
index += 2;
|
|
||||||
continue;
|
|
||||||
}
|
|
||||||
if (first >= 0xf0 && first <= 0xf4) {
|
|
||||||
const second = bytes[index + 1];
|
|
||||||
if (
|
|
||||||
second === undefined ||
|
|
||||||
(first === 0xf0 && (second < 0x90 || second > 0xbf)) ||
|
|
||||||
(first === 0xf4 && (second < 0x80 || second > 0x8f)) ||
|
|
||||||
(first !== 0xf0 &&
|
|
||||||
first !== 0xf4 &&
|
|
||||||
(second < 0x80 || second > 0xbf)) ||
|
|
||||||
!continuation(index + 2) ||
|
|
||||||
!continuation(index + 3)
|
|
||||||
)
|
|
||||||
return index;
|
|
||||||
index += 3;
|
|
||||||
continue;
|
|
||||||
}
|
|
||||||
return index;
|
|
||||||
}
|
|
||||||
return undefined;
|
|
||||||
}
|
|
||||||
|
|
||||||
function inspectByteNewlines(bytes: Uint8Array) {
|
|
||||||
let crlf = 0;
|
|
||||||
let bareLf = 0;
|
|
||||||
let bareCr = 0;
|
|
||||||
for (let index = 0; index < bytes.length; index += 1) {
|
|
||||||
if (bytes[index] === 0x0d && bytes[index + 1] === 0x0a) {
|
|
||||||
crlf += 1;
|
|
||||||
index += 1;
|
|
||||||
} else if (bytes[index] === 0x0a) bareLf += 1;
|
|
||||||
else if (bytes[index] === 0x0d) bareCr += 1;
|
|
||||||
}
|
|
||||||
return { crlf, bareLf, bareCr };
|
|
||||||
}
|
|
||||||
|
|
||||||
export function decodeTextWithEvidence(
|
|
||||||
bytes: Uint8Array,
|
|
||||||
selectedEncoding: TextEncoding,
|
|
||||||
fatalDecode: boolean,
|
|
||||||
): { text: string; evidence: TextByteEvidence } {
|
|
||||||
if (bytes.byteLength > 16 * 1024 * 1024)
|
|
||||||
throw new RangeError("Text byte evidence is limited to 16 MiB.");
|
|
||||||
const detectedBom = bom(bytes);
|
|
||||||
const invalidOffset = utf8InvalidOffset(bytes);
|
|
||||||
let zeroTotal = 0;
|
|
||||||
let zeroEven = 0;
|
|
||||||
let zeroOdd = 0;
|
|
||||||
bytes.forEach((value, index) => {
|
|
||||||
if (value !== 0) return;
|
|
||||||
zeroTotal += 1;
|
|
||||||
if (index % 2) zeroOdd += 1;
|
|
||||||
else zeroEven += 1;
|
|
||||||
});
|
|
||||||
const text = decodeText(bytes, selectedEncoding, fatalDecode);
|
|
||||||
const warnings: string[] = [];
|
|
||||||
if (detectedBom && detectedBom.encoding !== selectedEncoding)
|
|
||||||
warnings.push(
|
|
||||||
`The ${detectedBom.encoding.toUpperCase()} BOM conflicts with the selected ${selectedEncoding.toUpperCase()} decoder.`,
|
|
||||||
);
|
|
||||||
if (!detectedBom)
|
|
||||||
warnings.push(
|
|
||||||
"No byte-order mark is present; the selected encoding is an explicit user choice, not a detection claim.",
|
|
||||||
);
|
|
||||||
if (selectedEncoding.startsWith("utf-16") && bytes.length % 2)
|
|
||||||
warnings.push("UTF-16 input has an odd trailing byte.");
|
|
||||||
if (selectedEncoding !== "utf-8" && invalidOffset === undefined)
|
|
||||||
warnings.push(
|
|
||||||
"The same bytes are also well-formed UTF-8; encoding intent cannot be inferred from validity alone.",
|
|
||||||
);
|
|
||||||
const replacementCharacters = [...text].filter(
|
|
||||||
(character) => character === "\uFFFD",
|
|
||||||
).length;
|
|
||||||
if (replacementCharacters)
|
|
||||||
warnings.push(
|
|
||||||
`${replacementCharacters} replacement character(s) appear in decoded text; they may be source data or decoder substitutions.`,
|
|
||||||
);
|
|
||||||
return {
|
|
||||||
text,
|
|
||||||
evidence: Object.freeze({
|
|
||||||
schemaVersion: 1,
|
|
||||||
byteLength: bytes.byteLength,
|
|
||||||
selectedEncoding,
|
|
||||||
fatalDecode,
|
|
||||||
...(detectedBom ? { bom: detectedBom } : {}),
|
|
||||||
utf8: {
|
|
||||||
valid: invalidOffset === undefined,
|
|
||||||
...(invalidOffset === undefined
|
|
||||||
? {}
|
|
||||||
: { firstInvalidOffset: invalidOffset }),
|
|
||||||
},
|
|
||||||
zeroBytes: {
|
|
||||||
total: zeroTotal,
|
|
||||||
evenOffsets: zeroEven,
|
|
||||||
oddOffsets: zeroOdd,
|
|
||||||
},
|
|
||||||
byteNewlines: inspectByteNewlines(bytes),
|
|
||||||
replacementCharacters,
|
|
||||||
warnings: Object.freeze(warnings),
|
|
||||||
}),
|
|
||||||
};
|
|
||||||
}
|
|
||||||
|
|
||||||
export async function createTextArtifactEvidence(input: {
|
|
||||||
sourceName: string;
|
|
||||||
sourceEvidence?: TextByteEvidence;
|
|
||||||
sourceText: string;
|
|
||||||
outputName: string;
|
|
||||||
outputEncoding: TextEncoding;
|
|
||||||
outputBytes: Uint8Array;
|
|
||||||
pipeline: PipelineResult;
|
|
||||||
steps: readonly TransformStep[];
|
|
||||||
}): Promise<TextArtifactEvidence> {
|
|
||||||
if (input.outputBytes.byteLength > 32 * 1024 * 1024)
|
|
||||||
throw new RangeError("Artifact output exceeds the 32 MiB evidence limit.");
|
|
||||||
const canonicalSource = encodeText(input.sourceText, "utf-8");
|
|
||||||
return Object.freeze({
|
|
||||||
schemaVersion: 1,
|
|
||||||
artifactType: "de.add-ideas.toolbox.text/v1",
|
|
||||||
createdBy: {
|
|
||||||
app: "text-tools" as const,
|
|
||||||
version: "0.2.0" as const,
|
|
||||||
},
|
|
||||||
source: {
|
|
||||||
name: input.sourceName,
|
|
||||||
canonicalUtf8Sha256: await digestHex(
|
|
||||||
canonicalSource,
|
|
||||||
"SHA-256",
|
|
||||||
32 * 1024 * 1024,
|
|
||||||
),
|
|
||||||
inventory: textInventory(input.sourceText),
|
|
||||||
...(input.sourceEvidence ? { byteEvidence: input.sourceEvidence } : {}),
|
|
||||||
},
|
|
||||||
output: {
|
|
||||||
name: input.outputName,
|
|
||||||
mediaType: "text/plain" as const,
|
|
||||||
encoding: input.outputEncoding,
|
|
||||||
bytes: input.outputBytes.byteLength,
|
|
||||||
sha256: await digestHex(input.outputBytes, "SHA-256", 32 * 1024 * 1024),
|
|
||||||
inventory: textInventory(input.pipeline.output),
|
|
||||||
},
|
|
||||||
pipeline: {
|
|
||||||
steps: Object.freeze(
|
|
||||||
input.steps.map((step) => Object.freeze({ ...step })),
|
|
||||||
),
|
|
||||||
reports: Object.freeze(
|
|
||||||
input.pipeline.steps.map((report) => Object.freeze({ ...report })),
|
|
||||||
),
|
|
||||||
warnings: Object.freeze([...input.pipeline.warnings]),
|
|
||||||
},
|
|
||||||
handoff: {
|
|
||||||
supportedByThisBuild: false as const,
|
|
||||||
note: "The portable files are ready for explicit local transfer. SDK 0.3.0 provides the shared contract, but Open With remains disabled until the coordinated Portal consumer rollout.",
|
|
||||||
},
|
|
||||||
});
|
|
||||||
}
|
|
||||||
@@ -1,515 +0,0 @@
|
|||||||
import {
|
|
||||||
base64ToBytes,
|
|
||||||
bytesToBase64,
|
|
||||||
bytesToHex,
|
|
||||||
convertLineEndings,
|
|
||||||
decodeText,
|
|
||||||
encodeText,
|
|
||||||
hexToBytes,
|
|
||||||
normalizeUnicode,
|
|
||||||
parseCsv,
|
|
||||||
stringifyCsv,
|
|
||||||
transformCase,
|
|
||||||
type CaseTransform,
|
|
||||||
type LineEnding,
|
|
||||||
} from "@add-ideas/toolbox-helpers";
|
|
||||||
|
|
||||||
export type StepType =
|
|
||||||
| "line-endings"
|
|
||||||
| "trim-lines"
|
|
||||||
| "trim-document"
|
|
||||||
| "collapse-whitespace"
|
|
||||||
| "sort-lines"
|
|
||||||
| "dedupe-lines"
|
|
||||||
| "case"
|
|
||||||
| "normalize"
|
|
||||||
| "transliterate"
|
|
||||||
| "escape"
|
|
||||||
| "unescape"
|
|
||||||
| "wrap"
|
|
||||||
| "columns"
|
|
||||||
| "replace-literal"
|
|
||||||
| "prefix-lines"
|
|
||||||
| "suffix-lines"
|
|
||||||
| "filter-lines"
|
|
||||||
| "number-lines"
|
|
||||||
| "join-lines"
|
|
||||||
| "reverse-lines";
|
|
||||||
export interface TransformStep {
|
|
||||||
id: string;
|
|
||||||
type: StepType;
|
|
||||||
enabled: boolean;
|
|
||||||
option: string;
|
|
||||||
}
|
|
||||||
export interface StepReport {
|
|
||||||
id: string;
|
|
||||||
type: StepType;
|
|
||||||
beforeUnits: number;
|
|
||||||
afterUnits: number;
|
|
||||||
changed: boolean;
|
|
||||||
warning?: string;
|
|
||||||
}
|
|
||||||
export interface PipelineResult {
|
|
||||||
output: string;
|
|
||||||
steps: StepReport[];
|
|
||||||
warnings: string[];
|
|
||||||
}
|
|
||||||
|
|
||||||
const MAX_INPUT = 2_000_000;
|
|
||||||
const MAX_OUTPUT = 32 * 1024 * 1024;
|
|
||||||
let nextStepId = 0;
|
|
||||||
|
|
||||||
function lines(value: string): string[] {
|
|
||||||
return value.replaceAll("\r\n", "\n").replaceAll("\r", "\n").split("\n");
|
|
||||||
}
|
|
||||||
function htmlEscape(value: string): string {
|
|
||||||
return value
|
|
||||||
.replaceAll("&", "&")
|
|
||||||
.replaceAll("<", "<")
|
|
||||||
.replaceAll(">", ">")
|
|
||||||
.replaceAll('"', """)
|
|
||||||
.replaceAll("'", "'");
|
|
||||||
}
|
|
||||||
function stableUnique(values: string[]): string[] {
|
|
||||||
const seen = new Set<string>();
|
|
||||||
return values.filter((value) => !seen.has(value) && !!seen.add(value));
|
|
||||||
}
|
|
||||||
|
|
||||||
function wrapText(value: string, width: number): string {
|
|
||||||
if (!Number.isSafeInteger(width) || width < 1 || width > 10_000)
|
|
||||||
throw new Error("Wrap width must be 1–10,000 code points.");
|
|
||||||
return lines(value)
|
|
||||||
.flatMap((line) => {
|
|
||||||
const words = line.split(/\s+/u);
|
|
||||||
const output: string[] = [];
|
|
||||||
let current = "";
|
|
||||||
for (const word of words) {
|
|
||||||
if (!word) continue;
|
|
||||||
if (!current) current = word;
|
|
||||||
else if ([...current, " ", ...word].length <= width)
|
|
||||||
current += ` ${word}`;
|
|
||||||
else {
|
|
||||||
output.push(current);
|
|
||||||
current = word;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
output.push(current);
|
|
||||||
return output;
|
|
||||||
})
|
|
||||||
.join("\n");
|
|
||||||
}
|
|
||||||
|
|
||||||
function transformColumns(
|
|
||||||
value: string,
|
|
||||||
option: string,
|
|
||||||
): {
|
|
||||||
value: string;
|
|
||||||
warning?: string;
|
|
||||||
} {
|
|
||||||
const pieces = option.split("|");
|
|
||||||
const mode = pieces.length >= 3 ? pieces[0] : "literal";
|
|
||||||
const delimiterInput = pieces.length >= 3 ? pieces[1] : pieces[0];
|
|
||||||
const order = pieces.length >= 3 ? pieces.slice(2).join("|") : pieces[1];
|
|
||||||
const delimiter = delimiterInput === "\\t" ? "\t" : delimiterInput || ",";
|
|
||||||
if (!delimiter || delimiter.length > 8)
|
|
||||||
throw new Error("Column delimiter must contain 1–8 characters.");
|
|
||||||
if (mode !== "literal" && mode !== "csv")
|
|
||||||
throw new Error("Column parsing mode must be csv or literal.");
|
|
||||||
if (mode === "csv" && delimiter.length !== 1)
|
|
||||||
throw new Error("Quoted CSV mode requires a one-character delimiter.");
|
|
||||||
const indices = (order || "1")
|
|
||||||
.split(",")
|
|
||||||
.map((entry) => Number(entry.trim()) - 1);
|
|
||||||
if (
|
|
||||||
!indices.length ||
|
|
||||||
indices.some(
|
|
||||||
(index) => !Number.isSafeInteger(index) || index < 0 || index > 999,
|
|
||||||
)
|
|
||||||
)
|
|
||||||
throw new Error("Column order uses 1-based indices such as 3,1,2.");
|
|
||||||
if (mode === "literal")
|
|
||||||
return {
|
|
||||||
value: lines(value)
|
|
||||||
.map((line) => {
|
|
||||||
const cells = line.split(delimiter);
|
|
||||||
return indices.map((index) => cells[index] ?? "").join(delimiter);
|
|
||||||
})
|
|
||||||
.join("\n"),
|
|
||||||
};
|
|
||||||
const rows = parseCsv(value, {
|
|
||||||
delimiter,
|
|
||||||
maxRows: 200_000,
|
|
||||||
maxColumns: 1_000,
|
|
||||||
maxFieldChars: 2_000_000,
|
|
||||||
});
|
|
||||||
return {
|
|
||||||
value: stringifyCsv(
|
|
||||||
rows.map((row) => indices.map((index) => row[index] ?? "")),
|
|
||||||
{
|
|
||||||
delimiter,
|
|
||||||
maxRows: 200_000,
|
|
||||||
maxColumns: 1_000,
|
|
||||||
maxFieldChars: 2_000_000,
|
|
||||||
},
|
|
||||||
),
|
|
||||||
warning:
|
|
||||||
"Quoted CSV rows were parsed across embedded delimiters/newlines and serialized canonically with CRLF row endings.",
|
|
||||||
};
|
|
||||||
}
|
|
||||||
|
|
||||||
function strictHtmlUnescape(value: string): string {
|
|
||||||
const entities: Record<string, string> = {
|
|
||||||
"&": "&",
|
|
||||||
"<": "<",
|
|
||||||
">": ">",
|
|
||||||
""": '"',
|
|
||||||
"'": "'",
|
|
||||||
};
|
|
||||||
let output = "";
|
|
||||||
for (let index = 0; index < value.length; index += 1) {
|
|
||||||
if (value[index] !== "&") {
|
|
||||||
output += value[index];
|
|
||||||
continue;
|
|
||||||
}
|
|
||||||
const end = value.indexOf(";", index + 1);
|
|
||||||
if (end < 0 || end - index > 6)
|
|
||||||
throw new SyntaxError(
|
|
||||||
`HTML entity at UTF-16 index ${index} is malformed.`,
|
|
||||||
);
|
|
||||||
const entity = value.slice(index, end + 1);
|
|
||||||
const decoded = entities[entity];
|
|
||||||
if (decoded === undefined)
|
|
||||||
throw new SyntaxError(
|
|
||||||
`HTML entity ${entity} is not emitted by the matching escape stage.`,
|
|
||||||
);
|
|
||||||
output += decoded;
|
|
||||||
index = end;
|
|
||||||
}
|
|
||||||
return output;
|
|
||||||
}
|
|
||||||
|
|
||||||
function strictJsonUnescape(value: string): string {
|
|
||||||
const parsed = JSON.parse(`"${value}"`) as unknown;
|
|
||||||
if (typeof parsed !== "string")
|
|
||||||
throw new SyntaxError("JSON escape input is invalid.");
|
|
||||||
return parsed;
|
|
||||||
}
|
|
||||||
|
|
||||||
function unescapeValue(value: string, option: string): string {
|
|
||||||
if (option === "json") return strictJsonUnescape(value);
|
|
||||||
if (option === "html") return strictHtmlUnescape(value);
|
|
||||||
if (option === "url") return decodeURIComponent(value);
|
|
||||||
if (option === "base64")
|
|
||||||
return decodeText(
|
|
||||||
base64ToBytes(value, { maxOutputBytes: MAX_OUTPUT }),
|
|
||||||
"utf-8",
|
|
||||||
true,
|
|
||||||
);
|
|
||||||
if (option === "hex")
|
|
||||||
return decodeText(
|
|
||||||
hexToBytes(value, {
|
|
||||||
maxOutputBytes: MAX_OUTPUT,
|
|
||||||
allowWhitespace: false,
|
|
||||||
allowPrefix: false,
|
|
||||||
}),
|
|
||||||
"utf-8",
|
|
||||||
true,
|
|
||||||
);
|
|
||||||
throw new Error("Unsupported unescape source.");
|
|
||||||
}
|
|
||||||
|
|
||||||
function transliterate(value: string): string {
|
|
||||||
return value
|
|
||||||
.normalize("NFKD")
|
|
||||||
.replaceAll(/\p{Mark}+/gu, "")
|
|
||||||
.replaceAll("ß", "ss")
|
|
||||||
.replaceAll("Æ", "AE")
|
|
||||||
.replaceAll("æ", "ae")
|
|
||||||
.replaceAll("Ø", "O")
|
|
||||||
.replaceAll("ø", "o")
|
|
||||||
.replaceAll("Ł", "L")
|
|
||||||
.replaceAll("ł", "l");
|
|
||||||
}
|
|
||||||
|
|
||||||
function parseReplacement(option: string): readonly [string, string] {
|
|
||||||
let parsed: unknown;
|
|
||||||
try {
|
|
||||||
parsed = JSON.parse(option);
|
|
||||||
} catch {
|
|
||||||
throw new SyntaxError("Literal replacement options are invalid.");
|
|
||||||
}
|
|
||||||
if (
|
|
||||||
!Array.isArray(parsed) ||
|
|
||||||
parsed.length !== 2 ||
|
|
||||||
parsed.some((value) => typeof value !== "string")
|
|
||||||
)
|
|
||||||
throw new SyntaxError(
|
|
||||||
"Literal replacement requires [search, replacement].",
|
|
||||||
);
|
|
||||||
const [search, replacement] = parsed as [string, string];
|
|
||||||
if (!search) throw new SyntaxError("Literal search text must not be empty.");
|
|
||||||
if (search.length > 100_000 || replacement.length > 1_000_000)
|
|
||||||
throw new RangeError(
|
|
||||||
"Literal replacement option exceeds its safety limit.",
|
|
||||||
);
|
|
||||||
return [search, replacement];
|
|
||||||
}
|
|
||||||
|
|
||||||
function decodedDelimiter(option: string): string {
|
|
||||||
if (option.length > 1_000)
|
|
||||||
throw new RangeError("Line join delimiter exceeds 1,000 characters.");
|
|
||||||
return option.replaceAll("\\n", "\n").replaceAll("\\t", "\t");
|
|
||||||
}
|
|
||||||
|
|
||||||
function assertProjectedOutput(units: number): void {
|
|
||||||
if (!Number.isSafeInteger(units) || units > MAX_OUTPUT)
|
|
||||||
throw new RangeError(
|
|
||||||
"Transformation would exceed the 32 MiB output limit.",
|
|
||||||
);
|
|
||||||
}
|
|
||||||
|
|
||||||
function replaceLiteral(
|
|
||||||
value: string,
|
|
||||||
search: string,
|
|
||||||
replacement: string,
|
|
||||||
): string {
|
|
||||||
let matches = 0;
|
|
||||||
let offset = 0;
|
|
||||||
while ((offset = value.indexOf(search, offset)) >= 0) {
|
|
||||||
matches += 1;
|
|
||||||
offset += search.length;
|
|
||||||
assertProjectedOutput(
|
|
||||||
value.length + matches * (replacement.length - search.length),
|
|
||||||
);
|
|
||||||
}
|
|
||||||
return value.replaceAll(search, replacement);
|
|
||||||
}
|
|
||||||
|
|
||||||
function decorateLines(
|
|
||||||
value: string,
|
|
||||||
addition: string,
|
|
||||||
side: "prefix" | "suffix",
|
|
||||||
): string {
|
|
||||||
if (addition.length > 100_000)
|
|
||||||
throw new RangeError("Line decoration exceeds 100,000 UTF-16 units.");
|
|
||||||
const values = lines(value);
|
|
||||||
assertProjectedOutput(value.length + values.length * addition.length);
|
|
||||||
return values
|
|
||||||
.map((line) => (side === "prefix" ? addition + line : line + addition))
|
|
||||||
.join("\n");
|
|
||||||
}
|
|
||||||
|
|
||||||
function applyStep(
|
|
||||||
value: string,
|
|
||||||
step: TransformStep,
|
|
||||||
): { value: string; warning?: string } {
|
|
||||||
switch (step.type) {
|
|
||||||
case "line-endings":
|
|
||||||
return { value: convertLineEndings(value, step.option as LineEnding) };
|
|
||||||
case "trim-lines":
|
|
||||||
return {
|
|
||||||
value: lines(value)
|
|
||||||
.map((line) => line.trim())
|
|
||||||
.join("\n"),
|
|
||||||
};
|
|
||||||
case "trim-document":
|
|
||||||
return { value: value.trim() };
|
|
||||||
case "collapse-whitespace":
|
|
||||||
return {
|
|
||||||
value:
|
|
||||||
step.option === "all"
|
|
||||||
? value.replaceAll(/\s+/gu, " ")
|
|
||||||
: lines(value)
|
|
||||||
.map((line) => line.replaceAll(/[\t ]+/gu, " "))
|
|
||||||
.join("\n"),
|
|
||||||
};
|
|
||||||
case "sort-lines": {
|
|
||||||
const locale = step.option || "en";
|
|
||||||
return {
|
|
||||||
value: lines(value)
|
|
||||||
.map((line, index) => ({ line, index }))
|
|
||||||
.sort(
|
|
||||||
(left, right) =>
|
|
||||||
left.line.localeCompare(right.line, locale, {
|
|
||||||
numeric: true,
|
|
||||||
sensitivity: "variant",
|
|
||||||
}) || left.index - right.index,
|
|
||||||
)
|
|
||||||
.map(({ line }) => line)
|
|
||||||
.join("\n"),
|
|
||||||
warning: `Sort order uses the host Intl.Collator for locale ${locale}.`,
|
|
||||||
};
|
|
||||||
}
|
|
||||||
case "dedupe-lines":
|
|
||||||
return { value: stableUnique(lines(value)).join("\n") };
|
|
||||||
case "case":
|
|
||||||
return { value: transformCase(value, step.option as CaseTransform) };
|
|
||||||
case "normalize":
|
|
||||||
return {
|
|
||||||
value: normalizeUnicode(
|
|
||||||
value,
|
|
||||||
step.option as "NFC" | "NFD" | "NFKC" | "NFKD",
|
|
||||||
),
|
|
||||||
};
|
|
||||||
case "transliterate":
|
|
||||||
return {
|
|
||||||
value: transliterate(value),
|
|
||||||
warning:
|
|
||||||
"Best-effort Latin transliteration is lossy and incomplete; it is not language-aware.",
|
|
||||||
};
|
|
||||||
case "escape": {
|
|
||||||
if (step.option === "json")
|
|
||||||
return { value: JSON.stringify(value).slice(1, -1) };
|
|
||||||
if (step.option === "html") return { value: htmlEscape(value) };
|
|
||||||
if (step.option === "url") return { value: encodeURIComponent(value) };
|
|
||||||
if (step.option === "base64")
|
|
||||||
return { value: bytesToBase64(encodeText(value)) };
|
|
||||||
if (step.option === "hex")
|
|
||||||
return { value: bytesToHex(encodeText(value)) };
|
|
||||||
throw new Error("Unsupported escape target.");
|
|
||||||
}
|
|
||||||
case "unescape":
|
|
||||||
return {
|
|
||||||
value: unescapeValue(value, step.option),
|
|
||||||
warning:
|
|
||||||
step.option === "base64" || step.option === "hex"
|
|
||||||
? "Decoded bytes are required to be well-formed UTF-8 text; arbitrary binary is rejected."
|
|
||||||
: undefined,
|
|
||||||
};
|
|
||||||
case "wrap":
|
|
||||||
return { value: wrapText(value, Number(step.option)) };
|
|
||||||
case "columns":
|
|
||||||
return transformColumns(value, step.option);
|
|
||||||
case "replace-literal": {
|
|
||||||
const [search, replacement] = parseReplacement(step.option);
|
|
||||||
return { value: replaceLiteral(value, search, replacement) };
|
|
||||||
}
|
|
||||||
case "prefix-lines":
|
|
||||||
return { value: decorateLines(value, step.option, "prefix") };
|
|
||||||
case "suffix-lines":
|
|
||||||
return { value: decorateLines(value, step.option, "suffix") };
|
|
||||||
case "filter-lines": {
|
|
||||||
if (!step.option || step.option.length > 100_000)
|
|
||||||
throw new SyntaxError("Line filter text must not be empty.");
|
|
||||||
return {
|
|
||||||
value: lines(value)
|
|
||||||
.filter((line) => line.includes(step.option))
|
|
||||||
.join("\n"),
|
|
||||||
warning: "Line filtering removes every line without the literal text.",
|
|
||||||
};
|
|
||||||
}
|
|
||||||
case "number-lines": {
|
|
||||||
const start = Number(step.option || "1");
|
|
||||||
if (!Number.isSafeInteger(start) || Math.abs(start) > 1_000_000_000)
|
|
||||||
throw new RangeError("Line-number start must be a bounded integer.");
|
|
||||||
return {
|
|
||||||
value: lines(value)
|
|
||||||
.map((line, index) => `${start + index}. ${line}`)
|
|
||||||
.join("\n"),
|
|
||||||
};
|
|
||||||
}
|
|
||||||
case "join-lines": {
|
|
||||||
const values = lines(value);
|
|
||||||
const delimiter = decodedDelimiter(step.option);
|
|
||||||
assertProjectedOutput(
|
|
||||||
values.reduce((total, line) => total + line.length, 0) +
|
|
||||||
Math.max(0, values.length - 1) * delimiter.length,
|
|
||||||
);
|
|
||||||
return { value: values.join(delimiter) };
|
|
||||||
}
|
|
||||||
case "reverse-lines":
|
|
||||||
return { value: lines(value).reverse().join("\n") };
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
export function applyPipeline(
|
|
||||||
input: string,
|
|
||||||
steps: readonly TransformStep[],
|
|
||||||
): PipelineResult {
|
|
||||||
if (input.length > MAX_INPUT)
|
|
||||||
throw new Error(
|
|
||||||
`Input exceeds ${MAX_INPUT.toLocaleString()} UTF-16 units.`,
|
|
||||||
);
|
|
||||||
let output = input;
|
|
||||||
const reports: StepReport[] = [];
|
|
||||||
const warnings: string[] = [];
|
|
||||||
for (const step of steps) {
|
|
||||||
if (!step.enabled) continue;
|
|
||||||
const before = output;
|
|
||||||
const result = applyStep(before, step);
|
|
||||||
output = result.value;
|
|
||||||
if (
|
|
||||||
output.length > MAX_OUTPUT ||
|
|
||||||
output.length > Math.max(1024, input.length * 8)
|
|
||||||
)
|
|
||||||
throw new Error(
|
|
||||||
"Pipeline output exceeded its 8× / 32 MiB expansion limit.",
|
|
||||||
);
|
|
||||||
reports.push({
|
|
||||||
id: step.id,
|
|
||||||
type: step.type,
|
|
||||||
beforeUnits: before.length,
|
|
||||||
afterUnits: output.length,
|
|
||||||
changed: before !== output,
|
|
||||||
warning: result.warning,
|
|
||||||
});
|
|
||||||
if (result.warning) warnings.push(result.warning);
|
|
||||||
}
|
|
||||||
return { output, steps: reports, warnings: [...new Set(warnings)] };
|
|
||||||
}
|
|
||||||
|
|
||||||
export function createStep(type: StepType): TransformStep {
|
|
||||||
const defaults: Record<StepType, string> = {
|
|
||||||
"line-endings": "lf",
|
|
||||||
"trim-lines": "",
|
|
||||||
"trim-document": "",
|
|
||||||
"collapse-whitespace": "line",
|
|
||||||
"sort-lines": "en",
|
|
||||||
"dedupe-lines": "",
|
|
||||||
case: "lower",
|
|
||||||
normalize: "NFC",
|
|
||||||
transliterate: "",
|
|
||||||
escape: "json",
|
|
||||||
unescape: "json",
|
|
||||||
wrap: "80",
|
|
||||||
columns: "csv|,|1",
|
|
||||||
"replace-literal": JSON.stringify(["old", "new"]),
|
|
||||||
"prefix-lines": "> ",
|
|
||||||
"suffix-lines": "",
|
|
||||||
"filter-lines": "text",
|
|
||||||
"number-lines": "1",
|
|
||||||
"join-lines": " ",
|
|
||||||
"reverse-lines": "",
|
|
||||||
};
|
|
||||||
return {
|
|
||||||
id: `step-${++nextStepId}`,
|
|
||||||
type,
|
|
||||||
enabled: true,
|
|
||||||
option: defaults[type],
|
|
||||||
};
|
|
||||||
}
|
|
||||||
|
|
||||||
export function textInventory(value: string) {
|
|
||||||
let codePoints = 0;
|
|
||||||
for (let index = 0; index < value.length; index += 1) {
|
|
||||||
const unit = value.charCodeAt(index);
|
|
||||||
if (
|
|
||||||
unit >= 0xd800 &&
|
|
||||||
unit <= 0xdbff &&
|
|
||||||
index + 1 < value.length &&
|
|
||||||
value.charCodeAt(index + 1) >= 0xdc00 &&
|
|
||||||
value.charCodeAt(index + 1) <= 0xdfff
|
|
||||||
)
|
|
||||||
index += 1;
|
|
||||||
codePoints += 1;
|
|
||||||
}
|
|
||||||
return {
|
|
||||||
utf16Units: value.length,
|
|
||||||
codePoints,
|
|
||||||
lines: value ? value.split(/\r\n|\r|\n/u).length : 0,
|
|
||||||
crlf: (value.match(/\r\n/gu) ?? []).length,
|
|
||||||
bareLf: (value.match(/(?<!\r)\n/gu) ?? []).length,
|
|
||||||
bareCr: (value.match(/\r(?!\n)/gu) ?? []).length,
|
|
||||||
finalNewline: /(?:\r\n|\r|\n)$/u.test(value),
|
|
||||||
};
|
|
||||||
}
|
|
||||||
@@ -1,22 +1,21 @@
|
|||||||
{
|
{
|
||||||
"$schema": "https://git.add-ideas.de/lotobo/toolbox-sdk/raw/branch/main/schemas/toolbox-app.v1.schema.json",
|
"$schema": "https://git.add-ideas.de/lotobo/toolbox-sdk/raw/branch/main/schemas/toolbox-app.v1.schema.json",
|
||||||
"schemaVersion": 1,
|
"schemaVersion": 1,
|
||||||
"id": "de.add-ideas.text-tools",
|
"id": "de.add-ideas.translator-tools",
|
||||||
"name": "Text Tools",
|
"name": "Translator Tools",
|
||||||
"version": "0.2.0",
|
"version": "0.1.0",
|
||||||
"description": "Compose text transforms and export encoding evidence locally.",
|
"description": "Review and edit translation bundles in the browser with glossary suggestions.",
|
||||||
"entry": "./",
|
"entry": "./",
|
||||||
"icon": "./favicon.svg",
|
"icon": "./favicon.svg",
|
||||||
"categories": ["text", "developer", "productivity"],
|
"categories": ["text", "developer", "productivity", "i18n"],
|
||||||
"tags": [
|
"tags": [
|
||||||
"text",
|
"translation",
|
||||||
"unicode",
|
"i18n",
|
||||||
"pipeline",
|
"locale",
|
||||||
"normalize",
|
"glossary",
|
||||||
"sort",
|
"review",
|
||||||
"escape",
|
"json",
|
||||||
"encoding",
|
"localization"
|
||||||
"newline"
|
|
||||||
],
|
],
|
||||||
"integration": {
|
"integration": {
|
||||||
"contextVersion": 1,
|
"contextVersion": 1,
|
||||||
@@ -26,32 +25,30 @@
|
|||||||
"requirements": {
|
"requirements": {
|
||||||
"secureContext": false,
|
"secureContext": false,
|
||||||
"workers": false,
|
"workers": false,
|
||||||
"indexedDb": false,
|
"indexedDb": true,
|
||||||
"crossOriginIsolated": false,
|
"crossOriginIsolated": false,
|
||||||
"topLevelContext": false
|
"topLevelContext": false
|
||||||
},
|
},
|
||||||
"io": {
|
"io": {
|
||||||
"accepts": [
|
"accepts": [
|
||||||
{
|
{
|
||||||
"mediaType": "text/*",
|
"mediaType": "application/json",
|
||||||
"extensions": [".txt", ".csv", ".md", ".log"],
|
"extensions": [".json", ".js"],
|
||||||
"label": "Bounded text files"
|
"label": "Locale JSON or i18next-like translation files"
|
||||||
}
|
}
|
||||||
],
|
],
|
||||||
"produces": [
|
"produces": [
|
||||||
{
|
|
||||||
"mediaType": "text/plain",
|
|
||||||
"extensions": [".txt"],
|
|
||||||
"label": "Transformed text"
|
|
||||||
},
|
|
||||||
{
|
{
|
||||||
"mediaType": "application/json",
|
"mediaType": "application/json",
|
||||||
"extensions": [".json"],
|
"extensions": [".json"],
|
||||||
"label": "Recipe and artifact evidence"
|
"label": "Reviewed translation bundle"
|
||||||
}
|
}
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
"capabilities": { "required": [], "optional": ["web-crypto"] },
|
"capabilities": {
|
||||||
|
"required": [],
|
||||||
|
"optional": ["indexeddb", "local-artifact-handoff"]
|
||||||
|
},
|
||||||
"privacy": {
|
"privacy": {
|
||||||
"processing": "local",
|
"processing": "local",
|
||||||
"fileUploads": true,
|
"fileUploads": true,
|
||||||
@@ -59,14 +56,14 @@
|
|||||||
"label": "Inputs stay in this browser; nothing is uploaded."
|
"label": "Inputs stay in this browser; nothing is uploaded."
|
||||||
},
|
},
|
||||||
"source": {
|
"source": {
|
||||||
"repository": "https://git.add-ideas.de/lotobo/text-tools",
|
"repository": "https://git.add-ideas.de/lotobo/translator-tools",
|
||||||
"license": "GPL-3.0-or-later"
|
"license": "GPL-3.0-or-later"
|
||||||
},
|
},
|
||||||
"actions": [
|
"actions": [
|
||||||
{
|
{
|
||||||
"id": "source",
|
"id": "source",
|
||||||
"label": "Source",
|
"label": "Source",
|
||||||
"url": "https://git.add-ideas.de/lotobo/text-tools"
|
"url": "https://git.add-ideas.de/lotobo/translator-tools"
|
||||||
}
|
}
|
||||||
]
|
]
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -0,0 +1,146 @@
|
|||||||
|
import {
|
||||||
|
TOOLBOX_TRANSFER_QUERY_PARAMETER,
|
||||||
|
consumeToolboxTransfer,
|
||||||
|
createToolboxTransfer,
|
||||||
|
createToolboxTransferUrl,
|
||||||
|
loadToolboxContext,
|
||||||
|
readToolboxTransferToken,
|
||||||
|
type ToolboxTransfer,
|
||||||
|
} from "@add-ideas/toolbox-contract";
|
||||||
|
import { APP_VERSION } from "../version";
|
||||||
|
|
||||||
|
export const TRANS_TOOL_ID = "de.add-ideas.translator-tools";
|
||||||
|
export const PORTAL_APP_ID = "de.add-ideas.toolbox-portal";
|
||||||
|
export const MAX_TRANSLATOR_TRANSFER_BYTES = 5 * 1024 * 1024;
|
||||||
|
|
||||||
|
interface ReceiveDependencies {
|
||||||
|
consume: typeof consumeToolboxTransfer;
|
||||||
|
readToken: typeof readToolboxTransferToken;
|
||||||
|
replaceUrl: (url: string) => void;
|
||||||
|
}
|
||||||
|
|
||||||
|
const receiveDefaults: ReceiveDependencies = {
|
||||||
|
consume: consumeToolboxTransfer,
|
||||||
|
readToken: readToolboxTransferToken,
|
||||||
|
replaceUrl: (url) =>
|
||||||
|
window.history.replaceState(window.history.state, "", url),
|
||||||
|
};
|
||||||
|
|
||||||
|
export type IncomingTranslatorTransfer =
|
||||||
|
| { status: "none" | "missing"; error?: string }
|
||||||
|
| { status: "ready"; file: File; transfer: ToolboxTransfer };
|
||||||
|
|
||||||
|
export async function receiveTranslatorTransfer(
|
||||||
|
href: string | URL = window.location.href,
|
||||||
|
dependencies: ReceiveDependencies = receiveDefaults,
|
||||||
|
): Promise<IncomingTranslatorTransfer> {
|
||||||
|
const url = new URL(href);
|
||||||
|
if (!url.searchParams.has(TOOLBOX_TRANSFER_QUERY_PARAMETER))
|
||||||
|
return { status: "none" };
|
||||||
|
|
||||||
|
let token: string | undefined;
|
||||||
|
try {
|
||||||
|
token = dependencies.readToken(url);
|
||||||
|
} finally {
|
||||||
|
url.searchParams.delete(TOOLBOX_TRANSFER_QUERY_PARAMETER);
|
||||||
|
dependencies.replaceUrl(`${url.pathname}${url.search}${url.hash}`);
|
||||||
|
}
|
||||||
|
|
||||||
|
if (!token) return { status: "missing" };
|
||||||
|
|
||||||
|
const transfer = await dependencies.consume(token, TRANS_TOOL_ID);
|
||||||
|
if (!transfer) return { status: "missing" };
|
||||||
|
|
||||||
|
if (transfer.files.length !== 1)
|
||||||
|
throw new RangeError(
|
||||||
|
"Translator Tools accepts exactly one file per handoff.",
|
||||||
|
);
|
||||||
|
|
||||||
|
const source = transfer.files[0];
|
||||||
|
if (!source) throw new RangeError("The handoff did not contain a file.");
|
||||||
|
|
||||||
|
if (!source.size || source.size > MAX_TRANSLATOR_TRANSFER_BYTES)
|
||||||
|
throw new RangeError(
|
||||||
|
"Transferred translation files must be between 1 byte and 5 MiB.",
|
||||||
|
);
|
||||||
|
|
||||||
|
if (source.size !== source.blob.size)
|
||||||
|
throw new RangeError("Transferred size evidence does not match the file.");
|
||||||
|
|
||||||
|
const mediaType = source.mediaType.split(";", 1)[0]!.toLowerCase();
|
||||||
|
if (
|
||||||
|
!mediaType.startsWith("application/json") &&
|
||||||
|
!/\.(?:json|js|txt)$/iu.test(source.name)
|
||||||
|
) {
|
||||||
|
throw new TypeError(
|
||||||
|
"The transferred file is not a supported locale/translation document.",
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
return {
|
||||||
|
status: "ready",
|
||||||
|
file: new File([source.blob], source.name, {
|
||||||
|
type: source.mediaType || source.blob.type,
|
||||||
|
}),
|
||||||
|
transfer,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
interface SendDependencies {
|
||||||
|
createTransfer: typeof createToolboxTransfer;
|
||||||
|
loadContext: typeof loadToolboxContext;
|
||||||
|
}
|
||||||
|
|
||||||
|
const sendDefaults: SendDependencies = {
|
||||||
|
createTransfer: createToolboxTransfer,
|
||||||
|
loadContext: loadToolboxContext,
|
||||||
|
};
|
||||||
|
|
||||||
|
export async function prepareTranslatorPortalHandoff(
|
||||||
|
blob: Blob,
|
||||||
|
name: string,
|
||||||
|
dependencies: SendDependencies = sendDefaults,
|
||||||
|
): Promise<URL> {
|
||||||
|
if (!blob.size || blob.size > MAX_TRANSLATOR_TRANSFER_BYTES)
|
||||||
|
throw new RangeError(
|
||||||
|
"Translator exports must be between 1 byte and 5 MiB.",
|
||||||
|
);
|
||||||
|
|
||||||
|
const context = await dependencies.loadContext();
|
||||||
|
if (context.status !== "ready")
|
||||||
|
throw new Error(
|
||||||
|
"Open Translator Tools from the Toolbox Portal to send this output locally.",
|
||||||
|
);
|
||||||
|
|
||||||
|
const transfer = await dependencies.createTransfer({
|
||||||
|
source: { appId: TRANS_TOOL_ID, appVersion: APP_VERSION },
|
||||||
|
targetAppId: PORTAL_APP_ID,
|
||||||
|
files: [
|
||||||
|
{
|
||||||
|
blob,
|
||||||
|
name,
|
||||||
|
mediaType: blob.type || "application/json",
|
||||||
|
size: blob.size,
|
||||||
|
},
|
||||||
|
],
|
||||||
|
evidence: {
|
||||||
|
formatVersion: "1",
|
||||||
|
operation: "translator-workspace-output",
|
||||||
|
settings: {
|
||||||
|
preferredDestinationAppIds: [
|
||||||
|
"de.add-ideas.regex-tools",
|
||||||
|
"de.add-ideas.data-tools",
|
||||||
|
],
|
||||||
|
},
|
||||||
|
warnings: [
|
||||||
|
"Translator output uses your current local edits; review authoritative and target pairs before downstream use.",
|
||||||
|
],
|
||||||
|
},
|
||||||
|
});
|
||||||
|
|
||||||
|
return createToolboxTransferUrl(
|
||||||
|
context.context.catalog.homeUrl,
|
||||||
|
transfer.token,
|
||||||
|
{ location: context.context.catalog.homeUrl },
|
||||||
|
);
|
||||||
|
}
|
||||||
+1
-1
@@ -1 +1 @@
|
|||||||
export const APP_VERSION = "0.2.0";
|
export const APP_VERSION = "0.1.0";
|
||||||
|
|||||||
@@ -1,91 +0,0 @@
|
|||||||
import { expect, test, type Page } from "@playwright/test";
|
|
||||||
import { Buffer } from "node:buffer";
|
|
||||||
|
|
||||||
const ORIGIN = "http://127.0.0.1:4173";
|
|
||||||
async function localOnly(page: Page) {
|
|
||||||
const external: string[] = [];
|
|
||||||
await page.route("**/*", async (route) => {
|
|
||||||
const url = new URL(route.request().url());
|
|
||||||
if (url.origin !== ORIGIN) {
|
|
||||||
external.push(url.href);
|
|
||||||
await route.abort();
|
|
||||||
} else await route.continue();
|
|
||||||
});
|
|
||||||
return external;
|
|
||||||
}
|
|
||||||
|
|
||||||
test("runs from a nested path without external requests", async ({ page }) => {
|
|
||||||
const errors: string[] = [];
|
|
||||||
page.on("pageerror", (error) => errors.push(error.message));
|
|
||||||
page.on("console", (message) => {
|
|
||||||
if (message.type() === "error") errors.push(message.text());
|
|
||||||
});
|
|
||||||
const external = await localOnly(page);
|
|
||||||
await page.goto("/deep/nested/text/");
|
|
||||||
await expect(page.getByRole("heading", { name: "Text Tools" })).toBeVisible();
|
|
||||||
expect(external).toEqual([]);
|
|
||||||
expect(errors).toEqual([]);
|
|
||||||
});
|
|
||||||
|
|
||||||
test("applies the visible text pipeline and preserves its last output", async ({
|
|
||||||
page,
|
|
||||||
}) => {
|
|
||||||
const external = await localOnly(page);
|
|
||||||
await page.goto("/deep/nested/text/");
|
|
||||||
await page.getByLabel("Text source").fill(" Alpha \r\nAlpha\n");
|
|
||||||
await page.getByRole("button", { name: "Apply pipeline" }).click();
|
|
||||||
await expect(page.getByLabel("Transformed output")).toHaveValue("Alpha\n");
|
|
||||||
await expect(
|
|
||||||
page.getByRole("cell", { name: "Trim every line" }),
|
|
||||||
).toBeVisible();
|
|
||||||
await expect(
|
|
||||||
page.getByRole("cell", { name: "Deduplicate lines" }),
|
|
||||||
).toBeVisible();
|
|
||||||
expect(external).toEqual([]);
|
|
||||||
});
|
|
||||||
|
|
||||||
test("shows byte evidence and composes an added literal transform", async ({
|
|
||||||
page,
|
|
||||||
}) => {
|
|
||||||
await page.goto("/deep/nested/text/");
|
|
||||||
await page.getByLabel("Decode as").selectOption("utf-8");
|
|
||||||
await page.locator('input[type="file"]').setInputFiles({
|
|
||||||
name: "bom.txt",
|
|
||||||
mimeType: "text/plain",
|
|
||||||
buffer: Buffer.from([0xef, 0xbb, 0xbf, ...Buffer.from("old\r\nold")]),
|
|
||||||
});
|
|
||||||
await page.getByText("Byte decoding evidence").click();
|
|
||||||
await expect(page.getByText("utf-8 / utf-8")).toBeVisible();
|
|
||||||
await expect(
|
|
||||||
page
|
|
||||||
.locator("details")
|
|
||||||
.filter({ hasText: "Byte decoding evidence" })
|
|
||||||
.getByText("1 / 0 / 0", { exact: true }),
|
|
||||||
).toBeVisible();
|
|
||||||
await page.getByLabel("New transformation").selectOption("replace-literal");
|
|
||||||
await page.getByRole("button", { name: "Add step" }).click();
|
|
||||||
await page.getByLabel("Literal search text").fill("old");
|
|
||||||
await page.getByLabel("Literal replacement text").fill("new");
|
|
||||||
await page.getByRole("button", { name: "Apply pipeline" }).click();
|
|
||||||
await expect(page.getByLabel("Transformed output")).toHaveValue("new");
|
|
||||||
await expect(
|
|
||||||
page.getByRole("button", { name: "Export artifact + evidence" }),
|
|
||||||
).toBeVisible();
|
|
||||||
});
|
|
||||||
|
|
||||||
test("serves the release identity and hardened headers", async ({
|
|
||||||
request,
|
|
||||||
}) => {
|
|
||||||
const index = await request.get("/deep/nested/text/");
|
|
||||||
expect(index.ok()).toBe(true);
|
|
||||||
expect(index.headers()["content-security-policy"]).toContain(
|
|
||||||
"default-src 'self'",
|
|
||||||
);
|
|
||||||
expect(await index.text()).not.toMatch(/\b(?:src|href)=["']\//u);
|
|
||||||
const manifest = await request.get("/deep/nested/text/toolbox-app.json");
|
|
||||||
await expect(manifest.json()).resolves.toMatchObject({
|
|
||||||
id: "de.add-ideas.text-tools",
|
|
||||||
version: "0.2.0",
|
|
||||||
entry: "./",
|
|
||||||
});
|
|
||||||
});
|
|
||||||
@@ -1,18 +0,0 @@
|
|||||||
import { expect, test } from "@playwright/test";
|
|
||||||
|
|
||||||
test("keeps the primary workspace inside a narrow viewport", async ({
|
|
||||||
page,
|
|
||||||
}) => {
|
|
||||||
await page.goto("/deep/nested/text/");
|
|
||||||
await expect(page.locator("main").first()).toBeVisible();
|
|
||||||
await expect(
|
|
||||||
page.locator("main .loading, main .workbench-loading"),
|
|
||||||
).toHaveCount(0);
|
|
||||||
|
|
||||||
const widths = await page.evaluate(() => ({
|
|
||||||
content: document.documentElement.scrollWidth,
|
|
||||||
viewport: document.documentElement.clientWidth,
|
|
||||||
}));
|
|
||||||
expect(widths.viewport).toBeLessThanOrEqual(430);
|
|
||||||
expect(widths.content).toBeLessThanOrEqual(widths.viewport + 1);
|
|
||||||
});
|
|
||||||
@@ -1,17 +0,0 @@
|
|||||||
import { render, screen } from "@testing-library/react";
|
|
||||||
import { describe, expect, it, vi } from "vitest";
|
|
||||||
import { App } from "../../src/App";
|
|
||||||
|
|
||||||
describe("Text Tools", () => {
|
|
||||||
it("renders the local workbench and standard shell", async () => {
|
|
||||||
vi.stubGlobal(
|
|
||||||
"fetch",
|
|
||||||
vi.fn(async () => new Response("Not found", { status: 404 })),
|
|
||||||
);
|
|
||||||
render(<App />);
|
|
||||||
expect(
|
|
||||||
await screen.findByRole("heading", { name: "Text Tools" }),
|
|
||||||
).toBeVisible();
|
|
||||||
expect(await screen.findByText("Browser-local")).toBeVisible();
|
|
||||||
});
|
|
||||||
});
|
|
||||||
@@ -0,0 +1,13 @@
|
|||||||
|
import { describe, expect, it } from "vitest";
|
||||||
|
import { manifest } from "../src/toolbox/manifest";
|
||||||
|
import { APP_VERSION } from "../src/version";
|
||||||
|
|
||||||
|
describe("translator-tools smoke", () => {
|
||||||
|
it("exposes a coherent toolbox manifest", () => {
|
||||||
|
expect(manifest.id).toBe("de.add-ideas.translator-tools");
|
||||||
|
expect(manifest.name).toBe("Translator Tools");
|
||||||
|
expect(manifest.version).toBe(APP_VERSION);
|
||||||
|
expect(manifest.categories).toContain("text");
|
||||||
|
expect(manifest.entry).toBe("./");
|
||||||
|
});
|
||||||
|
});
|
||||||
@@ -1,147 +0,0 @@
|
|||||||
import { describe, expect, it } from "vitest";
|
|
||||||
import {
|
|
||||||
applyPipeline,
|
|
||||||
textInventory,
|
|
||||||
type TransformStep,
|
|
||||||
} from "../../src/text/pipeline";
|
|
||||||
import {
|
|
||||||
createTextArtifactEvidence,
|
|
||||||
decodeTextWithEvidence,
|
|
||||||
} from "../../src/text/evidence";
|
|
||||||
|
|
||||||
const step = (type: TransformStep["type"], option = ""): TransformStep => ({
|
|
||||||
id: type,
|
|
||||||
type,
|
|
||||||
option,
|
|
||||||
enabled: true,
|
|
||||||
});
|
|
||||||
describe("text pipelines", () => {
|
|
||||||
it("applies transformations in the declared order", () =>
|
|
||||||
expect(
|
|
||||||
applyPipeline(" b \nA\na", [
|
|
||||||
step("trim-lines"),
|
|
||||||
step("dedupe-lines"),
|
|
||||||
step("sort-lines", "en"),
|
|
||||||
]).output,
|
|
||||||
).toBe("a\nA\nb"));
|
|
||||||
it("keeps disabled steps out of the report", () =>
|
|
||||||
expect(
|
|
||||||
applyPipeline("A", [{ ...step("case", "lower"), enabled: false }]),
|
|
||||||
).toEqual({ output: "A", steps: [], warnings: [] }));
|
|
||||||
it("escapes HTML as data", () =>
|
|
||||||
expect(
|
|
||||||
applyPipeline("<img src=x onerror=1>", [step("escape", "html")]).output,
|
|
||||||
).toBe("<img src=x onerror=1>"));
|
|
||||||
it("strictly reverses each escape/encoding stage", () => {
|
|
||||||
const source = 'café <x> & "quoted"\n';
|
|
||||||
for (const option of ["json", "html", "url", "base64", "hex"])
|
|
||||||
expect(
|
|
||||||
applyPipeline(applyPipeline(source, [step("escape", option)]).output, [
|
|
||||||
step("unescape", option),
|
|
||||||
]).output,
|
|
||||||
).toBe(source);
|
|
||||||
expect(() => applyPipeline("©", [step("unescape", "html")])).toThrow(
|
|
||||||
/not emitted/u,
|
|
||||||
);
|
|
||||||
expect(() => applyPipeline("w6k", [step("unescape", "base64")])).toThrow(
|
|
||||||
/canonical|padding|UTF-8/u,
|
|
||||||
);
|
|
||||||
expect(() => applyPipeline("%ZZ", [step("unescape", "url")])).toThrow();
|
|
||||||
});
|
|
||||||
it("selects columns from quoted CSV records including embedded newlines", () => {
|
|
||||||
const result = applyPipeline(
|
|
||||||
'name,note,id\r\nAlice,"hello, world",1\r\nBob,"two\nlines",2\r\n',
|
|
||||||
[step("columns", "csv|,|3,1,2")],
|
|
||||||
);
|
|
||||||
expect(result.output).toContain('1,Alice,"hello, world"');
|
|
||||||
expect(result.output).toContain('2,Bob,"two\nlines"');
|
|
||||||
expect(result.warnings[0]).toMatch(/Quoted CSV/u);
|
|
||||||
});
|
|
||||||
it("retains multi-character literal-delimiter mode", () =>
|
|
||||||
expect(
|
|
||||||
applyPipeline("left::right\na::b", [step("columns", "literal|::|2,1")])
|
|
||||||
.output,
|
|
||||||
).toBe("right::left\nb::a"));
|
|
||||||
it("reports lossy transliteration", () =>
|
|
||||||
expect(applyPipeline("Crème", [step("transliterate")]).warnings[0]).toMatch(
|
|
||||||
/lossy/u,
|
|
||||||
));
|
|
||||||
it("tracks exact line endings", () =>
|
|
||||||
expect(textInventory("a\r\nb\nc\r")).toMatchObject({
|
|
||||||
crlf: 1,
|
|
||||||
bareLf: 1,
|
|
||||||
bareCr: 1,
|
|
||||||
finalNewline: true,
|
|
||||||
}));
|
|
||||||
it("composes literal replacement, filtering, numbering and line decoration", () => {
|
|
||||||
const result = applyPipeline("old apple\nold pear\nbanana", [
|
|
||||||
step("replace-literal", JSON.stringify(["old", "fresh"])),
|
|
||||||
step("filter-lines", "fresh"),
|
|
||||||
step("prefix-lines", "- "),
|
|
||||||
step("number-lines", "10"),
|
|
||||||
step("suffix-lines", "!"),
|
|
||||||
step("join-lines", " | "),
|
|
||||||
]);
|
|
||||||
expect(result.output).toBe("10. - fresh apple! | 11. - fresh pear!");
|
|
||||||
expect(result.warnings).toEqual(
|
|
||||||
expect.arrayContaining([expect.stringMatching(/filtering removes/u)]),
|
|
||||||
);
|
|
||||||
});
|
|
||||||
|
|
||||||
it("reverses lines without reversing grapheme contents", () =>
|
|
||||||
expect(applyPipeline("😀a\nb", [step("reverse-lines")]).output).toBe(
|
|
||||||
"b\n😀a",
|
|
||||||
));
|
|
||||||
|
|
||||||
it("preflights expanding transforms before allocating their output", () => {
|
|
||||||
expect(() =>
|
|
||||||
applyPipeline("x".repeat(40), [
|
|
||||||
step("replace-literal", JSON.stringify(["x", "y".repeat(1_000_000)])),
|
|
||||||
]),
|
|
||||||
).toThrow(/32 MiB/u);
|
|
||||||
expect(() =>
|
|
||||||
applyPipeline("a\nb", [step("prefix-lines", "x".repeat(100_001))]),
|
|
||||||
).toThrow(/100,000/u);
|
|
||||||
});
|
|
||||||
});
|
|
||||||
|
|
||||||
describe("byte and portable artifact evidence", () => {
|
|
||||||
it("records BOM conflicts, UTF-8 validity and byte newline forms", () => {
|
|
||||||
const bytes = Uint8Array.of(0xef, 0xbb, 0xbf, 0x61, 0x0d, 0x0a, 0x62);
|
|
||||||
const decoded = decodeTextWithEvidence(bytes, "latin1", true);
|
|
||||||
expect(decoded.evidence).toMatchObject({
|
|
||||||
byteLength: 7,
|
|
||||||
bom: { encoding: "utf-8", bytes: 3 },
|
|
||||||
utf8: { valid: true },
|
|
||||||
byteNewlines: { crlf: 1, bareLf: 0, bareCr: 0 },
|
|
||||||
});
|
|
||||||
expect(decoded.evidence.warnings.join(" ")).toMatch(/conflicts/u);
|
|
||||||
const malformed = decodeTextWithEvidence(
|
|
||||||
Uint8Array.of(0xc3, 0x28),
|
|
||||||
"latin1",
|
|
||||||
true,
|
|
||||||
);
|
|
||||||
expect(malformed.evidence.utf8).toEqual({
|
|
||||||
valid: false,
|
|
||||||
firstInvalidOffset: 0,
|
|
||||||
});
|
|
||||||
});
|
|
||||||
|
|
||||||
it("emits exact output hashes and an honest handoff marker", async () => {
|
|
||||||
const steps = [step("case", "upper")];
|
|
||||||
const pipeline = applyPipeline("hello", steps);
|
|
||||||
const bytes = new TextEncoder().encode(pipeline.output);
|
|
||||||
const evidence = await createTextArtifactEvidence({
|
|
||||||
sourceName: "input.txt",
|
|
||||||
sourceText: "hello",
|
|
||||||
outputName: "output.txt",
|
|
||||||
outputEncoding: "utf-8",
|
|
||||||
outputBytes: bytes,
|
|
||||||
pipeline,
|
|
||||||
steps,
|
|
||||||
});
|
|
||||||
expect(evidence.output.sha256).toMatch(/^[a-f0-9]{64}$/u);
|
|
||||||
expect(evidence.handoff.supportedByThisBuild).toBe(false);
|
|
||||||
expect(evidence.pipeline.steps[0]).toMatchObject({ type: "case" });
|
|
||||||
});
|
|
||||||
});
|
|
||||||
Reference in New Issue
Block a user