Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
 zemion
|
424605e61f |
@@ -0,0 +1,39 @@
|
||||
name: Verify
|
||||
|
||||
on:
|
||||
push:
|
||||
branches: [main]
|
||||
pull_request:
|
||||
workflow_dispatch:
|
||||
|
||||
concurrency:
|
||||
group: verify-${{ gitea.repository }}-${{ gitea.ref }}
|
||||
cancel-in-progress: true
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
jobs:
|
||||
verify:
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 45
|
||||
env:
|
||||
CI: "true"
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
- uses: actions/setup-node@v4
|
||||
with:
|
||||
node-version: "22"
|
||||
cache: npm
|
||||
- name: Select declared npm version
|
||||
run: npm install --global npm@11.17.0
|
||||
- name: Install dependencies
|
||||
run: npm ci
|
||||
- name: Audit runtime dependencies
|
||||
run: npm audit --omit=dev --audit-level=moderate
|
||||
- name: Check, test, and build
|
||||
run: npm run check
|
||||
- name: Install browser engines
|
||||
run: npx playwright install --with-deps chromium firefox webkit
|
||||
- name: Browser tests
|
||||
run: npm run test:browser
|
||||
@@ -1,5 +1,10 @@
|
||||
# Changelog
|
||||
|
||||
## 0.2.0 - 2026-09-02
|
||||
|
||||
- Added lazy, pinned UTS #39 Identifier_Status/Identifier_Type evidence, skeleton comparison, restriction estimates, and structured confusable/invisible/control diagnostics.
|
||||
- Added detailed grapheme/emoji structure inspection and bounded presentation-selector, skin-tone and ZWJ construction with explicit non-RGI/font-support limits; added Toolbox I/O/capability metadata.
|
||||
|
||||
## 0.1.0 - 2026-09-01
|
||||
|
||||
- Added the initial local-first Unicode Tools workbench.
|
||||
|
||||
@@ -4,26 +4,36 @@ Explore Unicode characters, scripts, emoji and text locally.
|
||||
|
||||
Unicode Tools is a standalone local-first application in the [add·ideas Toolbox](https://git.add-ideas.de/lotobo/toolbox-portal). Inputs are processed in the browser and are not uploaded.
|
||||
|
||||
## Version 0.1.0 scope
|
||||
## Version 0.2.0 scope
|
||||
|
||||
- Search 159,866 assigned Unicode scalar records by official name, alias,
|
||||
literal character, or code point, with script, block and category filters.
|
||||
- Traverse any block or hexadecimal range in a 16-column virtualized code
|
||||
chart with roving keyboard navigation and property cards. Only visible rows
|
||||
are materialized, including when the entire U+0000–U+10FFFF space is open.
|
||||
- Browse and copy 3,952 fully-qualified emoji and components, inspect named
|
||||
sequences, and open practical symbol collections.
|
||||
sequences, open practical symbol collections, and construct bounded
|
||||
presentation-selector, skin-tone and ZWJ variants without claiming RGI
|
||||
validity or font support.
|
||||
- Inspect text as Unicode scalar values, UTF-16 positions, UTF-8 bytes,
|
||||
JavaScript and HTML escapes, and browser-segmented grapheme clusters.
|
||||
JavaScript and HTML escapes, and browser-segmented grapheme clusters with
|
||||
emoji/ZWJ/variation/modifier/flag/keycap/tag evidence.
|
||||
- Decode lists of hexadecimal scalar values and compare NFC, NFD, NFKC and
|
||||
NFKD normalization without losing sight of the original value.
|
||||
- Compute a UTS #39 confusable skeleton, show applied mappings, and flag mixed
|
||||
primary scripts as a review signal rather than a security verdict.
|
||||
- Compute UTS #39 confusable skeletons, compare skeleton collisions, display
|
||||
pinned Identifier_Status/Identifier_Type evidence, and flag mixed scripts,
|
||||
controls, default ignorables, combining repetition and non-NFC text as
|
||||
review signals rather than security verdicts.
|
||||
- Inspect character properties and optionally load selected Unihan readings,
|
||||
definitions and radical/stroke fields for Han ideographs.
|
||||
- Download an auditable source-identity report for every pinned data input.
|
||||
|
||||
The catalog is generated from checksum-pinned Unicode Consortium 17.0 data. It
|
||||
does not scrape or redistribute descriptions, artwork, or annotations from
|
||||
Symbl or another symbol website. Private-use code points and UTF-16 surrogates
|
||||
remain visible as ranges, but are not presented as assigned characters.
|
||||
Symbl or another symbol website. Unassigned scalars, private-use scalars,
|
||||
permanently reserved noncharacters and UTF-16 surrogate code units are labelled
|
||||
as distinct chart states; none is presented as an assigned character. Lone
|
||||
surrogate copy is disabled because a surrogate is not a Unicode scalar value.
|
||||
|
||||
The application treats every input as untrusted, applies explicit resource
|
||||
limits, and never interprets text as active content. The checked-in catalog can
|
||||
@@ -44,7 +54,7 @@ npm run test:browser
|
||||
|
||||
## Release
|
||||
|
||||
`npm run release:artifact` creates a deterministic `release/unicode-tools-0.1.0.zip` and checksum sidecar.
|
||||
`npm run release:artifact` creates a deterministic `release/unicode-tools-0.2.0.zip` and checksum sidecar.
|
||||
|
||||
## Licence
|
||||
|
||||
|
||||
@@ -1,8 +1,8 @@
|
||||
# Corresponding source
|
||||
|
||||
The corresponding source for Unicode Tools 0.1.0 is available at:
|
||||
The corresponding source for Unicode Tools 0.2.0 is available at:
|
||||
|
||||
https://git.add-ideas.de/lotobo/unicode-tools/src/tag/v0.1.0
|
||||
https://git.add-ideas.de/lotobo/unicode-tools/src/tag/v0.2.0
|
||||
|
||||
Build with Node.js 22, npm 11, `npm ci`, and `npm run release:artifact`.
|
||||
The preferred source includes the generated JSON catalog in `public/data/` so
|
||||
|
||||
@@ -14,7 +14,7 @@ Inc.
|
||||
## Runtime code
|
||||
|
||||
Runtime dependency licences are reproduced in
|
||||
`LICENSES/npm-runtime-licenses.txt` in release artifacts. Toolbox SDK packages
|
||||
are Apache-2.0. The reusable Toolbox Helpers package is GPL-3.0-or-later.
|
||||
`LICENSES/npm-runtime-licenses.txt` in release artifacts. Toolbox SDK 0.3.0
|
||||
packages are Apache-2.0. Toolbox Helpers 0.2.0 is GPL-3.0-or-later.
|
||||
React and React DOM are MIT. The build-only `fflate` dependency is MIT and is
|
||||
used by the catalog generator, not shipped as runtime code.
|
||||
|
||||
+15
-2
@@ -17,8 +17,21 @@ persistent search history.
|
||||
archives and text files into deterministic checked-in source data. It derives
|
||||
names, aliases, general categories, scripts, blocks, ages, bidi and case
|
||||
properties, emoji, named sequences, selected Unihan fields, and UTS #39
|
||||
confusable mappings. Private-use and surrogate ranges are not expanded into
|
||||
synthetic catalog records.
|
||||
confusable mappings. The code chart virtualizes complete Unicode block or
|
||||
custom-range traversal into fixed row windows, requesting at most 512 entries
|
||||
from the worker. Assigned records come from the pinned catalog; missing entries
|
||||
are classified explicitly as unassigned, private-use, surrogate, or
|
||||
noncharacter instead of being omitted. Private-use and surrogate ranges are not
|
||||
expanded into the checked-in catalog itself.
|
||||
|
||||
The generated catalog also includes lazy worker shards for the checksum-pinned
|
||||
UTS #39 `IdentifierStatus.txt` and `IdentifierType.txt` ranges. Worker analysis
|
||||
loads them with the confusable table only when requested, applies those exact
|
||||
ranges per scalar, creates NFD skeletons, and emits structured diagnostics plus
|
||||
an intentionally conservative restriction-level estimate. `unicode/graphemes.ts` layers bounded structural
|
||||
inspection and small sequence-editing operations on the shared
|
||||
`Intl.Segmenter` wrapper; those operations construct code-point sequences but
|
||||
do not alter or validate the pinned RGI emoji catalog.
|
||||
|
||||
Rendering, normalization and grapheme segmentation use the current browser and
|
||||
operating-system Unicode implementation, which can differ from the pinned
|
||||
|
||||
@@ -6,16 +6,26 @@ versioned static catalog files and never sends search terms or inspected text.
|
||||
Input is rendered as React text, not executable HTML.
|
||||
|
||||
Text inspection is limited to 1,000,000 UTF-16 code units, scalar decoding to
|
||||
10,000 values, catalog results to 180, emoji results to 240, and displayed
|
||||
confusable mappings to 500. Large static datasets are held in a worker so the
|
||||
UI can remain responsive. Unihan data loads only on demand.
|
||||
10,000 values, catalog results to 180, emoji results to 240, displayed
|
||||
confusable mappings to 500, and each virtual code-chart worker request to 512
|
||||
entries. Large static datasets are held in a worker and chart rows are
|
||||
virtualized so the UI can remain responsive. Unihan data loads only on demand.
|
||||
|
||||
UTS #39 skeleton equivalence and mixed-script detection are review signals.
|
||||
They are not proof of spoofing, maliciousness, safety, or identifier validity;
|
||||
applications still need a context-specific identifier profile and policy.
|
||||
UTS #39 skeleton equivalence, pinned identifier status/type, mixed-script
|
||||
detection, bidi/default-ignorable controls, repeated combining marks and the
|
||||
displayed restriction-level estimate are review signals. They are not proof of
|
||||
spoofing, maliciousness, safety, or identifier validity; applications still
|
||||
need a context-specific identifier profile and policy. The comparison field
|
||||
tests skeleton equivalence only and does not perform account or domain lookup.
|
||||
Character glyphs come from the user's fonts, so a missing or surprising glyph
|
||||
does not alter the underlying code point.
|
||||
|
||||
Emoji editing removes or inserts specific variation selectors and skin-tone
|
||||
modifiers or joins 2–8 grapheme parts with ZWJ. It intentionally refuses
|
||||
ambiguous selector editing inside an existing ZWJ sequence, and it does not
|
||||
promise that constructed output is RGI, semantically appropriate, supported by
|
||||
the current font, or rendered as a single glyph.
|
||||
|
||||
Developers can regenerate the catalog, but every upstream byte sequence must
|
||||
match a hard-coded SHA-256 before parsing. The public release performs no such
|
||||
download. Data source identities are inspectable and exportable from the app.
|
||||
|
||||
Generated
+22
-23
@@ -1,22 +1,22 @@
|
||||
{
|
||||
"name": "unicode-tools",
|
||||
"version": "0.1.0",
|
||||
"version": "0.2.0",
|
||||
"lockfileVersion": 3,
|
||||
"requires": true,
|
||||
"packages": {
|
||||
"": {
|
||||
"name": "unicode-tools",
|
||||
"version": "0.1.0",
|
||||
"version": "0.2.0",
|
||||
"license": "GPL-3.0-or-later",
|
||||
"dependencies": {
|
||||
"@add-ideas/toolbox-contract": "0.2.3",
|
||||
"@add-ideas/toolbox-helpers": "0.1.0",
|
||||
"@add-ideas/toolbox-shell-react": "0.2.3",
|
||||
"@add-ideas/toolbox-contract": "0.3.0",
|
||||
"@add-ideas/toolbox-helpers": "0.2.0",
|
||||
"@add-ideas/toolbox-shell-react": "0.3.0",
|
||||
"react": "19.2.8",
|
||||
"react-dom": "19.2.8"
|
||||
},
|
||||
"devDependencies": {
|
||||
"@add-ideas/toolbox-testkit": "0.2.3",
|
||||
"@add-ideas/toolbox-testkit": "0.3.0",
|
||||
"@eslint/js": "10.0.1",
|
||||
"@playwright/test": "1.62.1",
|
||||
"@testing-library/jest-dom": "6.9.1",
|
||||
@@ -43,24 +43,24 @@
|
||||
}
|
||||
},
|
||||
"node_modules/@add-ideas/toolbox-contract": {
|
||||
"version": "0.2.3",
|
||||
"license": "Apache-2.0",
|
||||
"engines": {
|
||||
"node": ">=20"
|
||||
}
|
||||
"version": "0.3.0",
|
||||
"resolved": "https://git.add-ideas.de/api/packages/lotobo/npm/%40add-ideas%2Ftoolbox-contract/-/0.3.0/toolbox-contract-0.3.0.tgz",
|
||||
"integrity": "sha512-dKrK7BjOFwqJaBfJuhKxZKIld4sH0AKjEn6a0yLnbdMUFY+fFv4VSLGV2tNSBD016gumc2iNqOjUj/ld7x4rtA==",
|
||||
"license": "Apache-2.0"
|
||||
},
|
||||
"node_modules/@add-ideas/toolbox-helpers": {
|
||||
"version": "0.1.0",
|
||||
"license": "GPL-3.0-or-later",
|
||||
"engines": {
|
||||
"node": ">=22"
|
||||
}
|
||||
"version": "0.2.0",
|
||||
"resolved": "https://git.add-ideas.de/api/packages/lotobo/npm/%40add-ideas%2Ftoolbox-helpers/-/0.2.0/toolbox-helpers-0.2.0.tgz",
|
||||
"integrity": "sha512-SdOqkw+P+3J3fa5iVkzb5P15rVepB001GNV21Oh8w0CZcVL+YRltgD/s+MVcTyrNijWQf3E5vtQON/3N2LLyKg==",
|
||||
"license": "GPL-3.0-or-later"
|
||||
},
|
||||
"node_modules/@add-ideas/toolbox-shell-react": {
|
||||
"version": "0.2.3",
|
||||
"version": "0.3.0",
|
||||
"resolved": "https://git.add-ideas.de/api/packages/lotobo/npm/%40add-ideas%2Ftoolbox-shell-react/-/0.3.0/toolbox-shell-react-0.3.0.tgz",
|
||||
"integrity": "sha512-74p6JzAOG0YCAKdlc1hLofV4ZIko7vb448S75cIiM88PKm93EHl5VD7g8YVyfM56Ui97UY9dmy+Whiq4sGzpsg==",
|
||||
"license": "Apache-2.0",
|
||||
"dependencies": {
|
||||
"@add-ideas/toolbox-contract": "0.2.3"
|
||||
"@add-ideas/toolbox-contract": "0.3.0"
|
||||
},
|
||||
"peerDependencies": {
|
||||
"react": ">=18 <20",
|
||||
@@ -68,17 +68,16 @@
|
||||
}
|
||||
},
|
||||
"node_modules/@add-ideas/toolbox-testkit": {
|
||||
"version": "0.2.3",
|
||||
"version": "0.3.0",
|
||||
"resolved": "https://git.add-ideas.de/api/packages/lotobo/npm/%40add-ideas%2Ftoolbox-testkit/-/0.3.0/toolbox-testkit-0.3.0.tgz",
|
||||
"integrity": "sha512-4Fk+oSvZFspOMIXr8Xy040nhAaBsIQAzsGyXWSpjn3+k3yBKq7nB1r5zCHhsXzfdLzvPDAx2KcmSNOhM330D9w==",
|
||||
"dev": true,
|
||||
"license": "Apache-2.0",
|
||||
"dependencies": {
|
||||
"@add-ideas/toolbox-contract": "0.2.3"
|
||||
"@add-ideas/toolbox-contract": "0.3.0"
|
||||
},
|
||||
"bin": {
|
||||
"toolbox-check": "dist/cli.js"
|
||||
},
|
||||
"engines": {
|
||||
"node": ">=20"
|
||||
}
|
||||
},
|
||||
"node_modules/@adobe/css-tools": {
|
||||
|
||||
+5
-5
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "unicode-tools",
|
||||
"version": "0.1.0",
|
||||
"version": "0.2.0",
|
||||
"description": "Explore Unicode characters, scripts, emoji and text locally.",
|
||||
"license": "GPL-3.0-or-later",
|
||||
"author": "Albrecht Degering",
|
||||
@@ -40,14 +40,14 @@
|
||||
"release:artifact": "npm run check && npm run test:browser && npm run package:release -- --force"
|
||||
},
|
||||
"dependencies": {
|
||||
"@add-ideas/toolbox-contract": "0.2.3",
|
||||
"@add-ideas/toolbox-helpers": "0.1.0",
|
||||
"@add-ideas/toolbox-shell-react": "0.2.3",
|
||||
"@add-ideas/toolbox-contract": "0.3.0",
|
||||
"@add-ideas/toolbox-helpers": "0.2.0",
|
||||
"@add-ideas/toolbox-shell-react": "0.3.0",
|
||||
"react": "19.2.8",
|
||||
"react-dom": "19.2.8"
|
||||
},
|
||||
"devDependencies": {
|
||||
"@add-ideas/toolbox-testkit": "0.2.3",
|
||||
"@add-ideas/toolbox-testkit": "0.3.0",
|
||||
"@eslint/js": "10.0.1",
|
||||
"@playwright/test": "1.62.1",
|
||||
"@testing-library/jest-dom": "6.9.1",
|
||||
|
||||
+20
-2
@@ -15,7 +15,25 @@ export default defineConfig({
|
||||
timeout: 180_000,
|
||||
},
|
||||
projects: [
|
||||
{ name: "chromium", use: { ...devices["Desktop Chrome"] } },
|
||||
{ name: "firefox", use: { ...devices["Desktop Firefox"] } },
|
||||
{
|
||||
name: "chromium",
|
||||
testIgnore: /responsive\.spec\.ts/,
|
||||
use: { ...devices["Desktop Chrome"] },
|
||||
},
|
||||
{
|
||||
name: "firefox",
|
||||
testIgnore: /responsive\.spec\.ts/,
|
||||
use: { ...devices["Desktop Firefox"] },
|
||||
},
|
||||
{
|
||||
name: "webkit",
|
||||
testIgnore: /responsive\.spec\.ts/,
|
||||
use: { ...devices["Desktop Safari"] },
|
||||
},
|
||||
{
|
||||
name: "mobile-chromium",
|
||||
testMatch: /responsive\.spec\.ts/,
|
||||
use: { ...devices["Pixel 5"] },
|
||||
},
|
||||
],
|
||||
});
|
||||
|
||||
@@ -1,5 +1,10 @@
|
||||
# Changelog
|
||||
|
||||
## 0.2.0 - 2026-09-02
|
||||
|
||||
- Added lazy, pinned UTS #39 Identifier_Status/Identifier_Type evidence, skeleton comparison, restriction estimates, and structured confusable/invisible/control diagnostics.
|
||||
- Added detailed grapheme/emoji structure inspection and bounded presentation-selector, skin-tone and ZWJ construction with explicit non-RGI/font-support limits; added Toolbox I/O/capability metadata.
|
||||
|
||||
## 0.1.0 - 2026-09-01
|
||||
|
||||
- Added the initial local-first Unicode Tools workbench.
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
==============================================================================
|
||||
@add-ideas/toolbox-contract@0.2.3
|
||||
@add-ideas/toolbox-contract@0.3.0
|
||||
Declared licence: Apache-2.0
|
||||
==============================================================================
|
||||
--- LICENSE ---
|
||||
@@ -198,7 +198,7 @@ Declared licence: Apache-2.0
|
||||
|
||||
|
||||
==============================================================================
|
||||
@add-ideas/toolbox-helpers@0.1.0
|
||||
@add-ideas/toolbox-helpers@0.2.0
|
||||
Declared licence: GPL-3.0-or-later
|
||||
==============================================================================
|
||||
--- LICENSE ---
|
||||
@@ -879,7 +879,7 @@ Public License instead of this License. But first, please read
|
||||
|
||||
|
||||
==============================================================================
|
||||
@add-ideas/toolbox-shell-react@0.2.3
|
||||
@add-ideas/toolbox-shell-react@0.3.0
|
||||
Declared licence: Apache-2.0
|
||||
==============================================================================
|
||||
--- LICENSE ---
|
||||
|
||||
+18
-8
@@ -4,26 +4,36 @@ Explore Unicode characters, scripts, emoji and text locally.
|
||||
|
||||
Unicode Tools is a standalone local-first application in the [add·ideas Toolbox](https://git.add-ideas.de/lotobo/toolbox-portal). Inputs are processed in the browser and are not uploaded.
|
||||
|
||||
## Version 0.1.0 scope
|
||||
## Version 0.2.0 scope
|
||||
|
||||
- Search 159,866 assigned Unicode scalar records by official name, alias,
|
||||
literal character, or code point, with script, block and category filters.
|
||||
- Traverse any block or hexadecimal range in a 16-column virtualized code
|
||||
chart with roving keyboard navigation and property cards. Only visible rows
|
||||
are materialized, including when the entire U+0000–U+10FFFF space is open.
|
||||
- Browse and copy 3,952 fully-qualified emoji and components, inspect named
|
||||
sequences, and open practical symbol collections.
|
||||
sequences, open practical symbol collections, and construct bounded
|
||||
presentation-selector, skin-tone and ZWJ variants without claiming RGI
|
||||
validity or font support.
|
||||
- Inspect text as Unicode scalar values, UTF-16 positions, UTF-8 bytes,
|
||||
JavaScript and HTML escapes, and browser-segmented grapheme clusters.
|
||||
JavaScript and HTML escapes, and browser-segmented grapheme clusters with
|
||||
emoji/ZWJ/variation/modifier/flag/keycap/tag evidence.
|
||||
- Decode lists of hexadecimal scalar values and compare NFC, NFD, NFKC and
|
||||
NFKD normalization without losing sight of the original value.
|
||||
- Compute a UTS #39 confusable skeleton, show applied mappings, and flag mixed
|
||||
primary scripts as a review signal rather than a security verdict.
|
||||
- Compute UTS #39 confusable skeletons, compare skeleton collisions, display
|
||||
pinned Identifier_Status/Identifier_Type evidence, and flag mixed scripts,
|
||||
controls, default ignorables, combining repetition and non-NFC text as
|
||||
review signals rather than security verdicts.
|
||||
- Inspect character properties and optionally load selected Unihan readings,
|
||||
definitions and radical/stroke fields for Han ideographs.
|
||||
- Download an auditable source-identity report for every pinned data input.
|
||||
|
||||
The catalog is generated from checksum-pinned Unicode Consortium 17.0 data. It
|
||||
does not scrape or redistribute descriptions, artwork, or annotations from
|
||||
Symbl or another symbol website. Private-use code points and UTF-16 surrogates
|
||||
remain visible as ranges, but are not presented as assigned characters.
|
||||
Symbl or another symbol website. Unassigned scalars, private-use scalars,
|
||||
permanently reserved noncharacters and UTF-16 surrogate code units are labelled
|
||||
as distinct chart states; none is presented as an assigned character. Lone
|
||||
surrogate copy is disabled because a surrogate is not a Unicode scalar value.
|
||||
|
||||
The application treats every input as untrusted, applies explicit resource
|
||||
limits, and never interprets text as active content. The checked-in catalog can
|
||||
@@ -44,7 +54,7 @@ npm run test:browser
|
||||
|
||||
## Release
|
||||
|
||||
`npm run release:artifact` creates a deterministic `release/unicode-tools-0.1.0.zip` and checksum sidecar.
|
||||
`npm run release:artifact` creates a deterministic `release/unicode-tools-0.2.0.zip` and checksum sidecar.
|
||||
|
||||
## Licence
|
||||
|
||||
|
||||
+2
-2
@@ -1,8 +1,8 @@
|
||||
# Corresponding source
|
||||
|
||||
The corresponding source for Unicode Tools 0.1.0 is available at:
|
||||
The corresponding source for Unicode Tools 0.2.0 is available at:
|
||||
|
||||
https://git.add-ideas.de/lotobo/unicode-tools/src/tag/v0.1.0
|
||||
https://git.add-ideas.de/lotobo/unicode-tools/src/tag/v0.2.0
|
||||
|
||||
Build with Node.js 22, npm 11, `npm ci`, and `npm run release:artifact`.
|
||||
The preferred source includes the generated JSON catalog in `public/data/` so
|
||||
|
||||
@@ -14,7 +14,7 @@ Inc.
|
||||
## Runtime code
|
||||
|
||||
Runtime dependency licences are reproduced in
|
||||
`LICENSES/npm-runtime-licenses.txt` in release artifacts. Toolbox SDK packages
|
||||
are Apache-2.0. The reusable Toolbox Helpers package is GPL-3.0-or-later.
|
||||
`LICENSES/npm-runtime-licenses.txt` in release artifacts. Toolbox SDK 0.3.0
|
||||
packages are Apache-2.0. Toolbox Helpers 0.2.0 is GPL-3.0-or-later.
|
||||
React and React DOM are MIT. The build-only `fflate` dependency is MIT and is
|
||||
used by the catalog generator, not shipped as runtime code.
|
||||
|
||||
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
@@ -13803,9 +13803,13 @@
|
||||
"value": "Inherited"
|
||||
}
|
||||
],
|
||||
"identifierStatusFile": "identifier-status.json",
|
||||
"identifierTypeFile": "identifier-types.json",
|
||||
"emojiCount": 3952,
|
||||
"namedSequenceCount": 461,
|
||||
"confusableMappingCount": 6565,
|
||||
"identifierStatusRangeCount": 1649,
|
||||
"identifierTypeRangeCount": 5104,
|
||||
"unihanCount": 102998,
|
||||
"sourceIdentity": [
|
||||
{
|
||||
|
||||
@@ -17,8 +17,21 @@ persistent search history.
|
||||
archives and text files into deterministic checked-in source data. It derives
|
||||
names, aliases, general categories, scripts, blocks, ages, bidi and case
|
||||
properties, emoji, named sequences, selected Unihan fields, and UTS #39
|
||||
confusable mappings. Private-use and surrogate ranges are not expanded into
|
||||
synthetic catalog records.
|
||||
confusable mappings. The code chart virtualizes complete Unicode block or
|
||||
custom-range traversal into fixed row windows, requesting at most 512 entries
|
||||
from the worker. Assigned records come from the pinned catalog; missing entries
|
||||
are classified explicitly as unassigned, private-use, surrogate, or
|
||||
noncharacter instead of being omitted. Private-use and surrogate ranges are not
|
||||
expanded into the checked-in catalog itself.
|
||||
|
||||
The generated catalog also includes lazy worker shards for the checksum-pinned
|
||||
UTS #39 `IdentifierStatus.txt` and `IdentifierType.txt` ranges. Worker analysis
|
||||
loads them with the confusable table only when requested, applies those exact
|
||||
ranges per scalar, creates NFD skeletons, and emits structured diagnostics plus
|
||||
an intentionally conservative restriction-level estimate. `unicode/graphemes.ts` layers bounded structural
|
||||
inspection and small sequence-editing operations on the shared
|
||||
`Intl.Segmenter` wrapper; those operations construct code-point sequences but
|
||||
do not alter or validate the pinned RGI emoji catalog.
|
||||
|
||||
Rendering, normalization and grapheme segmentation use the current browser and
|
||||
operating-system Unicode implementation, which can differ from the pinned
|
||||
|
||||
@@ -6,16 +6,26 @@ versioned static catalog files and never sends search terms or inspected text.
|
||||
Input is rendered as React text, not executable HTML.
|
||||
|
||||
Text inspection is limited to 1,000,000 UTF-16 code units, scalar decoding to
|
||||
10,000 values, catalog results to 180, emoji results to 240, and displayed
|
||||
confusable mappings to 500. Large static datasets are held in a worker so the
|
||||
UI can remain responsive. Unihan data loads only on demand.
|
||||
10,000 values, catalog results to 180, emoji results to 240, displayed
|
||||
confusable mappings to 500, and each virtual code-chart worker request to 512
|
||||
entries. Large static datasets are held in a worker and chart rows are
|
||||
virtualized so the UI can remain responsive. Unihan data loads only on demand.
|
||||
|
||||
UTS #39 skeleton equivalence and mixed-script detection are review signals.
|
||||
They are not proof of spoofing, maliciousness, safety, or identifier validity;
|
||||
applications still need a context-specific identifier profile and policy.
|
||||
UTS #39 skeleton equivalence, pinned identifier status/type, mixed-script
|
||||
detection, bidi/default-ignorable controls, repeated combining marks and the
|
||||
displayed restriction-level estimate are review signals. They are not proof of
|
||||
spoofing, maliciousness, safety, or identifier validity; applications still
|
||||
need a context-specific identifier profile and policy. The comparison field
|
||||
tests skeleton equivalence only and does not perform account or domain lookup.
|
||||
Character glyphs come from the user's fonts, so a missing or surprising glyph
|
||||
does not alter the underlying code point.
|
||||
|
||||
Emoji editing removes or inserts specific variation selectors and skin-tone
|
||||
modifiers or joins 2–8 grapheme parts with ZWJ. It intentionally refuses
|
||||
ambiguous selector editing inside an existing ZWJ sequence, and it does not
|
||||
promise that constructed output is RGI, semantically appropriate, supported by
|
||||
the current font, or rendered as a single glyph.
|
||||
|
||||
Developers can regenerate the catalog, but every upstream byte sequence must
|
||||
match a hard-coded SHA-256 before parsing. The public release performs no such
|
||||
download. Data source identities are inspectable and exportable from the app.
|
||||
|
||||
+1
-1
@@ -1,5 +1,5 @@
|
||||
const CACHE_PREFIX = "unicode-tools-shell-";
|
||||
const CACHE_NAME = CACHE_PREFIX + "0.1.0";
|
||||
const CACHE_NAME = CACHE_PREFIX + "0.2.0";
|
||||
const CORE = ["./", "./manifest.webmanifest", "./favicon.svg"];
|
||||
self.addEventListener("install", (event) => {
|
||||
event.waitUntil(
|
||||
|
||||
+27
-3
@@ -3,12 +3,22 @@
|
||||
"schemaVersion": 1,
|
||||
"id": "de.add-ideas.unicode-tools",
|
||||
"name": "Unicode Tools",
|
||||
"version": "0.1.0",
|
||||
"description": "Explore Unicode characters, scripts, emoji and text locally.",
|
||||
"version": "0.2.0",
|
||||
"description": "Explore Unicode, emoji, graphemes and text security locally.",
|
||||
"entry": "./",
|
||||
"icon": "./favicon.svg",
|
||||
"categories": ["unicode", "text", "developer"],
|
||||
"tags": ["unicode", "emoji", "character", "codepoint", "script", "block"],
|
||||
"tags": [
|
||||
"unicode",
|
||||
"emoji",
|
||||
"grapheme",
|
||||
"confusable",
|
||||
"security",
|
||||
"character",
|
||||
"codepoint",
|
||||
"script",
|
||||
"block"
|
||||
],
|
||||
"integration": {
|
||||
"contextVersion": 1,
|
||||
"launchModes": ["navigate", "new-tab"],
|
||||
@@ -21,6 +31,20 @@
|
||||
"crossOriginIsolated": false,
|
||||
"topLevelContext": false
|
||||
},
|
||||
"io": {
|
||||
"accepts": [],
|
||||
"produces": [
|
||||
{
|
||||
"mediaType": "application/json",
|
||||
"extensions": [".json"],
|
||||
"label": "Pinned data identity report"
|
||||
}
|
||||
]
|
||||
},
|
||||
"capabilities": {
|
||||
"required": ["workers"],
|
||||
"optional": ["clipboard-write", "intl-segmenter"]
|
||||
},
|
||||
"privacy": {
|
||||
"processing": "local",
|
||||
"fileUploads": false,
|
||||
|
||||
@@ -327,6 +327,12 @@ const aliases = parseAliases(textFromArchive(ucd, "NameAliases.txt"));
|
||||
const emojiFlags = parseEmojiFlags(
|
||||
textFromArchive(ucd, "emoji/emoji-data.txt"),
|
||||
);
|
||||
const identifierStatusRanges = parseRanges(
|
||||
textFromArchive(security, "IdentifierStatus.txt"),
|
||||
);
|
||||
const identifierTypeRanges = parseRanges(
|
||||
textFromArchive(security, "IdentifierType.txt"),
|
||||
);
|
||||
|
||||
const dictionaries = {
|
||||
categories: { values: [], index: new Map() },
|
||||
@@ -416,6 +422,14 @@ await writeFile(
|
||||
join(outputDirectory, "confusables.json"),
|
||||
JSON.stringify(confusables),
|
||||
);
|
||||
await writeFile(
|
||||
join(outputDirectory, "identifier-status.json"),
|
||||
JSON.stringify(identifierStatusRanges),
|
||||
);
|
||||
await writeFile(
|
||||
join(outputDirectory, "identifier-types.json"),
|
||||
JSON.stringify(identifierTypeRanges),
|
||||
);
|
||||
await writeFile(
|
||||
join(outputDirectory, "unihan.json"),
|
||||
JSON.stringify(unihanRecords),
|
||||
@@ -432,9 +446,13 @@ const manifest = {
|
||||
),
|
||||
blockRanges: blocks,
|
||||
scriptRanges: scripts,
|
||||
identifierStatusFile: "identifier-status.json",
|
||||
identifierTypeFile: "identifier-types.json",
|
||||
emojiCount: emoji.length,
|
||||
namedSequenceCount: namedSequences.length,
|
||||
confusableMappingCount: confusables.length,
|
||||
identifierStatusRangeCount: identifierStatusRanges.length,
|
||||
identifierTypeRangeCount: identifierTypeRanges.length,
|
||||
unihanCount: unihanRecords.length,
|
||||
sourceIdentity: sources.map(({ id, filename, url, sha256 }) => ({
|
||||
id,
|
||||
|
||||
+654
-16
@@ -1,14 +1,22 @@
|
||||
import { useEffect, useMemo, useState } from "react";
|
||||
import { useEffect, useMemo, useRef, useState } from "react";
|
||||
import { triggerBlobDownload } from "@add-ideas/toolbox-helpers";
|
||||
import {
|
||||
analyzeConfusables,
|
||||
characterDetail,
|
||||
codeChartRange,
|
||||
loadManifest,
|
||||
searchCatalog,
|
||||
unihanDetail,
|
||||
} from "../unicode/catalog";
|
||||
import {
|
||||
inspectGraphemeClusters,
|
||||
joinEmojiWithZwj,
|
||||
setEmojiPresentation,
|
||||
setEmojiSkinTone,
|
||||
} from "../unicode/graphemes";
|
||||
import {
|
||||
codePointLabel,
|
||||
parseCodeChartRange,
|
||||
compareNormalization,
|
||||
decodeCodePointList,
|
||||
inspectText,
|
||||
@@ -17,15 +25,23 @@ import {
|
||||
import type {
|
||||
CatalogResult,
|
||||
CharacterDetail,
|
||||
CodeChartItem,
|
||||
EmojiSummary,
|
||||
UnicodeManifest,
|
||||
UnihanDetail,
|
||||
} from "../unicode/types";
|
||||
|
||||
type Tab =
|
||||
"catalog" | "emoji" | "text" | "normalization" | "collections" | "about";
|
||||
| "catalog"
|
||||
| "chart"
|
||||
| "emoji"
|
||||
| "text"
|
||||
| "normalization"
|
||||
| "collections"
|
||||
| "about";
|
||||
const tabs: ReadonlyArray<readonly [Tab, string]> = [
|
||||
["catalog", "Character catalog"],
|
||||
["chart", "Code chart"],
|
||||
["emoji", "Emoji"],
|
||||
["text", "Text inspector"],
|
||||
["normalization", "Normalize & confusables"],
|
||||
@@ -63,6 +79,406 @@ const categoryNames: Record<string, string> = {
|
||||
Zs: "Space separator",
|
||||
};
|
||||
|
||||
const CHART_COLUMNS = 16;
|
||||
const CHART_ROW_HEIGHT = 76;
|
||||
const CHART_OVERSCAN_ROWS = 3;
|
||||
|
||||
function CodePointPropertyCard({
|
||||
item,
|
||||
detail,
|
||||
}: {
|
||||
item: CodeChartItem;
|
||||
detail?: CharacterDetail;
|
||||
}) {
|
||||
const statusNotes: Record<CodeChartItem["status"], string> = {
|
||||
assigned:
|
||||
"This scalar value has an assigned Unicode character property record.",
|
||||
unassigned:
|
||||
"This is a valid Unicode scalar value with no character assigned in Unicode 17.0. Its future meaning is not stable.",
|
||||
"private-use":
|
||||
"This is a valid private-use scalar. Its meaning and glyph are defined only by a private agreement or font.",
|
||||
surrogate:
|
||||
"This is a UTF-16 surrogate code unit, not a Unicode scalar value or independently encodable character.",
|
||||
noncharacter:
|
||||
"This is a permanently reserved Unicode noncharacter scalar and is not intended for open interchange.",
|
||||
};
|
||||
const canCopy = item.status !== "surrogate";
|
||||
return (
|
||||
<section className="panel chart-detail" aria-live="polite">
|
||||
<div
|
||||
className={`chart-detail-glyph is-${item.status}`}
|
||||
aria-hidden="true"
|
||||
>
|
||||
{item.status === "assigned"
|
||||
? printableGlyph(item.character, item.category, item.name)
|
||||
: item.status === "private-use"
|
||||
? "PUA"
|
||||
: item.status === "surrogate"
|
||||
? "UTF-16"
|
||||
: item.status === "noncharacter"
|
||||
? "N/A"
|
||||
: "—"}
|
||||
</div>
|
||||
<div className="detail-copy">
|
||||
<p className="eyebrow">{item.status.replace("-", " ")}</p>
|
||||
<h2>{item.name}</h2>
|
||||
<p className="code">{item.hex}</p>
|
||||
<p className="notice">{statusNotes[item.status]}</p>
|
||||
<div className="actions">
|
||||
<button
|
||||
type="button"
|
||||
disabled={!canCopy}
|
||||
onClick={() => void navigator.clipboard.writeText(item.character)}
|
||||
>
|
||||
Copy {item.status === "assigned" ? "character" : "scalar"}
|
||||
</button>
|
||||
<button
|
||||
type="button"
|
||||
onClick={() => void navigator.clipboard.writeText(item.hex)}
|
||||
>
|
||||
Copy code point
|
||||
</button>
|
||||
</div>
|
||||
<dl className="property-grid">
|
||||
<div>
|
||||
<dt>Status / category</dt>
|
||||
<dd>
|
||||
{item.status} / {item.category}
|
||||
</dd>
|
||||
</div>
|
||||
<div>
|
||||
<dt>Script / block</dt>
|
||||
<dd>
|
||||
{item.script} / {item.block}
|
||||
</dd>
|
||||
</div>
|
||||
<div>
|
||||
<dt>Unicode age</dt>
|
||||
<dd>{detail?.age ?? item.age}</dd>
|
||||
</div>
|
||||
<div>
|
||||
<dt>Encoding forms</dt>
|
||||
<dd>
|
||||
{item.status === "surrogate"
|
||||
? `\\u${item.point.toString(16).toUpperCase().padStart(4, "0")} · no UTF-8 or HTML scalar encoding`
|
||||
: `${item.point <= 0xffff ? `\\u${item.point.toString(16).toUpperCase().padStart(4, "0")}` : `\\u{${item.point.toString(16).toUpperCase()}}`} · &#x${item.point.toString(16).toUpperCase()};`}
|
||||
</dd>
|
||||
</div>
|
||||
{detail && (
|
||||
<>
|
||||
<div>
|
||||
<dt>Bidi / mirrored</dt>
|
||||
<dd>
|
||||
{detail.bidiClass} / {detail.mirrored ? "yes" : "no"}
|
||||
</dd>
|
||||
</div>
|
||||
<div>
|
||||
<dt>Combining class / decomposition</dt>
|
||||
<dd>
|
||||
{detail.combiningClass} / {detail.decomposition || "none"}
|
||||
</dd>
|
||||
</div>
|
||||
<div>
|
||||
<dt>Aliases</dt>
|
||||
<dd>{detail.aliases.join("; ") || "none"}</dd>
|
||||
</div>
|
||||
</>
|
||||
)}
|
||||
</dl>
|
||||
</div>
|
||||
</section>
|
||||
);
|
||||
}
|
||||
|
||||
function CodeChartPanel({ manifest }: { manifest?: UnicodeManifest }) {
|
||||
const [range, setRange] = useState({ start: 0, end: 0x7f });
|
||||
const [startInput, setStartInput] = useState("U+0000");
|
||||
const [endInput, setEndInput] = useState("U+007F");
|
||||
const [scrollTop, setScrollTop] = useState(0);
|
||||
const [items, setItems] = useState<CodeChartItem[]>([]);
|
||||
const [selected, setSelected] = useState<CodeChartItem>();
|
||||
const [detail, setDetail] = useState<CharacterDetail>();
|
||||
const [busy, setBusy] = useState(true);
|
||||
const [error, setError] = useState("");
|
||||
const viewport = useRef<HTMLDivElement>(null);
|
||||
const selectedRef = useRef<CodeChartItem | undefined>(undefined);
|
||||
const pendingFocus = useRef<number | undefined>(undefined);
|
||||
const total = range.end - range.start + 1;
|
||||
const rows = Math.ceil(total / CHART_COLUMNS);
|
||||
const firstRow = Math.max(
|
||||
0,
|
||||
Math.floor(scrollTop / CHART_ROW_HEIGHT) - CHART_OVERSCAN_ROWS,
|
||||
);
|
||||
const visibleRows = Math.min(rows - firstRow, 13);
|
||||
const windowStart = range.start + firstRow * CHART_COLUMNS;
|
||||
const windowCount = Math.max(
|
||||
1,
|
||||
Math.min(512, range.end - windowStart + 1, visibleRows * CHART_COLUMNS),
|
||||
);
|
||||
|
||||
useEffect(() => {
|
||||
let active = true;
|
||||
void codeChartRange(windowStart, windowCount)
|
||||
.then((value) => {
|
||||
if (!active) return;
|
||||
setItems(value);
|
||||
if (!selectedRef.current && value[0]) {
|
||||
selectedRef.current = value[0];
|
||||
setSelected(value[0]);
|
||||
if (value[0].status === "assigned")
|
||||
void characterDetail(value[0].point).then((record) => {
|
||||
if (active) setDetail(record);
|
||||
});
|
||||
}
|
||||
setError("");
|
||||
})
|
||||
.catch((reason: unknown) => {
|
||||
if (active)
|
||||
setError(
|
||||
reason instanceof Error ? reason.message : "Code chart failed.",
|
||||
);
|
||||
})
|
||||
.finally(() => {
|
||||
if (active) setBusy(false);
|
||||
});
|
||||
return () => {
|
||||
active = false;
|
||||
};
|
||||
}, [windowCount, windowStart]);
|
||||
|
||||
useEffect(() => {
|
||||
const point = pendingFocus.current;
|
||||
if (point === undefined) return;
|
||||
const target = viewport.current?.querySelector<HTMLButtonElement>(
|
||||
`[data-code-point="${point}"]`,
|
||||
);
|
||||
if (target) {
|
||||
target.focus();
|
||||
pendingFocus.current = undefined;
|
||||
}
|
||||
}, [items]);
|
||||
|
||||
const choose = async (item: CodeChartItem, focus = false) => {
|
||||
setSelected(item);
|
||||
selectedRef.current = item;
|
||||
setDetail(undefined);
|
||||
if (item.status === "assigned") {
|
||||
try {
|
||||
setDetail(await characterDetail(item.point));
|
||||
} catch (reason) {
|
||||
setError(
|
||||
reason instanceof Error ? reason.message : "Property lookup failed.",
|
||||
);
|
||||
}
|
||||
}
|
||||
if (focus) {
|
||||
pendingFocus.current = item.point;
|
||||
window.requestAnimationFrame(() => {
|
||||
const target = viewport.current?.querySelector<HTMLButtonElement>(
|
||||
`[data-code-point="${item.point}"]`,
|
||||
);
|
||||
if (target) {
|
||||
target.focus();
|
||||
pendingFocus.current = undefined;
|
||||
}
|
||||
});
|
||||
}
|
||||
};
|
||||
|
||||
const choosePoint = async (point: number) => {
|
||||
const next = Math.max(range.start, Math.min(range.end, point));
|
||||
const row = Math.floor((next - range.start) / CHART_COLUMNS);
|
||||
const node = viewport.current;
|
||||
if (node) {
|
||||
const top = row * CHART_ROW_HEIGHT;
|
||||
if (top < node.scrollTop) node.scrollTop = top;
|
||||
else if (top + CHART_ROW_HEIGHT > node.scrollTop + node.clientHeight)
|
||||
node.scrollTop = top - node.clientHeight + CHART_ROW_HEIGHT;
|
||||
setScrollTop(node.scrollTop);
|
||||
}
|
||||
try {
|
||||
const [item] = await codeChartRange(next, 1);
|
||||
if (item) await choose(item, true);
|
||||
} catch (reason) {
|
||||
setError(
|
||||
reason instanceof Error ? reason.message : "Property lookup failed.",
|
||||
);
|
||||
}
|
||||
};
|
||||
|
||||
const applyRange = (next: { start: number; end: number }) => {
|
||||
setRange(next);
|
||||
setStartInput(codePointLabel(next.start));
|
||||
setEndInput(codePointLabel(next.end));
|
||||
setItems([]);
|
||||
setSelected(undefined);
|
||||
selectedRef.current = undefined;
|
||||
pendingFocus.current = undefined;
|
||||
setDetail(undefined);
|
||||
setScrollTop(0);
|
||||
setBusy(true);
|
||||
if (viewport.current) viewport.current.scrollTop = 0;
|
||||
};
|
||||
|
||||
return (
|
||||
<>
|
||||
<section className="panel workspace">
|
||||
<div className="panel-heading">
|
||||
<div>
|
||||
<p className="eyebrow">Virtualized Unicode code space</p>
|
||||
<h2>Browse a block or exact range</h2>
|
||||
</div>
|
||||
<span className="count">{total.toLocaleString()} code points</span>
|
||||
</div>
|
||||
<div className="form-grid">
|
||||
<label className="field">
|
||||
<span>Unicode block</span>
|
||||
<select
|
||||
value=""
|
||||
disabled={!manifest}
|
||||
onChange={(event) => {
|
||||
const block = manifest?.blockRanges[Number(event.target.value)];
|
||||
if (block) applyRange({ start: block.start, end: block.end });
|
||||
}}
|
||||
>
|
||||
<option value="">Choose a block…</option>
|
||||
{manifest?.blockRanges.map((block, index) => (
|
||||
<option value={index} key={`${block.start}-${block.value}`}>
|
||||
{block.value} ({codePointLabel(block.start)}–
|
||||
{codePointLabel(block.end)})
|
||||
</option>
|
||||
))}
|
||||
</select>
|
||||
</label>
|
||||
<label className="field">
|
||||
<span>Range start</span>
|
||||
<input
|
||||
value={startInput}
|
||||
onChange={(event) => setStartInput(event.target.value)}
|
||||
/>
|
||||
</label>
|
||||
<label className="field">
|
||||
<span>Range end</span>
|
||||
<input
|
||||
value={endInput}
|
||||
onChange={(event) => setEndInput(event.target.value)}
|
||||
/>
|
||||
</label>
|
||||
</div>
|
||||
<div className="actions">
|
||||
<button
|
||||
className="primary"
|
||||
type="button"
|
||||
onClick={() => {
|
||||
try {
|
||||
applyRange(parseCodeChartRange(startInput, endInput));
|
||||
setError("");
|
||||
} catch (reason) {
|
||||
setError(
|
||||
reason instanceof Error
|
||||
? reason.message
|
||||
: "Range is invalid.",
|
||||
);
|
||||
}
|
||||
}}
|
||||
>
|
||||
Open range
|
||||
</button>
|
||||
<button
|
||||
type="button"
|
||||
onClick={() => applyRange({ start: 0, end: 0x10ffff })}
|
||||
>
|
||||
Entire code space
|
||||
</button>
|
||||
</div>
|
||||
<p className="muted">
|
||||
Only visible rows are materialized. Use arrow keys for adjacent code
|
||||
points, Page Up/Down for eight rows, and Home/End for range bounds.
|
||||
Private-use scalars, noncharacters, unassigned scalars and UTF-16
|
||||
surrogates are labelled explicitly rather than shown as characters.
|
||||
</p>
|
||||
<ErrorMessage value={error} />
|
||||
</section>
|
||||
<section className="panel chart-panel" aria-label="Unicode code chart">
|
||||
<div className="chart-column-headings" aria-hidden="true">
|
||||
{Array.from({ length: CHART_COLUMNS }, (_, index) => (
|
||||
<span key={index}>{index.toString(16).toUpperCase()}</span>
|
||||
))}
|
||||
</div>
|
||||
<div
|
||||
className="chart-viewport"
|
||||
ref={viewport}
|
||||
role="grid"
|
||||
aria-rowcount={rows}
|
||||
aria-colcount={CHART_COLUMNS}
|
||||
aria-busy={busy}
|
||||
onScroll={(event) => {
|
||||
setBusy(true);
|
||||
setScrollTop(event.currentTarget.scrollTop);
|
||||
}}
|
||||
>
|
||||
<div
|
||||
className="chart-spacer"
|
||||
style={{ height: rows * CHART_ROW_HEIGHT }}
|
||||
>
|
||||
<div
|
||||
className="chart-window"
|
||||
style={{ top: firstRow * CHART_ROW_HEIGHT }}
|
||||
>
|
||||
{items.map((item) => (
|
||||
<button
|
||||
type="button"
|
||||
role="gridcell"
|
||||
className={`chart-cell is-${item.status}`}
|
||||
aria-selected={selected?.point === item.point}
|
||||
aria-label={`${item.hex} ${item.name}, ${item.status}`}
|
||||
tabIndex={selected?.point === item.point ? 0 : -1}
|
||||
data-code-point={item.point}
|
||||
key={item.point}
|
||||
onClick={() => void choose(item)}
|
||||
onKeyDown={(event) => {
|
||||
let next: number | undefined;
|
||||
if (event.key === "ArrowRight") next = item.point + 1;
|
||||
else if (event.key === "ArrowLeft") next = item.point - 1;
|
||||
else if (event.key === "ArrowDown")
|
||||
next = item.point + CHART_COLUMNS;
|
||||
else if (event.key === "ArrowUp")
|
||||
next = item.point - CHART_COLUMNS;
|
||||
else if (event.key === "PageDown")
|
||||
next = item.point + CHART_COLUMNS * 8;
|
||||
else if (event.key === "PageUp")
|
||||
next = item.point - CHART_COLUMNS * 8;
|
||||
else if (event.key === "Home") next = range.start;
|
||||
else if (event.key === "End") next = range.end;
|
||||
if (next !== undefined) {
|
||||
event.preventDefault();
|
||||
void choosePoint(next);
|
||||
}
|
||||
}}
|
||||
>
|
||||
<span className="chart-glyph" aria-hidden="true">
|
||||
{item.status === "assigned"
|
||||
? printableGlyph(item.character, item.category, item.name)
|
||||
: item.status === "private-use"
|
||||
? "PUA"
|
||||
: item.status === "surrogate"
|
||||
? "S"
|
||||
: item.status === "noncharacter"
|
||||
? "N"
|
||||
: "·"}
|
||||
</span>
|
||||
<small>{item.hex.slice(2)}</small>
|
||||
</button>
|
||||
))}
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</section>
|
||||
{selected && <CodePointPropertyCard item={selected} detail={detail} />}
|
||||
</>
|
||||
);
|
||||
}
|
||||
|
||||
function ErrorMessage({ value }: { value: string }) {
|
||||
return value ? (
|
||||
<p className="error" role="alert">
|
||||
@@ -453,6 +869,12 @@ function EmojiPanel({ manifest }: { manifest?: UnicodeManifest }) {
|
||||
const [results, setResults] = useState<CatalogResult[]>([]);
|
||||
const [busy, setBusy] = useState(false);
|
||||
const [error, setError] = useState("");
|
||||
const [builder, setBuilder] = useState("👍");
|
||||
const [tone, setTone] = useState<0 | 1 | 2 | 3 | 4 | 5>(0);
|
||||
const [presentation, setPresentation] = useState<
|
||||
"default" | "text" | "emoji"
|
||||
>("default");
|
||||
const [zwjParts, setZwjParts] = useState("👩\n💻");
|
||||
const run = async () => {
|
||||
setBusy(true);
|
||||
setError("");
|
||||
@@ -551,6 +973,135 @@ function EmojiPanel({ manifest }: { manifest?: UnicodeManifest }) {
|
||||
Search the pinned Emoji 17.0 catalog. Click a result to copy it.
|
||||
</p>
|
||||
)}
|
||||
<section
|
||||
className="panel workspace"
|
||||
aria-labelledby="emoji-builder-heading"
|
||||
>
|
||||
<div>
|
||||
<p className="eyebrow">Grapheme-aware emoji lab</p>
|
||||
<h2 id="emoji-builder-heading">
|
||||
Presentation, tone and ZWJ composition
|
||||
</h2>
|
||||
</div>
|
||||
<label className="field">
|
||||
<span>One emoji grapheme</span>
|
||||
<input
|
||||
maxLength={256}
|
||||
value={builder}
|
||||
onChange={(event) => setBuilder(event.target.value)}
|
||||
/>
|
||||
</label>
|
||||
<div className="form-grid">
|
||||
<label className="field">
|
||||
<span>Presentation selector</span>
|
||||
<select
|
||||
value={presentation}
|
||||
onChange={(event) =>
|
||||
setPresentation(event.target.value as typeof presentation)
|
||||
}
|
||||
>
|
||||
<option value="default">Default</option>
|
||||
<option value="text">Text (VS15)</option>
|
||||
<option value="emoji">Emoji (VS16)</option>
|
||||
</select>
|
||||
</label>
|
||||
<button
|
||||
type="button"
|
||||
onClick={() => {
|
||||
try {
|
||||
setBuilder(setEmojiPresentation(builder, presentation));
|
||||
setError("");
|
||||
} catch (reason) {
|
||||
setError(
|
||||
reason instanceof Error
|
||||
? reason.message
|
||||
: "Presentation edit failed.",
|
||||
);
|
||||
}
|
||||
}}
|
||||
>
|
||||
Apply presentation
|
||||
</button>
|
||||
<label className="field">
|
||||
<span>Skin-tone modifier</span>
|
||||
<select
|
||||
value={tone}
|
||||
onChange={(event) =>
|
||||
setTone(Number(event.target.value) as typeof tone)
|
||||
}
|
||||
>
|
||||
<option value="0">None</option>
|
||||
<option value="1">Light</option>
|
||||
<option value="2">Medium-light</option>
|
||||
<option value="3">Medium</option>
|
||||
<option value="4">Medium-dark</option>
|
||||
<option value="5">Dark</option>
|
||||
</select>
|
||||
</label>
|
||||
<button
|
||||
type="button"
|
||||
onClick={() => {
|
||||
try {
|
||||
setBuilder(setEmojiSkinTone(builder, tone));
|
||||
setError("");
|
||||
} catch (reason) {
|
||||
setError(
|
||||
reason instanceof Error
|
||||
? reason.message
|
||||
: "Tone edit failed.",
|
||||
);
|
||||
}
|
||||
}}
|
||||
>
|
||||
Apply tone
|
||||
</button>
|
||||
</div>
|
||||
<label className="field">
|
||||
<span>ZWJ parts, one grapheme per line (2–8)</span>
|
||||
<textarea
|
||||
className="short"
|
||||
maxLength={2048}
|
||||
value={zwjParts}
|
||||
onChange={(event) => setZwjParts(event.target.value)}
|
||||
/>
|
||||
</label>
|
||||
<button
|
||||
type="button"
|
||||
onClick={() => {
|
||||
try {
|
||||
setBuilder(
|
||||
joinEmojiWithZwj(zwjParts.split(/\r?\n/u).filter(Boolean)),
|
||||
);
|
||||
setError("");
|
||||
} catch (reason) {
|
||||
setError(
|
||||
reason instanceof Error
|
||||
? reason.message
|
||||
: "ZWJ composition failed.",
|
||||
);
|
||||
}
|
||||
}}
|
||||
>
|
||||
Compose with ZWJ
|
||||
</button>
|
||||
<div className="copy-output">
|
||||
<span className="large-text">{builder}</span>
|
||||
<code>
|
||||
{inspectGraphemeClusters(builder)[0]?.codePoints.join(" ")}
|
||||
</code>
|
||||
<button
|
||||
type="button"
|
||||
onClick={() => void navigator.clipboard.writeText(builder)}
|
||||
>
|
||||
Copy edited grapheme
|
||||
</button>
|
||||
</div>
|
||||
<p className="muted">
|
||||
These controls construct Unicode sequences; they do not guarantee that
|
||||
a sequence is RGI, supported by the current font, or rendered as one
|
||||
glyph.
|
||||
</p>
|
||||
</section>
|
||||
</>
|
||||
);
|
||||
}
|
||||
@@ -561,10 +1112,13 @@ function TextPanel() {
|
||||
const [decoded, setDecoded] = useState("");
|
||||
const [inspection, setInspection] =
|
||||
useState<ReturnType<typeof inspectText>>();
|
||||
const [clusterDetails, setClusterDetails] =
|
||||
useState<ReturnType<typeof inspectGraphemeClusters>>();
|
||||
const [error, setError] = useState("");
|
||||
const inspect = () => {
|
||||
try {
|
||||
setInspection(inspectText(input));
|
||||
setClusterDetails(inspectGraphemeClusters(input));
|
||||
setError("");
|
||||
} catch (reason) {
|
||||
setError(reason instanceof Error ? reason.message : "Inspection failed.");
|
||||
@@ -636,6 +1190,51 @@ function TextPanel() {
|
||||
</span>
|
||||
))}
|
||||
</div>
|
||||
{clusterDetails && (
|
||||
<div className="table-scroll">
|
||||
<table>
|
||||
<thead>
|
||||
<tr>
|
||||
<th>Grapheme</th>
|
||||
<th>UTF-16 range</th>
|
||||
<th>Code points</th>
|
||||
<th>Emoji structure</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
{clusterDetails.slice(0, 10_000).map((cluster) => (
|
||||
<tr key={`${cluster.utf16Start}-${cluster.index}`}>
|
||||
<td className="table-glyph">{cluster.value}</td>
|
||||
<td>
|
||||
{cluster.utf16Start}–{cluster.utf16End}
|
||||
</td>
|
||||
<td>
|
||||
<code>{cluster.codePoints.join(" ")}</code>
|
||||
</td>
|
||||
<td>
|
||||
{[
|
||||
cluster.extendedPictographic && "pictographic",
|
||||
cluster.containsZwj && "ZWJ",
|
||||
cluster.variation !== "none" &&
|
||||
`${cluster.variation} selector`,
|
||||
cluster.skinTones.length &&
|
||||
`${cluster.skinTones.length} tone modifier(s)`,
|
||||
cluster.regionalIndicators &&
|
||||
`${cluster.regionalIndicators} regional indicator(s)`,
|
||||
cluster.keycap && "keycap",
|
||||
cluster.tagCharacters &&
|
||||
`${cluster.tagCharacters} tag character(s)`,
|
||||
cluster.unpairedSurrogate && "unpaired surrogate",
|
||||
]
|
||||
.filter(Boolean)
|
||||
.join(" · ") || "plain cluster"}
|
||||
</td>
|
||||
</tr>
|
||||
))}
|
||||
</tbody>
|
||||
</table>
|
||||
</div>
|
||||
)}
|
||||
<div className="table-scroll">
|
||||
<table>
|
||||
<thead>
|
||||
@@ -682,6 +1281,7 @@ function TextPanel() {
|
||||
|
||||
function NormalizationPanel() {
|
||||
const [input, setInput] = useState("раypal é ①");
|
||||
const [compareInput, setCompareInput] = useState("paypal é 1");
|
||||
const [comparison, setComparison] =
|
||||
useState<ReturnType<typeof compareNormalization>>();
|
||||
const [security, setSecurity] =
|
||||
@@ -702,7 +1302,7 @@ function NormalizationPanel() {
|
||||
setBusy(true);
|
||||
setError("");
|
||||
try {
|
||||
setSecurity(await analyzeConfusables(input));
|
||||
setSecurity(await analyzeConfusables(input, compareInput));
|
||||
} catch (reason) {
|
||||
setError(
|
||||
reason instanceof Error
|
||||
@@ -725,6 +1325,13 @@ function NormalizationPanel() {
|
||||
onChange={(event) => setInput(event.target.value)}
|
||||
/>
|
||||
</label>
|
||||
<label className="field">
|
||||
<span>Compare identifier (optional)</span>
|
||||
<input
|
||||
value={compareInput}
|
||||
onChange={(event) => setCompareInput(event.target.value)}
|
||||
/>
|
||||
</label>
|
||||
<div className="actions">
|
||||
<button className="primary" type="button" onClick={normalize}>
|
||||
Compare normalization
|
||||
@@ -734,9 +1341,10 @@ function NormalizationPanel() {
|
||||
</button>
|
||||
</div>
|
||||
<p className="warning">
|
||||
A matching skeleton or mixed-script flag is a review signal, not proof
|
||||
that text is malicious or safe. This lab does not implement a complete
|
||||
identifier policy.
|
||||
Identifier_Status and Identifier_Type come from the pinned Unicode
|
||||
17.0 UTS #39 data. Restriction level, skeleton collisions and
|
||||
invisible controls remain review evidence, not proof that text is
|
||||
malicious or safe.
|
||||
</p>
|
||||
<ErrorMessage value={error} />
|
||||
</section>
|
||||
@@ -784,7 +1392,40 @@ function NormalizationPanel() {
|
||||
<dt>Mappings applied</dt>
|
||||
<dd>{security.mappings.length}</dd>
|
||||
</div>
|
||||
<div>
|
||||
<dt>Restriction level</dt>
|
||||
<dd>{security.restrictionLevel}</dd>
|
||||
</div>
|
||||
{security.comparison && (
|
||||
<div>
|
||||
<dt>Comparison skeleton collision</dt>
|
||||
<dd>{security.comparison.collision ? "yes — review" : "no"}</dd>
|
||||
</div>
|
||||
)}
|
||||
</dl>
|
||||
{security.diagnostics.map((diagnostic) => (
|
||||
<p
|
||||
className={diagnostic.severity === "danger" ? "error" : "warning"}
|
||||
key={diagnostic.code}
|
||||
>
|
||||
<strong>{diagnostic.code}</strong>: {diagnostic.message}{" "}
|
||||
{diagnostic.points.join(" ")}
|
||||
</p>
|
||||
))}
|
||||
<details>
|
||||
<summary>
|
||||
Identifier status/type evidence (
|
||||
{security.identifierStatus.length})
|
||||
</summary>
|
||||
<div className="mapping-list">
|
||||
{security.identifierStatus.slice(0, 2_000).map((item, index) => (
|
||||
<span key={`${item.point}-${index}`}>
|
||||
<code>{item.point}</code> {item.character} · {item.status} ·{" "}
|
||||
{item.types.join(", ")}
|
||||
</span>
|
||||
))}
|
||||
</div>
|
||||
</details>
|
||||
{security.mappings.length > 0 && (
|
||||
<div className="mapping-list">
|
||||
{security.mappings.slice(0, 500).map((item, index) => (
|
||||
@@ -959,6 +1600,8 @@ export function Workbench() {
|
||||
manifest={manifest}
|
||||
initial={collection}
|
||||
/>
|
||||
) : tab === "chart" ? (
|
||||
<CodeChartPanel manifest={manifest} />
|
||||
) : tab === "emoji" ? (
|
||||
<EmojiPanel manifest={manifest} />
|
||||
) : tab === "text" ? (
|
||||
@@ -985,24 +1628,19 @@ export function Workbench() {
|
||||
<p className="eyebrow">Pinned Unicode 17.0 catalog</p>
|
||||
<h1>Unicode Tools</h1>
|
||||
<p>
|
||||
Find and copy characters, browse emoji and collections, inspect
|
||||
exact text structure, compare normalization, and review
|
||||
confusables—all locally.
|
||||
Find and copy characters, traverse a virtualized Unicode code chart,
|
||||
browse emoji and collections, inspect exact text structure, compare
|
||||
normalization, and review confusables—all locally.
|
||||
</p>
|
||||
</div>
|
||||
<span className="privacy-pill">Official data · local search</span>
|
||||
</header>
|
||||
<ErrorMessage value={manifestError} />
|
||||
<nav
|
||||
className="panel workspace-tabs"
|
||||
role="tablist"
|
||||
aria-label="Unicode workspaces"
|
||||
>
|
||||
<nav className="panel workspace-tabs" aria-label="Unicode workspaces">
|
||||
{tabs.map(([value, label]) => (
|
||||
<button
|
||||
type="button"
|
||||
role="tab"
|
||||
aria-selected={tab === value}
|
||||
aria-pressed={tab === value}
|
||||
key={value}
|
||||
onClick={() => setTab(value)}
|
||||
>
|
||||
|
||||
+100
-1
@@ -153,7 +153,7 @@ textarea {
|
||||
overflow-x: auto;
|
||||
padding: 0.75rem;
|
||||
}
|
||||
.workspace-tabs button[aria-selected="true"] {
|
||||
.workspace-tabs button[aria-pressed="true"] {
|
||||
border-color: var(--toolbox-accent);
|
||||
background: var(--toolbox-accent);
|
||||
color: var(--toolbox-accent-contrast);
|
||||
@@ -201,6 +201,102 @@ textarea {
|
||||
grid-template-columns: repeat(auto-fill, minmax(min(100%, 15rem), 1fr));
|
||||
gap: 0.7rem;
|
||||
}
|
||||
.chart-panel {
|
||||
min-width: 0;
|
||||
overflow: hidden;
|
||||
}
|
||||
.chart-column-headings,
|
||||
.chart-window {
|
||||
display: grid;
|
||||
grid-template-columns: repeat(16, minmax(3.25rem, 1fr));
|
||||
}
|
||||
.chart-column-headings {
|
||||
padding: 0 0.2rem 0.35rem;
|
||||
color: var(--toolbox-muted);
|
||||
font-family: ui-monospace, SFMono-Regular, Consolas, monospace;
|
||||
font-size: 0.7rem;
|
||||
text-align: center;
|
||||
}
|
||||
.chart-viewport {
|
||||
height: min(36rem, 70vh);
|
||||
overflow: auto;
|
||||
border: 1px solid var(--toolbox-border);
|
||||
border-radius: 0.7rem;
|
||||
}
|
||||
.chart-spacer {
|
||||
position: relative;
|
||||
min-width: 52rem;
|
||||
}
|
||||
.chart-window {
|
||||
position: absolute;
|
||||
right: 0;
|
||||
left: 0;
|
||||
grid-auto-rows: 4.75rem;
|
||||
padding: 0.15rem;
|
||||
}
|
||||
.chart-cell {
|
||||
min-width: 0;
|
||||
min-height: 0;
|
||||
display: grid;
|
||||
gap: 0.15rem;
|
||||
align-content: center;
|
||||
padding: 0.25rem;
|
||||
border-radius: 0.35rem;
|
||||
font-weight: 500;
|
||||
}
|
||||
.chart-cell[aria-selected="true"] {
|
||||
border-color: var(--toolbox-accent);
|
||||
background: var(--toolbox-accent-soft);
|
||||
}
|
||||
.chart-cell.is-unassigned,
|
||||
.chart-cell.is-noncharacter {
|
||||
color: var(--toolbox-muted);
|
||||
background: repeating-linear-gradient(
|
||||
-45deg,
|
||||
transparent,
|
||||
transparent 5px,
|
||||
var(--toolbox-surface-soft) 5px,
|
||||
var(--toolbox-surface-soft) 10px
|
||||
);
|
||||
}
|
||||
.chart-cell.is-private-use {
|
||||
border-style: dashed;
|
||||
}
|
||||
.chart-cell.is-surrogate {
|
||||
border-color: var(--toolbox-danger);
|
||||
color: var(--toolbox-danger);
|
||||
}
|
||||
.chart-glyph {
|
||||
overflow: hidden;
|
||||
font-family: "Noto Sans", "Segoe UI Symbol", sans-serif;
|
||||
font-size: 1.45rem;
|
||||
line-height: 1.15;
|
||||
}
|
||||
.chart-cell:not(.is-assigned) .chart-glyph {
|
||||
font-family: ui-monospace, SFMono-Regular, Consolas, monospace;
|
||||
font-size: 0.72rem;
|
||||
font-weight: 800;
|
||||
}
|
||||
.chart-detail {
|
||||
display: grid;
|
||||
grid-template-columns: minmax(8rem, 13rem) 1fr;
|
||||
gap: 1rem;
|
||||
}
|
||||
.chart-detail-glyph {
|
||||
min-height: 11rem;
|
||||
display: grid;
|
||||
place-items: center;
|
||||
border-radius: 0.8rem;
|
||||
background: var(--toolbox-surface-soft);
|
||||
font-family: "Noto Sans", "Segoe UI Symbol", sans-serif;
|
||||
font-size: clamp(3rem, 9vw, 7rem);
|
||||
}
|
||||
.chart-detail-glyph:not(.is-assigned) {
|
||||
color: var(--toolbox-muted);
|
||||
font-family: ui-monospace, SFMono-Regular, Consolas, monospace;
|
||||
font-size: 1.1rem;
|
||||
font-weight: 800;
|
||||
}
|
||||
.character-card {
|
||||
min-height: 8rem;
|
||||
justify-content: start;
|
||||
@@ -502,6 +598,9 @@ th {
|
||||
.detail {
|
||||
grid-template-columns: 1fr;
|
||||
}
|
||||
.chart-detail {
|
||||
grid-template-columns: 1fr;
|
||||
}
|
||||
.detail-glyph {
|
||||
min-height: 8rem;
|
||||
}
|
||||
|
||||
@@ -3,12 +3,22 @@
|
||||
"schemaVersion": 1,
|
||||
"id": "de.add-ideas.unicode-tools",
|
||||
"name": "Unicode Tools",
|
||||
"version": "0.1.0",
|
||||
"description": "Explore Unicode characters, scripts, emoji and text locally.",
|
||||
"version": "0.2.0",
|
||||
"description": "Explore Unicode, emoji, graphemes and text security locally.",
|
||||
"entry": "./",
|
||||
"icon": "./favicon.svg",
|
||||
"categories": ["unicode", "text", "developer"],
|
||||
"tags": ["unicode", "emoji", "character", "codepoint", "script", "block"],
|
||||
"tags": [
|
||||
"unicode",
|
||||
"emoji",
|
||||
"grapheme",
|
||||
"confusable",
|
||||
"security",
|
||||
"character",
|
||||
"codepoint",
|
||||
"script",
|
||||
"block"
|
||||
],
|
||||
"integration": {
|
||||
"contextVersion": 1,
|
||||
"launchModes": ["navigate", "new-tab"],
|
||||
@@ -21,6 +31,20 @@
|
||||
"crossOriginIsolated": false,
|
||||
"topLevelContext": false
|
||||
},
|
||||
"io": {
|
||||
"accepts": [],
|
||||
"produces": [
|
||||
{
|
||||
"mediaType": "application/json",
|
||||
"extensions": [".json"],
|
||||
"label": "Pinned data identity report"
|
||||
}
|
||||
]
|
||||
},
|
||||
"capabilities": {
|
||||
"required": ["workers"],
|
||||
"optional": ["clipboard-write", "intl-segmenter"]
|
||||
},
|
||||
"privacy": {
|
||||
"processing": "local",
|
||||
"fileUploads": false,
|
||||
|
||||
+21
-9
@@ -1,6 +1,8 @@
|
||||
import type {
|
||||
CatalogResult,
|
||||
CharacterDetail,
|
||||
CodeChartItem,
|
||||
ConfusableAnalysis,
|
||||
UnicodeManifest,
|
||||
UnihanDetail,
|
||||
} from "./types";
|
||||
@@ -100,6 +102,20 @@ export async function characterDetail(
|
||||
return request<CharacterDetail | undefined>("detail", { point });
|
||||
}
|
||||
|
||||
export async function codeChartRange(
|
||||
start: number,
|
||||
count: number,
|
||||
): Promise<CodeChartItem[]> {
|
||||
if (!Number.isSafeInteger(start) || start < 0 || start > 0x10ffff)
|
||||
throw new RangeError("Chart start is outside the Unicode code space.");
|
||||
if (!Number.isSafeInteger(count) || count < 1 || count > 512)
|
||||
throw new RangeError(
|
||||
"Chart window must contain between 1 and 512 code points.",
|
||||
);
|
||||
await initializeCatalog();
|
||||
return request<CodeChartItem[]>("range", { start, count });
|
||||
}
|
||||
|
||||
export async function unihanDetail(
|
||||
point: number,
|
||||
): Promise<UnihanDetail | undefined> {
|
||||
@@ -107,14 +123,10 @@ export async function unihanDetail(
|
||||
return request<UnihanDetail | undefined>("unihan", { point });
|
||||
}
|
||||
|
||||
export async function analyzeConfusables(input: string): Promise<{
|
||||
input: string;
|
||||
normalized: string;
|
||||
skeleton: string;
|
||||
scripts: string[];
|
||||
mixedScripts: boolean;
|
||||
mappings: Array<{ source: string; target: string; point: string }>;
|
||||
}> {
|
||||
export async function analyzeConfusables(
|
||||
input: string,
|
||||
comparison = "",
|
||||
): Promise<ConfusableAnalysis> {
|
||||
await initializeCatalog();
|
||||
return request("confusables", { input });
|
||||
return request("confusables", { input, comparison });
|
||||
}
|
||||
|
||||
@@ -0,0 +1,133 @@
|
||||
import { segmentGraphemes } from "@add-ideas/toolbox-helpers";
|
||||
|
||||
import { codePointLabel } from "./model";
|
||||
|
||||
export interface GraphemeDiagnostic {
|
||||
readonly index: number;
|
||||
readonly utf16Start: number;
|
||||
readonly utf16End: number;
|
||||
readonly value: string;
|
||||
readonly codePoints: readonly string[];
|
||||
readonly utf8Bytes: number;
|
||||
readonly extendedPictographic: boolean;
|
||||
readonly containsZwj: boolean;
|
||||
readonly variation: "text" | "emoji" | "none";
|
||||
readonly skinTones: readonly string[];
|
||||
readonly regionalIndicators: number;
|
||||
readonly keycap: boolean;
|
||||
readonly tagCharacters: number;
|
||||
readonly unpairedSurrogate: boolean;
|
||||
}
|
||||
|
||||
export function inspectGraphemeClusters(
|
||||
input: string,
|
||||
): readonly GraphemeDiagnostic[] {
|
||||
if (input.length > 1_000_000)
|
||||
throw new RangeError(
|
||||
"Grapheme inspection is limited to 1,000,000 UTF-16 units.",
|
||||
);
|
||||
const clusters = segmentGraphemes(input);
|
||||
if (clusters.length > 200_000)
|
||||
throw new RangeError(
|
||||
"Grapheme inspection exceeds the 200,000-cluster limit.",
|
||||
);
|
||||
let cursor = 0;
|
||||
return Object.freeze(
|
||||
clusters.map(({ segment: value }, index) => {
|
||||
const start = cursor;
|
||||
cursor += value.length;
|
||||
const points = [...value].map((character) => character.codePointAt(0)!);
|
||||
const skinTones = points
|
||||
.filter((point) => point >= 0x1f3fb && point <= 0x1f3ff)
|
||||
.map(codePointLabel);
|
||||
return Object.freeze({
|
||||
index: index + 1,
|
||||
utf16Start: start,
|
||||
utf16End: cursor,
|
||||
value,
|
||||
codePoints: Object.freeze(points.map(codePointLabel)),
|
||||
utf8Bytes: new TextEncoder().encode(value).byteLength,
|
||||
extendedPictographic: /\p{Extended_Pictographic}/u.test(value),
|
||||
containsZwj: points.includes(0x200d),
|
||||
variation: points.includes(0xfe0f)
|
||||
? "emoji"
|
||||
: points.includes(0xfe0e)
|
||||
? "text"
|
||||
: "none",
|
||||
skinTones: Object.freeze(skinTones),
|
||||
regionalIndicators: points.filter(
|
||||
(point) => point >= 0x1f1e6 && point <= 0x1f1ff,
|
||||
).length,
|
||||
keycap: points.includes(0x20e3),
|
||||
tagCharacters: points.filter(
|
||||
(point) => point >= 0xe0020 && point <= 0xe007f,
|
||||
).length,
|
||||
unpairedSurrogate: [...value].some((character) => {
|
||||
const point = character.codePointAt(0)!;
|
||||
return point >= 0xd800 && point <= 0xdfff;
|
||||
}),
|
||||
});
|
||||
}),
|
||||
);
|
||||
}
|
||||
|
||||
function oneCluster(input: string): string {
|
||||
if (input.length > 256)
|
||||
throw new RangeError("Emoji editing is limited to 256 UTF-16 units.");
|
||||
const clusters = segmentGraphemes(input);
|
||||
if (clusters.length !== 1)
|
||||
throw new RangeError(
|
||||
"Emoji editing requires exactly one grapheme cluster.",
|
||||
);
|
||||
return clusters[0]!.segment;
|
||||
}
|
||||
|
||||
export function setEmojiPresentation(
|
||||
input: string,
|
||||
presentation: "default" | "text" | "emoji",
|
||||
): string {
|
||||
const cluster = oneCluster(input);
|
||||
const points = [...cluster]
|
||||
.map((character) => character.codePointAt(0)!)
|
||||
.filter((point) => point !== 0xfe0e && point !== 0xfe0f);
|
||||
if (points.includes(0x200d))
|
||||
throw new RangeError(
|
||||
"Variation-selector editing of ZWJ sequences is withheld because selectors apply to individual components.",
|
||||
);
|
||||
if (points.some((point) => point >= 0x1f3fb && point <= 0x1f3ff))
|
||||
throw new RangeError(
|
||||
"Variation-selector editing of an emoji-modifier sequence is withheld because selector ordering is sequence-specific.",
|
||||
);
|
||||
if (presentation === "default") return String.fromCodePoint(...points);
|
||||
if (!points.length) throw new RangeError("Emoji base is missing.");
|
||||
points.splice(1, 0, presentation === "emoji" ? 0xfe0f : 0xfe0e);
|
||||
return String.fromCodePoint(...points);
|
||||
}
|
||||
|
||||
export function setEmojiSkinTone(
|
||||
input: string,
|
||||
tone: 0 | 1 | 2 | 3 | 4 | 5,
|
||||
): string {
|
||||
const cluster = oneCluster(input);
|
||||
const characters = [...cluster].filter((character) => {
|
||||
const point = character.codePointAt(0)!;
|
||||
return (
|
||||
(point < 0x1f3fb || point > 0x1f3ff) &&
|
||||
(tone === 0 || (point !== 0xfe0e && point !== 0xfe0f))
|
||||
);
|
||||
});
|
||||
const baseIndex = characters.findIndex((character) =>
|
||||
/\p{Extended_Pictographic}/u.test(character),
|
||||
);
|
||||
if (baseIndex < 0)
|
||||
throw new RangeError("No extended-pictographic emoji base was found.");
|
||||
if (tone)
|
||||
characters.splice(baseIndex + 1, 0, String.fromCodePoint(0x1f3fa + tone));
|
||||
return characters.join("");
|
||||
}
|
||||
|
||||
export function joinEmojiWithZwj(parts: readonly string[]): string {
|
||||
if (parts.length < 2 || parts.length > 8)
|
||||
throw new RangeError("ZWJ composition requires 2–8 grapheme parts.");
|
||||
return parts.map(oneCluster).join("\u200D");
|
||||
}
|
||||
@@ -9,6 +9,44 @@ export function codePointLabel(point: number): string {
|
||||
return `U+${point.toString(16).toUpperCase().padStart(4, "0")}`;
|
||||
}
|
||||
|
||||
export function codePointFallbackStatus(
|
||||
point: number,
|
||||
): "unassigned" | "private-use" | "surrogate" | "noncharacter" {
|
||||
if (!Number.isSafeInteger(point) || point < 0 || point > 0x10ffff)
|
||||
throw new RangeError("Code point must be from U+0000 to U+10FFFF.");
|
||||
if (point >= 0xd800 && point <= 0xdfff) return "surrogate";
|
||||
if (
|
||||
(point >= 0xe000 && point <= 0xf8ff) ||
|
||||
(point >= 0xf0000 && point <= 0xffffd) ||
|
||||
(point >= 0x100000 && point <= 0x10fffd)
|
||||
)
|
||||
return "private-use";
|
||||
if ((point & 0xffff) >= 0xfffe || (point >= 0xfdd0 && point <= 0xfdef))
|
||||
return "noncharacter";
|
||||
return "unassigned";
|
||||
}
|
||||
|
||||
export function parseCodeChartRange(
|
||||
startInput: string,
|
||||
endInput: string,
|
||||
): { start: number; end: number } {
|
||||
const parse = (input: string, label: string) => {
|
||||
const value = input.trim().replace(/^U\+/iu, "").replace(/^0x/iu, "");
|
||||
if (!/^[0-9a-f]{1,6}$/iu.test(value))
|
||||
throw new SyntaxError(
|
||||
`${label} must be a hexadecimal Unicode code point.`,
|
||||
);
|
||||
const point = Number.parseInt(value, 16);
|
||||
if (point > 0x10ffff) throw new RangeError(`${label} exceeds U+10FFFF.`);
|
||||
return point;
|
||||
};
|
||||
const start = parse(startInput, "Range start");
|
||||
const end = parse(endInput, "Range end");
|
||||
if (start > end)
|
||||
throw new RangeError("Range start must not follow range end.");
|
||||
return { start, end };
|
||||
}
|
||||
|
||||
export function parseCodePointList(input: string): number[] {
|
||||
const value = input.trim();
|
||||
if (!value) return [];
|
||||
|
||||
@@ -38,9 +38,13 @@ export interface UnicodeManifest {
|
||||
};
|
||||
blockRanges: UnicodeRange[];
|
||||
scriptRanges: UnicodeRange[];
|
||||
identifierStatusFile: string;
|
||||
identifierTypeFile: string;
|
||||
emojiCount: number;
|
||||
namedSequenceCount: number;
|
||||
confusableMappingCount: number;
|
||||
identifierStatusRangeCount: number;
|
||||
identifierTypeRangeCount: number;
|
||||
unihanCount: number;
|
||||
sourceIdentity: Array<{
|
||||
id: string;
|
||||
@@ -50,6 +54,40 @@ export interface UnicodeManifest {
|
||||
}>;
|
||||
}
|
||||
|
||||
export interface UnicodeSecurityDiagnostic {
|
||||
severity: "info" | "warning" | "danger";
|
||||
code: string;
|
||||
message: string;
|
||||
points: string[];
|
||||
}
|
||||
|
||||
export interface ConfusableAnalysis {
|
||||
input: string;
|
||||
normalized: string;
|
||||
skeleton: string;
|
||||
scripts: string[];
|
||||
mixedScripts: boolean;
|
||||
restrictionLevel:
|
||||
| "ASCII"
|
||||
| "single-script"
|
||||
| "highly-restrictive-script-mix"
|
||||
| "mixed-script-review"
|
||||
| "restricted-code-point";
|
||||
identifierStatus: Array<{
|
||||
character: string;
|
||||
point: string;
|
||||
status: string;
|
||||
types: string[];
|
||||
}>;
|
||||
mappings: Array<{ source: string; target: string; point: string }>;
|
||||
diagnostics: UnicodeSecurityDiagnostic[];
|
||||
comparison?: {
|
||||
input: string;
|
||||
skeleton: string;
|
||||
collision: boolean;
|
||||
};
|
||||
}
|
||||
|
||||
export interface CharacterSummary {
|
||||
kind: "character";
|
||||
point: number;
|
||||
@@ -76,6 +114,22 @@ export interface CharacterDetail extends CharacterSummary {
|
||||
titlecase?: string;
|
||||
}
|
||||
|
||||
export type CodePointStatus =
|
||||
"assigned" | "unassigned" | "private-use" | "surrogate" | "noncharacter";
|
||||
|
||||
export interface CodeChartItem {
|
||||
point: number;
|
||||
hex: string;
|
||||
character: string;
|
||||
name: string;
|
||||
status: CodePointStatus;
|
||||
category: string;
|
||||
script: string;
|
||||
block: string;
|
||||
age: string;
|
||||
assigned?: CharacterSummary;
|
||||
}
|
||||
|
||||
export interface EmojiSummary {
|
||||
kind: "emoji";
|
||||
points: number[];
|
||||
|
||||
+1
-1
@@ -1 +1 @@
|
||||
export const APP_VERSION = "0.1.0";
|
||||
export const APP_VERSION = "0.2.0";
|
||||
|
||||
+271
-19
@@ -4,11 +4,15 @@ import type {
|
||||
CharacterDetail,
|
||||
CharacterSummary,
|
||||
CharacterTuple,
|
||||
CodeChartItem,
|
||||
ConfusableAnalysis,
|
||||
EmojiSummary,
|
||||
NamedSequenceSummary,
|
||||
UnicodeManifest,
|
||||
UnihanDetail,
|
||||
UnicodeRange,
|
||||
} from "../unicode/types";
|
||||
import { codePointFallbackStatus } from "../unicode/model";
|
||||
|
||||
declare const self: DedicatedWorkerGlobalScope;
|
||||
|
||||
@@ -44,6 +48,8 @@ let sequences: SequenceTuple[] = [];
|
||||
let byPoint = new Map<number, CharacterTuple>();
|
||||
let unihan: Map<number, UnihanTuple> | undefined;
|
||||
let confusables: Map<string, string> | undefined;
|
||||
let identifierStatusRanges: UnicodeRange[] | undefined;
|
||||
let identifierTypeRanges: UnicodeRange[] | undefined;
|
||||
let initialization: Promise<UnicodeManifest> | undefined;
|
||||
|
||||
async function fetchJson<T>(filename: string): Promise<T> {
|
||||
@@ -121,6 +127,93 @@ function detail(record: CharacterTuple): CharacterDetail {
|
||||
};
|
||||
}
|
||||
|
||||
function blockFor(point: number): string {
|
||||
const ranges = requireManifest().blockRanges;
|
||||
let low = 0;
|
||||
let high = ranges.length - 1;
|
||||
while (low <= high) {
|
||||
const middle = (low + high) >> 1;
|
||||
const range = ranges[middle]!;
|
||||
if (point < range.start) high = middle - 1;
|
||||
else if (point > range.end) low = middle + 1;
|
||||
else return range.value;
|
||||
}
|
||||
return "No_Block";
|
||||
}
|
||||
|
||||
function rangeValue(
|
||||
ranges: readonly { start: number; end: number; value: string }[],
|
||||
point: number,
|
||||
fallback: string,
|
||||
): string {
|
||||
let low = 0;
|
||||
let high = ranges.length - 1;
|
||||
while (low <= high) {
|
||||
const middle = (low + high) >> 1;
|
||||
const range = ranges[middle]!;
|
||||
if (point < range.start) high = middle - 1;
|
||||
else if (point > range.end) low = middle + 1;
|
||||
else return range.value;
|
||||
}
|
||||
return fallback;
|
||||
}
|
||||
|
||||
function chartRange(startInput: number, countInput: number): CodeChartItem[] {
|
||||
const start = Math.trunc(startInput);
|
||||
const count = Math.trunc(countInput);
|
||||
if (!Number.isSafeInteger(start) || start < 0 || start > 0x10ffff)
|
||||
throw new RangeError("Chart start is outside the Unicode code space.");
|
||||
if (!Number.isSafeInteger(count) || count < 1 || count > 512)
|
||||
throw new RangeError(
|
||||
"Chart window must contain between 1 and 512 code points.",
|
||||
);
|
||||
const end = Math.min(0x10ffff, start + count - 1);
|
||||
const output: CodeChartItem[] = [];
|
||||
for (let point = start; point <= end; point += 1) {
|
||||
const record = byPoint.get(point);
|
||||
if (record) {
|
||||
const assigned = summarize(record);
|
||||
output.push({
|
||||
point,
|
||||
hex: assigned.hex,
|
||||
character: assigned.character,
|
||||
name: assigned.name,
|
||||
status: "assigned",
|
||||
category: assigned.category,
|
||||
script: assigned.script,
|
||||
block: assigned.block,
|
||||
age: assigned.age,
|
||||
assigned,
|
||||
});
|
||||
continue;
|
||||
}
|
||||
const status = codePointFallbackStatus(point);
|
||||
output.push({
|
||||
point,
|
||||
hex: label(point),
|
||||
character:
|
||||
status === "surrogate"
|
||||
? String.fromCharCode(point)
|
||||
: String.fromCodePoint(point),
|
||||
name:
|
||||
status === "private-use"
|
||||
? "PRIVATE-USE CODE POINT"
|
||||
: status === "surrogate"
|
||||
? "UTF-16 SURROGATE CODE UNIT"
|
||||
: status === "noncharacter"
|
||||
? "UNICODE NONCHARACTER"
|
||||
: "UNASSIGNED CODE POINT",
|
||||
status,
|
||||
category:
|
||||
status === "private-use" ? "Co" : status === "surrogate" ? "Cs" : "Cn",
|
||||
script: "Unknown",
|
||||
block: blockFor(point),
|
||||
age: "Unassigned",
|
||||
});
|
||||
}
|
||||
return output;
|
||||
}
|
||||
|
||||
function directPoint(query: string): number | undefined {
|
||||
const value = query.trim();
|
||||
const explicit = /^(?:U\+|0x|\\u\{)([0-9a-f]{1,6})\}?$/iu.exec(value);
|
||||
@@ -236,23 +329,16 @@ async function getUnihan(point: number): Promise<UnihanDetail | undefined> {
|
||||
};
|
||||
}
|
||||
|
||||
async function confusableAnalysis(input: string) {
|
||||
if (input.length > 100_000)
|
||||
throw new RangeError(
|
||||
"Confusable analysis is limited to 100,000 UTF-16 code units.",
|
||||
);
|
||||
if (!confusables)
|
||||
confusables = new Map(
|
||||
(await fetchJson<ConfusableTuple[]>("confusables.json")).map(
|
||||
([source, target]) => [source, target],
|
||||
),
|
||||
);
|
||||
function confusableSkeleton(input: string): {
|
||||
normalized: string;
|
||||
skeleton: string;
|
||||
mappings: Array<{ source: string; target: string; point: string }>;
|
||||
} {
|
||||
const normalized = input.normalize("NFD");
|
||||
const mappings = [];
|
||||
let skeleton = "";
|
||||
const scripts = new Set<string>();
|
||||
for (const character of normalized) {
|
||||
const target = confusables.get(character) ?? character;
|
||||
const target = confusables?.get(character) ?? character;
|
||||
skeleton += target;
|
||||
if (target !== character)
|
||||
mappings.push({
|
||||
@@ -260,21 +346,182 @@ async function confusableAnalysis(input: string) {
|
||||
target,
|
||||
point: label(character.codePointAt(0) ?? 0),
|
||||
});
|
||||
const record = byPoint.get(character.codePointAt(0) ?? -1);
|
||||
}
|
||||
return { normalized, skeleton: skeleton.normalize("NFD"), mappings };
|
||||
}
|
||||
|
||||
function restrictionLevel(
|
||||
input: string,
|
||||
scripts: ReadonlySet<string>,
|
||||
restricted: number,
|
||||
): ConfusableAnalysis["restrictionLevel"] {
|
||||
if (restricted) return "restricted-code-point";
|
||||
if ([...input].every((character) => (character.codePointAt(0) ?? 128) < 128))
|
||||
return "ASCII";
|
||||
if (scripts.size <= 1) return "single-script";
|
||||
const values = [...scripts];
|
||||
const allowedMixes = [
|
||||
new Set(["Han", "Hiragana", "Katakana"]),
|
||||
new Set(["Han", "Bopomofo"]),
|
||||
new Set(["Han", "Hangul"]),
|
||||
];
|
||||
if (
|
||||
allowedMixes.some((allowed) =>
|
||||
values.every((script) => allowed.has(script)),
|
||||
)
|
||||
)
|
||||
return "highly-restrictive-script-mix";
|
||||
return "mixed-script-review";
|
||||
}
|
||||
|
||||
async function confusableAnalysis(
|
||||
input: string,
|
||||
comparisonInput = "",
|
||||
): Promise<ConfusableAnalysis> {
|
||||
if (input.length > 100_000)
|
||||
throw new RangeError(
|
||||
"Confusable analysis is limited to 100,000 UTF-16 code units.",
|
||||
);
|
||||
if (!confusables || !identifierStatusRanges || !identifierTypeRanges)
|
||||
[confusables, identifierStatusRanges, identifierTypeRanges] =
|
||||
await Promise.all([
|
||||
fetchJson<ConfusableTuple[]>("confusables.json").then(
|
||||
(records) =>
|
||||
new Map(records.map(([source, target]) => [source, target])),
|
||||
),
|
||||
fetchJson<UnicodeRange[]>(requireManifest().identifierStatusFile),
|
||||
fetchJson<UnicodeRange[]>(requireManifest().identifierTypeFile),
|
||||
]);
|
||||
if (comparisonInput.length > 100_000)
|
||||
throw new RangeError(
|
||||
"Confusable comparison is limited to 100,000 UTF-16 code units.",
|
||||
);
|
||||
const skeletonResult = confusableSkeleton(input);
|
||||
const scripts = new Set<string>();
|
||||
const identifierStatus: ConfusableAnalysis["identifierStatus"] = [];
|
||||
const diagnostics: ConfusableAnalysis["diagnostics"] = [];
|
||||
const bidiPoints: string[] = [];
|
||||
const ignorablePoints: string[] = [];
|
||||
const repeatedMarks: string[] = [];
|
||||
for (const character of input) {
|
||||
const point = character.codePointAt(0) ?? 0;
|
||||
const record = byPoint.get(point);
|
||||
if (record) {
|
||||
const script =
|
||||
requireManifest().dictionaries.scripts[record[3]] ?? "Unknown";
|
||||
if (script !== "Common" && script !== "Inherited" && script !== "Unknown")
|
||||
scripts.add(script);
|
||||
}
|
||||
const status = rangeValue(identifierStatusRanges!, point, "Restricted");
|
||||
const types = rangeValue(
|
||||
identifierTypeRanges!,
|
||||
point,
|
||||
"Not_Character",
|
||||
).split(/\s+/u);
|
||||
identifierStatus.push({ character, point: label(point), status, types });
|
||||
if (
|
||||
(point >= 0x202a && point <= 0x202e) ||
|
||||
(point >= 0x2066 && point <= 0x2069) ||
|
||||
point === 0x061c ||
|
||||
point === 0x200e ||
|
||||
point === 0x200f
|
||||
)
|
||||
bidiPoints.push(label(point));
|
||||
if (/\p{Default_Ignorable_Code_Point}/u.test(character))
|
||||
ignorablePoints.push(label(point));
|
||||
}
|
||||
const previousMarks = new Set<number>();
|
||||
for (const character of input.normalize("NFD")) {
|
||||
const point = character.codePointAt(0) ?? 0;
|
||||
const record = byPoint.get(point);
|
||||
if (record?.[6]) {
|
||||
if (previousMarks.has(point)) repeatedMarks.push(label(point));
|
||||
previousMarks.add(point);
|
||||
} else previousMarks.clear();
|
||||
}
|
||||
const restricted = identifierStatus.filter(
|
||||
(entry) => entry.status !== "Allowed",
|
||||
);
|
||||
if (restricted.length)
|
||||
diagnostics.push({
|
||||
severity: "danger",
|
||||
code: "restricted-identifier-status",
|
||||
message: `${restricted.length} code point(s) are Restricted by Unicode 17.0 Identifier_Status.`,
|
||||
points: [...new Set(restricted.map((entry) => entry.point))],
|
||||
});
|
||||
if (scripts.size > 1)
|
||||
diagnostics.push({
|
||||
severity: "warning",
|
||||
code: "mixed-primary-scripts",
|
||||
message: `Identifier uses primary scripts ${[...scripts].sort().join(", ")}.`,
|
||||
points: [],
|
||||
});
|
||||
if (skeletonResult.mappings.length)
|
||||
diagnostics.push({
|
||||
severity: "info",
|
||||
code: "confusable-mappings",
|
||||
message: `${skeletonResult.mappings.length} UTS #39 confusable mapping(s) contribute to the skeleton.`,
|
||||
points: [...new Set(skeletonResult.mappings.map((item) => item.point))],
|
||||
});
|
||||
if (bidiPoints.length)
|
||||
diagnostics.push({
|
||||
severity: "danger",
|
||||
code: "bidi-controls",
|
||||
message:
|
||||
"Bidirectional formatting/control characters can change visual order.",
|
||||
points: [...new Set(bidiPoints)],
|
||||
});
|
||||
if (ignorablePoints.length)
|
||||
diagnostics.push({
|
||||
severity: "warning",
|
||||
code: "default-ignorable",
|
||||
message:
|
||||
"Default-ignorable code points may be invisible in ordinary rendering.",
|
||||
points: [...new Set(ignorablePoints)],
|
||||
});
|
||||
if (repeatedMarks.length)
|
||||
diagnostics.push({
|
||||
severity: "warning",
|
||||
code: "repeated-combining-mark",
|
||||
message: "A combining mark repeats within one combining sequence.",
|
||||
points: [...new Set(repeatedMarks)],
|
||||
});
|
||||
if (input !== input.normalize("NFC"))
|
||||
diagnostics.push({
|
||||
severity: "info",
|
||||
code: "not-nfc",
|
||||
message: "Input is not NFC-normalized.",
|
||||
points: [],
|
||||
});
|
||||
const comparison = comparisonInput
|
||||
? {
|
||||
input: comparisonInput,
|
||||
skeleton: confusableSkeleton(comparisonInput).skeleton,
|
||||
collision:
|
||||
comparisonInput.normalize("NFD") !== skeletonResult.normalized &&
|
||||
confusableSkeleton(comparisonInput).skeleton ===
|
||||
skeletonResult.skeleton,
|
||||
}
|
||||
: undefined;
|
||||
if (comparison?.collision)
|
||||
diagnostics.push({
|
||||
severity: "danger",
|
||||
code: "skeleton-collision",
|
||||
message:
|
||||
"The two different inputs have the same UTS #39 confusable skeleton.",
|
||||
points: [],
|
||||
});
|
||||
return {
|
||||
input,
|
||||
normalized,
|
||||
skeleton: skeleton.normalize("NFD"),
|
||||
normalized: skeletonResult.normalized,
|
||||
skeleton: skeletonResult.skeleton,
|
||||
scripts: [...scripts].sort(),
|
||||
mixedScripts: scripts.size > 1,
|
||||
mappings,
|
||||
restrictionLevel: restrictionLevel(input, scripts, restricted.length),
|
||||
identifierStatus,
|
||||
mappings: skeletonResult.mappings,
|
||||
diagnostics,
|
||||
...(comparison ? { comparison } : {}),
|
||||
};
|
||||
}
|
||||
|
||||
@@ -297,10 +544,15 @@ self.addEventListener(
|
||||
else if (type === "detail") {
|
||||
const record = byPoint.get(Number(payload.point));
|
||||
value = record ? detail(record) : undefined;
|
||||
} else if (type === "unihan")
|
||||
} else if (type === "range")
|
||||
value = chartRange(Number(payload.start), Number(payload.count));
|
||||
else if (type === "unihan")
|
||||
value = await getUnihan(Number(payload.point));
|
||||
else if (type === "confusables")
|
||||
value = await confusableAnalysis(String(payload.input ?? ""));
|
||||
value = await confusableAnalysis(
|
||||
String(payload.input ?? ""),
|
||||
String(payload.comparison ?? ""),
|
||||
);
|
||||
else throw new Error(`Unknown catalog operation: ${type}`);
|
||||
self.postMessage({ id, ok: true, value });
|
||||
} catch (reason) {
|
||||
|
||||
@@ -40,7 +40,7 @@ test("serves the release identity and hardened headers", async ({
|
||||
const manifest = await request.get("/deep/nested/unicode/toolbox-app.json");
|
||||
await expect(manifest.json()).resolves.toMatchObject({
|
||||
id: "de.add-ideas.unicode-tools",
|
||||
version: "0.1.0",
|
||||
version: "0.2.0",
|
||||
entry: "./",
|
||||
});
|
||||
});
|
||||
@@ -59,3 +59,62 @@ test("searches the pinned catalog in a worker and shows character properties", a
|
||||
await expect(page.getByText("Common / Emoticons")).toBeVisible();
|
||||
expect(external).toEqual([]);
|
||||
});
|
||||
|
||||
test("virtualizes a range and labels surrogate code units explicitly", async ({
|
||||
page,
|
||||
}) => {
|
||||
const external = await localOnly(page);
|
||||
await page.goto("/deep/nested/unicode/");
|
||||
await page.getByRole("button", { name: "Code chart" }).click();
|
||||
await page.getByLabel("Range start").fill("U+D7F0");
|
||||
await page.getByLabel("Range end").fill("U+D810");
|
||||
await page.getByRole("button", { name: "Open range" }).click();
|
||||
const surrogate = page.getByRole("gridcell", {
|
||||
name: /U\+D800 UTF-16 SURROGATE CODE UNIT, surrogate/u,
|
||||
});
|
||||
await expect(surrogate).toBeVisible({ timeout: 60_000 });
|
||||
await surrogate.click();
|
||||
await expect(page.getByText(/not a Unicode scalar value/u)).toBeVisible();
|
||||
await expect(
|
||||
page.getByRole("button", { name: "Copy scalar" }),
|
||||
).toBeDisabled();
|
||||
await surrogate.press("ArrowRight");
|
||||
await expect(
|
||||
page.getByRole("gridcell", { name: /U\+D801 UTF-16 SURROGATE/u }),
|
||||
).toBeFocused();
|
||||
expect(external).toEqual([]);
|
||||
});
|
||||
|
||||
test("inspects graphemes and compares UTS #39 confusable skeletons locally", async ({
|
||||
page,
|
||||
}) => {
|
||||
const external = await localOnly(page);
|
||||
await page.goto("/deep/nested/unicode/");
|
||||
await page.getByRole("button", { name: "Text inspector" }).click();
|
||||
await page.getByLabel("Text to inspect").fill("👨👩👧👦 🇩🇪");
|
||||
await page.getByRole("button", { name: "Inspect text" }).click();
|
||||
await expect(page.getByText("pictographic · ZWJ")).toBeVisible();
|
||||
await expect(page.getByText("2 regional indicator(s)")).toBeVisible();
|
||||
|
||||
await page.getByRole("button", { name: "Normalize & confusables" }).click();
|
||||
const normalization = page
|
||||
.getByRole("heading", {
|
||||
name: "Compare forms and compute a confusable skeleton",
|
||||
})
|
||||
.locator("..");
|
||||
await normalization
|
||||
.getByRole("textbox", { name: "Text", exact: true })
|
||||
.fill("раypal");
|
||||
await normalization
|
||||
.getByRole("textbox", { name: "Compare identifier (optional)" })
|
||||
.fill("paypal");
|
||||
await page.getByRole("button", { name: "Analyze confusables" }).click();
|
||||
await expect(
|
||||
page.getByRole("heading", { name: "UTS #39 confusable analysis" }),
|
||||
).toBeVisible({ timeout: 60_000 });
|
||||
await expect(
|
||||
page.getByText("yes — review", { exact: true }).last(),
|
||||
).toBeVisible();
|
||||
await expect(page.getByText(/mixed-script/u).first()).toBeVisible();
|
||||
expect(external).toEqual([]);
|
||||
});
|
||||
|
||||
@@ -0,0 +1,18 @@
|
||||
import { expect, test } from "@playwright/test";
|
||||
|
||||
test("keeps the primary workspace inside a narrow viewport", async ({
|
||||
page,
|
||||
}) => {
|
||||
await page.goto("/deep/nested/unicode/");
|
||||
await expect(page.locator("main").first()).toBeVisible();
|
||||
await expect(
|
||||
page.locator("main .loading, main .workbench-loading"),
|
||||
).toHaveCount(0);
|
||||
|
||||
const widths = await page.evaluate(() => ({
|
||||
content: document.documentElement.scrollWidth,
|
||||
viewport: document.documentElement.clientWidth,
|
||||
}));
|
||||
expect(widths.viewport).toBeLessThanOrEqual(430);
|
||||
expect(widths.content).toBeLessThanOrEqual(widths.viewport + 1);
|
||||
});
|
||||
@@ -1,9 +1,17 @@
|
||||
import { describe, expect, it } from "vitest";
|
||||
import {
|
||||
inspectGraphemeClusters,
|
||||
joinEmojiWithZwj,
|
||||
setEmojiPresentation,
|
||||
setEmojiSkinTone,
|
||||
} from "../../src/unicode/graphemes";
|
||||
import {
|
||||
codePointFallbackStatus,
|
||||
codePointLabel,
|
||||
compareNormalization,
|
||||
decodeCodePointList,
|
||||
inspectText,
|
||||
parseCodeChartRange,
|
||||
parseCodePointList,
|
||||
} from "../../src/unicode/model";
|
||||
|
||||
@@ -26,4 +34,48 @@ describe("Unicode helper integration", () => {
|
||||
expect(result.forms.find((item) => item.form === "NFC")?.value).toBe("é");
|
||||
expect(codePointLabel(0x1f600)).toBe("U+1F600");
|
||||
});
|
||||
|
||||
it("classifies non-character code-space regions explicitly", () => {
|
||||
expect(codePointFallbackStatus(0x0378)).toBe("unassigned");
|
||||
expect(codePointFallbackStatus(0xe000)).toBe("private-use");
|
||||
expect(codePointFallbackStatus(0xd800)).toBe("surrogate");
|
||||
expect(codePointFallbackStatus(0xfdd0)).toBe("noncharacter");
|
||||
expect(codePointFallbackStatus(0x10ffff)).toBe("noncharacter");
|
||||
});
|
||||
|
||||
it("accepts bounded hexadecimal chart ranges", () => {
|
||||
expect(parseCodeChartRange("U+D7F0", "0xE010")).toEqual({
|
||||
start: 0xd7f0,
|
||||
end: 0xe010,
|
||||
});
|
||||
expect(() => parseCodeChartRange("110000", "110001")).toThrow(/U\+10FFFF/u);
|
||||
expect(() => parseCodeChartRange("100", "FF")).toThrow(/must not follow/u);
|
||||
});
|
||||
|
||||
it("reports emoji structure at grapheme-cluster boundaries", () => {
|
||||
const result = inspectGraphemeClusters("👨👩👧👦 🇩🇪 1️⃣");
|
||||
expect(result[0]).toMatchObject({
|
||||
value: "👨👩👧👦",
|
||||
containsZwj: true,
|
||||
extendedPictographic: true,
|
||||
});
|
||||
expect(result.find((item) => item.value === "🇩🇪")?.regionalIndicators).toBe(
|
||||
2,
|
||||
);
|
||||
expect(result.find((item) => item.value === "1️⃣")?.keycap).toBe(true);
|
||||
});
|
||||
|
||||
it("edits bounded emoji sequences without claiming RGI validity", () => {
|
||||
expect(setEmojiPresentation("☕", "emoji")).toBe("☕️");
|
||||
expect(setEmojiSkinTone("👍", 1)).toBe("👍🏻");
|
||||
expect(setEmojiSkinTone("☝️", 1)).toBe("☝🏻");
|
||||
expect(joinEmojiWithZwj(["👩", "💻"])).toBe("👩💻");
|
||||
expect(() => setEmojiPresentation("👩💻", "text")).toThrow(
|
||||
/individual components/u,
|
||||
);
|
||||
expect(() => setEmojiPresentation("👍🏻", "text")).toThrow(
|
||||
/sequence-specific/u,
|
||||
);
|
||||
expect(() => joinEmojiWithZwj(["👩"])).toThrow(/2–8/u);
|
||||
});
|
||||
});
|
||||
|
||||
Reference in New Issue
Block a user