56 lines
4.3 KiB
Markdown
56 lines
4.3 KiB
Markdown
# Data Tools
|
|
|
|
Inspect, format, query, flatten and convert structured data locally in the browser. Data Tools is a standalone application in the [add·ideas Toolbox](https://git.add-ideas.de/lotobo/toolbox-portal); input text and files never leave the current browser.
|
|
|
|
## Version 0.2.0
|
|
|
|
- Detect or explicitly select JSON, YAML 1.2, TOML, XML, CSV, TSV and NDJSON.
|
|
- Keep JSON integer and decimal lexemes exact, including values beyond IEEE-754 range.
|
|
- Inspect a bounded typed tree, a tabular projection and flattened scalar leaves.
|
|
- Query with RFC 6901 JSON Pointer or a deliberately small, non-evaluating path subset.
|
|
- Infer a bounded, sample-labelled JSON Schema with field presence, nullability, mixed types and exact-number provenance.
|
|
- Preview immutable RFC 6901 add/replace/remove edits, inspect the source-format serialization disclosure, and explicitly accept any loss before applying it.
|
|
- Format and convert between every supported output with a visible loss/coercion report.
|
|
- Drive import, export and format selectors from one ordered adapter catalogue with a concrete parser and writer registry for every advertised format.
|
|
- Export CSV/TSV with opt-in spreadsheet formula neutralisation, enabled by default.
|
|
- Open UTF-8 files or edit text directly; parsing runs through the shared
|
|
cancellable disposable-worker protocol with a hard deadline, while the
|
|
previous valid result remains visible until a replacement is ready.
|
|
|
|
### Format semantics
|
|
|
|
| Format | v0.1 behaviour |
|
|
| --------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
| JSON | Duplicate keys and prototype-affecting keys are rejected. Number lexemes are stored and emitted exactly. |
|
|
| YAML | YAML 1.2 core schema, unique string keys, BigInt integers, maximum 50 alias expansions. Custom tags and merge keys are rejected. |
|
|
| TOML | Integers are BigInt-backed and dates/times retain their TOML kind. Floats use the browser's binary representation and are diagnosed. |
|
|
| XML | `DOCTYPE`, entity declarations and XInclude are rejected before inert parsing. Elements, namespaces, attributes, text, CDATA, comments and processing instructions use an explicit ordered record model. |
|
|
| CSV / TSV | Fields remain strings. Quoting and line endings are parsed without dynamic typing. The first row can be used as table headings. |
|
|
| NDJSON | Every non-empty line is parsed as lossless JSON. Blank lines and the exact failing line are reported. |
|
|
|
|
The safe path subset supports `$`, `.key`, `['key']`, `[index]` and `[*]`. It intentionally has no recursive descent, filters, expressions, functions or script evaluation.
|
|
|
|
### Resource limits
|
|
|
|
The current production limits are 16 MiB of UTF-8 input, 8 million JavaScript characters, 100,000 model nodes, depth 128, 50,000 delimited rows, 2,000 columns, 250,000 cells, 1 million characters per field/edit value, 5,000 inference samples/fields, 4,096 query or edit-pointer characters, 1,000 query matches and 16 million generated characters. Disposable parses have a 30-second deadline. UI previews are smaller than the model and state when they are truncated.
|
|
|
|
See [Architecture](docs/ARCHITECTURE.md), [Privacy and security](docs/PRIVACY-SECURITY.md) and [Accessibility](docs/ACCESSIBILITY.md) for the design and threat model.
|
|
|
|
## Development
|
|
|
|
Requires Node.js 22 and npm 11.
|
|
|
|
```sh
|
|
npm ci
|
|
npm run check
|
|
npm run test:browser
|
|
```
|
|
|
|
## Release
|
|
|
|
`npm run release:artifact` creates a deterministic `release/data-tools-0.2.0.zip` and checksum sidecar.
|
|
|
|
## Licence
|
|
|
|
GPL-3.0-or-later. Runtime dependencies retain their own licences; see [THIRD_PARTY_NOTICES.md](THIRD_PARTY_NOTICES.md).
|