41 lines
3.3 KiB
Markdown
41 lines
3.3 KiB
Markdown
# Log Tools
|
|
|
|
Stream, inspect, correlate, redact, and export logs locally in the browser.
|
|
|
|
Log Tools is a standalone local-first application in the [add·ideas Toolbox](https://git.add-ideas.de/lotobo/toolbox-portal). Local files are consumed through `File.stream()` and a streaming `TextDecoder`; the application never turns a complete file into one string and never uploads it.
|
|
|
|
## Version 0.1 scope
|
|
|
|
- Incremental UTF-8 file scanning with byte/line progress, cancellation, a five-million-line stop, and an eight-GiB input gate
|
|
- Bounded pasted/sample fallback and automatic or explicit plain, JSON Lines, nginx common/combined, syslog, and logfmt parsing
|
|
- Configurable streaming assembly for Java, .NET, Python, and generic indented multiline records, retaining their physical start/end lines
|
|
- Normalized level, timestamp, source, message, and common structured fields with malformed-line diagnostics
|
|
- Whole-scan bounded level/source/hour aggregation plus a 10,000-record retained preview for literal search and field filters
|
|
- Multi-file temporal merging for up to 16 separately streamed logs, with first-event offsets, backward-time diagnostics, W3C trace/span and common request-ID grouping, cross-file duration/source summaries, and retained-evidence caveats
|
|
- Focused OTLP/JSON log export with nanosecond timestamp strings, severity mapping, string attributes, trace/span context, and no invented timestamp for undated events
|
|
- Deterministic email, IP, UUID, credential/query-secret, long-number, and exact-literal redaction recipes with generic or salted correlation tokens
|
|
- Filtered CSV, normalized NDJSON, or original-line text export from the retained preview, plus a complete second streaming pass with cancellation and progress; supported browsers write directly to a selected file and others use a 256 MiB Blob fallback
|
|
- Inert ANSI/HTML handling, responsive Toolbox shell themes, nested-path offline PWA support, and deterministic release archives
|
|
|
|
The preview retains up to the first 10,000 nonblank parsed or malformed records and stops earlier at a 32 MiB retained-character budget; counts and timeline buckets continue across the scanned portion. Multi-file mode retains at most 3,000 records per file (50,000/128 MiB across the analysis, 20,000 merged table events), so missing parent spans mean “not present in retained logs,” not necessarily a broken trace. OTLP/JSON export is capped at 20,000 records/64 MiB of retained text. Interactive search and redaction preview operate on bounded sets, while complete single-source export reapplies the same filters and recipes during a second pass over every scanned physical line. Auto-detection and automatic multiline classification are heuristic. The syslog parser covers RFC 5424 and a common RFC 3164 shape, nginx assumes the standard common/combined field order, and JSONL flattening is capped.
|
|
|
|
See [Architecture](docs/ARCHITECTURE.md) and [Privacy and security](docs/PRIVACY-SECURITY.md) for exact bounds and trust assumptions.
|
|
|
|
## Development
|
|
|
|
Requires Node.js 22 and npm 11.
|
|
|
|
```sh
|
|
npm ci
|
|
npm run check
|
|
npm run test:browser
|
|
```
|
|
|
|
## Release
|
|
|
|
`npm run release:artifact` creates deterministic `release/log-tools-0.2.0.zip` and checksum files.
|
|
|
|
## Licence
|
|
|
|
GPL-3.0-or-later. Runtime dependencies retain their own licences; see [THIRD_PARTY_NOTICES.md](THIRD_PARTY_NOTICES.md).
|