@@ -1,5 +1,13 @@
|
||||
# Changelog
|
||||
|
||||
## 0.2.0 - 2026-09-02
|
||||
|
||||
- Added configurable multiline assembly, multi-file temporal merge, trace/span
|
||||
and request-ID correlation with bounded retained evidence.
|
||||
- Added OTLP/JSON output and a complete second streaming pass for filtered,
|
||||
redacted export, with direct file-system writing where available and a
|
||||
bounded Blob fallback.
|
||||
|
||||
## 0.1.0 - 2026-09-01
|
||||
|
||||
- Added genuinely incremental `File.stream()` / `TextDecoder` scanning with progress, cancellation, line framing, and bounded memory.
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
==============================================================================
|
||||
@add-ideas/toolbox-contract@0.2.3
|
||||
@add-ideas/toolbox-contract@0.3.0
|
||||
Declared licence: Apache-2.0
|
||||
==============================================================================
|
||||
--- LICENSE ---
|
||||
@@ -198,7 +198,7 @@ Declared licence: Apache-2.0
|
||||
|
||||
|
||||
==============================================================================
|
||||
@add-ideas/toolbox-helpers@0.1.0
|
||||
@add-ideas/toolbox-helpers@0.2.0
|
||||
Declared licence: GPL-3.0-or-later
|
||||
==============================================================================
|
||||
--- LICENSE ---
|
||||
@@ -879,7 +879,7 @@ Public License instead of this License. But first, please read
|
||||
|
||||
|
||||
==============================================================================
|
||||
@add-ideas/toolbox-shell-react@0.2.3
|
||||
@add-ideas/toolbox-shell-react@0.3.0
|
||||
Declared licence: Apache-2.0
|
||||
==============================================================================
|
||||
--- LICENSE ---
|
||||
|
||||
+6
-3
@@ -8,13 +8,16 @@ Log Tools is a standalone local-first application in the [add·ideas Toolbox](ht
|
||||
|
||||
- Incremental UTF-8 file scanning with byte/line progress, cancellation, a five-million-line stop, and an eight-GiB input gate
|
||||
- Bounded pasted/sample fallback and automatic or explicit plain, JSON Lines, nginx common/combined, syslog, and logfmt parsing
|
||||
- Configurable streaming assembly for Java, .NET, Python, and generic indented multiline records, retaining their physical start/end lines
|
||||
- Normalized level, timestamp, source, message, and common structured fields with malformed-line diagnostics
|
||||
- Whole-scan bounded level/source/hour aggregation plus a 10,000-record retained preview for literal search and field filters
|
||||
- Multi-file temporal merging for up to 16 separately streamed logs, with first-event offsets, backward-time diagnostics, W3C trace/span and common request-ID grouping, cross-file duration/source summaries, and retained-evidence caveats
|
||||
- Focused OTLP/JSON log export with nanosecond timestamp strings, severity mapping, string attributes, trace/span context, and no invented timestamp for undated events
|
||||
- Deterministic email, IP, UUID, credential/query-secret, long-number, and exact-literal redaction recipes with generic or salted correlation tokens
|
||||
- Filtered CSV, normalized NDJSON, or original-line text export from the retained preview, with row/text caps and spreadsheet-safe CSV cells
|
||||
- Filtered CSV, normalized NDJSON, or original-line text export from the retained preview, plus a complete second streaming pass with cancellation and progress; supported browsers write directly to a selected file and others use a 256 MiB Blob fallback
|
||||
- Inert ANSI/HTML handling, responsive Toolbox shell themes, nested-path offline PWA support, and deterministic release archives
|
||||
|
||||
The preview retains up to the first 10,000 nonblank parsed or malformed lines and stops earlier at a 32 MiB retained-character budget; counts and timeline buckets continue across the scanned portion. Search, redaction preview, and export therefore cannot include later records dropped from the preview. Auto-detection is heuristic. The syslog parser covers RFC 5424 and a common RFC 3164 shape, nginx assumes the standard common/combined field order, JSONL flattening is capped, and multiline stack traces remain separate plain records.
|
||||
The preview retains up to the first 10,000 nonblank parsed or malformed records and stops earlier at a 32 MiB retained-character budget; counts and timeline buckets continue across the scanned portion. Multi-file mode retains at most 3,000 records per file (50,000/128 MiB across the analysis, 20,000 merged table events), so missing parent spans mean “not present in retained logs,” not necessarily a broken trace. OTLP/JSON export is capped at 20,000 records/64 MiB of retained text. Interactive search and redaction preview operate on bounded sets, while complete single-source export reapplies the same filters and recipes during a second pass over every scanned physical line. Auto-detection and automatic multiline classification are heuristic. The syslog parser covers RFC 5424 and a common RFC 3164 shape, nginx assumes the standard common/combined field order, and JSONL flattening is capped.
|
||||
|
||||
See [Architecture](docs/ARCHITECTURE.md) and [Privacy and security](docs/PRIVACY-SECURITY.md) for exact bounds and trust assumptions.
|
||||
|
||||
@@ -30,7 +33,7 @@ npm run test:browser
|
||||
|
||||
## Release
|
||||
|
||||
`npm run release:artifact` creates deterministic `release/log-tools-0.1.0.zip` and checksum files.
|
||||
`npm run release:artifact` creates deterministic `release/log-tools-0.2.0.zip` and checksum files.
|
||||
|
||||
## Licence
|
||||
|
||||
|
||||
+2
-2
@@ -1,7 +1,7 @@
|
||||
# Corresponding source
|
||||
|
||||
The corresponding source for Log Tools 0.1.0 is available at:
|
||||
The corresponding source for Log Tools 0.2.0 is available at:
|
||||
|
||||
https://git.add-ideas.de/lotobo/log-tools/src/tag/v0.1.0
|
||||
https://git.add-ideas.de/lotobo/log-tools/src/tag/v0.2.0
|
||||
|
||||
Build with Node.js 22, npm 11, `npm ci`, and `npm run release:artifact`.
|
||||
|
||||
@@ -1,12 +1,12 @@
|
||||
# Third-party notices
|
||||
|
||||
Log Tools 0.1.0 directly depends on these runtime packages:
|
||||
Log Tools 0.2.0 directly depends on these runtime packages:
|
||||
|
||||
| Package | Pinned version | Declared licence |
|
||||
| -------------------------------- | -------------: | ---------------- |
|
||||
| `@add-ideas/toolbox-contract` | 0.2.3 | Apache-2.0 |
|
||||
| `@add-ideas/toolbox-helpers` | 0.1.0 | GPL-3.0-or-later |
|
||||
| `@add-ideas/toolbox-shell-react` | 0.2.3 | Apache-2.0 |
|
||||
| `@add-ideas/toolbox-contract` | 0.3.0 | Apache-2.0 |
|
||||
| `@add-ideas/toolbox-helpers` | 0.2.0 | GPL-3.0-or-later |
|
||||
| `@add-ideas/toolbox-shell-react` | 0.3.0 | Apache-2.0 |
|
||||
| `react` | 19.2.8 | MIT |
|
||||
| `react-dom` | 19.2.8 | MIT |
|
||||
|
||||
|
||||
@@ -1,9 +1,11 @@
|
||||
# Architecture
|
||||
|
||||
Log Tools is a static React/Vite application inside the shared Toolbox shell. `src/log/model.ts` contains the streaming and bounded trust boundary. `scanLogBlob()` obtains a `ReadableStream` from the supplied `Blob`/`File`, reads byte chunks, decodes UTF-8 incrementally with `TextDecoder.decode(..., {stream:true})`, and feeds a line framer that retains at most 256 KiB for one physical line. It never calls `File.text()`, `arrayBuffer()`, or constructs a complete-file string.
|
||||
Log Tools is a static React/Vite application inside the shared Toolbox shell. `src/log/model.ts` contains the streaming and bounded trust boundary. `scanLogBlob()` obtains a `ReadableStream` from the supplied `Blob`/`File`, reads byte chunks, decodes UTF-8 incrementally with `TextDecoder.decode(..., {stream:true})`, and feeds a line framer that retains at most 256 KiB for one physical line. A bounded multiline assembler groups configured stack-trace continuations and records their first and last physical lines. It never calls `File.text()`, `arrayBuffer()`, or constructs a complete-file string.
|
||||
|
||||
Automatic detection buffers only the first 30 nonblank bounded lines, chooses among JSON Lines, nginx common/combined, RFC 5424/common RFC 3164 syslog, logfmt, and plain text, then replays that small buffer through the selected parser. Explicit selection skips detection. Parsing continues up to five million lines or eight GiB. At most the first 10,000 nonblank records and 32 MiB of retained characters are kept; individual retained raw/message strings, fields, level/source maps, hourly timestamp buckets, field names, counters, and progress are independently capped. Reads yield between progress intervals so cancellation and UI updates can run.
|
||||
|
||||
Filtering operates on the retained preview. Correlation counts describe the scanned portion. Redaction recipes transform raw text, normalized messages, sources, and extracted values using fixed, linear-oriented patterns and an optional exact literal. Expanded redaction output is capped per value and truncation is reported. Salted pseudonyms use a stable non-cryptographic hash: useful for local equality correlation, not encryption or resistance to guessing. Exports use the filtered retained preview, at most 50,000 rows and 10 MiB of text. CSV prefixes spreadsheet-formula-leading cells.
|
||||
Filtering operates interactively on the retained preview. Correlation counts describe the scanned portion. Redaction recipes transform raw text, normalized messages, sources, and extracted values using fixed, linear-oriented patterns and an optional exact literal. Expanded redaction output is capped per value and truncation is reported. Salted pseudonyms use a stable non-cryptographic hash: useful for local equality correlation, not encryption or resistance to guessing. Quick exports use the preview; `exportLogBlob()` performs a second cancellable streaming pass and reapplies parsing, filters, and redaction to all records. It writes through a supplied `WritableStream`, or returns a Blob under a 256 MiB cap. CSV prefixes spreadsheet-formula-leading cells.
|
||||
|
||||
Multi-file analysis streams each selected file through the same scanner, retains at most 3,000 records per file, and merges only those retained records. `analyzeLogSources()` caps file count, record count, retained characters, merged events, and correlation groups. It recognizes valid non-zero W3C-sized trace/span IDs plus bounded common request identifiers, reports unresolved parent references as retained-evidence gaps, and never adjusts timestamps. `exportOtlpJson()` produces a focused bounded OTLP/JSON request from retained records; it does not claim to reconstruct complete traces or export the unretained source.
|
||||
|
||||
React renders log values through text nodes only. ANSI escape sequences are removed and other control bytes are replaced; HTML-like content is never interpreted. Relative assets and a same-origin scoped service worker keep the build relocatable below nested portal routes. There is no telemetry, database, server API, worker, or remote dependency at runtime.
|
||||
|
||||
+1
-1
@@ -1,5 +1,5 @@
|
||||
const CACHE_PREFIX = "log-tools-shell-";
|
||||
const CACHE_NAME = CACHE_PREFIX + "0.1.0";
|
||||
const CACHE_NAME = CACHE_PREFIX + "0.2.0";
|
||||
const CORE = ["./", "./manifest.webmanifest", "./favicon.svg"];
|
||||
self.addEventListener("install", (event) => {
|
||||
event.waitUntil(
|
||||
|
||||
+18
-1
@@ -3,7 +3,7 @@
|
||||
"schemaVersion": 1,
|
||||
"id": "de.add-ideas.log-tools",
|
||||
"name": "Log Tools",
|
||||
"version": "0.1.0",
|
||||
"version": "0.2.0",
|
||||
"description": "Stream, inspect, correlate, redact, and export logs locally.",
|
||||
"entry": "./",
|
||||
"icon": "./favicon.svg",
|
||||
@@ -21,6 +21,23 @@
|
||||
"crossOriginIsolated": false,
|
||||
"topLevelContext": false
|
||||
},
|
||||
"io": {
|
||||
"accepts": [
|
||||
{ "mediaType": "text/plain", "extensions": [".log", ".txt"] },
|
||||
{
|
||||
"mediaType": "application/x-ndjson",
|
||||
"extensions": [".ndjson", ".jsonl"]
|
||||
},
|
||||
{ "mediaType": "application/json", "extensions": [".json"] }
|
||||
],
|
||||
"produces": [
|
||||
{ "mediaType": "text/plain", "extensions": [".log", ".txt"] },
|
||||
{ "mediaType": "application/x-ndjson", "extensions": [".ndjson"] },
|
||||
{ "mediaType": "text/csv", "extensions": [".csv"] },
|
||||
{ "mediaType": "application/json", "extensions": [".json"] }
|
||||
]
|
||||
},
|
||||
"capabilities": { "required": [], "optional": ["file-system-access"] },
|
||||
"privacy": {
|
||||
"processing": "local",
|
||||
"fileUploads": true,
|
||||
|
||||
Reference in New Issue
Block a user