fix(xrechnung): bound validator output and report reads

Enforce capture limits while draining both subprocess pipes, kill and reap interrupted validators, and bound report allocation before parsing. Preserve result precedence and add synthetic-process regressions with EN/DE documentation.

Refs #2
This commit is contained in:
2026-09-08 05:32:37 +02:00
parent c2be55d46e
commit b73709c65a
4 changed files with 358 additions and 9 deletions
+45
View File
@@ -31,6 +31,51 @@ The configuration tree must be installed through deployment governance. A newer
Inbound XML is size-bounded and parsed with external entities and DTD processing disabled before Java runs. KoSIT runs without a shell, with an absolute executable, fixed argument vector, bounded time, bounded captured output, and an isolated result directory.
The validator's stdout and stderr share a fixed 2 MiB capture limit, including
the single separator byte used when inspecting their combined output. Both
streams are drained while the validator runs. If the next byte exceeds that
limit, GovOPlaN kills and reaps the direct validator process and returns
`failed` with the reason `KoSIT runner output exceeded the safety limit.`
Conformance and assessment remain `unknown`, and no handoff is allowed even
if a valid-looking report was already written. The existing timeout (120 seconds
by default, configurable from 1 to 600 seconds) also kills and reaps the
validator; cancellation propagates after the same cleanup. Output is not
included in these technical-failure reasons. Operators should check the
approved engine and configuration using synthetic inputs before retrying an
output-limit failure. The pipe runner uses POSIX process facilities; deployment
resource limits and descendant-process isolation remain outside this capture
bound.
Report reads are bounded to the existing 16 MiB limit plus one probe byte.
Reports exactly at the limit remain eligible for normal interpretation;
oversized reports cannot supply trusted evidence and yield `incomplete` with
unknown conformance unless an earlier runner failure already determines the
outcome. An output-limit failure skips reading the report entirely. This read
bound does not limit how much report data the validator can write to disk.
Standardausgabe und Standardfehlerausgabe des Validators teilen eine feste
Erfassungsgrenze von 2 MiB einschließlich eines Trennbytes. Beide Ausgaben
werden während des Laufs gelesen. Bei Überschreitung beendet GovOPlaN den
direkten Validatorprozess und wartet dessen Abschluss ab. Das Ergebnis lautet
`failed` mit unbekannter Konformität und Bewertung (`unknown`); auch ein
scheinbar gültiger Bericht erlaubt keine Übergabe. Das bestehende Zeitlimit
(standardmäßig 120 Sekunden, konfigurierbar von 1 bis 600 Sekunden) beendet den
Prozess ebenfalls; ein Abbruch wird nach der Prozessbereinigung weitergegeben.
Diese technischen Fehlergründe enthalten keine Runner-Ausgabe. Betreibende
sollten das freigegebene Profil vor einem erneuten Versuch mit synthetischen
Eingaben prüfen. Der Pipe-Runner verwendet POSIX-Prozessfunktionen;
Ressourcengrenzen der Installation und die Isolation von Kindprozessen bleiben
außerhalb dieser Erfassungsgrenze.
Berichte werden höchstens bis zur bestehenden Grenze von 16 MiB zuzüglich
eines Prüfbytes gelesen. Berichte genau an der Grenze werden normal
ausgewertet; übergroße Berichte liefern keinen vertrauenswürdigen Nachweis und
ergeben `incomplete` mit unbekannter Konformität, sofern nicht bereits ein
vorrangiger Runner-Fehler das Ergebnis bestimmt. Bei Überschreitung der
Runner-Ausgabegrenze wird der Bericht gar nicht gelesen. Die Lesegrenze
begrenzt nicht die Berichtsmenge, die der Validator auf Datenträger schreiben
kann.
GovOPlaN does not trust the report alone. A technically complete result requires a zero process status, no technical error marker in runner output, a bounded well-formed VARL report, a matched scenario, at least the configured number of complete validation steps, and exactly one assessment. Formal validity and the accept/reject recommendation are retained separately because warnings can make them differ. This compensates for the known risk that a partial report can look valid after a transformation failure.
Semantic invalidity is different from technical failure. A complete reject report is `invalid`; a timeout, crash, partial report, or inconsistent assessment is `unknown`. Neither result may be handed off as valid.
+21 -2
View File
@@ -115,11 +115,14 @@ manifest = ModuleManifest(
"No XRechnung/configuration version is activated by default in this release.",
"The connector creates a handoff contract but does not own payable approval or booking.",
"A syntactically valid report is not trusted when runner output signals a technical failure.",
"Runner stdout and stderr share a fixed 2 MiB capture limit, including one separator byte; the limit is enforced while the validator runs.",
"Reports are limited to 16 MiB and read with at most one extra byte to detect overflow. Pipe capture requires POSIX; deployment resource limits and descendant-process isolation remain separate.",
],
"consequences": [
"Changing any pinned artifact changes the profile digest and requires a new validation.",
"Invalid invoices retain diagnostics but cannot enter the valid-invoice handoff.",
"Technical failure produces unknown conformance, never a semantic rejection or acceptance.",
"Exceeding the output limit stops and reaps the validator and reports technical failure with unknown conformance; operators should check the approved profile using synthetic inputs before retrying.",
],
"verification": "Confirm artifact and profile digests, technical outcome, validation-step count, formal validity, VARL assessment, invoice digest, and report digest before following the handoff reference.",
},
@@ -143,11 +146,14 @@ manifest = ModuleManifest(
"Diese Version aktiviert standardmäßig keine XRechnung- oder Konfigurationsversion.",
"Der Konnektor erzeugt einen Übergabevertrag, führt aber weder Freigabe der Verbindlichkeit noch Buchung.",
"Ein syntaktisch gültiger Bericht ist nicht vertrauenswürdig, wenn die Runner-Ausgabe einen technischen Fehler meldet.",
"Standardausgabe und Standardfehlerausgabe teilen eine feste Erfassungsgrenze von 2 MiB einschließlich eines Trennbytes; die Grenze wird während des Validatorlaufs durchgesetzt.",
"Berichte sind auf 16 MiB begrenzt und werden mit höchstens einem zusätzlichen Byte zur Erkennung einer Überschreitung gelesen. Die Pipe-Erfassung erfordert POSIX; Ressourcengrenzen der Installation und die Isolation nachgelagerter Prozesse bleiben getrennt.",
],
"consequences": [
"Jede Änderung eines festgelegten Artefakts ändert die Profilprüfsumme und erfordert eine neue Validierung.",
"Ungültige Rechnungen bewahren Diagnosen, dürfen aber nicht in die gültige Rechnungsübergabe gelangen.",
"Technischer Fehler ergibt unbekannte Konformität, niemals semantische Ablehnung oder Annahme.",
"Bei Überschreitung der Ausgabegrenze wird der Validator beendet und sein Prozessabschluss abgewartet; das Ergebnis ist ein technischer Fehler mit unbekannter Konformität. Betreibende sollten das freigegebene Profil vor einem erneuten Versuch mit synthetischen Eingaben prüfen.",
],
"verification": "Vor dem Folgen des Übergabeverweises Artefakt- und Profilprüfsummen, technisches Ergebnis, Prüfschrittzahl, formale Gültigkeit, VARL-Bewertung, Rechnungsprüfsumme und Berichtsprüfsumme bestätigen.",
}
@@ -167,7 +173,12 @@ manifest = ModuleManifest(
"approved, suspended, or retired state. Selection uses the invoice receive time and "
"never infers a product version. Unknown, not-yet-valid, expired, suspended, retired, "
"or digest-mismatched profiles fail closed. Validation evidence distinguishes runner "
"completion, report completeness, formal validity, and semantic assessment. A handoff "
"completion, report completeness, formal validity, and semantic assessment. Stdout and "
"stderr share a fixed 2 MiB capture limit, including one separator byte, enforced during "
"execution. Overflow stops and reaps the validator; the result is technical failure "
"with unknown conformance and no handoff, even if a report looks valid. Reports are "
"read with a 16 MiB limit plus one probe byte; oversized reports cannot supply trusted "
"validation evidence. A handoff "
"binds the exact invoice, report, and profile digests but does not approve a payable, "
"book an invoice, retain the source file, or replace Records custody."
),
@@ -194,7 +205,13 @@ manifest = ModuleManifest(
"oder ausgemustert. Die Auswahl verwendet den Eingangszeitpunkt der Rechnung und leitet niemals eine "
"Produktversion ab. Unbekannte, noch nicht gültige, abgelaufene, gesperrte, ausgemusterte oder bei der "
"Prüfsumme abweichende Profile werden geschlossen abgewiesen. Der Validierungsnachweis unterscheidet "
"Runner-Abschluss, Berichtsvollständigkeit, formale Gültigkeit und semantische Bewertung. Eine Übergabe "
"Runner-Abschluss, Berichtsvollständigkeit, formale Gültigkeit und semantische Bewertung. Standardausgabe "
"und Standardfehlerausgabe teilen eine feste Erfassungsgrenze von 2 MiB einschließlich eines Trennbytes, "
"die während der Ausführung durchgesetzt wird. Bei Überschreitung wird der Validator beendet und sein "
"Prozessabschluss abgewartet; das Ergebnis ist ein technischer Fehler mit unbekannter Konformität ohne "
"Übergabe, selbst wenn ein Bericht gültig erscheint. Berichte werden mit einer Grenze von 16 MiB "
"zuzüglich eines Prüfbytes gelesen; übergroße Berichte liefern keinen vertrauenswürdigen "
"Validierungsnachweis. Eine Übergabe "
"bindet exakte Rechnungs-, Berichts- und Profilprüfsummen, genehmigt aber keine Verbindlichkeit, bucht keine "
"Rechnung, bewahrt keine Quelldatei und ersetzt nicht die Verwahrung durch Records."
),
@@ -206,6 +223,7 @@ manifest = ModuleManifest(
"approve_profile": "Makes one exact verified artifact set eligible only in its receive-time window.",
"suspend_or_retire_profile": "Blocks selection without rewriting prior validation evidence.",
"fail_closed": "Treats unknown profile or technical outcome as untrusted rather than valid or invalid.",
"exceed_output_limit": "Stops and reaps the validator with a technical-failure reason and blocks handoff regardless of report content.",
"create_handoff": "Binds valid evidence for an owner workflow without approving, booking, or storing the invoice.",
},
},
@@ -216,6 +234,7 @@ manifest = ModuleManifest(
"approve_profile": "Macht genau einen geprüften Artefaktsatz ausschließlich in seinem Eingangszeitfenster auswählbar.",
"suspend_or_retire_profile": "Sperrt die Auswahl, ohne frühere Validierungsnachweise umzuschreiben.",
"fail_closed": "Behandelt unbekanntes Profil oder technisches Ergebnis als nicht vertrauenswürdig statt als gültig oder ungültig.",
"exceed_output_limit": "Beendet den Validator, wartet seinen Prozessabschluss ab, meldet einen technischen Fehler und sperrt die Übergabe unabhängig vom Berichtsinhalt.",
"create_handoff": "Bindet gültige Nachweise für einen Eigentümerablauf, ohne die Rechnung zu genehmigen, zu buchen oder zu speichern.",
}
}
+59 -7
View File
@@ -7,8 +7,10 @@ import json
import os
from pathlib import Path
import re
import selectors
import subprocess
import tempfile
import time
from typing import Literal
from defusedxml import ElementTree
@@ -31,6 +33,10 @@ class XRechnungValidationError(RuntimeError):
"""Stable validation or configuration error without invoice contents."""
class _RunnerOutputLimitExceeded(Exception):
"""The shared stdout/stderr budget was exhausted during execution."""
@dataclass(frozen=True, slots=True)
class KoSITValidationProfile:
"""Exact locally installed validator and rule-set selection."""
@@ -158,13 +164,10 @@ class KoSITValidator:
if os.environ.get("JAVA_HOME"):
env["JAVA_HOME"] = os.environ["JAVA_HOME"]
try:
completed = subprocess.run( # noqa: S603 - absolute, digest-pinned executable and artifacts; no shell.
completed = _run_with_bounded_output(
command,
cwd=self.profile.configuration_root,
env=env,
stdin=subprocess.DEVNULL,
capture_output=True,
check=False,
timeout=self.timeout_seconds,
)
except subprocess.TimeoutExpired:
@@ -175,8 +178,7 @@ class KoSITValidator:
"failed",
"KoSIT validation timed out; no handoff is allowed.",
)
output = completed.stdout + b"\n" + completed.stderr
if len(output) > MAX_RUNNER_OUTPUT_BYTES:
except _RunnerOutputLimitExceeded:
return _technical_result(
self.profile,
profile_sha256,
@@ -184,8 +186,12 @@ class KoSITValidator:
"failed",
"KoSIT runner output exceeded the safety limit.",
)
output = completed.stdout + b"\n" + completed.stderr
report_path = workdir / "invoice-report.xml"
report = report_path.read_bytes() if report_path.is_file() else None
report = None
if report_path.is_file():
with report_path.open("rb") as report_file:
report = report_file.read(MAX_REPORT_BYTES + 1)
return interpret_kosit_result(
profile=self.profile,
profile_sha256=profile_sha256,
@@ -196,6 +202,52 @@ class KoSITValidator:
)
def _run_with_bounded_output(
command: list[str], *, cwd: Path, env: dict[str, str], timeout: int
) -> subprocess.CompletedProcess[bytes]:
"""Drain both pipes within one budget, killing and reaping on any interruption."""
with subprocess.Popen( # noqa: S603 - absolute executable, pinned artifacts, fixed argv; no shell.
command,
cwd=cwd,
env=env,
stdin=subprocess.DEVNULL,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
bufsize=0,
) as process:
try:
deadline = time.monotonic() + timeout
stdout, stderr = bytearray(), bytearray()
# Preserve the existing combined-output cap, including its separator.
captured_bytes = 1
with selectors.DefaultSelector() as selector:
selector.register(process.stdout, selectors.EVENT_READ, stdout)
selector.register(process.stderr, selectors.EVENT_READ, stderr)
while selector.get_map():
remaining = deadline - time.monotonic()
if remaining <= 0:
raise subprocess.TimeoutExpired(command, timeout)
for key, _events in selector.select(remaining):
# Read at most one byte beyond the budget to detect overflow.
chunk = os.read(
key.fd, min(64 * 1024, MAX_RUNNER_OUTPUT_BYTES - captured_bytes + 1)
)
if not chunk:
selector.unregister(key.fileobj)
continue
captured_bytes += len(chunk)
if captured_bytes > MAX_RUNNER_OUTPUT_BYTES:
raise _RunnerOutputLimitExceeded
key.data.extend(chunk)
return_code = process.wait(timeout=max(0, deadline - time.monotonic()))
except BaseException:
# Do not communicate() here: draining after termination would be unbounded.
process.kill()
process.wait()
raise
return subprocess.CompletedProcess(command, return_code, bytes(stdout), bytes(stderr))
def verify_profile(profile: KoSITValidationProfile) -> str:
if not profile.java_executable.is_file() or not os.access(profile.java_executable, os.X_OK):
raise XRechnungValidationError("The pinned Java executable is unavailable or not executable.")
+233
View File
@@ -1,15 +1,23 @@
from __future__ import annotations
from collections.abc import Iterator
from datetime import UTC, datetime
import hashlib
import os
from pathlib import Path
import subprocess
import sys
import time
import pytest
from govoplan_xrechnung.backend import validation
from govoplan_xrechnung.backend.manifest import get_manifest
from govoplan_xrechnung.backend.validation import (
InboundInvoice,
KoSITValidationProfile,
KoSITValidator,
MAX_RUNNER_OUTPUT_BYTES,
XRechnungValidationError,
configuration_tree_sha256,
create_validated_handoff,
@@ -70,6 +78,231 @@ def _invoice() -> InboundInvoice:
)
def _runner_profile(tmp_path: Path, script: str) -> KoSITValidationProfile:
"""Install a synthetic executable; never invoke Java or a real invoice validator."""
profile = _profile(tmp_path)
profile.java_executable.write_text(
f"#!{sys.executable}\n"
"import os, sys, time\n"
"from pathlib import Path\n"
f"Path(sys.argv[-1]).with_name('invoice-report.xml').write_bytes({_report(valid=True)!r})\n"
+ script,
encoding="utf-8",
)
return profile
@pytest.fixture
def runner_processes(monkeypatch: pytest.MonkeyPatch) -> Iterator[list[subprocess.Popen[bytes]]]:
processes: list[subprocess.Popen[bytes]] = []
real_popen = subprocess.Popen
def start(*args, **kwargs):
process = real_popen(*args, **kwargs)
processes.append(process)
return process
monkeypatch.setattr(validation.subprocess, "Popen", start)
try:
yield processes
finally:
for process in processes:
if process.poll() is None:
process.kill()
process.wait()
def _assert_runner_reaped(processes: list[subprocess.Popen[bytes]]) -> None:
assert len(processes) == 1
process = processes[0]
assert process.returncode is not None
assert process.stdout is not None and process.stdout.closed
assert process.stderr is not None and process.stderr.closed
with pytest.raises(ChildProcessError):
os.waitpid(process.pid, os.WNOHANG)
@pytest.fixture
def report_read_sizes(monkeypatch: pytest.MonkeyPatch) -> list[int]:
sizes: list[int] = []
real_open = Path.open
class ReportReader:
def __init__(self, stream):
self.stream = stream
def __enter__(self):
return self
def __exit__(self, *_args):
self.stream.close()
def read(self, size=-1):
sizes.append(size)
assert size == validation.MAX_REPORT_BYTES + 1
return self.stream.read(size)
def open_file(path, mode="r", *args, **kwargs):
stream = real_open(path, mode, *args, **kwargs)
if path.name == "invoice-report.xml" and mode == "rb":
return ReportReader(stream)
return stream
monkeypatch.setattr(Path, "open", open_file)
return sizes
@pytest.mark.parametrize("stream", ["stdout", "stderr", "both"])
def test_noisy_runner_is_stopped_at_shared_output_limit(
tmp_path: Path,
runner_processes: list[subprocess.Popen[bytes]],
report_read_sizes: list[int],
stream: str,
) -> None:
count = MAX_RUNNER_OUTPUT_BYTES
writes = {
"stdout": f"sys.stdout.buffer.write(b'x' * {count})\nsys.stdout.flush()\n",
"stderr": f"sys.stderr.buffer.write(b'x' * {count})\nsys.stderr.flush()\n",
"both": (
f"sys.stdout.buffer.write(b'x' * {count // 2})\nsys.stdout.flush()\n"
f"sys.stderr.buffer.write(b'x' * {count // 2})\nsys.stderr.flush()\n"
),
}
profile = _runner_profile(tmp_path, writes[stream] + "time.sleep(30)\n")
result = KoSITValidator(profile, timeout_seconds=2).validate(_invoice())
assert result.technical_outcome == "failed"
assert result.technical_reason == "KoSIT runner output exceeded the safety limit."
assert result.conformance == result.assessment == "unknown"
assert not result.handoff_allowed
assert result.report_sha256 is None
assert result.diagnostics == ()
assert report_read_sizes == []
_assert_runner_reaped(runner_processes)
@pytest.mark.parametrize(("extra_bytes", "exit_code"), [(0, 0), (1, 0), (100_000, 0), (100_000, 7)])
def test_report_reads_are_bounded_and_preserve_exact_limit(
tmp_path: Path,
runner_processes: list[subprocess.Popen[bytes]],
report_read_sizes: list[int],
monkeypatch: pytest.MonkeyPatch,
extra_bytes: int,
exit_code: int,
) -> None:
report_limit = len(_report(valid=True))
monkeypatch.setattr(validation, "MAX_REPORT_BYTES", report_limit)
profile = _runner_profile(
tmp_path,
"with Path(sys.argv[-1]).with_name('invoice-report.xml').open('ab') as report:\n"
f" report.write(b' ' * {extra_bytes})\n"
f"sys.exit({exit_code})\n",
)
result = KoSITValidator(profile, timeout_seconds=2).validate(_invoice())
assert report_read_sizes == [report_limit + 1]
if extra_bytes:
assert result.technical_outcome == ("failed" if exit_code else "incomplete")
assert result.technical_reason == (
f"KoSIT runner exited with status {exit_code}; report semantics are not trusted."
if exit_code else "KoSIT did not produce a bounded XML report."
)
assert result.conformance == result.assessment == "unknown"
assert result.report_sha256 is None
assert not result.handoff_allowed
else:
assert result.technical_outcome == "complete"
assert result.report_sha256 == hashlib.sha256(_report(valid=True)).hexdigest()
assert result.handoff_allowed
_assert_runner_reaped(runner_processes)
@pytest.mark.parametrize("excess_bytes", [0, 1])
def test_completed_runner_output_preserves_exact_combined_limit(
tmp_path: Path, runner_processes: list[subprocess.Popen[bytes]], excess_bytes: int
) -> None:
stdout_size = MAX_RUNNER_OUTPUT_BYTES // 2
stderr_size = MAX_RUNNER_OUTPUT_BYTES - stdout_size - 1 + excess_bytes
profile = _runner_profile(
tmp_path,
f"sys.stdout.buffer.write(b'x' * {stdout_size})\n"
f"sys.stderr.buffer.write(b'x' * {stderr_size})\n",
)
result = KoSITValidator(profile, timeout_seconds=2).validate(_invoice())
assert result.technical_outcome == ("failed" if excess_bytes else "complete")
assert result.handoff_allowed is (excess_bytes == 0)
_assert_runner_reaped(runner_processes)
@pytest.mark.parametrize("close_pipes", [False, True])
def test_hanging_runner_times_out_and_is_reaped(
tmp_path: Path, runner_processes: list[subprocess.Popen[bytes]], close_pipes: bool
) -> None:
script = "os.close(1)\nos.close(2)\n" if close_pipes else ""
profile = _runner_profile(tmp_path, script + "time.sleep(30)\n")
started = time.monotonic()
result = KoSITValidator(profile, timeout_seconds=1).validate(_invoice())
assert time.monotonic() - started < 5
assert result.technical_outcome == "failed"
assert result.technical_reason == "KoSIT validation timed out; no handoff is allowed."
assert result.conformance == result.assessment == "unknown"
assert not result.handoff_allowed
_assert_runner_reaped(runner_processes)
@pytest.mark.parametrize(
("script", "outcome", "reason"),
[
("sys.stderr.write('synthetic failure')\nsys.exit(7)\n", "failed", "status 7"),
(
"sys.stdout.buffer.write(b'x' * 65532 + b'\\nERROR synthetic failure')\n",
"incomplete",
"technical error",
),
("sys.stderr.write('ERROR synthetic failure')\n", "incomplete", "technical error"),
],
)
def test_runner_failure_and_technical_output_never_trust_valid_report(
tmp_path: Path,
runner_processes: list[subprocess.Popen[bytes]],
script: str,
outcome: str,
reason: str,
) -> None:
profile = _runner_profile(tmp_path, script)
result = KoSITValidator(profile, timeout_seconds=2).validate(_invoice())
assert result.technical_outcome == outcome
assert reason in result.technical_reason
assert result.conformance == result.assessment == "unknown"
assert not result.handoff_allowed
_assert_runner_reaped(runner_processes)
def test_runner_cancellation_kills_and_reaps_before_propagating(
tmp_path: Path,
runner_processes: list[subprocess.Popen[bytes]],
monkeypatch: pytest.MonkeyPatch,
) -> None:
profile = _runner_profile(tmp_path, "time.sleep(30)\n")
def cancel(_selector, _timeout):
raise KeyboardInterrupt
monkeypatch.setattr(validation.selectors.DefaultSelector, "select", cancel)
with pytest.raises(KeyboardInterrupt):
KoSITValidator(profile, timeout_seconds=2).validate(_invoice())
_assert_runner_reaped(runner_processes)
def test_profile_verifies_exact_engine_and_complete_configuration_tree(tmp_path: Path) -> None:
profile = _profile(tmp_path)
first = verify_profile(profile)