fix(security): isolate template rendering with hard resource limits
This commit is contained in:
@@ -36,3 +36,21 @@ Apply the module Alembic migration before startup. Monitor rejected renders for
|
|||||||
contract drift, output limits, missing Files permission, and reused idempotency
|
contract drift, output limits, missing Files permission, and reused idempotency
|
||||||
keys. HTML is designed for browser/OS printing; do not treat it as a signed PDF
|
keys. HTML is designed for browser/OS printing; do not treat it as a signed PDF
|
||||||
or proof of physical printer delivery.
|
or proof of physical printer delivery.
|
||||||
|
|
||||||
|
Pure rendering runs in a disposable Core worker after authorization, revision
|
||||||
|
selection and idempotency lookup. Each worker has a 15-second wall limit,
|
||||||
|
10 CPU seconds, 512 MiB address space and no file output. Typed input transport
|
||||||
|
is limited to 32 MiB and result transport to 8 MiB, including serialization
|
||||||
|
overhead; nesting is limited to 64 levels and 1,000,000 value nodes. The existing
|
||||||
|
5 MiB final output and 5,000-item limits remain. Only data DTOs cross the
|
||||||
|
boundary; principals, SQL sessions, credentials and artifact writes stay in the
|
||||||
|
parent. Completed idempotent renders return before a worker is started.
|
||||||
|
|
||||||
|
`GOVOPLAN_ISOLATED_PROCESS_CONCURRENCY` in Core controls active isolated work per
|
||||||
|
API/worker process without queuing. Busy capacity produces a sanitized retryable
|
||||||
|
render error. CPU, memory, transport or time failures stop the entire render
|
||||||
|
before artifact persistence; reduce the workload before retrying. POSIX process
|
||||||
|
resource controls are required. Missing controls, cancellation or worker failure
|
||||||
|
fails closed with no in-process fallback. Budget aggregate memory across all
|
||||||
|
slots and API/worker replicas; monitor limit and overload errors separately
|
||||||
|
from template compatibility errors.
|
||||||
|
|||||||
@@ -21,3 +21,13 @@ Open **Templates** to create or select a reusable definition.
|
|||||||
Render evidence shows the exact revision and abbreviated template, input, and
|
Render evidence shows the exact revision and abbreviated template, input, and
|
||||||
output hashes. A consumer such as Campaign can submit many frozen recipients;
|
output hashes. A consumer such as Campaign can submit many frozen recipients;
|
||||||
the UI sample intentionally validates one representative item.
|
the UI sample intentionally validates one representative item.
|
||||||
|
|
||||||
|
Rendering is also limited to 15 seconds elapsed time, 10 CPU seconds, 512 MiB
|
||||||
|
process memory, 32 MiB serialized input and 8 MiB serialized result transport;
|
||||||
|
the final output remains limited to 5 MiB and 5,000 items. Deeply nested data
|
||||||
|
or more than 1,000,000 transported value nodes is rejected. A limit error creates
|
||||||
|
no partial output or render evidence: reduce the selected input or template
|
||||||
|
complexity and retry. If processing capacity is busy, retry later. Unavailable
|
||||||
|
worker controls or a worker failure require operator investigation. Existing
|
||||||
|
successful output with the same idempotency key is reused without rendering
|
||||||
|
again, provided its input and authorization still match.
|
||||||
|
|||||||
@@ -102,6 +102,43 @@ ROLE_TEMPLATES = (
|
|||||||
)
|
)
|
||||||
|
|
||||||
DOCUMENTATION = (
|
DOCUMENTATION = (
|
||||||
|
DocumentationTopic(
|
||||||
|
id="templates.render-worker-limits",
|
||||||
|
title="Rendering resource and capacity limits",
|
||||||
|
summary="Render in a disposable process and reject incomplete output before persistence.",
|
||||||
|
body=(
|
||||||
|
"Template substitution and document composition run in a fresh process with a 15-second wall limit, 10 CPU seconds, "
|
||||||
|
"512 MiB address space, no file output, 32 MiB typed input and 8 MiB typed result transport. The existing final output "
|
||||||
|
"limit remains 5 MiB and the item limit remains 5,000. Transport permits only explicit data types, at most 64 nesting "
|
||||||
|
"levels and 1,000,000 value nodes. Exceeding a limit fails the entire render before Files or render evidence is written; "
|
||||||
|
"reduce the input or template complexity and retry. Core GOVOPLAN_ISOLATED_PROCESS_CONCURRENCY limits simultaneous work per "
|
||||||
|
"API/worker process without queuing; a busy error can be retried later. Missing POSIX resource controls, cancellation "
|
||||||
|
"and worker failures are explicit sanitized render errors, with no in-process fallback. Authorization, revision "
|
||||||
|
"selection, idempotency lookup, SQL sessions and artifact persistence remain in the parent. Existing completed "
|
||||||
|
"idempotent output is reused before starting a worker. Operators must budget aggregate memory across process slots."
|
||||||
|
),
|
||||||
|
layer="static",
|
||||||
|
documentation_types=("user", "admin"),
|
||||||
|
audience=("user", "module_admin", "operator"),
|
||||||
|
order=30,
|
||||||
|
translations={"de": {
|
||||||
|
"title": "Ressourcen- und Kapazitätsgrenzen beim Rendern",
|
||||||
|
"summary": "In einem kurzlebigen Prozess rendern und unvollständige Ausgaben vor dem Speichern ablehnen.",
|
||||||
|
"body": (
|
||||||
|
"Platzhalterersetzung und Dokumentaufbau laufen in einem neuen Prozess mit höchstens 15 Sekunden Gesamtdauer, "
|
||||||
|
"10 CPU-Sekunden, 512 MiB Adressraum, keiner Dateiausgabe sowie 32 MiB typisierter Eingabe und 8 MiB Ergebnistransport. "
|
||||||
|
"Die endgültige Ausgabe bleibt auf 5 MiB und die Elementzahl auf 5.000 begrenzt. Der Transport erlaubt nur ausdrückliche "
|
||||||
|
"Datentypen, höchstens 64 Verschachtelungsebenen und 1.000.000 Wertknoten. Eine Grenzverletzung bricht den gesamten "
|
||||||
|
"Vorgang vor dem Speichern in Files oder als Render-Nachweis ab; reduzieren Sie Eingaben oder Vorlagenkomplexität "
|
||||||
|
"und versuchen Sie es erneut. Core GOVOPLAN_ISOLATED_PROCESS_CONCURRENCY begrenzt gleichzeitige Arbeiten pro API-/Worker-Prozess "
|
||||||
|
"ohne Warteschlange; bei belegter Kapazität kann später erneut versucht werden. Fehlende POSIX-Ressourcenbegrenzungen, "
|
||||||
|
"Abbruch und Worker-Fehler sind ausdrückliche bereinigte Render-Fehler ohne Rückfall auf den Elternprozess. "
|
||||||
|
"Berechtigungsprüfung, Revisionsauswahl, Idempotenzprüfung, SQL-Sitzungen und Artefaktspeicherung bleiben im Elternprozess. "
|
||||||
|
"Bereits abgeschlossene idempotente Ausgaben werden vor einem Worker-Start wiederverwendet. Betreiber müssen den "
|
||||||
|
"Gesamtspeicher aller Prozessplätze berücksichtigen."
|
||||||
|
),
|
||||||
|
}},
|
||||||
|
),
|
||||||
DocumentationTopic(
|
DocumentationTopic(
|
||||||
id="templates.workspace-layout",
|
id="templates.workspace-layout",
|
||||||
title="Templates workspace actions",
|
title="Templates workspace actions",
|
||||||
|
|||||||
@@ -25,6 +25,12 @@ from govoplan_core.core.templates import (
|
|||||||
TemplateRenderRequest,
|
TemplateRenderRequest,
|
||||||
TemplateRenderResult,
|
TemplateRenderResult,
|
||||||
)
|
)
|
||||||
|
from govoplan_core.security.bounded_process import (
|
||||||
|
ProcessBudgetError,
|
||||||
|
ProcessLimits,
|
||||||
|
run_bounded_operation,
|
||||||
|
)
|
||||||
|
from govoplan_core.security.worker_payload import decode_worker_payload, encode_worker_payload
|
||||||
from govoplan_templates.backend.db.models import (
|
from govoplan_templates.backend.db.models import (
|
||||||
TemplateDefinition,
|
TemplateDefinition,
|
||||||
TemplateRender,
|
TemplateRender,
|
||||||
@@ -42,9 +48,36 @@ from govoplan_templates.backend.service import (
|
|||||||
RENDERER_VERSION = "templates-html-1"
|
RENDERER_VERSION = "templates-html-1"
|
||||||
MAX_OUTPUT_BYTES = 5 * 1024 * 1024
|
MAX_OUTPUT_BYTES = 5 * 1024 * 1024
|
||||||
MAX_ITEMS = 5_000
|
MAX_ITEMS = 5_000
|
||||||
|
RENDER_PROCESS_LIMITS = ProcessLimits(
|
||||||
|
wall_seconds=15,
|
||||||
|
cpu_seconds=10,
|
||||||
|
memory_bytes=512 * 1024 * 1024,
|
||||||
|
input_bytes=32 * 1024 * 1024,
|
||||||
|
output_bytes=8 * 1024 * 1024,
|
||||||
|
)
|
||||||
_TOKEN_PATTERN = re.compile(r"{{\s*([A-Za-z_][A-Za-z0-9_.-]*)\s*}}")
|
_TOKEN_PATTERN = re.compile(r"{{\s*([A-Za-z_][A-Za-z0-9_.-]*)\s*}}")
|
||||||
|
|
||||||
|
|
||||||
|
@dataclasses.dataclass(frozen=True, slots=True)
|
||||||
|
class _RenderDefinition:
|
||||||
|
name: str
|
||||||
|
|
||||||
|
|
||||||
|
@dataclasses.dataclass(frozen=True, slots=True)
|
||||||
|
class _RenderRevision:
|
||||||
|
content_text: str | None
|
||||||
|
content_html: str | None
|
||||||
|
template_type: str
|
||||||
|
layout: dict[str, object]
|
||||||
|
output_profiles: list[dict[str, object]]
|
||||||
|
|
||||||
|
|
||||||
|
@dataclasses.dataclass(frozen=True, slots=True)
|
||||||
|
class _RenderRequest:
|
||||||
|
output_format: str
|
||||||
|
parameters: dict[str, object]
|
||||||
|
|
||||||
|
|
||||||
class SqlTemplateRenderer:
|
class SqlTemplateRenderer:
|
||||||
def __init__(self, registry: object | None = None) -> None:
|
def __init__(self, registry: object | None = None) -> None:
|
||||||
self.registry = registry
|
self.registry = registry
|
||||||
@@ -359,6 +392,68 @@ def _render_payload(
|
|||||||
*,
|
*,
|
||||||
request: TemplateRenderRequest,
|
request: TemplateRenderRequest,
|
||||||
items: Sequence[Mapping[str, object]],
|
items: Sequence[Mapping[str, object]],
|
||||||
|
) -> tuple[bytes, str, int]:
|
||||||
|
# Authoritative selection/idempotency precede this call. Transfer only the
|
||||||
|
# fields needed for pure rendering, never ORM objects, principals or Files.
|
||||||
|
try:
|
||||||
|
data = encode_worker_payload({
|
||||||
|
"definition": {"name": definition.name},
|
||||||
|
"revision": {
|
||||||
|
"content_text": revision.content_text,
|
||||||
|
"content_html": revision.content_html,
|
||||||
|
"template_type": revision.template_type,
|
||||||
|
"layout": dict(revision.layout),
|
||||||
|
"output_profiles": list(revision.output_profiles),
|
||||||
|
},
|
||||||
|
"request": {
|
||||||
|
"output_format": request.output_format,
|
||||||
|
"parameters": dict(request.parameters),
|
||||||
|
},
|
||||||
|
"items": tuple(dict(item) for item in items),
|
||||||
|
}, max_bytes=RENDER_PROCESS_LIMITS.input_bytes)
|
||||||
|
result = decode_worker_payload(
|
||||||
|
run_bounded_operation(_render_payload_worker, data, limits=RENDER_PROCESS_LIMITS),
|
||||||
|
max_bytes=RENDER_PROCESS_LIMITS.output_bytes,
|
||||||
|
)
|
||||||
|
except ProcessBudgetError as exc:
|
||||||
|
raise TemplateRenderError(f"Template rendering failed ({exc.code}): {exc}") from exc
|
||||||
|
except (TypeError, ValueError, RecursionError) as exc:
|
||||||
|
raise TemplateRenderError("Template input or output could not be safely transferred.") from exc
|
||||||
|
if not isinstance(result, dict):
|
||||||
|
raise TemplateRenderError("Template worker returned an invalid result.")
|
||||||
|
if "render_error" in result:
|
||||||
|
raise TemplateRenderError(str(result["render_error"]))
|
||||||
|
payload, content_type, page_count = result.get("payload"), result.get("content_type"), result.get("page_count")
|
||||||
|
if not isinstance(payload, bytes) or not isinstance(content_type, str) or type(page_count) is not int:
|
||||||
|
raise TemplateRenderError("Template worker returned an invalid result.")
|
||||||
|
if len(payload) > MAX_OUTPUT_BYTES:
|
||||||
|
_output_limit_exceeded()
|
||||||
|
return payload, content_type, page_count
|
||||||
|
|
||||||
|
|
||||||
|
def _render_payload_worker(payload: bytes) -> bytes:
|
||||||
|
data = decode_worker_payload(payload, max_bytes=RENDER_PROCESS_LIMITS.input_bytes)
|
||||||
|
definition = _RenderDefinition(**data["definition"])
|
||||||
|
revision = _RenderRevision(**data["revision"])
|
||||||
|
request = _RenderRequest(**data["request"])
|
||||||
|
try:
|
||||||
|
payload, content_type, page_count = _render_payload_content(
|
||||||
|
definition, revision, request=request, items=data["items"]
|
||||||
|
)
|
||||||
|
except TemplateRenderError as exc:
|
||||||
|
return encode_worker_payload({"render_error": str(exc)})
|
||||||
|
return encode_worker_payload(
|
||||||
|
{"payload": payload, "content_type": content_type, "page_count": page_count},
|
||||||
|
max_bytes=RENDER_PROCESS_LIMITS.output_bytes,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _render_payload_content(
|
||||||
|
definition: _RenderDefinition,
|
||||||
|
revision: _RenderRevision,
|
||||||
|
*,
|
||||||
|
request: _RenderRequest,
|
||||||
|
items: Sequence[Mapping[str, object]],
|
||||||
) -> tuple[bytes, str, int]:
|
) -> tuple[bytes, str, int]:
|
||||||
if request.output_format == "text":
|
if request.output_format == "text":
|
||||||
body = revision.content_text or _html_to_text(revision.content_html or "")
|
body = revision.content_text or _html_to_text(revision.content_html or "")
|
||||||
|
|||||||
@@ -1,6 +1,7 @@
|
|||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
import unittest
|
import unittest
|
||||||
|
from dataclasses import replace
|
||||||
from types import SimpleNamespace
|
from types import SimpleNamespace
|
||||||
from unittest.mock import patch
|
from unittest.mock import patch
|
||||||
|
|
||||||
@@ -47,6 +48,43 @@ class TemplateRenderLimitTests(unittest.TestCase):
|
|||||||
with self.assertRaises(TemplateRenderError):
|
with self.assertRaises(TemplateRenderError):
|
||||||
rendering._render_payload(definition, revision, request=request, items=[{}])
|
rendering._render_payload(definition, revision, request=request, items=[{}])
|
||||||
|
|
||||||
|
def test_fresh_worker_preserves_exact_text_bytes_and_page_count(self):
|
||||||
|
definition = SimpleNamespace(name="Example")
|
||||||
|
revision = SimpleNamespace(content_html=None, content_text="Hi {{name}}: {{flag}}", template_type="serial_letter", layout={}, output_profiles=[])
|
||||||
|
request = SimpleNamespace(output_format="text", parameters={"flag": True})
|
||||||
|
with patch.object(rendering, "_render_payload_content", side_effect=AssertionError("parent renderer ran")):
|
||||||
|
result = rendering._render_payload(definition, revision, request=request, items=[{"name": "Äda"}, {"name": "Grace"}])
|
||||||
|
self.assertEqual(("Hi Äda: true\n\n---\n\nHi Grace: true".encode(), "text/plain; charset=utf-8", 2), result)
|
||||||
|
|
||||||
|
def test_real_worker_timeout_fails_without_parent_fallback(self):
|
||||||
|
definition = SimpleNamespace(name="Example")
|
||||||
|
revision = SimpleNamespace(content_html=None, content_text="Hello", template_type="serial_letter", layout={}, output_profiles=[])
|
||||||
|
request = SimpleNamespace(output_format="text", parameters={})
|
||||||
|
limits = replace(rendering.RENDER_PROCESS_LIMITS, wall_seconds=0.001)
|
||||||
|
with patch.object(rendering, "RENDER_PROCESS_LIMITS", limits), patch.object(
|
||||||
|
rendering, "_render_payload_content", side_effect=AssertionError("parent renderer ran")
|
||||||
|
):
|
||||||
|
with self.assertRaisesRegex(TemplateRenderError, "timeout"):
|
||||||
|
rendering._render_payload(definition, revision, request=request, items=[{}])
|
||||||
|
|
||||||
|
def test_worker_html_bytes_match_existing_composition(self):
|
||||||
|
definition = SimpleNamespace(name="Letters <archive>")
|
||||||
|
revision = SimpleNamespace(content_html="<p>{{name}}: {{settings}}</p>", content_text=None, template_type="label_sheet", layout={"page_size": "A5", "columns": 2, "rows": 2}, output_profiles=[])
|
||||||
|
request = SimpleNamespace(output_format="html", parameters={"settings": {"b": True, "a": "ü"}})
|
||||||
|
items = [{"name": "<Ada>"}, {"name": "Grace & Co"}]
|
||||||
|
expected = rendering._render_payload_content(definition, revision, request=request, items=items)
|
||||||
|
actual = rendering._render_payload(definition, revision, request=request, items=items)
|
||||||
|
self.assertEqual(expected, actual)
|
||||||
|
|
||||||
|
def test_real_worker_output_transport_limit_is_enforced(self):
|
||||||
|
definition = SimpleNamespace(name="Example")
|
||||||
|
revision = SimpleNamespace(content_html=None, content_text="x" * 4096, template_type="serial_letter", layout={}, output_profiles=[])
|
||||||
|
request = SimpleNamespace(output_format="text", parameters={})
|
||||||
|
limits = replace(rendering.RENDER_PROCESS_LIMITS, output_bytes=1024)
|
||||||
|
with patch.object(rendering, "RENDER_PROCESS_LIMITS", limits):
|
||||||
|
with self.assertRaisesRegex(TemplateRenderError, "output_limit"):
|
||||||
|
rendering._render_payload(definition, revision, request=request, items=[{}])
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
if __name__ == "__main__":
|
||||||
unittest.main()
|
unittest.main()
|
||||||
|
|||||||
+31
-1
@@ -2,6 +2,7 @@ from __future__ import annotations
|
|||||||
|
|
||||||
import hashlib
|
import hashlib
|
||||||
import unittest
|
import unittest
|
||||||
|
from dataclasses import replace
|
||||||
from unittest.mock import patch
|
from unittest.mock import patch
|
||||||
|
|
||||||
from govoplan_core.auth import ApiPrincipal
|
from govoplan_core.auth import ApiPrincipal
|
||||||
@@ -19,6 +20,7 @@ from govoplan_core.core.templates import (
|
|||||||
)
|
)
|
||||||
from govoplan_core.db.base import Base
|
from govoplan_core.db.base import Base
|
||||||
from govoplan_core.db.session import configure_database, reset_database
|
from govoplan_core.db.session import configure_database, reset_database
|
||||||
|
from govoplan_templates.backend import rendering
|
||||||
from govoplan_templates.backend.capabilities import (
|
from govoplan_templates.backend.capabilities import (
|
||||||
SqlTemplateCatalog,
|
SqlTemplateCatalog,
|
||||||
SqlTemplateContentLibrary,
|
SqlTemplateContentLibrary,
|
||||||
@@ -231,7 +233,8 @@ class TemplateServiceTests(unittest.TestCase):
|
|||||||
idempotency_key="campaign-1:postal-output-1",
|
idempotency_key="campaign-1:postal-output-1",
|
||||||
)
|
)
|
||||||
first = render_template(session, principal(), registry=_Registry(), request=request)
|
first = render_template(session, principal(), registry=_Registry(), request=request)
|
||||||
second = render_template(session, principal(), registry=_Registry(), request=request)
|
with patch.object(rendering, "run_bounded_operation", side_effect=AssertionError("idempotent render started a worker")):
|
||||||
|
second = render_template(session, principal(), registry=_Registry(), request=request)
|
||||||
session.commit()
|
session.commit()
|
||||||
|
|
||||||
self.assertEqual(first.render_id, second.render_id)
|
self.assertEqual(first.render_id, second.render_id)
|
||||||
@@ -243,6 +246,33 @@ class TemplateServiceTests(unittest.TestCase):
|
|||||||
self.assertIn(b"Ada", first.payload)
|
self.assertIn(b"Ada", first.payload)
|
||||||
self.assertIn(b"Grace", first.payload)
|
self.assertIn(b"Grace", first.payload)
|
||||||
|
|
||||||
|
def test_real_worker_timeout_writes_neither_artifact_nor_render_evidence(self) -> None:
|
||||||
|
store = _ArtifactStore()
|
||||||
|
with self.database.session() as session:
|
||||||
|
item, _ = create_template(session, principal(), payload())
|
||||||
|
request = TemplateRenderRequest(
|
||||||
|
template_id=item.id,
|
||||||
|
usage="campaign.postal",
|
||||||
|
items=({"name": "Ada", "postal": {"address": "Street 1"}},),
|
||||||
|
persist_to_files=True,
|
||||||
|
)
|
||||||
|
limits = replace(rendering.RENDER_PROCESS_LIMITS, wall_seconds=0.001)
|
||||||
|
with patch.object(rendering, "RENDER_PROCESS_LIMITS", limits):
|
||||||
|
with self.assertRaisesRegex(TemplateRenderError, "timeout"):
|
||||||
|
render_template(session, principal(), registry=_Registry(store), request=request)
|
||||||
|
self.assertIsNone(store.request)
|
||||||
|
self.assertEqual([], list_renders(session, principal()))
|
||||||
|
|
||||||
|
def test_incompatible_input_is_rejected_before_worker_admission(self) -> None:
|
||||||
|
with self.database.session() as session:
|
||||||
|
item, _ = create_template(session, principal(), payload())
|
||||||
|
request = TemplateRenderRequest(
|
||||||
|
template_id=item.id, usage="campaign.postal", items=({"name": "Missing address"},),
|
||||||
|
)
|
||||||
|
with patch.object(rendering, "run_bounded_operation", side_effect=AssertionError("incompatible render started a worker")):
|
||||||
|
with self.assertRaises(TemplateCompatibilityError):
|
||||||
|
render_template(session, principal(), registry=_Registry(), request=request)
|
||||||
|
|
||||||
def test_bounded_render_history_and_payload_are_owner_scoped(self) -> None:
|
def test_bounded_render_history_and_payload_are_owner_scoped(self) -> None:
|
||||||
owner = principal(admin=False)
|
owner = principal(admin=False)
|
||||||
other = principal(account_id="account-2", admin=False)
|
other = principal(account_id="account-2", admin=False)
|
||||||
|
|||||||
Reference in New Issue
Block a user