diff --git a/docs/CALENDAR_INTEGRATION_CONCEPT.md b/docs/CALENDAR_INTEGRATION_CONCEPT.md index f3d57fe..6181082 100644 --- a/docs/CALENDAR_INTEGRATION_CONCEPT.md +++ b/docs/CALENDAR_INTEGRATION_CONCEPT.md @@ -24,10 +24,88 @@ The first standalone module provides: - API endpoints for listing calendars, creating/updating/deleting events, importing iCalendar, and exporting event ICS - free/busy API primitive with recurrence expansion for scheduling and appointment conflict checks - calendar-owned CalDAV sync sources with credential references, scheduled due-sync metadata, full/incremental inbound sync, two-way PUT/DELETE writes, and ETag conflict handling +- a Calendar-owned durable desired-state outbox for two-way CalDAV writes. Event + state and the exact external resource snapshot commit atomically; workers use + deterministic hrefs, conditional requests, expiring leases, bounded + exponential retry, and semantic GET reconciliation after ambiguous outcomes - WebUI views: month, week, workweek, day, and continuous week-row scrolling The first implementation is not yet a full CalDAV network server. It is the internal calendar storage, sync, availability, and UI foundation on which Open-Xchange integration, richer recurrence editing, scheduling inbox/outbox behavior, and CalDAV server endpoints can be built. +## Outbound delivery operations + +No event API or cross-module capability performs CalDAV network I/O inside the +caller's transaction. A local mutation instead creates a +`calendar_outbox_operations` row containing the exact desired ICS resource, +target href, expected ETag, conflict-policy snapshot, and idempotency key. + +The registered `govoplan.calendar.dispatch_outbox` worker commits an expiring +lease before doing network I/O and commits each outcome independently. If a +worker loses the database connection after a successful remote write, the next +attempt reads the remote object and compares semantic ICS fingerprints. A +matching PUT or an already absent DELETE completes successfully without a blind +duplicate write. Pending updates for one href supersede older never-attempted +rows; attempted predecessors are reconciled first, and updates queued behind an +in-flight write inherit the ETag produced by that write. Delivery and inbound +REPORT application take the same source-row lock, +and only one operation per source is leased at a time. This deliberately trades +per-source throughput for a simple ordering guarantee: a stale inbound report +cannot land after a newer outbound result, and queued leases do not expire while +waiting behind another network request for the same source. + +Operators can list, dispatch, retry, reconcile, and discard tenant operations +through the `/calendar/caldav/outbox` administration endpoints. Terminal ETag +conflicts and exhausted retries remain the current local desired state and +shield that resource from inbound overwrite until explicitly resolved. +Discarding is the administrator's accept-remote transition: it is allowed only +for the latest generation, atomically cancels its unresolved predecessor chain, +marks the event projection as discarded, clears the sync token, and schedules a +full inbound reconciliation so an unchanged remote object is not missed. + +Disabling outbound delivery or switching to inbound-only is rejected while +unresolved desired state exists; retirement is the explicit exception and +cancels unresolved work. Endpoint/calendar changes also require no active event +bindings or unresolved delivery. Credential rotation does not discard committed +desired state. Public event mutation cannot set sync-owned source hrefs, kinds, +or ETags. Deleting a synchronized collection or retiring its source is a local +unlink: it never deletes the remote collection or its remaining remote events. +The current singular ownership model permits one active sync source per +calendar; multi-source fan-in will require per-event source routing. Celery beat +triggers recovery every minute, while root-transaction after-commit dispatch +provides the normal low-latency path. + +### Collection retirement and bulk event moves + +Collection deletion accepts two explicit, non-destructive external actions for +`event_action="move"`: + +- `detach_keep_remote` applies only when the source collection is synchronized + and the target is local. It moves the local event projections, clears their + sync-owned source kind, href, ETag, and CalDAV projection metadata, retires + the source locally, and cancels undelivered outbox state. An active delivery + lease blocks the transaction. The action never enqueues a remote DELETE, so + the remote collection and remote events remain unchanged. +- `copy_to_remote` applies only when the source is local and the target has an + active, enabled, two-way CalDAV source. It moves the local events and commits + destination PUT desired states in the durable outbox in the same database + transaction. Remote failures therefore remain visible and retryable without + rolling back or hiding the local move. + +Local-to-local moves retain their existing behavior and do not accept an +`external_action`. Implicit or mismatched actions, inbound-only destinations, +and synchronized-to-synchronized moves are rejected. `remote_move` is a known +but deliberately unsupported action: implementing it requires a separately +approved saga for destination reconciliation, conditional source deletion, +collision handling, and concurrent-edit policy. The WebUI does not offer that +destructive mode. + +Resolved terminal rows (`succeeded`, `superseded`, and `cancelled`) are removed +in bounded batches after `CALENDAR_OUTBOX_TERMINAL_RETENTION_DAYS` (90 days by +default; `0` disables cleanup). `conflict` and `dead` rows are never removed by +this cleanup because they still carry unresolved local desired state. Each +periodic or after-commit dispatch also performs one bounded cleanup batch for +the same tenant scope. + ## Integration Points ### Scheduling diff --git a/src/govoplan_calendar/backend/caldav.py b/src/govoplan_calendar/backend/caldav.py index 5f564cd..1a67a2d 100644 --- a/src/govoplan_calendar/backend/caldav.py +++ b/src/govoplan_calendar/backend/caldav.py @@ -1,6 +1,7 @@ from __future__ import annotations import base64 +import posixpath import urllib.error import urllib.parse import urllib.request @@ -241,8 +242,23 @@ class CalDAVClient: return parse_multistatus(payload) def fetch_object(self, href: str) -> str: - payload = self.request("GET", self.object_url(href), body=None, depth=None, expected={200}) - return payload.decode("utf-8") + return self.fetch_object_state(href).calendar_data or "" + + def fetch_object_state(self, href: str) -> CalDAVObject: + """Fetch a resource together with its ETag for outbox reconciliation.""" + + _status, headers, payload = self.request_raw( + "GET", + self.object_url(href), + body=None, + depth=None, + expected={200}, + ) + return CalDAVObject( + href=href, + etag=response_etag(headers), + calendar_data=payload.decode("utf-8"), + ) def put_object(self, href: str, ics: str, *, etag: str | None = None, create: bool = False, overwrite: bool = False) -> CalDAVWriteResult: headers = {"Content-Type": "text/calendar; charset=utf-8"} @@ -279,7 +295,32 @@ class CalDAVClient: return CalDAVWriteResult(href=href, etag=response_etag(response_headers), status=status) def object_url(self, href: str) -> str: - return urllib.parse.urljoin(self.collection_url, href) + candidate = urllib.parse.urljoin(self.collection_url, href) + collection_parts = urllib.parse.urlparse(self.collection_url) + candidate_parts = urllib.parse.urlparse(candidate) + if ( + candidate_parts.username + or candidate_parts.password + or ( + candidate_parts.scheme.lower(), + candidate_parts.hostname, + candidate_parts.port, + ) + != ( + collection_parts.scheme.lower(), + collection_parts.hostname, + collection_parts.port, + ) + ): + raise CalDAVError("CalDAV object href must use the configured collection origin") + collection_path = posixpath.normpath(urllib.parse.unquote(collection_parts.path)) + candidate_path = posixpath.normpath(urllib.parse.unquote(candidate_parts.path)) + collection_prefix = collection_path.rstrip("/") + "/" + if not candidate_path.startswith(collection_prefix) or candidate_path == collection_path: + raise CalDAVError("CalDAV object href must remain inside the configured collection path") + if candidate_parts.query or candidate_parts.fragment: + raise CalDAVError("CalDAV object href must not contain a query or fragment") + return urllib.parse.urlunparse(candidate_parts) def request(self, method: str, url: str, *, body: bytes | None, depth: str | None, expected: set[int]) -> bytes: _status, _headers, payload = self.request_raw(method, url, body=body, depth=depth, expected=expected) diff --git a/src/govoplan_calendar/backend/capabilities.py b/src/govoplan_calendar/backend/capabilities.py new file mode 100644 index 0000000..2825b3f --- /dev/null +++ b/src/govoplan_calendar/backend/capabilities.py @@ -0,0 +1,82 @@ +from __future__ import annotations + +from collections.abc import Mapping, Sequence +from datetime import datetime + +from govoplan_core.core.calendar import ( + CalendarCapabilityError, + CalendarEventRef, + CalendarEventRequest, + CalendarSchedulingProvider, +) +from govoplan_calendar.backend.schemas import CalendarEventCreateRequest +from govoplan_calendar.backend.service import CalendarError, create_event, list_freebusy + + +class SqlCalendarSchedulingProvider(CalendarSchedulingProvider): + def list_freebusy( + self, + session: object, + *, + tenant_id: str, + start_at: datetime, + end_at: datetime, + calendar_ids: Sequence[str] | None = None, + ) -> tuple[Mapping[str, object], ...]: + try: + blocks = list_freebusy( + session, + tenant_id=tenant_id, + start_at=start_at, + end_at=end_at, + calendar_ids=list(calendar_ids) if calendar_ids is not None else None, + ) + except CalendarError as exc: + raise CalendarCapabilityError(str(exc)) from exc + return tuple(blocks) + + def create_event( + self, + session: object, + *, + tenant_id: str, + user_id: str | None, + request: CalendarEventRequest, + ) -> CalendarEventRef: + try: + payload = CalendarEventCreateRequest( + calendar_id=request.calendar_id, + summary=request.summary, + description=request.description, + location=request.location, + status=request.status, + transparency=request.transparency, + classification=request.classification, + start_at=request.start_at, + end_at=request.end_at, + timezone=request.timezone, + attendees=[dict(item) for item in request.attendees], + categories=list(request.categories), + related_to=[dict(item) for item in request.related_to], + metadata=dict(request.metadata), + ) + except (TypeError, ValueError) as exc: + raise CalendarCapabilityError(str(exc)) from exc + try: + event = create_event(session, tenant_id=tenant_id, user_id=user_id, payload=payload) + except CalendarError as exc: + raise CalendarCapabilityError(str(exc)) from exc + caldav_metadata = (event.metadata_ or {}).get("caldav") + external_state = "local" + outbox_operation_id = None + if isinstance(caldav_metadata, dict): + external_state = str(caldav_metadata.get("external_state") or "queued") + raw_operation_id = caldav_metadata.get("outbox_operation_id") + outbox_operation_id = str(raw_operation_id) if raw_operation_id else None + return CalendarEventRef( + id=event.id, + calendar_id=event.calendar_id, + uid=event.uid, + external_state=external_state, + outbox_operation_id=outbox_operation_id, + ) diff --git a/src/govoplan_calendar/backend/db/models.py b/src/govoplan_calendar/backend/db/models.py index ad743c0..a110681 100644 --- a/src/govoplan_calendar/backend/db/models.py +++ b/src/govoplan_calendar/backend/db/models.py @@ -1,7 +1,7 @@ from __future__ import annotations import uuid -from datetime import datetime +from datetime import datetime, timezone from typing import Any from sqlalchemy import Boolean, DateTime, ForeignKey, Index, Integer, JSON, String, Text, UniqueConstraint, text @@ -151,4 +151,69 @@ class CalendarSyncCredential(Base, TimestampMixin): metadata_: Mapped[dict[str, Any] | None] = mapped_column("metadata", JSON, nullable=True) -__all__ = ["CalendarCollection", "CalendarEvent", "CalendarSyncCredential", "CalendarSyncSource", "new_uuid"] +class CalendarOutboxOperation(Base, TimestampMixin): + """Durable desired state for one external CalDAV resource. + + Rows are inserted in the same transaction as the local event mutation. A + dispatcher leases and commits the row before performing network I/O, so a + worker crash can be detected and reconciled without repeating a blind + write. + """ + + __tablename__ = "calendar_outbox_operations" + __table_args__ = ( + UniqueConstraint("idempotency_key", name="uq_calendar_outbox_operations_idempotency_key"), + Index("ix_calendar_outbox_due", "status", "available_at", "created_at"), + Index("ix_calendar_outbox_tenant_status", "tenant_id", "status"), + Index("ix_calendar_outbox_resource", "source_id", "resource_href", "created_at"), + Index("ix_calendar_outbox_lease", "status", "lease_expires_at"), + ) + + id: Mapped[str] = mapped_column(String(36), primary_key=True, default=new_uuid) + tenant_id: Mapped[str] = mapped_column(String(36), nullable=False, index=True) + source_id: Mapped[str] = mapped_column( + ForeignKey("calendar_sync_sources.id", ondelete="CASCADE"), + nullable=False, + index=True, + ) + event_id: Mapped[str | None] = mapped_column( + ForeignKey("calendar_events.id", ondelete="SET NULL"), + nullable=True, + index=True, + ) + operation_kind: Mapped[str] = mapped_column(String(20), nullable=False, index=True) + resource_href: Mapped[str] = mapped_column(String(1000), nullable=False) + payload_ics: Mapped[str | None] = mapped_column(Text) + payload_fingerprint: Mapped[str | None] = mapped_column(String(64), nullable=True, index=True) + expected_etag: Mapped[str | None] = mapped_column(String(255)) + idempotency_key: Mapped[str] = mapped_column(String(64), nullable=False) + status: Mapped[str] = mapped_column(String(30), default="pending", nullable=False, index=True) + attempt_count: Mapped[int] = mapped_column(Integer, default=0, nullable=False) + max_attempts: Mapped[int] = mapped_column(Integer, default=8, nullable=False) + available_at: Mapped[datetime] = mapped_column( + DateTime(timezone=True), + default=lambda: datetime.now(timezone.utc), + nullable=False, + index=True, + ) + last_attempt_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True) + lease_token: Mapped[str | None] = mapped_column(String(36), nullable=True, index=True) + lease_expires_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True, index=True) + completed_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True) + reconciled_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True) + remote_etag: Mapped[str | None] = mapped_column(String(255)) + last_error: Mapped[str | None] = mapped_column(Text) + metadata_: Mapped[dict[str, Any] | None] = mapped_column("metadata", JSON, nullable=True) + + source: Mapped[CalendarSyncSource] = relationship() + event: Mapped[CalendarEvent | None] = relationship() + + +__all__ = [ + "CalendarCollection", + "CalendarEvent", + "CalendarOutboxOperation", + "CalendarSyncCredential", + "CalendarSyncSource", + "new_uuid", +] diff --git a/src/govoplan_calendar/backend/manifest.py b/src/govoplan_calendar/backend/manifest.py index 978a527..6c0a84b 100644 --- a/src/govoplan_calendar/backend/manifest.py +++ b/src/govoplan_calendar/backend/manifest.py @@ -4,8 +4,23 @@ from pathlib import Path from govoplan_calendar.backend.db import models as calendar_models # noqa: F401 - populate Calendar ORM metadata from govoplan_core.core.access import CAPABILITY_AUTH_PERMISSION_EVALUATOR, CAPABILITY_AUTH_PRINCIPAL_RESOLVER +from govoplan_core.core.calendar import ( + CALENDAR_AVAILABILITY_READ_SCOPE, + CALENDAR_EVENT_WRITE_SCOPE, + CAPABILITY_CALENDAR_OUTBOX, + CAPABILITY_CALENDAR_SCHEDULING, +) from govoplan_core.core.module_guards import drop_table_retirement_provider, persistent_table_uninstall_guard -from govoplan_core.core.modules import FrontendModule, MigrationSpec, ModuleContext, ModuleManifest, NavItem, PermissionDefinition, RoleTemplate +from govoplan_core.core.modules import ( + FrontendModule, + MigrationSpec, + ModuleContext, + ModuleInterfaceProvider, + ModuleManifest, + NavItem, + PermissionDefinition, + RoleTemplate, +) from govoplan_core.db.base import Base @@ -28,11 +43,11 @@ PERMISSIONS = ( _permission("calendar:calendar:write", "Manage calendars", "Create and edit tenant calendar collections."), _permission("calendar:calendar:admin", "Administer calendars", "Delete calendars and manage tenant-level calendar settings."), _permission("calendar:event:read", "View calendar events", "List and inspect calendar events."), - _permission("calendar:event:write", "Manage calendar events", "Create and edit calendar events."), + _permission(CALENDAR_EVENT_WRITE_SCOPE, "Manage calendar events", "Create and edit calendar events."), _permission("calendar:event:delete", "Delete calendar events", "Delete or cancel calendar events where policy allows it."), _permission("calendar:event:import", "Import iCalendar events", "Import VEVENT data from iCalendar sources."), _permission("calendar:event:export", "Export iCalendar events", "Export events as text/calendar VEVENT data."), - _permission("calendar:availability:read", "Read availability", "Read free/busy and availability data for integrations."), + _permission(CALENDAR_AVAILABILITY_READ_SCOPE, "Read availability", "Read free/busy and availability data for integrations."), ) ROLE_TEMPLATES = ( @@ -44,11 +59,11 @@ ROLE_TEMPLATES = ( "calendar:calendar:read", "calendar:calendar:write", "calendar:event:read", - "calendar:event:write", + CALENDAR_EVENT_WRITE_SCOPE, "calendar:event:delete", "calendar:event:import", "calendar:event:export", - "calendar:availability:read", + CALENDAR_AVAILABILITY_READ_SCOPE, ), ), RoleTemplate( @@ -59,20 +74,32 @@ ROLE_TEMPLATES = ( "calendar:calendar:read", "calendar:event:read", "calendar:event:export", - "calendar:availability:read", + CALENDAR_AVAILABILITY_READ_SCOPE, ), ), ) def _tenant_summary(session, tenant_id: str) -> dict[str, int]: - from govoplan_calendar.backend.db.models import CalendarCollection, CalendarEvent, CalendarSyncCredential, CalendarSyncSource + from govoplan_calendar.backend.db.models import ( + CalendarCollection, + CalendarEvent, + CalendarOutboxOperation, + CalendarSyncCredential, + CalendarSyncSource, + ) return { "calendars": session.query(CalendarCollection).filter(CalendarCollection.tenant_id == tenant_id, CalendarCollection.deleted_at.is_(None)).count(), "calendar_events": session.query(CalendarEvent).filter(CalendarEvent.tenant_id == tenant_id, CalendarEvent.deleted_at.is_(None)).count(), "calendar_sync_sources": session.query(CalendarSyncSource).filter(CalendarSyncSource.tenant_id == tenant_id, CalendarSyncSource.deleted_at.is_(None)).count(), "calendar_sync_credentials": session.query(CalendarSyncCredential).filter(CalendarSyncCredential.tenant_id == tenant_id, CalendarSyncCredential.deleted_at.is_(None)).count(), + "calendar_outbox_pending": session.query(CalendarOutboxOperation) + .filter( + CalendarOutboxOperation.tenant_id == tenant_id, + CalendarOutboxOperation.status.in_(("pending", "retry", "in_progress")), + ) + .count(), } @@ -85,16 +112,46 @@ def _calendar_router(context: ModuleContext): return router +def _calendar_scheduling_provider(context: ModuleContext) -> object: + del context + from govoplan_calendar.backend.capabilities import SqlCalendarSchedulingProvider + + return SqlCalendarSchedulingProvider() + + +def _calendar_outbox_provider(context: ModuleContext) -> object: + from govoplan_calendar.backend.outbox import ( + OUTBOX_DEFAULT_TERMINAL_RETENTION_DAYS, + SqlCalendarOutboxProvider, + ) + + return SqlCalendarOutboxProvider( + terminal_retention_days=getattr( + context.settings, + "calendar_outbox_terminal_retention_days", + OUTBOX_DEFAULT_TERMINAL_RETENTION_DAYS, + ) + ) + + manifest = ModuleManifest( id="calendar", name="Calendar", version="0.1.8", required_capabilities=(CAPABILITY_AUTH_PRINCIPAL_RESOLVER, CAPABILITY_AUTH_PERMISSION_EVALUATOR), optional_dependencies=("mail", "tasks", "scheduling", "appointments", "workflow", "notifications", "dms", "connectors"), + provides_interfaces=( + ModuleInterfaceProvider(name="calendar.outbox", version="0.1.8"), + ModuleInterfaceProvider(name="calendar.scheduling", version="0.1.8"), + ), permissions=PERMISSIONS, route_factory=_calendar_router, role_templates=ROLE_TEMPLATES, tenant_summary_providers=(_tenant_summary,), + capability_factories={ + CAPABILITY_CALENDAR_OUTBOX: _calendar_outbox_provider, + CAPABILITY_CALENDAR_SCHEDULING: _calendar_scheduling_provider, + }, nav_items=(NavItem(path="/calendar", label="Calendar", icon="calendar", required_any=("calendar:event:read",), order=55),), frontend=FrontendModule( module_id="calendar", @@ -109,6 +166,7 @@ manifest = ModuleManifest( retirement_provider=drop_table_retirement_provider( calendar_models.CalendarCollection, calendar_models.CalendarEvent, + calendar_models.CalendarOutboxOperation, calendar_models.CalendarSyncCredential, calendar_models.CalendarSyncSource, label="Calendar", @@ -119,6 +177,7 @@ manifest = ModuleManifest( persistent_table_uninstall_guard( calendar_models.CalendarCollection, calendar_models.CalendarEvent, + calendar_models.CalendarOutboxOperation, calendar_models.CalendarSyncCredential, calendar_models.CalendarSyncSource, label="Calendar", diff --git a/src/govoplan_calendar/backend/migrations/dev_versions/af1b2c3d4e5f_calendar_caldav_outbox.py b/src/govoplan_calendar/backend/migrations/dev_versions/af1b2c3d4e5f_calendar_caldav_outbox.py new file mode 100644 index 0000000..98d504b --- /dev/null +++ b/src/govoplan_calendar/backend/migrations/dev_versions/af1b2c3d4e5f_calendar_caldav_outbox.py @@ -0,0 +1,28 @@ +"""Add the durable CalDAV desired-state outbox on the development track. + +Revision ID: af1b2c3d4e5f +Revises: 9e0f1a2b3c4d +Create Date: 2026-07-20 00:00:00.000000 +""" +from __future__ import annotations + +from govoplan_calendar.backend.migrations.versions.af1b2c3d4e5f_calendar_caldav_outbox import ( + downgrade as _downgrade, +) +from govoplan_calendar.backend.migrations.versions.af1b2c3d4e5f_calendar_caldav_outbox import ( + upgrade as _upgrade, +) + + +revision = "af1b2c3d4e5f" +down_revision = "9e0f1a2b3c4d" +branch_labels = None +depends_on = "4f2a9c8e7b6d" + + +def upgrade() -> None: + _upgrade() + + +def downgrade() -> None: + _downgrade() diff --git a/src/govoplan_calendar/backend/migrations/versions/af1b2c3d4e5f_calendar_caldav_outbox.py b/src/govoplan_calendar/backend/migrations/versions/af1b2c3d4e5f_calendar_caldav_outbox.py new file mode 100644 index 0000000..4acc826 --- /dev/null +++ b/src/govoplan_calendar/backend/migrations/versions/af1b2c3d4e5f_calendar_caldav_outbox.py @@ -0,0 +1,114 @@ +"""Add the durable CalDAV desired-state outbox. + +Revision ID: af1b2c3d4e5f +Revises: 9e0f1a2b3c4d +Create Date: 2026-07-20 00:00:00.000000 +""" +from __future__ import annotations + +from alembic import op +import sqlalchemy as sa + + +revision = "af1b2c3d4e5f" +down_revision = "9e0f1a2b3c4d" +branch_labels = None +depends_on = None + + +def upgrade() -> None: + op.create_table( + "calendar_outbox_operations", + sa.Column("id", sa.String(length=36), nullable=False), + sa.Column("tenant_id", sa.String(length=36), nullable=False), + sa.Column("source_id", sa.String(length=36), nullable=False), + sa.Column("event_id", sa.String(length=36), nullable=True), + sa.Column("operation_kind", sa.String(length=20), nullable=False), + sa.Column("resource_href", sa.String(length=1000), nullable=False), + sa.Column("payload_ics", sa.Text(), nullable=True), + sa.Column("payload_fingerprint", sa.String(length=64), nullable=True), + sa.Column("expected_etag", sa.String(length=255), nullable=True), + sa.Column("idempotency_key", sa.String(length=64), nullable=False), + sa.Column("status", sa.String(length=30), nullable=False), + sa.Column("attempt_count", sa.Integer(), nullable=False), + sa.Column("max_attempts", sa.Integer(), nullable=False), + sa.Column("available_at", sa.DateTime(timezone=True), nullable=False), + sa.Column("last_attempt_at", sa.DateTime(timezone=True), nullable=True), + sa.Column("lease_token", sa.String(length=36), nullable=True), + sa.Column("lease_expires_at", sa.DateTime(timezone=True), nullable=True), + sa.Column("completed_at", sa.DateTime(timezone=True), nullable=True), + sa.Column("reconciled_at", sa.DateTime(timezone=True), nullable=True), + sa.Column("remote_etag", sa.String(length=255), nullable=True), + sa.Column("last_error", sa.Text(), nullable=True), + sa.Column("metadata", sa.JSON(), nullable=True), + sa.Column("created_at", sa.DateTime(timezone=True), nullable=False), + sa.Column("updated_at", sa.DateTime(timezone=True), nullable=False), + sa.ForeignKeyConstraint( + ["event_id"], + ["calendar_events.id"], + name=op.f("fk_calendar_outbox_operations_event_id_calendar_events"), + ondelete="SET NULL", + ), + sa.ForeignKeyConstraint( + ["source_id"], + ["calendar_sync_sources.id"], + name=op.f("fk_calendar_outbox_operations_source_id_calendar_sync_sources"), + ondelete="CASCADE", + ), + sa.ForeignKeyConstraint( + ["tenant_id"], + ["core_scopes.id"], + name=op.f("fk_calendar_outbox_operations_tenant_id_scopes"), + ondelete="CASCADE", + ), + sa.PrimaryKeyConstraint("id", name=op.f("pk_calendar_outbox_operations")), + sa.UniqueConstraint( + "idempotency_key", + name="uq_calendar_outbox_operations_idempotency_key", + ), + ) + op.create_index( + "ix_calendar_outbox_due", + "calendar_outbox_operations", + ["status", "available_at", "created_at"], + unique=False, + ) + op.create_index( + "ix_calendar_outbox_lease", + "calendar_outbox_operations", + ["status", "lease_expires_at"], + unique=False, + ) + op.create_index( + "ix_calendar_outbox_resource", + "calendar_outbox_operations", + ["source_id", "resource_href", "created_at"], + unique=False, + ) + op.create_index( + "ix_calendar_outbox_tenant_status", + "calendar_outbox_operations", + ["tenant_id", "status"], + unique=False, + ) + for column in ( + "available_at", + "event_id", + "lease_expires_at", + "lease_token", + "operation_kind", + "payload_fingerprint", + "source_id", + "status", + "tenant_id", + ): + op.create_index( + op.f(f"ix_calendar_outbox_operations_{column}"), + "calendar_outbox_operations", + [column], + unique=False, + ) + + +def downgrade() -> None: + op.drop_table("calendar_outbox_operations") diff --git a/src/govoplan_calendar/backend/outbox.py b/src/govoplan_calendar/backend/outbox.py new file mode 100644 index 0000000..a17806d --- /dev/null +++ b/src/govoplan_calendar/backend/outbox.py @@ -0,0 +1,1501 @@ +from __future__ import annotations + +import hashlib +import json +import uuid +from collections.abc import Callable, Mapping +from datetime import datetime, timedelta, timezone +from typing import Any + +from sqlalchemy import and_, event, or_ +from sqlalchemy.orm import Session + +from govoplan_calendar.backend.caldav import ( + CalDAVNotFound, + CalDAVObject, + CalDAVPreconditionFailed, +) +from govoplan_calendar.backend.db.models import ( + CalendarEvent, + CalendarOutboxOperation, + CalendarSyncSource, +) +from govoplan_calendar.backend.ical import events_to_ics, parse_vevents +from govoplan_core.db.base import utcnow + + +OUTBOX_PENDING_STATUSES = {"pending", "retry"} +OUTBOX_ACTIVE_STATUSES = OUTBOX_PENDING_STATUSES | {"in_progress"} +OUTBOX_TERMINAL_STATUSES = {"succeeded", "superseded", "cancelled", "conflict", "dead"} +OUTBOX_UNRESOLVED_DESIRED_STATUSES = OUTBOX_ACTIVE_STATUSES | {"conflict", "dead"} +OUTBOX_PURGEABLE_STATUSES = OUTBOX_TERMINAL_STATUSES - OUTBOX_UNRESOLVED_DESIRED_STATUSES +OUTBOX_DEFAULT_MAX_ATTEMPTS = 8 +OUTBOX_DEFAULT_TERMINAL_RETENTION_DAYS = 90 +OUTBOX_LEASE_SECONDS = 300 +OUTBOX_BACKOFF_BASE_SECONDS = 5 +OUTBOX_BACKOFF_MAX_SECONDS = 3600 +_AFTER_COMMIT_TENANTS = "govoplan_calendar_after_commit_outbox_tenants" + + +def _source_configuration_snapshot(source: CalendarSyncSource) -> dict[str, str]: + return { + "calendar_id": source.calendar_id, + "collection_url": source.collection_url, + } + + +def _operation_source_unavailable_reason( + operation: CalendarOutboxOperation, + source: CalendarSyncSource | None, +) -> str | None: + if source is None: + return "CalDAV source no longer exists" + if source.tenant_id != operation.tenant_id: + return "Calendar outbox source tenant does not match the operation tenant" + if source.deleted_at is not None: + return "CalDAV source was retired before this operation was delivered" + if not source.sync_enabled: + return "CalDAV source was disabled before this operation was delivered" + if source.sync_direction != "two_way": + return "CalDAV source was made inbound-only before this operation was delivered" + metadata = operation.metadata_ or {} + frozen_configuration = ( + metadata.get("source_configuration") if isinstance(metadata, dict) else None + ) + if ( + isinstance(frozen_configuration, dict) + and frozen_configuration != _source_configuration_snapshot(source) + ): + return "CalDAV source endpoint or calendar changed before this operation was delivered" + return None + + +def _as_utc(value: datetime) -> datetime: + if value.tzinfo is None: + return value.replace(tzinfo=timezone.utc) + return value.astimezone(timezone.utc) + + +@event.listens_for(Session, "after_commit") +def _dispatch_committed_calendar_outbox(session: Session) -> None: + # ``after_commit`` also fires when a SAVEPOINT is released. The durable + # row is not visible outside the outer transaction at that point, so keep + # the wake-up registered until the root transaction commits. + if session.in_nested_transaction(): + return + tenant_ids = tuple(session.info.pop(_AFTER_COMMIT_TENANTS, ())) + for tenant_id in tenant_ids: + _enqueue_celery_dispatch(tenant_id) + + +@event.listens_for(Session, "after_rollback") +def _discard_rolled_back_calendar_outbox(session: Session) -> None: + # A nested rollback must not discard wake-ups registered by work in the + # still-live outer transaction. A possibly superfluous wake-up after an + # outer commit is harmless; the dispatcher simply finds no due row. + if session.in_nested_transaction(): + return + session.info.pop(_AFTER_COMMIT_TENANTS, None) + + +def _schedule_dispatch_after_commit(session: Session, tenant_id: str) -> None: + tenants = session.info.setdefault(_AFTER_COMMIT_TENANTS, []) + if tenant_id not in tenants: + tenants.append(tenant_id) + + +def _enqueue_celery_dispatch(tenant_id: str) -> None: + try: + from govoplan_core.celery_app import celery + from govoplan_core.settings import settings + + if not getattr(settings, "celery_enabled", False): + return + celery.send_task( + "govoplan.calendar.dispatch_outbox", + args=[tenant_id, 50], + queue="calendar", + ) + except Exception: + # The committed outbox row is the source of truth. Broker/import + # failures are recovered by the periodic dispatcher. + return + + +def calendar_payload_fingerprint(ics: str) -> str: + """Return a semantic fingerprint resilient to harmless ICS formatting.""" + + try: + components: list[dict[str, Any]] = [] + for parsed in parse_vevents(ics): + normalized = {key: value for key, value in parsed.items() if key != "raw_ics"} + components.append(normalized) + material = json.dumps(components, sort_keys=True, separators=(",", ":"), default=str) + except Exception: + material = "\n".join(line.rstrip() for line in ics.replace("\r\n", "\n").splitlines()).strip() + return hashlib.sha256(material.encode("utf-8")).hexdigest() + + +def _operation_idempotency_key( + *, + source: CalendarSyncSource, + href: str, + operation_kind: str, + payload_fingerprint: str | None, + expected_etag: str | None, + generation: str, +) -> str: + material = "\0".join( + ( + source.id, + href, + operation_kind, + payload_fingerprint or "", + expected_etag or "", + generation, + ) + ) + return hashlib.sha256(material.encode("utf-8")).hexdigest() + + +def _event_generation(events: list[CalendarEvent], *, trigger_event: CalendarEvent | None) -> str: + parts = [f"{item.id}:{item.sequence}" for item in sorted(events, key=lambda item: item.id)] + if trigger_event is not None and trigger_event not in events: + parts.append(f"trigger:{trigger_event.id}:{trigger_event.sequence}") + return "|".join(parts) + + +def _supersede_pending_resource_operations( + session: Session, + *, + source_id: str, + href: str, + now: datetime, +) -> None: + for operation in ( + session.query(CalendarOutboxOperation) + .filter( + CalendarOutboxOperation.source_id == source_id, + CalendarOutboxOperation.resource_href == href, + or_( + and_( + CalendarOutboxOperation.status == "pending", + CalendarOutboxOperation.attempt_count == 0, + CalendarOutboxOperation.last_attempt_at.is_(None), + ), + CalendarOutboxOperation.status.in_(("conflict", "dead")), + ), + ) + .all() + ): + was_unattempted = operation.status == "pending" + operation.status = "superseded" + operation.completed_at = operation.completed_at or now + if was_unattempted: + operation.last_error = ( + "Superseded by a newer desired state for the same CalDAV resource" + ) + + +def _set_event_outbox_state( + event_model: CalendarEvent, + *, + source: CalendarSyncSource, + href: str, + operation: CalendarOutboxOperation, + state: str, + raw_ics: str | None, +) -> None: + event_model.source_kind = "caldav" + event_model.source_href = href + if raw_ics is not None: + event_model.raw_ics = raw_ics + metadata = dict(event_model.metadata_ or {}) + caldav_metadata = dict(metadata.get("caldav") or {}) + caldav_metadata.update( + { + "source_id": source.id, + "collection_url": source.collection_url, + "href": href, + "etag": event_model.etag, + "external_state": state, + "outbox_operation_id": operation.id, + } + ) + metadata["caldav"] = caldav_metadata + event_model.metadata_ = metadata + + +def enqueue_caldav_put( + session: Session, + *, + source: CalendarSyncSource, + event_model: CalendarEvent, +) -> CalendarOutboxOperation: + """Persist the exact desired remote resource in the caller transaction.""" + + from govoplan_calendar.backend.service import ( + caldav_resource_events, + caldav_resource_sort_key, + generated_caldav_href, + normalize_caldav_href, + resource_etag, + ) + + href = normalize_caldav_href( + source.collection_url, + event_model.source_href or generated_caldav_href(session, source=source, event=event_model), + ) + resource_events = caldav_resource_events(session, source=source, href=href) + if event_model not in resource_events: + resource_events.append(event_model) + resource_events = sorted(resource_events, key=caldav_resource_sort_key) + payload_ics = events_to_ics(resource_events) + return enqueue_caldav_desired_state( + session, + source=source, + trigger_event=event_model, + href=href, + resource_events=resource_events, + payload_ics=payload_ics, + expected_etag=resource_etag(resource_events), + ) + + +def enqueue_caldav_delete( + session: Session, + *, + source: CalendarSyncSource, + event_model: CalendarEvent, +) -> CalendarOutboxOperation | None: + """Persist the desired state after deleting one VEVENT component.""" + + if not event_model.source_href: + return None + from govoplan_calendar.backend.service import ( + caldav_resource_events, + caldav_resource_sort_key, + normalize_caldav_href, + resource_etag, + ) + + href = normalize_caldav_href(source.collection_url, event_model.source_href) + remaining = [ + item + for item in caldav_resource_events(session, source=source, href=href) + if item.id != event_model.id + ] + remaining = sorted(remaining, key=caldav_resource_sort_key) + payload_ics = events_to_ics(remaining) if remaining else None + expected_etag = event_model.etag or resource_etag(remaining) + return enqueue_caldav_desired_state( + session, + source=source, + trigger_event=event_model, + href=href, + resource_events=remaining, + payload_ics=payload_ics, + expected_etag=expected_etag, + ) + + +def enqueue_caldav_desired_state( + session: Session, + *, + source: CalendarSyncSource, + trigger_event: CalendarEvent | None, + href: str, + resource_events: list[CalendarEvent], + payload_ics: str | None, + expected_etag: str | None, +) -> CalendarOutboxOperation: + now = utcnow() + operation_kind = "put" if payload_ics is not None else "delete" + fingerprint = calendar_payload_fingerprint(payload_ics) if payload_ics is not None else None + idempotency_key = _operation_idempotency_key( + source=source, + href=href, + operation_kind=operation_kind, + payload_fingerprint=fingerprint, + expected_etag=expected_etag, + generation=_event_generation(resource_events, trigger_event=trigger_event), + ) + existing = ( + session.query(CalendarOutboxOperation) + .filter(CalendarOutboxOperation.idempotency_key == idempotency_key) + .first() + ) + if existing is not None: + return existing + _supersede_pending_resource_operations(session, source_id=source.id, href=href, now=now) + operation = CalendarOutboxOperation( + tenant_id=source.tenant_id, + source_id=source.id, + event_id=trigger_event.id if trigger_event is not None else None, + operation_kind=operation_kind, + resource_href=href, + payload_ics=payload_ics, + payload_fingerprint=fingerprint, + expected_etag=expected_etag, + idempotency_key=idempotency_key, + status="pending", + attempt_count=0, + max_attempts=OUTBOX_DEFAULT_MAX_ATTEMPTS, + available_at=now, + metadata_={ + "event_ids": [item.id for item in resource_events], + "overwrite": source.conflict_policy == "overwrite", + "source_configuration": _source_configuration_snapshot(source), + }, + ) + session.add(operation) + session.flush() + for item in resource_events: + _set_event_outbox_state( + item, + source=source, + href=href, + operation=operation, + state="queued", + raw_ics=payload_ics, + ) + if trigger_event is not None and trigger_event not in resource_events: + _set_event_outbox_state( + trigger_event, + source=source, + href=href, + operation=operation, + state="queued_delete", + raw_ics=None, + ) + session.flush() + _schedule_dispatch_after_commit(session, source.tenant_id) + return operation + + +def list_calendar_outbox_operations( + session: Session, + *, + tenant_id: str, + status: str | None = None, + source_id: str | None = None, + limit: int = 100, +) -> list[CalendarOutboxOperation]: + query = session.query(CalendarOutboxOperation).filter( + CalendarOutboxOperation.tenant_id == tenant_id + ) + if status: + query = query.filter(CalendarOutboxOperation.status == status) + if source_id: + query = query.filter(CalendarOutboxOperation.source_id == source_id) + return query.order_by( + CalendarOutboxOperation.created_at.desc(), + CalendarOutboxOperation.id.desc(), + ).limit(max(1, min(limit, 500))).all() + + +def get_calendar_outbox_operation( + session: Session, + *, + tenant_id: str, + operation_id: str, +) -> CalendarOutboxOperation: + operation = ( + session.query(CalendarOutboxOperation) + .filter( + CalendarOutboxOperation.tenant_id == tenant_id, + CalendarOutboxOperation.id == operation_id, + ) + .first() + ) + if operation is None: + raise ValueError("Calendar outbox operation not found") + return operation + + +def _lock_operation_source( + session: Session, + *, + tenant_id: str, + operation_id: str, +) -> tuple[CalendarOutboxOperation, CalendarSyncSource | None]: + """Lock source then operation, matching the outbound worker lock order.""" + + peek = ( + session.query(CalendarOutboxOperation.source_id) + .filter( + CalendarOutboxOperation.tenant_id == tenant_id, + CalendarOutboxOperation.id == operation_id, + ) + .first() + ) + if peek is None: + raise ValueError("Calendar outbox operation not found") + source = ( + session.query(CalendarSyncSource) + .filter(CalendarSyncSource.id == peek[0]) + .with_for_update() + .one_or_none() + ) + operation = ( + session.query(CalendarOutboxOperation) + .filter( + CalendarOutboxOperation.tenant_id == tenant_id, + CalendarOutboxOperation.id == operation_id, + ) + .with_for_update() + .one() + ) + if source is not None and source.id != operation.source_id: + raise ValueError("Calendar outbox source changed while acquiring the operation lock") + return operation, source + + +def _newer_resource_generation_exists( + session: Session, + operation: CalendarOutboxOperation, +) -> bool: + return ( + session.query(CalendarOutboxOperation.id) + .filter( + CalendarOutboxOperation.source_id == operation.source_id, + CalendarOutboxOperation.resource_href == operation.resource_href, + or_( + CalendarOutboxOperation.created_at > operation.created_at, + and_( + CalendarOutboxOperation.created_at == operation.created_at, + CalendarOutboxOperation.id > operation.id, + ), + ), + ) + .first() + is not None + ) + + +def _assert_latest_resource_generation( + session: Session, + operation: CalendarOutboxOperation, +) -> None: + if _newer_resource_generation_exists(session, operation): + raise ValueError( + "This Calendar outbox operation is stale because a newer desired state exists; " + "use the latest operation or make a new local edit" + ) + + +def _reclaim_expired_leases( + session: Session, + *, + now: datetime, + tenant_id: str | None = None, +) -> int: + query = session.query(CalendarOutboxOperation).filter( + CalendarOutboxOperation.status == "in_progress", + CalendarOutboxOperation.lease_expires_at.is_not(None), + CalendarOutboxOperation.lease_expires_at <= now, + ) + if tenant_id: + query = query.filter(CalendarOutboxOperation.tenant_id == tenant_id) + expired = query.with_for_update(skip_locked=True).all() + for operation in expired: + operation.status = "retry" + operation.available_at = now + operation.lease_token = None + operation.lease_expires_at = None + operation.last_error = "Worker lease expired; remote outcome will be reconciled before retry" + return len(expired) + + +def claim_next_calendar_outbox_operation( + session: Session, + *, + tenant_id: str | None = None, + now: datetime | None = None, + lease_seconds: int = OUTBOX_LEASE_SECONDS, +) -> CalendarOutboxOperation | None: + now = now or utcnow() + _reclaim_expired_leases(session, now=now, tenant_id=tenant_id) + query = session.query(CalendarOutboxOperation).filter( + CalendarOutboxOperation.status.in_(sorted(OUTBOX_PENDING_STATUSES)), + CalendarOutboxOperation.available_at <= now, + ) + if tenant_id: + query = query.filter(CalendarOutboxOperation.tenant_id == tenant_id) + candidates = query.order_by( + CalendarOutboxOperation.available_at.asc(), + CalendarOutboxOperation.created_at.asc(), + CalendarOutboxOperation.id.asc(), + ).with_for_update(skip_locked=True).limit(50).all() + for operation in candidates: + source_delivery_in_progress = ( + session.query(CalendarOutboxOperation.id) + .filter( + CalendarOutboxOperation.source_id == operation.source_id, + CalendarOutboxOperation.status == "in_progress", + ) + .first() + ) + if source_delivery_in_progress is not None: + # Delivery holds the source row lock across remote I/O. Leasing a + # second href for the same source would only make its lease age + # while it waits behind that lock. + continue + active_predecessor = ( + session.query(CalendarOutboxOperation.id) + .filter( + CalendarOutboxOperation.source_id == operation.source_id, + CalendarOutboxOperation.resource_href == operation.resource_href, + CalendarOutboxOperation.status.in_(sorted(OUTBOX_ACTIVE_STATUSES)), + or_( + CalendarOutboxOperation.created_at < operation.created_at, + and_( + CalendarOutboxOperation.created_at == operation.created_at, + CalendarOutboxOperation.id < operation.id, + ), + ), + ) + .first() + ) + if active_predecessor is not None: + continue + succeeded_predecessor = ( + session.query(CalendarOutboxOperation) + .filter( + CalendarOutboxOperation.source_id == operation.source_id, + CalendarOutboxOperation.resource_href == operation.resource_href, + CalendarOutboxOperation.status == "succeeded", + CalendarOutboxOperation.completed_at.is_not(None), + # Only repair the causal window where the newer row was + # created while its predecessor was still completing. A + # later inbound sync may legitimately have supplied a newer + # expected ETag and must not be overwritten here. + CalendarOutboxOperation.completed_at >= operation.created_at, + or_( + CalendarOutboxOperation.created_at < operation.created_at, + and_( + CalendarOutboxOperation.created_at == operation.created_at, + CalendarOutboxOperation.id < operation.id, + ), + ), + ) + .order_by( + CalendarOutboxOperation.created_at.desc(), + CalendarOutboxOperation.id.desc(), + ) + .first() + ) + if succeeded_predecessor is not None: + operation.expected_etag = ( + succeeded_predecessor.remote_etag + if succeeded_predecessor.operation_kind == "put" + else None + ) + operation.status = "in_progress" + operation.attempt_count += 1 + operation.last_attempt_at = now + operation.lease_token = str(uuid.uuid4()) + operation.lease_expires_at = now + timedelta(seconds=max(30, lease_seconds)) + session.flush() + return operation + session.flush() + return None + + +def _fetch_remote_object(client: object, href: str) -> CalDAVObject: + fetch_state = getattr(client, "fetch_object_state", None) + if callable(fetch_state): + result = fetch_state(href) + if isinstance(result, CalDAVObject): + return result + payload = client.fetch_object(href) + return CalDAVObject(href=href, calendar_data=payload) + + +def _remote_matches_operation(client: object, operation: CalendarOutboxOperation) -> tuple[bool, str | None]: + try: + remote = _fetch_remote_object(client, operation.resource_href) + except CalDAVNotFound: + return operation.operation_kind == "delete", None + if operation.operation_kind == "delete": + return False, remote.etag + if not remote.calendar_data or not operation.payload_fingerprint: + return False, remote.etag + return calendar_payload_fingerprint(remote.calendar_data) == operation.payload_fingerprint, remote.etag + + +def _operation_event_ids(operation: CalendarOutboxOperation) -> list[str]: + metadata = operation.metadata_ or {} + values = metadata.get("event_ids") if isinstance(metadata, dict) else None + return [str(value) for value in values or []] + + +def _event_points_to_operation(event_model: CalendarEvent, operation: CalendarOutboxOperation) -> bool: + metadata = event_model.metadata_ or {} + caldav = metadata.get("caldav") if isinstance(metadata, dict) else None + return bool( + isinstance(caldav, dict) + and caldav.get("outbox_operation_id") == operation.id + and caldav.get("source_id") == operation.source_id + and caldav.get("href") == operation.resource_href + and event_model.source_kind == "caldav" + and event_model.source_href == operation.resource_href + ) + + +def _pointed_operation_events( + session: Session, + operation: CalendarOutboxOperation, +) -> list[CalendarEvent]: + event_ids = set(_operation_event_ids(operation)) + if operation.event_id: + event_ids.add(operation.event_id) + if not event_ids: + return [] + return [ + event_model + for event_model in ( + session.query(CalendarEvent) + .filter( + CalendarEvent.tenant_id == operation.tenant_id, + CalendarEvent.id.in_(sorted(event_ids)), + ) + .all() + ) + if _event_points_to_operation(event_model, operation) + ] + + +def _record_event_projection_change( + session: Session, + *, + event_model: CalendarEvent, + previous: dict[str, Any], +) -> None: + from govoplan_calendar.backend.service import record_calendar_event_change + + record_calendar_event_change( + session, + event=event_model, + operation="deleted" if event_model.deleted_at is not None else "updated", + user_id=None, + previous=previous, + ) + + +def _set_pointed_event_outbox_state( + session: Session, + *, + operation: CalendarOutboxOperation, + source: CalendarSyncSource, + state: str, + raw_ics: str | None, + remote_etag: str | None = None, +) -> None: + from govoplan_calendar.backend.service import calendar_event_change_payload + + for event_model in _pointed_operation_events(session, operation): + previous = calendar_event_change_payload(event_model, prefix="previous_") + if remote_etag: + event_model.etag = remote_etag + _set_event_outbox_state( + event_model, + source=source, + href=operation.resource_href, + operation=operation, + state=state, + raw_ics=raw_ics, + ) + _record_event_projection_change( + session, + event_model=event_model, + previous=previous, + ) + + +def _complete_operation( + session: Session, + *, + operation: CalendarOutboxOperation, + source: CalendarSyncSource, + remote_etag: str | None, + reconciled: bool, +) -> None: + now = utcnow() + operation.status = "succeeded" + operation.completed_at = now + operation.reconciled_at = now if reconciled else operation.reconciled_at + operation.remote_etag = remote_etag + operation.last_error = None + operation.lease_token = None + operation.lease_expires_at = None + source.last_status = "outbound_ok" + source.last_error = None + for stale_operation in ( + session.query(CalendarOutboxOperation) + .filter( + CalendarOutboxOperation.source_id == operation.source_id, + CalendarOutboxOperation.resource_href == operation.resource_href, + CalendarOutboxOperation.status.in_(("conflict", "dead")), + or_( + CalendarOutboxOperation.created_at < operation.created_at, + and_( + CalendarOutboxOperation.created_at == operation.created_at, + CalendarOutboxOperation.id < operation.id, + ), + ), + ) + .all() + ): + # A successfully delivered newer desired state makes older failed + # snapshots audit history, not unresolved intent. + stale_operation.status = "superseded" + if operation.operation_kind == "put" and remote_etag: + # A newer desired state may have been queued while this operation was + # already in progress. Its snapshot can only know the predecessor + # ETag, not the ETag produced by this PUT. Carry the causal ETag forward + # so the newer PUT/DELETE is conditional on the state we just wrote. + later_operations = ( + session.query(CalendarOutboxOperation) + .filter( + CalendarOutboxOperation.source_id == operation.source_id, + CalendarOutboxOperation.resource_href == operation.resource_href, + CalendarOutboxOperation.id != operation.id, + CalendarOutboxOperation.status.in_(sorted(OUTBOX_ACTIVE_STATUSES)), + or_( + CalendarOutboxOperation.created_at > operation.created_at, + and_( + CalendarOutboxOperation.created_at == operation.created_at, + CalendarOutboxOperation.id > operation.id, + ), + ), + ) + .all() + ) + later_ids = {item.id for item in later_operations} + for later_operation in later_operations: + later_operation.expected_etag = remote_etag + if later_ids: + from govoplan_calendar.backend.service import calendar_event_change_payload + + for event_model in ( + session.query(CalendarEvent) + .filter( + CalendarEvent.tenant_id == operation.tenant_id, + CalendarEvent.source_href == operation.resource_href, + ) + .all() + ): + metadata = event_model.metadata_ or {} + caldav = metadata.get("caldav") if isinstance(metadata, dict) else None + if isinstance(caldav, dict) and caldav.get("outbox_operation_id") in later_ids: + previous = calendar_event_change_payload(event_model, prefix="previous_") + event_model.etag = remote_etag + _record_event_projection_change( + session, + event_model=event_model, + previous=previous, + ) + _set_pointed_event_outbox_state( + session, + operation=operation, + source=source, + state="synced", + raw_ics=operation.payload_ics, + remote_etag=remote_etag, + ) + session.flush() + + +def _retry_delay_seconds(attempt_count: int) -> int: + exponent = max(0, min(attempt_count - 1, 20)) + return min(OUTBOX_BACKOFF_MAX_SECONDS, OUTBOX_BACKOFF_BASE_SECONDS * (2**exponent)) + + +def _fail_operation( + session: Session, + *, + operation: CalendarOutboxOperation, + source: CalendarSyncSource, + error: BaseException | str, + terminal_status: str | None = None, +) -> None: + now = utcnow() + message = str(error)[:4000] + if terminal_status: + operation.status = terminal_status + operation.completed_at = now + elif operation.attempt_count >= operation.max_attempts: + operation.status = "dead" + operation.completed_at = now + else: + operation.status = "retry" + operation.available_at = now + timedelta(seconds=_retry_delay_seconds(operation.attempt_count)) + operation.last_error = message + operation.lease_token = None + operation.lease_expires_at = None + if operation.status in {"dead", "conflict"}: + source.last_status = "outbound_error" + elif operation.status == "cancelled": + source.last_status = "outbound_cancelled" + else: + source.last_status = "outbound_retry" + source.last_error = message + event_state = ( + "failed" + if operation.status in {"dead", "conflict"} + else "cancelled" + if operation.status == "cancelled" + else "retry" + ) + _set_pointed_event_outbox_state( + session, + operation=operation, + source=source, + state=event_state, + raw_ics=operation.payload_ics, + ) + session.flush() + + +def execute_calendar_outbox_operation( + session: Session, + *, + operation_id: str, + lease_token: str, + client_factory: Callable[[Session, CalendarSyncSource], object] | None = None, +) -> CalendarOutboxOperation: + operation = session.get(CalendarOutboxOperation, operation_id) + if operation is None or operation.status != "in_progress" or operation.lease_token != lease_token: + raise ValueError("Calendar outbox lease is no longer valid") + source = ( + session.query(CalendarSyncSource) + .filter(CalendarSyncSource.id == operation.source_id) + .with_for_update() + .one_or_none() + ) + operation = ( + session.query(CalendarOutboxOperation) + .filter(CalendarOutboxOperation.id == operation_id) + .with_for_update() + .one() + ) + if operation.status != "in_progress" or operation.lease_token != lease_token: + raise ValueError("Calendar outbox lease is no longer valid") + operation_metadata = operation.metadata_ or {} + unavailable_reason = _operation_source_unavailable_reason(operation, source) + if unavailable_reason: + if source is not None and source.tenant_id == operation.tenant_id: + _fail_operation( + session, + operation=operation, + source=source, + error=unavailable_reason, + terminal_status="cancelled", + ) + else: + operation.status = "cancelled" + operation.completed_at = utcnow() + operation.last_error = unavailable_reason + operation.lease_token = None + operation.lease_expires_at = None + session.flush() + return operation + + overwrite = bool(operation_metadata.get("overwrite")) if isinstance(operation_metadata, dict) else False + client: object | None = None + try: + if client_factory is None: + from govoplan_calendar.backend.service import caldav_client_for_source + + client = caldav_client_for_source(session, source) + else: + client = client_factory(session, source) + if operation.operation_kind == "put": + result = client.put_object( + operation.resource_href, + operation.payload_ics or "", + etag=operation.expected_etag, + create=not bool(operation.expected_etag), + overwrite=overwrite, + ) + remote_etag = result.etag + reconciled = False + if not remote_etag: + try: + matched, remote_etag = _remote_matches_operation(client, operation) + except Exception: + matched, remote_etag = False, None + if not matched or not remote_etag: + _fail_operation( + session, + operation=operation, + source=source, + error="CalDAV PUT succeeded without an ETag and could not be reconciled safely", + ) + return operation + reconciled = True + _complete_operation( + session, + operation=operation, + source=source, + remote_etag=remote_etag, + reconciled=reconciled, + ) + return operation + + if not operation.expected_etag and not overwrite: + matched, remote_etag = _remote_matches_operation(client, operation) + if matched: + _complete_operation( + session, + operation=operation, + source=source, + remote_etag=remote_etag, + reconciled=True, + ) + return operation + _fail_operation( + session, + operation=operation, + source=source, + error=( + "CalDAV DELETE has no known ETag and the remote resource exists; " + "refusing to delete an unknown remote version" + ), + terminal_status="conflict", + ) + return operation + result = client.delete_object( + operation.resource_href, + etag=operation.expected_etag, + overwrite=overwrite, + ) + _complete_operation( + session, + operation=operation, + source=source, + remote_etag=result.etag, + reconciled=result.status == 404, + ) + return operation + except CalDAVPreconditionFailed as exc: + assert client is not None + try: + matched, remote_etag = _remote_matches_operation(client, operation) + except Exception: + _fail_operation(session, operation=operation, source=source, error=exc) + else: + if matched and (operation.operation_kind == "delete" or remote_etag): + _complete_operation( + session, + operation=operation, + source=source, + remote_etag=remote_etag, + reconciled=True, + ) + else: + _fail_operation( + session, + operation=operation, + source=source, + error=( + "Matching CalDAV resource has no usable ETag; retrying reconciliation" + if matched + else "CalDAV resource changed remotely; reconcile or sync before retrying" + ), + terminal_status=None if matched else "conflict", + ) + return operation + except Exception as exc: + matched, remote_etag = False, None + if client is not None: + try: + matched, remote_etag = _remote_matches_operation(client, operation) + except Exception: + matched, remote_etag = False, None + if matched and (operation.operation_kind == "delete" or remote_etag): + _complete_operation( + session, + operation=operation, + source=source, + remote_etag=remote_etag, + reconciled=True, + ) + else: + _fail_operation(session, operation=operation, source=source, error=exc) + return operation + + +def dispatch_calendar_outbox( + session: Session, + *, + tenant_id: str | None = None, + limit: int = 50, + client_factory: Callable[[Session, CalendarSyncSource], object] | None = None, + now: datetime | None = None, +) -> dict[str, object]: + """Lease, deliver, and durably record up to ``limit`` operations.""" + + results: list[dict[str, object]] = [] + for _index in range(max(1, min(limit, 500))): + operation = claim_next_calendar_outbox_operation( + session, + tenant_id=tenant_id, + now=now, + ) + if operation is None: + session.commit() + break + operation_id = operation.id + lease_token = operation.lease_token or "" + # The lease must be visible before external I/O. + session.commit() + operation = execute_calendar_outbox_operation( + session, + operation_id=operation_id, + lease_token=lease_token, + client_factory=client_factory, + ) + results.append( + { + "id": operation.id, + "status": operation.status, + "attempt_count": operation.attempt_count, + "last_error": operation.last_error, + } + ) + session.commit() + return { + "processed": len(results), + "succeeded": sum(item["status"] == "succeeded" for item in results), + "retrying": sum(item["status"] == "retry" for item in results), + "failed": sum(item["status"] in {"conflict", "dead", "cancelled"} for item in results), + "operations": results, + } + + +def purge_terminal_calendar_outbox_operations( + session: Session, + *, + retention_days: int, + tenant_id: str | None = None, + now: datetime | None = None, + limit: int = 1000, +) -> int: + """Delete bounded, resolved outbox history older than the configured window. + + ``conflict`` and ``dead`` are terminal delivery outcomes but still carry + unresolved local desired state. They are intentionally excluded until an + administrator retries, reconciles, or discards them. A retention of zero + disables automatic cleanup. + """ + + if retention_days < 0: + raise ValueError("Calendar outbox terminal retention days must be zero or greater") + if retention_days == 0: + return 0 + cutoff = (now or utcnow()) - timedelta(days=retention_days) + query = session.query(CalendarOutboxOperation.id).filter( + CalendarOutboxOperation.status.in_(sorted(OUTBOX_PURGEABLE_STATUSES)), + CalendarOutboxOperation.completed_at.is_not(None), + CalendarOutboxOperation.completed_at <= cutoff, + ) + if tenant_id: + query = query.filter(CalendarOutboxOperation.tenant_id == tenant_id) + operation_ids = [ + operation_id + for (operation_id,) in ( + query.order_by( + CalendarOutboxOperation.completed_at.asc(), + CalendarOutboxOperation.id.asc(), + ) + .limit(max(1, min(limit, 10_000))) + .all() + ) + ] + if not operation_ids: + return 0 + deleted = ( + session.query(CalendarOutboxOperation) + .filter(CalendarOutboxOperation.id.in_(operation_ids)) + .delete(synchronize_session=False) + ) + session.flush() + return int(deleted) + + +def retry_calendar_outbox_operation( + session: Session, + *, + tenant_id: str, + operation_id: str, +) -> CalendarOutboxOperation: + operation, source = _lock_operation_source( + session, + tenant_id=tenant_id, + operation_id=operation_id, + ) + if operation.status in {"succeeded", "in_progress", "superseded", "cancelled"}: + raise ValueError(f"A {operation.status} Calendar outbox operation cannot be retried") + _assert_latest_resource_generation(session, operation) + unavailable_reason = _operation_source_unavailable_reason(operation, source) + if unavailable_reason: + raise ValueError(unavailable_reason) + assert source is not None + operation.status = "pending" + operation.attempt_count = 0 + operation.available_at = utcnow() + operation.completed_at = None + operation.last_error = None + operation.lease_token = None + operation.lease_expires_at = None + _set_pointed_event_outbox_state( + session, + operation=operation, + source=source, + state="queued", + raw_ics=operation.payload_ics, + ) + session.flush() + _schedule_dispatch_after_commit(session, tenant_id) + return operation + + +def reconcile_calendar_outbox_operation( + session: Session, + *, + tenant_id: str, + operation_id: str, + client_factory: Callable[[Session, CalendarSyncSource], object] | None = None, +) -> CalendarOutboxOperation: + operation, source = _lock_operation_source( + session, + tenant_id=tenant_id, + operation_id=operation_id, + ) + if operation.status in {"succeeded", "superseded", "cancelled"}: + raise ValueError(f"A {operation.status} Calendar outbox operation cannot be reconciled") + if ( + operation.status == "in_progress" + and operation.lease_expires_at is not None + and _as_utc(operation.lease_expires_at) > utcnow() + ): + raise ValueError("An actively leased Calendar outbox operation cannot be reconciled") + _assert_latest_resource_generation(session, operation) + unavailable_reason = _operation_source_unavailable_reason(operation, source) + if unavailable_reason: + raise ValueError(unavailable_reason) + assert source is not None + if client_factory is None: + from govoplan_calendar.backend.service import caldav_client_for_source + + client = caldav_client_for_source(session, source) + else: + client = client_factory(session, source) + matched, remote_etag = _remote_matches_operation(client, operation) + if matched and (operation.operation_kind == "delete" or remote_etag): + _complete_operation( + session, + operation=operation, + source=source, + remote_etag=remote_etag, + reconciled=True, + ) + elif matched: + _fail_operation( + session, + operation=operation, + source=source, + error="Matching CalDAV resource has no usable ETag; retrying reconciliation", + ) + else: + operation.reconciled_at = utcnow() + _fail_operation( + session, + operation=operation, + source=source, + error="Remote CalDAV resource does not match the queued desired state", + terminal_status="conflict", + ) + return operation + + +def discard_calendar_outbox_operation( + session: Session, + *, + tenant_id: str, + operation_id: str, +) -> CalendarOutboxOperation: + """Abandon local desired state so a later inbound sync may accept remote.""" + + operation, source = _lock_operation_source( + session, + tenant_id=tenant_id, + operation_id=operation_id, + ) + if operation.status in {"succeeded", "superseded", "cancelled"}: + raise ValueError(f"A {operation.status} Calendar outbox operation cannot be discarded") + if ( + operation.status == "in_progress" + and operation.lease_expires_at is not None + and _as_utc(operation.lease_expires_at) > utcnow() + ): + raise ValueError("An actively leased Calendar outbox operation cannot be discarded") + _assert_latest_resource_generation(session, operation) + unavailable_reason = _operation_source_unavailable_reason(operation, source) + if unavailable_reason and ( + source is None or source.tenant_id != operation.tenant_id + ): + raise ValueError(unavailable_reason) + assert source is not None + now = utcnow() + unresolved_chain = ( + session.query(CalendarOutboxOperation) + .filter( + CalendarOutboxOperation.source_id == operation.source_id, + CalendarOutboxOperation.resource_href == operation.resource_href, + CalendarOutboxOperation.status.in_(sorted(OUTBOX_UNRESOLVED_DESIRED_STATUSES)), + or_( + CalendarOutboxOperation.created_at < operation.created_at, + and_( + CalendarOutboxOperation.created_at == operation.created_at, + CalendarOutboxOperation.id <= operation.id, + ), + ), + ) + .order_by( + CalendarOutboxOperation.created_at.asc(), + CalendarOutboxOperation.id.asc(), + ) + .with_for_update() + .all() + ) + if any( + item.status == "in_progress" + and item.lease_expires_at is not None + and _as_utc(item.lease_expires_at) > now + for item in unresolved_chain + ): + raise ValueError( + "The Calendar resource has an actively leased predecessor and cannot be discarded" + ) + message = ( + "Local desired state was discarded by an administrator; accept remote on next sync" + ) + for item in unresolved_chain: + item.status = "cancelled" + item.completed_at = now + item.reconciled_at = now + item.last_error = message + item.lease_token = None + item.lease_expires_at = None + _set_pointed_event_outbox_state( + session, + operation=item, + source=source, + state="discarded", + raw_ics=item.payload_ics, + ) + source.last_status = "outbound_discarded" + source.last_error = message + # Incremental sync might not report an unchanged remote object. Force the + # next source run to perform a full collection reconciliation. + source.sync_token = None + source.next_sync_at = now + session.flush() + return operation + + +def cancel_calendar_outbox_for_source( + session: Session, + *, + source_id: str, + reason: str, +) -> int: + now = utcnow() + operations = ( + session.query(CalendarOutboxOperation) + .filter( + CalendarOutboxOperation.source_id == source_id, + CalendarOutboxOperation.status.in_(sorted(OUTBOX_UNRESOLVED_DESIRED_STATUSES)), + ) + .all() + ) + source = session.get(CalendarSyncSource, source_id) + for operation in operations: + operation.status = "cancelled" + operation.completed_at = now + operation.last_error = reason + operation.lease_token = None + operation.lease_expires_at = None + if source is not None and source.tenant_id == operation.tenant_id: + _set_pointed_event_outbox_state( + session, + operation=operation, + source=source, + state="cancelled", + raw_ics=operation.payload_ics, + ) + return len(operations) + + +def calendar_outbox_has_live_lease( + session: Session, + *, + source_id: str, + now: datetime | None = None, +) -> bool: + now = now or utcnow() + return ( + session.query(CalendarOutboxOperation.id) + .filter( + CalendarOutboxOperation.source_id == source_id, + CalendarOutboxOperation.status == "in_progress", + CalendarOutboxOperation.lease_expires_at > now, + ) + .first() + is not None + ) + + +def calendar_outbox_has_active_desired_state( + session: Session, + *, + source_id: str, + href: str, +) -> bool: + latest_status = ( + session.query(CalendarOutboxOperation.status) + .filter( + CalendarOutboxOperation.source_id == source_id, + CalendarOutboxOperation.resource_href == href, + ) + .order_by( + CalendarOutboxOperation.created_at.desc(), + CalendarOutboxOperation.id.desc(), + ) + .first() + ) + return bool( + latest_status is not None + and latest_status[0] in OUTBOX_UNRESOLVED_DESIRED_STATUSES + ) + + +def calendar_outbox_has_unresolved_desired_state( + session: Session, + *, + source_id: str, +) -> bool: + rows = ( + session.query( + CalendarOutboxOperation.resource_href, + CalendarOutboxOperation.status, + ) + .filter( + CalendarOutboxOperation.source_id == source_id, + ) + .order_by( + CalendarOutboxOperation.resource_href.asc(), + CalendarOutboxOperation.created_at.desc(), + CalendarOutboxOperation.id.desc(), + ) + .all() + ) + seen_hrefs: set[str] = set() + for href, status in rows: + if href in seen_hrefs: + continue + seen_hrefs.add(href) + if status in OUTBOX_UNRESOLVED_DESIRED_STATUSES: + return True + return False + + +def calendar_outbox_operation_response(operation: CalendarOutboxOperation) -> dict[str, Any]: + return { + "id": operation.id, + "tenant_id": operation.tenant_id, + "source_id": operation.source_id, + "event_id": operation.event_id, + "operation_kind": operation.operation_kind, + "resource_href": operation.resource_href, + "payload_fingerprint": operation.payload_fingerprint, + "expected_etag": operation.expected_etag, + "idempotency_key": operation.idempotency_key, + "status": operation.status, + "attempt_count": operation.attempt_count, + "max_attempts": operation.max_attempts, + "available_at": operation.available_at, + "last_attempt_at": operation.last_attempt_at, + "lease_expires_at": operation.lease_expires_at, + "completed_at": operation.completed_at, + "reconciled_at": operation.reconciled_at, + "remote_etag": operation.remote_etag, + "last_error": operation.last_error, + "created_at": operation.created_at, + "updated_at": operation.updated_at, + "metadata": dict(operation.metadata_ or {}), + } + + +class SqlCalendarOutboxProvider: + def __init__( + self, + *, + terminal_retention_days: int = OUTBOX_DEFAULT_TERMINAL_RETENTION_DAYS, + ) -> None: + if isinstance(terminal_retention_days, bool): + raise ValueError("Calendar outbox terminal retention days must be an integer") + try: + normalized_retention_days = int(terminal_retention_days) + except (TypeError, ValueError) as exc: + raise ValueError("Calendar outbox terminal retention days must be an integer") from exc + if normalized_retention_days < 0: + raise ValueError("Calendar outbox terminal retention days must be zero or greater") + self.terminal_retention_days = normalized_retention_days + + def dispatch_due( + self, + session: object, + *, + tenant_id: str | None = None, + limit: int = 50, + ) -> Mapping[str, object]: + if not isinstance(session, Session): + raise TypeError("Calendar outbox requires a SQLAlchemy Session") + result = dispatch_calendar_outbox(session, tenant_id=tenant_id, limit=limit) + result["purged"] = purge_terminal_calendar_outbox_operations( + session, + retention_days=self.terminal_retention_days, + tenant_id=tenant_id, + ) + return result + + +__all__ = [ + "OUTBOX_ACTIVE_STATUSES", + "OUTBOX_DEFAULT_MAX_ATTEMPTS", + "OUTBOX_DEFAULT_TERMINAL_RETENTION_DAYS", + "OUTBOX_PURGEABLE_STATUSES", + "OUTBOX_TERMINAL_STATUSES", + "OUTBOX_UNRESOLVED_DESIRED_STATUSES", + "SqlCalendarOutboxProvider", + "calendar_outbox_has_active_desired_state", + "calendar_outbox_has_live_lease", + "calendar_outbox_has_unresolved_desired_state", + "calendar_outbox_operation_response", + "calendar_payload_fingerprint", + "cancel_calendar_outbox_for_source", + "claim_next_calendar_outbox_operation", + "dispatch_calendar_outbox", + "discard_calendar_outbox_operation", + "enqueue_caldav_delete", + "enqueue_caldav_put", + "get_calendar_outbox_operation", + "list_calendar_outbox_operations", + "purge_terminal_calendar_outbox_operations", + "reconcile_calendar_outbox_operation", + "retry_calendar_outbox_operation", +] diff --git a/src/govoplan_calendar/backend/router.py b/src/govoplan_calendar/backend/router.py index e126ac1..330cb5a 100644 --- a/src/govoplan_calendar/backend/router.py +++ b/src/govoplan_calendar/backend/router.py @@ -7,6 +7,7 @@ from sqlalchemy.orm import Session from govoplan_core.auth import ApiPrincipal, get_api_principal, has_scope from govoplan_core.api.v1.schemas import DeltaDeletedItem +from govoplan_core.core.calendar import CALENDAR_AVAILABILITY_READ_SCOPE, CALENDAR_EVENT_WRITE_SCOPE from govoplan_core.core.change_sequence import decode_sequence_watermark, encode_sequence_watermark, max_sequence_id, sequence_entries_since, sequence_watermark_is_expired from govoplan_calendar.backend.db.models import CalendarEvent from govoplan_calendar.backend.ical import ICalendarError, event_to_ics, http_last_modified @@ -35,6 +36,9 @@ from govoplan_calendar.backend.schemas import ( CalendarFreeBusyRequest, CalendarFreeBusyResponse, CalendarIcsImportRequest, + CalendarOutboxDispatchResponse, + CalendarOutboxOperationListResponse, + CalendarOutboxOperationResponse, CalendarSyncDueSyncItemResponse, CalendarSyncDueSyncResponse, CalendarSyncSourceCreateRequest, @@ -44,6 +48,14 @@ from govoplan_calendar.backend.schemas import ( CalendarSyncSourceSyncResponse, CalendarSyncSourceUpdateRequest, ) +from govoplan_calendar.backend.outbox import ( + calendar_outbox_operation_response, + discard_calendar_outbox_operation, + dispatch_calendar_outbox, + list_calendar_outbox_operations, + reconcile_calendar_outbox_operation, + retry_calendar_outbox_operation, +) from govoplan_calendar.backend.service import ( CALENDAR_EVENTS_COLLECTION, CALENDAR_EVENT_RESOURCE, @@ -62,7 +74,6 @@ from govoplan_calendar.backend.service import ( delete_event, discover_caldav_calendars, event_response, - get_caldav_source, get_event, import_ics_event, list_freebusy, @@ -220,13 +231,25 @@ def _sync_source_response(source) -> CalendarSyncSourceResponse: return CalendarSyncSourceResponse.model_validate(caldav_source_response(source)) +def _outbox_operation_response(operation) -> CalendarOutboxOperationResponse: + return CalendarOutboxOperationResponse.model_validate( + calendar_outbox_operation_response(operation) + ) + + @router.get("/calendars", response_model=CalendarCollectionListResponse) def api_list_calendars( principal: ApiPrincipal = Depends(get_api_principal), session: Session = Depends(get_session), ): _require_scope(principal, "calendar:calendar:read") - calendars = list_calendars(session, tenant_id=principal.tenant_id, user_id=principal.user.id) + calendars = list_calendars( + session, + tenant_id=principal.tenant_id, + user_id=principal.user.id, + group_ids=principal.group_ids, + can_admin=principal.has("calendar:calendar:admin"), + ) session.commit() return CalendarCollectionListResponse(calendars=[_calendar_response(calendar) for calendar in calendars]) @@ -343,7 +366,12 @@ def api_delete_sync_source( return Response(status_code=status.HTTP_204_NO_CONTENT) except CalendarError as exc: session.rollback() - raise HTTPException(status_code=status.HTTP_404_NOT_FOUND, detail=str(exc)) from exc + error_status = ( + status.HTTP_404_NOT_FOUND + if "not found" in str(exc).lower() + else status.HTTP_409_CONFLICT + ) + raise HTTPException(status_code=error_status, detail=str(exc)) from exc @router.post("/sync-sources/{source_id}/sync", response_model=CalendarSyncSourceSyncResponse) @@ -473,7 +501,12 @@ def api_delete_caldav_source( return Response(status_code=status.HTTP_204_NO_CONTENT) except CalendarError as exc: session.rollback() - raise HTTPException(status_code=status.HTTP_404_NOT_FOUND, detail=str(exc)) from exc + error_status = ( + status.HTTP_404_NOT_FOUND + if "not found" in str(exc).lower() + else status.HTTP_409_CONFLICT + ) + raise HTTPException(status_code=error_status, detail=str(exc)) from exc @router.post("/caldav/sources/{source_id}/sync", response_model=CalendarCalDavSyncResponse) @@ -530,6 +563,117 @@ def api_sync_due_caldav_sources( ) +@router.get("/caldav/outbox", response_model=CalendarOutboxOperationListResponse) +def api_list_caldav_outbox( + operation_status: str | None = Query(default=None, alias="status"), + source_id: str | None = None, + limit: int = Query(default=100, ge=1, le=500), + principal: ApiPrincipal = Depends(get_api_principal), + session: Session = Depends(get_session), +): + _require_scope(principal, "calendar:calendar:admin") + operations = list_calendar_outbox_operations( + session, + tenant_id=principal.tenant_id, + status=operation_status, + source_id=source_id, + limit=limit, + ) + return CalendarOutboxOperationListResponse( + operations=[_outbox_operation_response(operation) for operation in operations] + ) + + +@router.post("/caldav/outbox/dispatch", response_model=CalendarOutboxDispatchResponse) +def api_dispatch_caldav_outbox( + limit: int = Query(default=50, ge=1, le=200), + principal: ApiPrincipal = Depends(get_api_principal), + session: Session = Depends(get_session), +): + _require_scope(principal, "calendar:calendar:admin") + return CalendarOutboxDispatchResponse.model_validate( + dispatch_calendar_outbox( + session, + tenant_id=principal.tenant_id, + limit=limit, + ) + ) + + +@router.post( + "/caldav/outbox/{operation_id}/retry", + response_model=CalendarOutboxOperationResponse, +) +def api_retry_caldav_outbox_operation( + operation_id: str, + principal: ApiPrincipal = Depends(get_api_principal), + session: Session = Depends(get_session), +): + _require_scope(principal, "calendar:calendar:admin") + try: + operation = retry_calendar_outbox_operation( + session, + tenant_id=principal.tenant_id, + operation_id=operation_id, + ) + session.commit() + session.refresh(operation) + return _outbox_operation_response(operation) + except ValueError as exc: + session.rollback() + raise HTTPException(status_code=status.HTTP_409_CONFLICT, detail=str(exc)) from exc + + +@router.post( + "/caldav/outbox/{operation_id}/reconcile", + response_model=CalendarOutboxOperationResponse, +) +def api_reconcile_caldav_outbox_operation( + operation_id: str, + principal: ApiPrincipal = Depends(get_api_principal), + session: Session = Depends(get_session), +): + _require_scope(principal, "calendar:calendar:admin") + try: + operation = reconcile_calendar_outbox_operation( + session, + tenant_id=principal.tenant_id, + operation_id=operation_id, + ) + session.commit() + session.refresh(operation) + return _outbox_operation_response(operation) + except (ValueError, CalDAVError, CalendarError) as exc: + session.rollback() + raise HTTPException(status_code=status.HTTP_422_UNPROCESSABLE_CONTENT, detail=str(exc)) from exc + + +@router.post( + "/caldav/outbox/{operation_id}/discard", + response_model=CalendarOutboxOperationResponse, +) +def api_discard_caldav_outbox_operation( + operation_id: str, + principal: ApiPrincipal = Depends(get_api_principal), + session: Session = Depends(get_session), +): + """Abandon local desired state and allow the next sync to accept remote.""" + + _require_scope(principal, "calendar:calendar:admin") + try: + operation = discard_calendar_outbox_operation( + session, + tenant_id=principal.tenant_id, + operation_id=operation_id, + ) + session.commit() + session.refresh(operation) + return _outbox_operation_response(operation) + except ValueError as exc: + session.rollback() + raise HTTPException(status_code=status.HTTP_409_CONFLICT, detail=str(exc)) from exc + + @router.get("/events", response_model=CalendarEventListResponse) def api_list_events( calendar_id: str | None = None, @@ -600,7 +744,7 @@ def api_create_event( principal: ApiPrincipal = Depends(get_api_principal), session: Session = Depends(get_session), ): - _require_scope(principal, "calendar:event:write") + _require_scope(principal, CALENDAR_EVENT_WRITE_SCOPE) try: event = create_event(session, tenant_id=principal.tenant_id, user_id=principal.user.id, payload=payload) session.commit() @@ -631,7 +775,7 @@ def api_update_event( principal: ApiPrincipal = Depends(get_api_principal), session: Session = Depends(get_session), ): - _require_scope(principal, "calendar:event:write") + _require_scope(principal, CALENDAR_EVENT_WRITE_SCOPE) try: event = update_event(session, tenant_id=principal.tenant_id, user_id=principal.user.id, event_id=event_id, payload=payload) session.commit() @@ -648,7 +792,7 @@ def api_delete_event( principal: ApiPrincipal = Depends(get_api_principal), session: Session = Depends(get_session), ): - _require_any_scope(principal, "calendar:event:delete", "calendar:event:write") + _require_any_scope(principal, "calendar:event:delete", CALENDAR_EVENT_WRITE_SCOPE) try: delete_event(session, tenant_id=principal.tenant_id, event_id=event_id, user_id=principal.user.id) session.commit() @@ -713,7 +857,7 @@ def api_freebusy( principal: ApiPrincipal = Depends(get_api_principal), session: Session = Depends(get_session), ): - _require_scope(principal, "calendar:availability:read") + _require_scope(principal, CALENDAR_AVAILABILITY_READ_SCOPE) try: busy = list_freebusy( session, diff --git a/src/govoplan_calendar/backend/schemas.py b/src/govoplan_calendar/backend/schemas.py index 057f85b..9516846 100644 --- a/src/govoplan_calendar/backend/schemas.py +++ b/src/govoplan_calendar/backend/schemas.py @@ -11,6 +11,11 @@ from govoplan_core.api.v1.schemas import DeltaDeletedItem CalendarOwnerType = Literal["tenant", "user", "group", "resource"] CalendarVisibility = Literal["private", "tenant", "shared", "public"] CalendarDeleteEventAction = Literal["delete", "move"] +CalendarBulkMoveExternalAction = Literal[ + "detach_keep_remote", + "copy_to_remote", + "remote_move", +] CalendarSyncAuthType = Literal["none", "basic", "bearer"] CalendarSyncDirection = Literal["inbound", "two_way"] CalendarSyncSourceKind = Literal["caldav", "ics", "webcal", "graph", "ews"] @@ -50,6 +55,7 @@ class CalendarCollectionDeleteRequest(BaseModel): event_action: CalendarDeleteEventAction = "delete" target_calendar_id: str | None = None make_target_default: bool = False + external_action: CalendarBulkMoveExternalAction | None = None class CalendarCollectionResponse(BaseModel): @@ -238,6 +244,50 @@ class CalendarCalDavDueSyncResponse(BaseModel): results: list[CalendarCalDavDueSyncItemResponse] = Field(default_factory=list) +class CalendarOutboxOperationResponse(BaseModel): + id: str + tenant_id: str + source_id: str + event_id: str | None = None + operation_kind: str + resource_href: str + payload_fingerprint: str | None = None + expected_etag: str | None = None + idempotency_key: str + status: str + attempt_count: int + max_attempts: int + available_at: datetime + last_attempt_at: datetime | None = None + lease_expires_at: datetime | None = None + completed_at: datetime | None = None + reconciled_at: datetime | None = None + remote_etag: str | None = None + last_error: str | None = None + created_at: datetime + updated_at: datetime + metadata: dict[str, Any] = Field(default_factory=dict) + + +class CalendarOutboxOperationListResponse(BaseModel): + operations: list[CalendarOutboxOperationResponse] = Field(default_factory=list) + + +class CalendarOutboxDispatchItemResponse(BaseModel): + id: str + status: str + attempt_count: int + last_error: str | None = None + + +class CalendarOutboxDispatchResponse(BaseModel): + processed: int = 0 + succeeded: int = 0 + retrying: int = 0 + failed: int = 0 + operations: list[CalendarOutboxDispatchItemResponse] = Field(default_factory=list) + + class CalendarEventCreateRequest(BaseModel): model_config = ConfigDict(extra="forbid") diff --git a/src/govoplan_calendar/backend/service.py b/src/govoplan_calendar/backend/service.py index 200e7cc..ae30d76 100644 --- a/src/govoplan_calendar/backend/service.py +++ b/src/govoplan_calendar/backend/service.py @@ -1,6 +1,7 @@ from __future__ import annotations import json +import posixpath import re import uuid import os @@ -9,16 +10,16 @@ import urllib.error import urllib.request from dataclasses import dataclass, field from datetime import datetime, timedelta, timezone -from typing import Any, Callable +from typing import Any, Callable, Iterable from zoneinfo import ZoneInfo, ZoneInfoNotFoundError from defusedxml import ElementTree as SafeElementTree from sqlalchemy import or_ from sqlalchemy.orm import Session -from govoplan_calendar.backend.caldav import CalDAVClient, CalDAVError, CalDAVNotFound, CalDAVPreconditionFailed, CalDAVReportResult, CalDAVSyncUnsupported, ensure_collection_url +from govoplan_calendar.backend.caldav import CalDAVClient, CalDAVError, CalDAVNotFound, CalDAVReportResult, CalDAVSyncUnsupported, ensure_collection_url from govoplan_calendar.backend.db.models import CalendarCollection, CalendarEvent, CalendarSyncCredential, CalendarSyncSource -from govoplan_calendar.backend.ical import events_to_ics, expand_event_occurrences, parse_vevent, parse_vevents +from govoplan_calendar.backend.ical import expand_event_occurrences, parse_vevent, parse_vevents from govoplan_calendar.backend.runtime import get_registry from govoplan_calendar.backend.schemas import ( CalendarCalDavDiscoveryRequest, @@ -58,12 +59,20 @@ SOURCE_EVENT_CLEANUP_BATCH_SIZE = 500 def calendar_event_change_payload(event: CalendarEvent, *, prefix: str = "") -> dict[str, Any]: + metadata = event.metadata_ or {} + caldav = metadata.get("caldav") if isinstance(metadata, dict) else None return { f"{prefix}calendar_id": event.calendar_id, f"{prefix}start_at": response_datetime(event.start_at).isoformat() if event.start_at else None, f"{prefix}end_at": response_datetime(event.end_at).isoformat() if event.end_at else None, f"{prefix}summary": event.summary, f"{prefix}status": event.status, + f"{prefix}external_state": ( + caldav.get("external_state") if isinstance(caldav, dict) else "local" + ), + f"{prefix}outbox_operation_id": ( + caldav.get("outbox_operation_id") if isinstance(caldav, dict) else None + ), } @@ -324,15 +333,63 @@ def get_default_calendar(session: Session, *, tenant_id: str) -> CalendarCollect ) -def list_calendars(session: Session, *, tenant_id: str, ensure_default: bool = False, user_id: str | None = None) -> list[CalendarCollection]: +def calendar_is_visible_to_principal( + calendar: CalendarCollection, + *, + user_id: str | None, + group_ids: Iterable[str] = (), + can_admin: bool = False, +) -> bool: + """Return whether a principal may discover a calendar collection. + + Collection visibility is deliberately enforced before serializing picker + metadata. ``shared`` currently means tenant-visible; explicit per-user + sharing can narrow this rule when Calendar gains a share relation. + """ + + if can_admin: + return True + if calendar.visibility in {"tenant", "shared", "public"}: + return True + if calendar.visibility != "private" or user_id is None: + return False + if calendar.created_by_user_id == user_id: + return True + if calendar.owner_type == "user": + return calendar.owner_id == user_id + if calendar.owner_type == "group": + return bool(calendar.owner_id and calendar.owner_id in set(group_ids)) + return False + + +def list_calendars( + session: Session, + *, + tenant_id: str, + ensure_default: bool = False, + user_id: str | None = None, + group_ids: Iterable[str] = (), + can_admin: bool = False, +) -> list[CalendarCollection]: if ensure_default: ensure_default_calendar(session, tenant_id=tenant_id, user_id=user_id) - return ( + calendars = ( session.query(CalendarCollection) .filter(CalendarCollection.tenant_id == tenant_id, CalendarCollection.deleted_at.is_(None)) .order_by(CalendarCollection.is_default.desc(), CalendarCollection.name.asc()) .all() ) + principal_group_ids = tuple(group_ids) + return [ + calendar + for calendar in calendars + if calendar_is_visible_to_principal( + calendar, + user_id=user_id, + group_ids=principal_group_ids, + can_admin=can_admin, + ) + ] def create_calendar(session: Session, *, tenant_id: str, user_id: str | None, payload: CalendarCollectionCreateRequest) -> CalendarCollection: @@ -387,14 +444,129 @@ def delete_calendar(session: Session, *, tenant_id: str, calendar_id: str, paylo if payload.target_calendar_id == calendar_id: raise CalendarError("Target calendar must be different from the deleted calendar") target_calendar = get_calendar(session, tenant_id=tenant_id, calendar_id=payload.target_calendar_id) - for event in calendar.events: - if event.deleted_at is None: - event.calendar_id = target_calendar.id + source = None + target_source = None + if hasattr(session, "query"): + # Source creation takes the collection row lock before linking a + # source. Lock both collections in stable order so the mode cannot + # change between validation and the local/outbox mutation. + locked_calendars = ( + session.query(CalendarCollection) + .filter( + CalendarCollection.tenant_id == tenant_id, + CalendarCollection.id.in_((calendar.id, target_calendar.id)), + CalendarCollection.deleted_at.is_(None), + ) + .order_by(CalendarCollection.id.asc()) + .populate_existing() + .with_for_update() + .all() + ) + locked_calendar_by_id = {item.id: item for item in locked_calendars} + if len(locked_calendar_by_id) != 2: + raise CalendarError("Source or target calendar is no longer available") + calendar = locked_calendar_by_id[calendar.id] + target_calendar = locked_calendar_by_id[target_calendar.id] + source = active_sync_source_for_calendar( + session, + tenant_id=tenant_id, + calendar_id=calendar.id, + ) + target_source = active_sync_source_for_calendar( + session, + tenant_id=tenant_id, + calendar_id=target_calendar.id, + ) + locked_sources = lock_sync_sources(session, source, target_source) + if source is not None: + source = locked_sources[source.id] + if target_source is not None: + target_source = locked_sources[target_source.id] + + if payload.external_action == "remote_move": + raise CalendarError( + "external_action='remote_move' is not supported; no destructive remote move was performed" + ) + if source is not None and target_source is not None: + raise CalendarError( + "Events cannot be bulk-moved between synchronized calendars; " + "external_action='remote_move' remains unsupported" + ) + active_events = [event for event in calendar.events if event.deleted_at is None] + previous_event_states = ( + { + event.id: calendar_event_change_payload(event, prefix="previous_") + for event in active_events + } + if hasattr(session, "query") + else {} + ) + if source is not None: + if payload.external_action != "detach_keep_remote": + raise CalendarError( + "Moving events from a synchronized calendar to a local calendar requires " + "external_action='detach_keep_remote'" + ) + # Retirement rejects a live delivery lease before any local event + # projection is changed, and cancels all remaining undelivered + # desired state. It never creates a remote DELETE operation. + retire_sync_source( + session, + tenant_id=tenant_id, + source=source, + deleted_at=deleted_at, + ) + elif target_source is not None: + if payload.external_action != "copy_to_remote": + raise CalendarError( + "Moving local events to a synchronized calendar requires " + "external_action='copy_to_remote'" + ) + if ( + target_source.source_kind != "caldav" + or target_source.sync_direction != "two_way" + or not target_source.sync_enabled + ): + raise CalendarError( + "external_action='copy_to_remote' requires an active two-way CalDAV target" + ) + assert_sync_mutation_allowed(target_source) + elif payload.external_action is not None: + raise CalendarError( + "external_action is only valid when moving events to or from a synchronized calendar" + ) + + for event in active_events: + if source is not None or target_source is not None: + event.source_kind = "local" + event.source_href = None + event.etag = None + metadata = dict(event.metadata_ or {}) + metadata.pop("caldav", None) + event.metadata_ = metadata + event.calendar_id = target_calendar.id + session.flush() + if target_source is not None: + from govoplan_calendar.backend.outbox import enqueue_caldav_put + + for event in active_events: + enqueue_caldav_put(session, source=target_source, event_model=event) + if previous_event_states: + for event in active_events: + record_calendar_event_change( + session, + event=event, + operation="updated", + user_id=None, + previous=previous_event_states[event.id], + ) if payload.make_target_default: clear_default_calendar(session, tenant_id=tenant_id) target_calendar.is_default = True elif payload.event_action != "delete": raise CalendarError(f"Unsupported calendar delete event action: {payload.event_action}") + elif payload.external_action is not None: + raise CalendarError("external_action is only valid when event_action='move'") if calendar.is_default: calendar.is_default = False calendar.deleted_at = deleted_at @@ -515,6 +687,30 @@ def discover_caldav_calendars(session: Session, *, tenant_id: str, payload: Cale def create_sync_source(session: Session, *, tenant_id: str, user_id: str | None, payload: CalendarSyncSourceCreateRequest) -> CalendarSyncSource: calendar = get_calendar(session, tenant_id=tenant_id, calendar_id=payload.calendar_id) + calendar = ( + session.query(CalendarCollection) + .filter( + CalendarCollection.id == calendar.id, + CalendarCollection.tenant_id == tenant_id, + CalendarCollection.deleted_at.is_(None), + ) + .populate_existing() + .with_for_update() + .one() + ) + existing_calendar_source = ( + session.query(CalendarSyncSource.id) + .filter( + CalendarSyncSource.tenant_id == tenant_id, + CalendarSyncSource.calendar_id == calendar.id, + CalendarSyncSource.deleted_at.is_(None), + ) + .first() + ) + if existing_calendar_source is not None: + raise CalendarError( + "A calendar can have only one active synchronization source" + ) source_kind = normalize_source_kind(payload.source_kind) collection_url = normalize_sync_source_url(source_kind, payload.collection_url) retire_stale_sync_sources_for_url(session, tenant_id=tenant_id, source_kind=source_kind, collection_url=collection_url) @@ -562,8 +758,123 @@ def create_caldav_source(session: Session, *, tenant_id: str, user_id: str | Non def update_sync_source(session: Session, *, tenant_id: str, source_id: str, payload: CalendarSyncSourceUpdateRequest) -> CalendarSyncSource: source = get_sync_source(session, tenant_id=tenant_id, source_id=source_id) + source = ( + session.query(CalendarSyncSource) + .filter( + CalendarSyncSource.id == source.id, + CalendarSyncSource.tenant_id == tenant_id, + CalendarSyncSource.deleted_at.is_(None), + ) + .populate_existing() + .with_for_update() + .one() + ) + normalized_collection_url = ( + normalize_sync_source_url(source.source_kind, payload.collection_url) + if payload.collection_url is not None + else source.collection_url + ) + materially_reconfigured = bool( + source.source_kind == "caldav" + and ( + (payload.calendar_id is not None and payload.calendar_id != source.calendar_id) + or normalized_collection_url != source.collection_url + or payload.sync_enabled is False + or payload.sync_direction == "inbound" + ) + ) + endpoint_or_calendar_changed = bool( + source.source_kind == "caldav" + and ( + (payload.calendar_id is not None and payload.calendar_id != source.calendar_id) + or normalized_collection_url != source.collection_url + ) + ) + if materially_reconfigured: + from govoplan_calendar.backend.outbox import ( + calendar_outbox_has_live_lease, + calendar_outbox_has_unresolved_desired_state, + cancel_calendar_outbox_for_source, + ) + + if calendar_outbox_has_live_lease(session, source_id=source.id): + raise CalendarError( + "CalDAV source cannot be materially changed while an outbound operation has an active lease" + ) + + unresolved_desired_state = calendar_outbox_has_unresolved_desired_state( + session, + source_id=source.id, + ) + stops_outbound_delivery = bool( + (payload.sync_enabled is False and source.sync_enabled) + or ( + payload.sync_direction == "inbound" + and source.sync_direction == "two_way" + ) + ) + if stops_outbound_delivery and unresolved_desired_state: + raise CalendarError( + "CalDAV source cannot disable outbound delivery while unresolved local desired " + "changes exist; deliver or explicitly discard them first" + ) + if endpoint_or_calendar_changed and unresolved_desired_state: + raise CalendarError( + "CalDAV source endpoint or calendar cannot change while unresolved local desired " + "changes exist; deliver or explicitly discard them first" + ) + + if endpoint_or_calendar_changed: + sourced_event_exists = ( + session.query(CalendarEvent.id) + .filter( + CalendarEvent.tenant_id == tenant_id, + CalendarEvent.calendar_id == source.calendar_id, + CalendarEvent.source_kind == "caldav", + CalendarEvent.source_href.is_not(None), + CalendarEvent.deleted_at.is_(None), + ) + .first() + is not None + ) + if sourced_event_exists: + raise CalendarError( + "CalDAV source endpoint or calendar cannot change while events still reference it; " + "detach or resync those events first" + ) + + cancel_calendar_outbox_for_source( + session, + source_id=source.id, + reason="CalDAV source endpoint/calendar changed, was disabled, or was made inbound-only", + ) if payload.calendar_id is not None: calendar = get_calendar(session, tenant_id=tenant_id, calendar_id=payload.calendar_id) + calendar = ( + session.query(CalendarCollection) + .filter( + CalendarCollection.id == calendar.id, + CalendarCollection.tenant_id == tenant_id, + CalendarCollection.deleted_at.is_(None), + ) + .populate_existing() + .with_for_update() + .one() + ) + conflicting_source = ( + session.query(CalendarSyncSource.id) + .filter( + CalendarSyncSource.tenant_id == tenant_id, + CalendarSyncSource.calendar_id == calendar.id, + CalendarSyncSource.id != source.id, + CalendarSyncSource.deleted_at.is_(None), + ) + .first() + ) + if conflicting_source is not None: + raise CalendarError( + "A calendar can have only one active synchronization source" + ) source.calendar_id = calendar.id else: calendar = get_calendar(session, tenant_id=tenant_id, calendar_id=source.calendar_id) @@ -583,7 +894,7 @@ def update_sync_source(session: Session, *, tenant_id: str, source_id: str, payl if value is not None: setattr(source, field_name, value) if payload.collection_url is not None: - source.collection_url = normalize_sync_source_url(source.source_kind, payload.collection_url) + source.collection_url = normalized_collection_url if payload.metadata is not None: source.metadata_ = payload.metadata if source.source_kind in READ_ONLY_SYNC_SOURCE_KINDS: @@ -684,6 +995,31 @@ def active_caldav_source_for_url(session: Session, *, tenant_id: str, collection def retire_sync_source(session: Session, *, tenant_id: str, source: CalendarSyncSource, deleted_at: datetime) -> None: + from govoplan_calendar.backend.outbox import ( + calendar_outbox_has_live_lease, + cancel_calendar_outbox_for_source, + ) + + source = ( + session.query(CalendarSyncSource) + .filter( + CalendarSyncSource.id == source.id, + CalendarSyncSource.tenant_id == tenant_id, + ) + .populate_existing() + .with_for_update() + .one() + ) + if calendar_outbox_has_live_lease(session, source_id=source.id): + raise CalendarError( + "CalDAV source cannot be retired while an outbound operation has an active lease" + ) + + cancel_calendar_outbox_for_source( + session, + source_id=source.id, + reason="CalDAV source was retired before queued external changes were delivered", + ) source.deleted_at = deleted_at delete_caldav_credential(session, tenant_id=tenant_id, credential_ref=source.credential_ref) @@ -933,6 +1269,22 @@ def sync_caldav_source( force_full: bool = False, ) -> tuple[CalendarSyncSource, CalendarCalDavSyncStats]: source = get_caldav_source(session, tenant_id=tenant_id, source_id=source_id) + # Serialize inbound application with local desired-state snapshots and + # outbound delivery for this source. The lock intentionally spans the + # REPORT and its application so an older report cannot land after a newer + # outbound write. + source = ( + session.query(CalendarSyncSource) + .filter( + CalendarSyncSource.id == source.id, + CalendarSyncSource.tenant_id == tenant_id, + CalendarSyncSource.source_kind == "caldav", + CalendarSyncSource.deleted_at.is_(None), + ) + .populate_existing() + .with_for_update() + .one() + ) get_calendar(session, tenant_id=tenant_id, calendar_id=source.calendar_id) client = client or caldav_client_for_source(session, source, password=password, bearer_token=bearer_token) stats = CalendarCalDavSyncStats() @@ -1452,6 +1804,15 @@ def soft_delete_unseen_source_events( .all() ) for event in events: + if effective_source_kind == "caldav" and event.source_href: + from govoplan_calendar.backend.outbox import calendar_outbox_has_active_desired_state + + if calendar_outbox_has_active_desired_state( + session, + source_id=source.id, + href=event.source_href, + ): + continue if event.source_href not in seen_hrefs: previous = calendar_event_change_payload(event, prefix="previous_") event.deleted_at = deleted_at @@ -1714,6 +2075,14 @@ def apply_caldav_report( seen_hrefs: set[str] = set() for item in report.objects: href = normalize_caldav_href(source.collection_url, item.href) + from govoplan_calendar.backend.outbox import calendar_outbox_has_active_desired_state + + if calendar_outbox_has_active_desired_state(session, source_id=source.id, href=href): + # Do not let an older remote representation erase a committed + # local desired state while its outbound operation is pending. + seen_hrefs.add(href) + stats.unchanged += 1 + continue if item.deleted: stats.deleted += soft_delete_caldav_href(session, source=source, href=href) continue @@ -1899,7 +2268,33 @@ def soft_delete_caldav_href(session: Session, *, source: CalendarSyncSource, hre def normalize_caldav_href(collection_url: str, href: str) -> str: - return urllib.parse.urljoin(ensure_collection_url(collection_url), href) + collection = ensure_collection_url(collection_url) + candidate = urllib.parse.urljoin(collection, href) + collection_parts = urllib.parse.urlparse(collection) + candidate_parts = urllib.parse.urlparse(candidate) + if ( + candidate_parts.username + or candidate_parts.password + or ( + candidate_parts.scheme.lower(), + candidate_parts.hostname, + candidate_parts.port, + ) + != ( + collection_parts.scheme.lower(), + collection_parts.hostname, + collection_parts.port, + ) + ): + raise CalendarError("CalDAV resource href must use the configured collection origin") + collection_path = posixpath.normpath(urllib.parse.unquote(collection_parts.path)) + candidate_path = posixpath.normpath(urllib.parse.unquote(candidate_parts.path)) + collection_prefix = collection_path.rstrip("/") + "/" + if not candidate_path.startswith(collection_prefix) or candidate_path == collection_path: + raise CalendarError("CalDAV resource href must remain inside the configured collection path") + if candidate_parts.query or candidate_parts.fragment: + raise CalendarError("CalDAV resource href must not contain a query or fragment") + return urllib.parse.urlunparse(candidate_parts) def mark_calendar_caldav(calendar: CalendarCollection, source: CalendarSyncSource) -> None: @@ -2044,8 +2439,25 @@ def create_event(session: Session, *, tenant_id: str, user_id: str | None, paylo if not calendar_id: raise CalendarError("calendar_id is required because no default calendar exists") get_calendar(session, tenant_id=tenant_id, calendar_id=calendar_id) - source = active_sync_source_for_calendar(session, tenant_id=tenant_id, calendar_id=calendar_id) + source = active_sync_source_for_calendar( + session, + tenant_id=tenant_id, + calendar_id=calendar_id, + for_update=True, + ) assert_sync_mutation_allowed(source) + supplied_sync_fields = bool( + payload.source_href is not None + or payload.etag is not None + or ( + "source_kind" in payload.model_fields_set + and payload.source_kind != "local" + ) + ) + if source is not None and supplied_sync_fields: + raise CalendarError( + "source_kind, source_href, and etag are sync-owned fields on synchronized calendars" + ) event = CalendarEvent( tenant_id=tenant_id, calendar_id=calendar_id, @@ -2084,24 +2496,63 @@ def create_event(session: Session, *, tenant_id: str, user_id: str | None, paylo session.add(event) session.flush() if should_push_event_to_caldav(source=source, event=event): - push_caldav_event_resource(session, source=source, event=event, create=True) + from govoplan_calendar.backend.outbox import enqueue_caldav_put + + enqueue_caldav_put(session, source=source, event_model=event) record_calendar_event_change(session, event=event, operation="created", user_id=user_id) return event def update_event(session: Session, *, tenant_id: str, user_id: str | None, event_id: str, payload: CalendarEventUpdateRequest) -> CalendarEvent: + event_locator = ( + session.query(CalendarEvent.calendar_id) + .filter( + CalendarEvent.tenant_id == tenant_id, + CalendarEvent.id == event_id, + CalendarEvent.deleted_at.is_(None), + ) + .first() + ) + if event_locator is None: + raise CalendarError("Calendar event not found") + original_calendar_id = event_locator[0] + target_calendar_id = payload.calendar_id or original_calendar_id + if payload.calendar_id is not None: + get_calendar(session, tenant_id=tenant_id, calendar_id=target_calendar_id) + original_source = active_sync_source_for_calendar( + session, + tenant_id=tenant_id, + calendar_id=original_calendar_id, + ) + target_source = ( + original_source + if target_calendar_id == original_calendar_id + else active_sync_source_for_calendar( + session, + tenant_id=tenant_id, + calendar_id=target_calendar_id, + ) + ) + locked_sources = lock_sync_sources(session, original_source, target_source) + if original_source is not None: + original_source = locked_sources[original_source.id] + if target_source is not None: + target_source = locked_sources[target_source.id] event = get_event(session, tenant_id=tenant_id, event_id=event_id) previous = calendar_event_change_payload(event, prefix="previous_") - original_calendar_id = event.calendar_id - original_source = active_sync_source_for_calendar(session, tenant_id=tenant_id, calendar_id=event.calendar_id) original_caldav = event.source_kind == "caldav" and bool(event.source_href) + supplied_sync_fields = {"source_kind", "source_href", "etag"} & payload.model_fields_set + if (original_source is not None or target_source is not None) and supplied_sync_fields: + raise CalendarError( + "source_kind, source_href, and etag are sync-owned fields on synchronized calendars" + ) if payload.calendar_id is not None: - get_calendar(session, tenant_id=tenant_id, calendar_id=payload.calendar_id) - target_source = active_sync_source_for_calendar(session, tenant_id=tenant_id, calendar_id=payload.calendar_id) assert_sync_mutation_allowed(target_source) if payload.calendar_id != event.calendar_id and original_source and original_caldav: assert_sync_mutation_allowed(original_source) - push_caldav_delete_for_event(session, source=original_source, event=event) + from govoplan_calendar.backend.outbox import enqueue_caldav_delete + + enqueue_caldav_delete(session, source=original_source, event_model=event) event.source_kind = "local" event.source_href = None event.etag = None @@ -2149,32 +2600,52 @@ def update_event(session: Session, *, tenant_id: str, user_id: str | None, event validate_event_time(event) event.raw_ics = None session.flush() - target_source = active_sync_source_for_calendar(session, tenant_id=tenant_id, calendar_id=event.calendar_id) if should_push_event_to_caldav(source=target_source, event=event): - push_caldav_event_resource( - session, - source=target_source, - event=event, - create=event.source_kind != "caldav" or not event.source_href or original_calendar_id != event.calendar_id, - ) + from govoplan_calendar.backend.outbox import enqueue_caldav_put + + enqueue_caldav_put(session, source=target_source, event_model=event) record_calendar_event_change(session, event=event, operation="updated", user_id=user_id, previous=previous) return event def delete_event(session: Session, *, tenant_id: str, event_id: str, user_id: str | None = None) -> None: + event_locator = ( + session.query(CalendarEvent.calendar_id) + .filter( + CalendarEvent.tenant_id == tenant_id, + CalendarEvent.id == event_id, + CalendarEvent.deleted_at.is_(None), + ) + .first() + ) + if event_locator is None: + raise CalendarError("Calendar event not found") + source = active_sync_source_for_calendar( + session, + tenant_id=tenant_id, + calendar_id=event_locator[0], + for_update=True, + ) event = get_event(session, tenant_id=tenant_id, event_id=event_id) previous = calendar_event_change_payload(event, prefix="previous_") - source = active_sync_source_for_calendar(session, tenant_id=tenant_id, calendar_id=event.calendar_id) assert_sync_mutation_allowed(source) if should_push_event_to_caldav(source=source, event=event): - push_caldav_delete_for_event(session, source=source, event=event) + from govoplan_calendar.backend.outbox import enqueue_caldav_delete + + enqueue_caldav_delete(session, source=source, event_model=event) event.deleted_at = utcnow() session.flush() record_calendar_event_change(session, event=event, operation="deleted", user_id=user_id, previous=previous) -def active_sync_source_for_calendar(session: Session, *, tenant_id: str, calendar_id: str) -> CalendarSyncSource | None: - return ( +def active_sync_source_for_calendar( + session: Session, + *, + tenant_id: str, + calendar_id: str, + for_update: bool = False, +) -> CalendarSyncSource | None: + query = ( session.query(CalendarSyncSource) .filter( CalendarSyncSource.tenant_id == tenant_id, @@ -2182,8 +2653,30 @@ def active_sync_source_for_calendar(session: Session, *, tenant_id: str, calenda CalendarSyncSource.deleted_at.is_(None), ) .order_by((CalendarSyncSource.sync_direction == "two_way").desc(), CalendarSyncSource.created_at.asc()) - .first() ) + if for_update: + query = query.populate_existing().with_for_update() + return query.first() + + +def lock_sync_sources( + session: Session, + *sources: CalendarSyncSource | None, +) -> dict[str, CalendarSyncSource]: + """Lock source rows in stable order before producing desired snapshots.""" + + source_ids = sorted({source.id for source in sources if source is not None}) + if not source_ids: + return {} + locked_sources = ( + session.query(CalendarSyncSource) + .filter(CalendarSyncSource.id.in_(source_ids)) + .order_by(CalendarSyncSource.id.asc()) + .populate_existing() + .with_for_update() + .all() + ) + return {source.id: source for source in locked_sources} def active_caldav_source_for_calendar(session: Session, *, tenant_id: str, calendar_id: str) -> CalendarSyncSource | None: @@ -2203,6 +2696,12 @@ def active_caldav_source_for_calendar(session: Session, *, tenant_id: str, calen def assert_sync_mutation_allowed(source: CalendarSyncSource | None) -> None: if source is None: return + if source.deleted_at is not None: + raise CalendarError("This calendar synchronization source is no longer active") + if not source.sync_enabled: + raise CalendarError( + "This calendar synchronization source is disabled and cannot accept local changes" + ) if source.source_kind != "caldav" or source.sync_direction != "two_way": raise CalendarError(f"This {sync_source_label(source.source_kind)} calendar is inbound-only and cannot be changed locally") @@ -2212,63 +2711,14 @@ def assert_caldav_mutation_allowed(source: CalendarSyncSource | None) -> None: def should_push_event_to_caldav(*, source: CalendarSyncSource | None, event: CalendarEvent) -> bool: - return bool(source and source.source_kind == "caldav" and source.sync_direction == "two_way" and event.deleted_at is None) - - -def push_caldav_event_resource(session: Session, *, source: CalendarSyncSource, event: CalendarEvent, create: bool) -> None: - href = event.source_href or generated_caldav_href(session, source=source, event=event) - full_href = normalize_caldav_href(source.collection_url, href) - resource_events = caldav_resource_events(session, source=source, href=full_href) if event.source_href else [event] - if event not in resource_events: - resource_events.append(event) - resource_events = sorted(resource_events, key=caldav_resource_sort_key) - ics = events_to_ics(resource_events) - client = caldav_client_for_source(session, source) - overwrite = source.conflict_policy == "overwrite" - etag = None if create else resource_etag(resource_events) - try: - result = client.put_object(full_href, ics, etag=etag, create=create, overwrite=overwrite) - except CalDAVPreconditionFailed as exc: - raise CalendarError("CalDAV resource changed remotely; sync the calendar before saving again") from exc - except CalDAVError as exc: - raise CalendarError(f"CalDAV write failed: {exc}") from exc - next_etag = result.etag - for item in resource_events: - apply_caldav_write_metadata(item, source=source, href=full_href, etag=next_etag, raw_ics=ics) - source.last_status = "outbound_ok" - source.last_error = None - schedule_next_caldav_sync(source) - session.flush() - - -def push_caldav_delete_for_event(session: Session, *, source: CalendarSyncSource, event: CalendarEvent) -> None: - if not event.source_href: - return - full_href = normalize_caldav_href(source.collection_url, event.source_href) - remaining = [ - item - for item in caldav_resource_events(session, source=source, href=full_href) - if item.id != event.id - ] - client = caldav_client_for_source(session, source) - overwrite = source.conflict_policy == "overwrite" - try: - if remaining: - remaining = sorted(remaining, key=caldav_resource_sort_key) - ics = events_to_ics(remaining) - result = client.put_object(full_href, ics, etag=event.etag or resource_etag(remaining), create=False, overwrite=overwrite) - for item in remaining: - apply_caldav_write_metadata(item, source=source, href=full_href, etag=result.etag, raw_ics=ics) - else: - client.delete_object(full_href, etag=event.etag, overwrite=overwrite) - except CalDAVPreconditionFailed as exc: - raise CalendarError("CalDAV resource changed remotely; sync the calendar before deleting again") from exc - except CalDAVError as exc: - raise CalendarError(f"CalDAV delete failed: {exc}") from exc - source.last_status = "outbound_ok" - source.last_error = None - schedule_next_caldav_sync(source) - session.flush() + return bool( + source + and source.deleted_at is None + and source.sync_enabled + and source.source_kind == "caldav" + and source.sync_direction == "two_way" + and event.deleted_at is None + ) def caldav_resource_events(session: Session, *, source: CalendarSyncSource, href: str) -> list[CalendarEvent]: @@ -2312,16 +2762,6 @@ def generated_caldav_href(session: Session, *, source: CalendarSyncSource, event return normalize_caldav_href(source.collection_url, f"{safe_uid}-{event.id}.ics") -def apply_caldav_write_metadata(event: CalendarEvent, *, source: CalendarSyncSource, href: str, etag: str | None, raw_ics: str) -> None: - event.source_kind = "caldav" - event.source_href = href - event.etag = etag - event.raw_ics = raw_ics - metadata = dict(event.metadata_ or {}) - metadata["caldav"] = {"source_id": source.id, "collection_url": source.collection_url, "href": href, "etag": etag} - event.metadata_ = metadata - - def import_ics_event( session: Session, *, @@ -2340,6 +2780,32 @@ def import_ics_event( target_calendar_id = calendar_id or (default_calendar.id if default_calendar else None) if not target_calendar_id: raise CalendarError("calendar_id is required because no default calendar exists") + target_calendar = get_calendar( + session, + tenant_id=tenant_id, + calendar_id=target_calendar_id, + ) + # Source creation also locks the calendar row. Holding it here makes the + # public import's provenance decision stable until create/update has queued + # its desired state. + ( + session.query(CalendarCollection) + .filter( + CalendarCollection.id == target_calendar.id, + CalendarCollection.tenant_id == tenant_id, + CalendarCollection.deleted_at.is_(None), + ) + .populate_existing() + .with_for_update() + .one() + ) + target_source = active_sync_source_for_calendar( + session, + tenant_id=tenant_id, + calendar_id=target_calendar_id, + for_update=True, + ) + synchronized_target = target_source is not None existing = None if upsert: existing = ( @@ -2353,16 +2819,27 @@ def import_ics_event( ) .first() ) - payload = CalendarEventCreateRequest( - calendar_id=target_calendar_id, - source_kind=source_kind, - source_href=source_href, - etag=etag, - metadata={}, + payload_values: dict[str, Any] = { + "calendar_id": target_calendar_id, + "metadata": {}, **parsed, - ) + } + if not synchronized_target: + payload_values.update( + { + "source_kind": source_kind, + "source_href": source_href, + "etag": etag, + } + ) + payload = CalendarEventCreateRequest(**payload_values) if existing: - update_payload = CalendarEventUpdateRequest(**payload.model_dump(exclude={"uid", "recurrence_id"})) + excluded_fields = {"uid", "recurrence_id"} + if synchronized_target: + excluded_fields.update({"source_kind", "source_href", "etag"}) + update_payload = CalendarEventUpdateRequest( + **payload.model_dump(exclude=excluded_fields) + ) event = update_event(session, tenant_id=tenant_id, user_id=user_id, event_id=existing.id, payload=update_payload) event.raw_ics = raw_ics return event diff --git a/tests/test_caldav.py b/tests/test_caldav.py index 2c39701..d96e775 100644 --- a/tests/test_caldav.py +++ b/tests/test_caldav.py @@ -9,7 +9,8 @@ from sqlalchemy.orm import sessionmaker from govoplan_access.backend.db import models as access_models # noqa: F401 - populate users/accounts tables from govoplan_calendar.backend.caldav import CalDAVClient, CalDAVError, CalDAVNotFound, CalDAVObject, CalDAVPreconditionFailed, CalDAVReportResult, CalDAVWriteResult, parse_multistatus -from govoplan_calendar.backend.db.models import CalendarEvent, CalendarSyncCredential +from govoplan_calendar.backend.db.models import CalendarEvent, CalendarOutboxOperation, CalendarSyncCredential +from govoplan_calendar.backend.outbox import dispatch_calendar_outbox from govoplan_calendar.backend.schemas import CalendarCalDavSourceCreateRequest, CalendarCollectionCreateRequest, CalendarEventCreateRequest, CalendarEventUpdateRequest from govoplan_calendar.backend.service import ( CALDAV_INTERNAL_CREDENTIAL_PREFIX, @@ -226,6 +227,18 @@ END:VCALENDAR with self.assertRaisesRegex(CalDAVError, "absolute HTTP"): CalDAVClient(collection_url="/remote.php/dav") + def test_object_url_rejects_off_origin_and_cross_collection_hrefs(self) -> None: + client = CalDAVClient(collection_url="https://dav.example.test/cal") + + with self.assertRaisesRegex(CalDAVError, "collection origin"): + client.object_url("https://evil.example.test/steal.ics") + with self.assertRaisesRegex(CalDAVError, "collection path"): + client.object_url("https://dav.example.test/other/steal.ics") + self.assertEqual( + client.object_url("/cal/event.ics"), + "https://dav.example.test/cal/event.ics", + ) + class CalDAVSyncTests(unittest.TestCase): def setUp(self) -> None: @@ -392,36 +405,39 @@ class CalDAVSyncTests(unittest.TestCase): self.assertEqual(len(provider.requests), 1) self.assertEqual(provider.requests[0].event_kind, "calendar.sync.ok") - def test_create_and_update_event_puts_caldav_resource_with_etag(self) -> None: + def test_create_and_update_event_coalesce_to_committed_outbox_state(self) -> None: session = self.Session() session.add(Tenant(id="tenant-1", slug="tenant-1", name="Tenant")) calendar = create_calendar(session, tenant_id="tenant-1", user_id=None, payload=CalendarCollectionCreateRequest(name="Remote")) create_caldav_source(session, tenant_id="tenant-1", user_id=None, payload=CalendarCalDavSourceCreateRequest(calendar_id=calendar.id, collection_url="https://dav.example.test/cal")) session.commit() - fake = FakeCalDAVClient(put_etags=['"etag-1"', '"etag-2"']) - with patch("govoplan_calendar.backend.service.caldav_client_for_source", return_value=fake): - event = create_event( - session, - tenant_id="tenant-1", - user_id=None, - payload=CalendarEventCreateRequest( - calendar_id=calendar.id, - uid="event-1@example.test", - summary="Planning", - start_at=datetime(2026, 7, 8, 9, 0, tzinfo=timezone.utc), - end_at=datetime(2026, 7, 8, 10, 0, tzinfo=timezone.utc), - ), - ) - update_event(session, tenant_id="tenant-1", user_id=None, event_id=event.id, payload=CalendarEventUpdateRequest(summary="Updated")) + fake = FakeCalDAVClient(put_etags=['"etag-1"']) + event = create_event( + session, + tenant_id="tenant-1", + user_id=None, + payload=CalendarEventCreateRequest( + calendar_id=calendar.id, + uid="event-1@example.test", + summary="Planning", + start_at=datetime(2026, 7, 8, 9, 0, tzinfo=timezone.utc), + end_at=datetime(2026, 7, 8, 10, 0, tzinfo=timezone.utc), + ), + ) + update_event(session, tenant_id="tenant-1", user_id=None, event_id=event.id, payload=CalendarEventUpdateRequest(summary="Updated")) + self.assertEqual(fake.puts, []) session.commit() - self.assertEqual(len(fake.puts), 2) + operations = session.query(CalendarOutboxOperation).all() + self.assertCountEqual([operation.status for operation in operations], ["superseded", "pending"]) + dispatch_calendar_outbox(session, tenant_id="tenant-1", client_factory=lambda _session, _source: fake) + + self.assertEqual(len(fake.puts), 1) self.assertTrue(fake.puts[0]["create"]) - self.assertEqual(fake.puts[1]["etag"], '"etag-1"') - self.assertFalse(fake.puts[1]["create"]) + self.assertIn("SUMMARY:Updated", fake.puts[0]["ics"]) self.assertEqual(event.source_kind, "caldav") - self.assertEqual(event.etag, '"etag-2"') + self.assertEqual(event.etag, '"etag-1"') self.assertIn("SUMMARY:Updated", event.raw_ics or "") def test_update_event_reports_remote_etag_conflict(self) -> None: @@ -433,10 +449,22 @@ class CalDAVSyncTests(unittest.TestCase): session.add(event) session.commit() - fake = FakeCalDAVClient(fail_precondition=True) - with patch("govoplan_calendar.backend.service.caldav_client_for_source", return_value=fake): - with self.assertRaisesRegex(CalendarError, "changed remotely"): - update_event(session, tenant_id="tenant-1", user_id=None, event_id=event.id, payload=CalendarEventUpdateRequest(summary="Updated")) + fake = FakeCalDAVClient( + fail_precondition=True, + objects={event.source_href: recurring_resource()}, + ) + update_event(session, tenant_id="tenant-1", user_id=None, event_id=event.id, payload=CalendarEventUpdateRequest(summary="Updated")) + session.commit() + result = dispatch_calendar_outbox( + session, + tenant_id="tenant-1", + client_factory=lambda _session, _source: fake, + ) + + self.assertEqual(result["failed"], 1) + operation = session.query(CalendarOutboxOperation).one() + self.assertEqual(operation.status, "conflict") + self.assertIn("changed remotely", operation.last_error or "") def test_delete_one_component_puts_remaining_resource_instead_of_deleting_object(self) -> None: session = self.Session() @@ -450,9 +478,9 @@ class CalDAVSyncTests(unittest.TestCase): session.commit() fake = FakeCalDAVClient(put_etags=['"etag-2"']) - with patch("govoplan_calendar.backend.service.caldav_client_for_source", return_value=fake): - delete_event(session, tenant_id="tenant-1", event_id=override.id) + delete_event(session, tenant_id="tenant-1", event_id=override.id) session.commit() + dispatch_calendar_outbox(session, tenant_id="tenant-1", client_factory=lambda _session, _source: fake) self.assertEqual(len(fake.puts), 1) self.assertEqual(fake.deletes, []) diff --git a/tests/test_capabilities.py b/tests/test_capabilities.py new file mode 100644 index 0000000..5ba318c --- /dev/null +++ b/tests/test_capabilities.py @@ -0,0 +1,27 @@ +from __future__ import annotations + +import unittest +from datetime import datetime, timezone + +from govoplan_calendar.backend.capabilities import SqlCalendarSchedulingProvider +from govoplan_core.core.calendar import CalendarCapabilityError, CalendarEventRequest + + +class CalendarSchedulingCapabilityTests(unittest.TestCase): + def test_event_request_validation_is_exposed_as_capability_error(self) -> None: + provider = SqlCalendarSchedulingProvider() + + with self.assertRaises(CalendarCapabilityError): + provider.create_event( + object(), + tenant_id="tenant-1", + user_id="user-1", + request=CalendarEventRequest( + summary="x" * 501, + start_at=datetime(2026, 7, 20, 9, tzinfo=timezone.utc), + ), + ) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_outbox.py b/tests/test_outbox.py new file mode 100644 index 0000000..036eb75 --- /dev/null +++ b/tests/test_outbox.py @@ -0,0 +1,1819 @@ +from __future__ import annotations + +import unittest +from datetime import datetime, timedelta, timezone +from unittest.mock import patch + +from sqlalchemy import create_engine +from sqlalchemy.orm import sessionmaker + +from govoplan_access.backend.db import models as access_models # noqa: F401 +from govoplan_calendar.backend.caldav import ( + CalDAVError, + CalDAVNotFound, + CalDAVObject, + CalDAVPreconditionFailed, + CalDAVWriteResult, +) +from govoplan_calendar.backend.db.models import CalendarOutboxOperation +from govoplan_calendar.backend.outbox import ( + SqlCalendarOutboxProvider, + claim_next_calendar_outbox_operation, + discard_calendar_outbox_operation, + dispatch_calendar_outbox, + execute_calendar_outbox_operation, + purge_terminal_calendar_outbox_operations, + reconcile_calendar_outbox_operation, + retry_calendar_outbox_operation, +) +from govoplan_calendar.backend.schemas import ( + CalendarCalDavSourceCreateRequest, + CalendarCalDavSourceUpdateRequest, + CalendarCollectionCreateRequest, + CalendarCollectionDeleteRequest, + CalendarEventCreateRequest, + CalendarEventUpdateRequest, +) +from govoplan_calendar.backend.service import ( + CalendarError, + apply_caldav_report, + create_caldav_source, + create_calendar, + create_event, + delete_caldav_source, + delete_calendar, + delete_event, + import_ics_event, + normalize_caldav_href, + update_caldav_source, + update_event, +) +from govoplan_calendar.backend.caldav import CalDAVReportResult +from govoplan_core.core.change_sequence import ChangeSequenceEntry +from govoplan_core.db.base import Base, utcnow +from govoplan_core.tenancy.scope import create_scope_tables +from govoplan_tenancy.backend.db.models import Tenant + + +class StatefulCalDAVClient: + def __init__(self) -> None: + self.resources: dict[str, tuple[str, str]] = {} + self.puts: list[dict[str, object]] = [] + self.deletes: list[dict[str, object]] = [] + self.next_etag = 1 + self.fail_before_write = False + self.fail_after_write = False + self.fetch_fails = False + self.omit_put_etag = False + + def _etag(self) -> str: + value = f'"etag-{self.next_etag}"' + self.next_etag += 1 + return value + + def put_object( + self, + href: str, + ics: str, + *, + etag: str | None = None, + create: bool = False, + overwrite: bool = False, + ) -> CalDAVWriteResult: + self.puts.append( + {"href": href, "ics": ics, "etag": etag, "create": create, "overwrite": overwrite} + ) + if self.fail_before_write: + raise CalDAVError("network unavailable") + current = self.resources.get(href) + if create and current is not None and not overwrite: + raise CalDAVPreconditionFailed("already exists") + if not create and not overwrite and (not etag or current is None or current[1] != etag): + raise CalDAVPreconditionFailed("changed") + next_etag = self._etag() + self.resources[href] = (ics, next_etag) + if self.fail_after_write: + raise CalDAVError("connection lost after remote commit") + return CalDAVWriteResult( + href=href, + etag=None if self.omit_put_etag else next_etag, + status=201 if create else 204, + ) + + def delete_object( + self, + href: str, + *, + etag: str | None = None, + overwrite: bool = False, + ) -> CalDAVWriteResult: + self.deletes.append({"href": href, "etag": etag, "overwrite": overwrite}) + if self.fail_before_write: + raise CalDAVError("network unavailable") + current = self.resources.get(href) + if current is None: + return CalDAVWriteResult(href=href, status=404) + if not overwrite and (not etag or current[1] != etag): + raise CalDAVPreconditionFailed("changed") + del self.resources[href] + if self.fail_after_write: + raise CalDAVError("connection lost after remote commit") + return CalDAVWriteResult(href=href, status=204) + + def fetch_object_state(self, href: str) -> CalDAVObject: + if self.fetch_fails: + raise CalDAVError("reconciliation unavailable") + current = self.resources.get(href) + if current is None: + raise CalDAVNotFound("not found") + return CalDAVObject(href=href, calendar_data=current[0], etag=current[1]) + + +class CalendarOutboxTests(unittest.TestCase): + def setUp(self) -> None: + self.engine = create_engine("sqlite+pysqlite:///:memory:", future=True) + create_scope_tables(self.engine) + Base.metadata.create_all(self.engine) + self.Session = sessionmaker(bind=self.engine, expire_on_commit=False) + self.session = self.Session() + self.session.add(Tenant(id="tenant-1", slug="tenant-1", name="Tenant")) + self.calendar = create_calendar( + self.session, + tenant_id="tenant-1", + user_id=None, + payload=CalendarCollectionCreateRequest(name="Remote"), + ) + self.source = create_caldav_source( + self.session, + tenant_id="tenant-1", + user_id=None, + payload=CalendarCalDavSourceCreateRequest( + calendar_id=self.calendar.id, + collection_url="https://dav.example.test/cal", + ), + ) + self.session.commit() + + def tearDown(self) -> None: + self.session.close() + self.engine.dispose() + + def create_local_event( + self, + *, + summary: str = "Planning", + uid: str = "event-1@example.test", + source_href: str | None = None, + ): + return create_event( + self.session, + tenant_id="tenant-1", + user_id=None, + payload=CalendarEventCreateRequest( + calendar_id=self.calendar.id, + uid=uid, + summary=summary, + start_at=datetime(2026, 7, 8, 9, 0, tzinfo=timezone.utc), + end_at=datetime(2026, 7, 8, 10, 0, tzinfo=timezone.utc), + source_href=source_href, + ), + ) + + def test_event_and_outbox_are_atomic_and_network_waits_for_commit(self) -> None: + event = self.create_local_event() + operation_id = (event.metadata_ or {})["caldav"]["outbox_operation_id"] + self.assertIsNotNone(self.session.get(CalendarOutboxOperation, operation_id)) + + self.session.rollback() + + self.assertIsNone(self.session.get(CalendarOutboxOperation, operation_id)) + self.assertEqual(self.session.query(CalendarOutboxOperation).count(), 0) + + def test_after_commit_enqueues_registered_worker_without_doing_network_io(self) -> None: + with ( + patch("govoplan_calendar.backend.outbox._enqueue_celery_dispatch") as enqueue, + patch("govoplan_calendar.backend.outbox._remote_matches_operation") as remote_probe, + ): + self.create_local_event() + self.session.commit() + + enqueue.assert_called_once_with("tenant-1") + remote_probe.assert_not_called() + + def test_savepoint_commit_does_not_dispatch_before_outer_commit(self) -> None: + with patch("govoplan_calendar.backend.outbox._enqueue_celery_dispatch") as enqueue: + self.create_local_event() + with self.session.begin_nested(): + self.session.flush() + enqueue.assert_not_called() + self.session.commit() + + enqueue.assert_called_once_with("tenant-1") + + def test_savepoint_rollback_preserves_outer_transaction_wakeup(self) -> None: + with patch("govoplan_calendar.backend.outbox._enqueue_celery_dispatch") as enqueue: + event = self.create_local_event(summary="Outer desired state") + try: + with self.session.begin_nested(): + update_event( + self.session, + tenant_id="tenant-1", + user_id=None, + event_id=event.id, + payload=CalendarEventUpdateRequest(summary="Rolled back desired state"), + ) + raise RuntimeError("roll back savepoint") + except RuntimeError: + pass + enqueue.assert_not_called() + self.session.commit() + + enqueue.assert_called_once_with("tenant-1") + self.assertEqual( + self.session.query(CalendarOutboxOperation).filter_by(status="pending").count(), + 1, + ) + + def test_terminal_retention_purges_only_resolved_old_history(self) -> None: + now = datetime(2026, 7, 20, 12, 0, tzinfo=timezone.utc) + operation_ids: dict[str, str] = {} + for index, status in enumerate( + ("succeeded", "superseded", "cancelled", "conflict", "dead", "pending") + ): + event = self.create_local_event( + summary=f"Retention {status}", + uid=f"retention-{index}@example.test", + ) + operation_id = (event.metadata_ or {})["caldav"]["outbox_operation_id"] + operation = self.session.get(CalendarOutboxOperation, operation_id) + assert operation is not None + operation.status = status + operation.completed_at = now - timedelta(days=31) + operation_ids[status] = operation.id + recent_event = self.create_local_event( + summary="Recent success", + uid="retention-recent@example.test", + ) + recent_operation_id = (recent_event.metadata_ or {})["caldav"]["outbox_operation_id"] + recent_operation = self.session.get(CalendarOutboxOperation, recent_operation_id) + assert recent_operation is not None + recent_operation.status = "succeeded" + recent_operation.completed_at = now - timedelta(days=29) + self.session.commit() + + purged = purge_terminal_calendar_outbox_operations( + self.session, + retention_days=30, + tenant_id="tenant-1", + now=now, + ) + + self.assertEqual(purged, 3) + for status in ("succeeded", "superseded", "cancelled"): + self.assertEqual( + self.session.query(CalendarOutboxOperation) + .filter(CalendarOutboxOperation.id == operation_ids[status]) + .count(), + 0, + ) + for status in ("conflict", "dead", "pending"): + self.assertEqual( + self.session.query(CalendarOutboxOperation) + .filter(CalendarOutboxOperation.id == operation_ids[status]) + .count(), + 1, + ) + self.assertEqual( + self.session.query(CalendarOutboxOperation) + .filter(CalendarOutboxOperation.id == recent_operation.id) + .count(), + 1, + ) + + def test_zero_retention_disables_cleanup_and_provider_validates_setting(self) -> None: + event = self.create_local_event(uid="retention-disabled@example.test") + operation_id = (event.metadata_ or {})["caldav"]["outbox_operation_id"] + operation = self.session.get(CalendarOutboxOperation, operation_id) + assert operation is not None + operation.status = "succeeded" + operation.completed_at = utcnow() - timedelta(days=365) + self.session.commit() + + provider = SqlCalendarOutboxProvider(terminal_retention_days=0) + result = provider.dispatch_due(self.session, tenant_id="tenant-1") + + self.assertEqual(result["purged"], 0) + self.assertIsNotNone(self.session.get(CalendarOutboxOperation, operation.id)) + with self.assertRaisesRegex(ValueError, "zero or greater"): + SqlCalendarOutboxProvider(terminal_retention_days=-1) + with self.assertRaisesRegex(ValueError, "must be an integer"): + SqlCalendarOutboxProvider(terminal_retention_days=True) + + def test_remote_success_before_local_completion_is_reconciled_idempotently(self) -> None: + self.create_local_event() + self.session.commit() + operation = claim_next_calendar_outbox_operation(self.session, tenant_id="tenant-1") + assert operation is not None + href = operation.resource_href + payload = operation.payload_ics or "" + operation_id = operation.id + self.session.commit() + + client = StatefulCalDAVClient() + client.put_object(href, payload, create=True) + operation = self.session.get(CalendarOutboxOperation, operation_id) + assert operation is not None + operation.lease_expires_at = utcnow() - timedelta(seconds=1) + self.session.commit() + + result = dispatch_calendar_outbox( + self.session, + tenant_id="tenant-1", + client_factory=lambda _session, _source: client, + ) + + operation = self.session.get(CalendarOutboxOperation, operation_id) + assert operation is not None + self.assertEqual(result["succeeded"], 1) + self.assertEqual(operation.status, "succeeded") + self.assertIsNotNone(operation.reconciled_at) + self.assertEqual(len(client.resources), 1) + + def test_transient_failure_retries_with_backoff_then_becomes_dead(self) -> None: + self.create_local_event() + self.session.commit() + operation = self.session.query(CalendarOutboxOperation).one() + operation.max_attempts = 2 + self.session.commit() + client = StatefulCalDAVClient() + client.fail_before_write = True + client.fetch_fails = True + + dispatch_calendar_outbox( + self.session, + tenant_id="tenant-1", + client_factory=lambda _session, _source: client, + ) + operation = self.session.get(CalendarOutboxOperation, operation.id) + assert operation is not None + self.assertEqual(operation.status, "retry") + self.assertGreater(operation.available_at.replace(tzinfo=timezone.utc), utcnow()) + operation.available_at = utcnow() - timedelta(seconds=1) + self.session.commit() + + dispatch_calendar_outbox( + self.session, + tenant_id="tenant-1", + client_factory=lambda _session, _source: client, + ) + operation = self.session.get(CalendarOutboxOperation, operation.id) + assert operation is not None + self.assertEqual(operation.status, "dead") + self.assertEqual(operation.attempt_count, 2) + + def test_newer_state_waits_behind_predecessor_backoff(self) -> None: + event = self.create_local_event(summary="First") + self.session.commit() + client = StatefulCalDAVClient() + client.fail_before_write = True + client.fetch_fails = True + dispatch_calendar_outbox( + self.session, + tenant_id="tenant-1", + client_factory=lambda _session, _source: client, + ) + first = self.session.query(CalendarOutboxOperation).one() + self.assertEqual(first.status, "retry") + + update_event( + self.session, + tenant_id="tenant-1", + user_id=None, + event_id=event.id, + payload=CalendarEventUpdateRequest(summary="Second"), + ) + self.session.commit() + second = ( + self.session.query(CalendarOutboxOperation) + .filter(CalendarOutboxOperation.id != first.id) + .one() + ) + client.fail_before_write = False + client.fetch_fails = False + + blocked = dispatch_calendar_outbox( + self.session, + tenant_id="tenant-1", + client_factory=lambda _session, _source: client, + ) + self.assertEqual(blocked["processed"], 0) + self.assertEqual(second.status, "pending") + + first.available_at = utcnow() - timedelta(seconds=1) + self.session.commit() + delivered = dispatch_calendar_outbox( + self.session, + tenant_id="tenant-1", + client_factory=lambda _session, _source: client, + ) + self.assertEqual(delivered["succeeded"], 2) + self.assertEqual(client.puts[-1]["etag"], '"etag-1"') + self.assertIn("SUMMARY:Second", client.puts[-1]["ics"]) + + def test_ambiguous_predecessor_is_reconciled_before_newer_state(self) -> None: + event = self.create_local_event(summary="First") + self.session.commit() + first = claim_next_calendar_outbox_operation(self.session, tenant_id="tenant-1") + assert first is not None + href = first.resource_href + client = StatefulCalDAVClient() + client.put_object(href, first.payload_ics or "", create=True) + first.lease_expires_at = utcnow() - timedelta(seconds=1) + self.session.commit() + + update_event( + self.session, + tenant_id="tenant-1", + user_id=None, + event_id=event.id, + payload=CalendarEventUpdateRequest(summary="Second"), + ) + self.session.commit() + second = ( + self.session.query(CalendarOutboxOperation) + .filter(CalendarOutboxOperation.id != first.id) + .one() + ) + + result = dispatch_calendar_outbox( + self.session, + tenant_id="tenant-1", + client_factory=lambda _session, _source: client, + ) + + self.assertEqual(result["succeeded"], 2) + self.assertIsNotNone(first.reconciled_at) + self.assertEqual(second.expected_etag, '"etag-1"') + self.assertEqual(client.puts[-1]["etag"], '"etag-1"') + self.assertIn("SUMMARY:Second", client.puts[-1]["ics"]) + + def test_only_one_resource_per_source_can_hold_a_delivery_lease(self) -> None: + self.create_local_event(uid="event-1@example.test") + self.create_local_event(uid="event-2@example.test") + self.session.commit() + + first = claim_next_calendar_outbox_operation(self.session, tenant_id="tenant-1") + assert first is not None + self.session.commit() + second = claim_next_calendar_outbox_operation(self.session, tenant_id="tenant-1") + + self.assertIsNone(second) + self.assertEqual( + self.session.query(CalendarOutboxOperation).filter_by(status="in_progress").count(), + 1, + ) + + def test_manual_retry_rejects_stale_resource_generation(self) -> None: + event = self.create_local_event(summary="First") + self.session.commit() + client = StatefulCalDAVClient() + client.fail_before_write = True + client.fetch_fails = True + dispatch_calendar_outbox( + self.session, + tenant_id="tenant-1", + client_factory=lambda _session, _source: client, + ) + first = self.session.query(CalendarOutboxOperation).one() + self.assertEqual(first.status, "retry") + update_event( + self.session, + tenant_id="tenant-1", + user_id=None, + event_id=event.id, + payload=CalendarEventUpdateRequest(summary="Second"), + ) + self.session.commit() + + with self.assertRaisesRegex(ValueError, "newer desired state"): + retry_calendar_outbox_operation( + self.session, + tenant_id="tenant-1", + operation_id=first.id, + ) + with self.assertRaisesRegex(ValueError, "newer desired state"): + discard_calendar_outbox_operation( + self.session, + tenant_id="tenant-1", + operation_id=first.id, + ) + + def test_discard_latest_cancels_attempted_predecessor_chain(self) -> None: + event = self.create_local_event(summary="First") + self.session.commit() + client = StatefulCalDAVClient() + client.fail_before_write = True + client.fetch_fails = True + dispatch_calendar_outbox( + self.session, + tenant_id="tenant-1", + client_factory=lambda _session, _source: client, + ) + first = self.session.query(CalendarOutboxOperation).one() + update_event( + self.session, + tenant_id="tenant-1", + user_id=None, + event_id=event.id, + payload=CalendarEventUpdateRequest(summary="Second"), + ) + self.session.commit() + latest = ( + self.session.query(CalendarOutboxOperation) + .filter(CalendarOutboxOperation.id != first.id) + .one() + ) + + discard_calendar_outbox_operation( + self.session, + tenant_id="tenant-1", + operation_id=latest.id, + ) + self.session.commit() + self.assertEqual(first.status, "cancelled") + self.assertEqual(latest.status, "cancelled") + self.assertEqual((event.metadata_ or {})["caldav"]["external_state"], "discarded") + client.fail_before_write = False + client.fetch_fails = False + result = dispatch_calendar_outbox( + self.session, + tenant_id="tenant-1", + client_factory=lambda _session, _source: client, + ) + self.assertEqual(result["processed"], 0) + + def test_client_or_credential_setup_failure_is_persisted_for_retry(self) -> None: + self.create_local_event() + self.session.commit() + + def unavailable_client(_session, _source): + raise CalendarError("credential temporarily unavailable") + + result = dispatch_calendar_outbox( + self.session, + tenant_id="tenant-1", + client_factory=unavailable_client, + ) + operation = self.session.query(CalendarOutboxOperation).one() + + self.assertEqual(result["retrying"], 1) + self.assertEqual(operation.status, "retry") + self.assertIn("credential temporarily unavailable", operation.last_error or "") + + def test_newer_update_uses_etag_from_older_in_progress_put(self) -> None: + event = self.create_local_event(summary="First") + self.session.commit() + first = claim_next_calendar_outbox_operation(self.session, tenant_id="tenant-1") + assert first is not None + first_id = first.id + first_lease = first.lease_token or "" + self.session.commit() + + update_event( + self.session, + tenant_id="tenant-1", + user_id=None, + event_id=event.id, + payload=CalendarEventUpdateRequest(summary="Second"), + ) + self.session.commit() + second = ( + self.session.query(CalendarOutboxOperation) + .filter(CalendarOutboxOperation.id != first_id) + .one() + ) + self.assertIsNone(second.expected_etag) + client = StatefulCalDAVClient() + + execute_calendar_outbox_operation( + self.session, + operation_id=first_id, + lease_token=first_lease, + client_factory=lambda _session, _source: client, + ) + self.session.commit() + self.session.refresh(second) + self.assertEqual(second.expected_etag, '"etag-1"') + self.assertEqual(event.etag, '"etag-1"') + self.assertEqual((event.metadata_ or {})["caldav"]["outbox_operation_id"], second.id) + + dispatch_calendar_outbox( + self.session, + tenant_id="tenant-1", + client_factory=lambda _session, _source: client, + ) + self.assertEqual(client.puts[-1]["etag"], '"etag-1"') + self.assertIn("SUMMARY:Second", client.puts[-1]["ics"]) + + def test_put_without_response_etag_fetches_it_before_succeeding(self) -> None: + event = self.create_local_event(summary="First") + self.session.commit() + client = StatefulCalDAVClient() + client.omit_put_etag = True + + result = dispatch_calendar_outbox( + self.session, + tenant_id="tenant-1", + client_factory=lambda _session, _source: client, + ) + first = self.session.query(CalendarOutboxOperation).one() + self.assertEqual(result["succeeded"], 1) + self.assertEqual(first.remote_etag, '"etag-1"') + self.assertIsNotNone(first.reconciled_at) + self.assertEqual(event.etag, '"etag-1"') + + update_event( + self.session, + tenant_id="tenant-1", + user_id=None, + event_id=event.id, + payload=CalendarEventUpdateRequest(summary="Second"), + ) + self.session.commit() + dispatch_calendar_outbox( + self.session, + tenant_id="tenant-1", + client_factory=lambda _session, _source: client, + ) + self.assertEqual(client.puts[-1]["etag"], '"etag-1"') + + def test_delete_without_known_etag_refuses_unknown_remote_version(self) -> None: + event = self.create_local_event() + self.session.commit() + href = event.source_href or "" + delete_event(self.session, tenant_id="tenant-1", event_id=event.id) + self.session.commit() + delete_operation = ( + self.session.query(CalendarOutboxOperation) + .filter_by(operation_kind="delete") + .one() + ) + self.assertIsNone(delete_operation.expected_etag) + client = StatefulCalDAVClient() + client.resources[href] = (different_event_ics(), '"unknown"') + + result = dispatch_calendar_outbox( + self.session, + tenant_id="tenant-1", + client_factory=lambda _session, _source: client, + ) + + self.assertEqual(result["failed"], 1) + self.assertEqual(delete_operation.status, "conflict") + self.assertEqual(client.deletes, []) + self.assertIn("no known ETag", delete_operation.last_error or "") + + def test_newer_delete_follows_in_progress_update_without_resurrecting_event(self) -> None: + event = self.create_local_event() + self.session.commit() + client = StatefulCalDAVClient() + dispatch_calendar_outbox( + self.session, + tenant_id="tenant-1", + client_factory=lambda _session, _source: client, + ) + update_event( + self.session, + tenant_id="tenant-1", + user_id=None, + event_id=event.id, + payload=CalendarEventUpdateRequest(summary="In flight"), + ) + self.session.commit() + update_operation = claim_next_calendar_outbox_operation(self.session, tenant_id="tenant-1") + assert update_operation is not None + update_id = update_operation.id + update_lease = update_operation.lease_token or "" + self.session.commit() + delete_event(self.session, tenant_id="tenant-1", event_id=event.id) + self.session.commit() + + execute_calendar_outbox_operation( + self.session, + operation_id=update_id, + lease_token=update_lease, + client_factory=lambda _session, _source: client, + ) + self.session.commit() + delete_operation = ( + self.session.query(CalendarOutboxOperation) + .filter(CalendarOutboxOperation.operation_kind == "delete") + .one() + ) + self.assertEqual(delete_operation.expected_etag, '"etag-2"') + dispatch_calendar_outbox( + self.session, + tenant_id="tenant-1", + client_factory=lambda _session, _source: client, + ) + + self.assertEqual(client.resources, {}) + self.assertEqual(delete_operation.status, "succeeded") + self.assertIsNotNone(event.deleted_at) + + def test_source_direction_change_rejects_unresolved_but_policy_and_credentials_do_not(self) -> None: + self.create_local_event() + self.session.commit() + operation = self.session.query(CalendarOutboxOperation).one() + update_caldav_source( + self.session, + tenant_id="tenant-1", + source_id=self.source.id, + payload=CalendarCalDavSourceUpdateRequest( + conflict_policy="overwrite", + username="rotated-user", + ), + ) + self.session.commit() + self.assertEqual(operation.status, "pending") + client = StatefulCalDAVClient() + dispatch_calendar_outbox( + self.session, + tenant_id="tenant-1", + client_factory=lambda _session, _source: client, + ) + self.assertFalse(client.puts[0]["overwrite"]) + + update_event( + self.session, + tenant_id="tenant-1", + user_id=None, + event_id=operation.event_id or "", + payload=CalendarEventUpdateRequest(summary="Pending again"), + ) + self.session.commit() + pending = ( + self.session.query(CalendarOutboxOperation) + .filter(CalendarOutboxOperation.status == "pending") + .one() + ) + with self.assertRaisesRegex(CalendarError, "unresolved local desired"): + update_caldav_source( + self.session, + tenant_id="tenant-1", + source_id=self.source.id, + payload=CalendarCalDavSourceUpdateRequest(sync_direction="inbound"), + ) + self.session.rollback() + self.assertEqual(pending.status, "pending") + self.assertEqual(self.source.sync_direction, "two_way") + + def test_disabled_source_rejects_local_edits_instead_of_losing_them(self) -> None: + event = self.create_local_event() + self.session.commit() + client = StatefulCalDAVClient() + dispatch_calendar_outbox( + self.session, + tenant_id="tenant-1", + client_factory=lambda _session, _source: client, + ) + update_caldav_source( + self.session, + tenant_id="tenant-1", + source_id=self.source.id, + payload=CalendarCalDavSourceUpdateRequest(sync_enabled=False), + ) + self.session.commit() + + with self.assertRaisesRegex(CalendarError, "disabled"): + update_event( + self.session, + tenant_id="tenant-1", + user_id=None, + event_id=event.id, + payload=CalendarEventUpdateRequest(summary="Would be lost"), + ) + self.session.rollback() + self.assertEqual(event.summary, "Planning") + self.assertEqual( + self.session.query(CalendarOutboxOperation).filter_by(status="pending").count(), + 0, + ) + + def test_source_endpoint_change_is_rejected_while_events_reference_it(self) -> None: + self.create_local_event() + self.session.commit() + client = StatefulCalDAVClient() + dispatch_calendar_outbox( + self.session, + tenant_id="tenant-1", + client_factory=lambda _session, _source: client, + ) + + with self.assertRaisesRegex(CalendarError, "events still reference"): + update_caldav_source( + self.session, + tenant_id="tenant-1", + source_id=self.source.id, + payload=CalendarCalDavSourceUpdateRequest( + collection_url="https://dav.example.test/new-calendar", + ), + ) + self.session.rollback() + self.assertEqual(self.source.collection_url, "https://dav.example.test/cal/") + + def test_calendar_rejects_a_second_active_sync_source(self) -> None: + with self.assertRaisesRegex(CalendarError, "only one active"): + create_caldav_source( + self.session, + tenant_id="tenant-1", + user_id=None, + payload=CalendarCalDavSourceCreateRequest( + calendar_id=self.calendar.id, + collection_url="https://dav.example.test/another", + ), + ) + + def test_soft_deleted_events_do_not_permanently_block_source_endpoint_change(self) -> None: + event = self.create_local_event() + self.session.commit() + client = StatefulCalDAVClient() + dispatch_calendar_outbox( + self.session, + tenant_id="tenant-1", + client_factory=lambda _session, _source: client, + ) + delete_event(self.session, tenant_id="tenant-1", event_id=event.id) + self.session.commit() + dispatch_calendar_outbox( + self.session, + tenant_id="tenant-1", + client_factory=lambda _session, _source: client, + ) + + update_caldav_source( + self.session, + tenant_id="tenant-1", + source_id=self.source.id, + payload=CalendarCalDavSourceUpdateRequest( + collection_url="https://dav.example.test/new-calendar", + ), + ) + + self.assertEqual( + self.source.collection_url, + "https://dav.example.test/new-calendar/", + ) + + def test_bulk_move_requires_the_matching_explicit_external_action(self) -> None: + event = self.create_local_event() + local_calendar = create_calendar( + self.session, + tenant_id="tenant-1", + user_id=None, + payload=CalendarCollectionCreateRequest(name="Local"), + ) + self.session.commit() + + with self.assertRaisesRegex(CalendarError, "detach_keep_remote"): + delete_calendar( + self.session, + tenant_id="tenant-1", + calendar_id=self.calendar.id, + payload=CalendarCollectionDeleteRequest( + event_action="move", + target_calendar_id=local_calendar.id, + ), + ) + self.session.rollback() + + self.assertEqual(event.calendar_id, self.calendar.id) + self.assertIsNone(self.calendar.deleted_at) + self.assertIsNone(self.source.deleted_at) + + another_local_calendar = create_calendar( + self.session, + tenant_id="tenant-1", + user_id=None, + payload=CalendarCollectionCreateRequest(name="Another local"), + ) + create_event( + self.session, + tenant_id="tenant-1", + user_id=None, + payload=CalendarEventCreateRequest( + calendar_id=another_local_calendar.id, + uid="local-event@example.test", + summary="Local event", + start_at=datetime(2026, 7, 9, 9, 0, tzinfo=timezone.utc), + ), + ) + self.session.commit() + with self.assertRaisesRegex(CalendarError, "copy_to_remote"): + delete_calendar( + self.session, + tenant_id="tenant-1", + calendar_id=another_local_calendar.id, + payload=CalendarCollectionDeleteRequest( + event_action="move", + target_calendar_id=self.calendar.id, + ), + ) + self.session.rollback() + self.assertIsNone(another_local_calendar.deleted_at) + + with self.assertRaisesRegex(CalendarError, "only valid"): + delete_calendar( + self.session, + tenant_id="tenant-1", + calendar_id=another_local_calendar.id, + payload=CalendarCollectionDeleteRequest( + event_action="move", + target_calendar_id=local_calendar.id, + external_action="detach_keep_remote", + ), + ) + self.session.rollback() + + with self.assertRaisesRegex(CalendarError, "remote_move.*not supported"): + delete_calendar( + self.session, + tenant_id="tenant-1", + calendar_id=self.calendar.id, + payload=CalendarCollectionDeleteRequest( + event_action="move", + target_calendar_id=local_calendar.id, + external_action="remote_move", + ), + ) + self.session.rollback() + + def test_detach_move_cancels_pending_state_and_never_deletes_remote_events(self) -> None: + delivered_event = self.create_local_event(uid="delivered@example.test") + self.session.commit() + client = StatefulCalDAVClient() + dispatch_calendar_outbox( + self.session, + tenant_id="tenant-1", + client_factory=lambda _session, _source: client, + ) + delivered_href = delivered_event.source_href or "" + self.assertIn(delivered_href, client.resources) + + pending_event = self.create_local_event(uid="pending@example.test") + pending_operation_id = (pending_event.metadata_ or {})["caldav"]["outbox_operation_id"] + local_calendar = create_calendar( + self.session, + tenant_id="tenant-1", + user_id=None, + payload=CalendarCollectionCreateRequest(name="Detached"), + ) + self.session.commit() + + delete_calendar( + self.session, + tenant_id="tenant-1", + calendar_id=self.calendar.id, + payload=CalendarCollectionDeleteRequest( + event_action="move", + target_calendar_id=local_calendar.id, + external_action="detach_keep_remote", + ), + ) + + pending_operation = self.session.get(CalendarOutboxOperation, pending_operation_id) + assert pending_operation is not None + self.assertEqual(pending_operation.status, "cancelled") + self.assertTrue(all(item.operation_kind != "delete" for item in self.session.query(CalendarOutboxOperation).all())) + for event in (delivered_event, pending_event): + self.assertEqual(event.calendar_id, local_calendar.id) + self.assertEqual(event.source_kind, "local") + self.assertIsNone(event.source_href) + self.assertIsNone(event.etag) + self.assertNotIn("caldav", event.metadata_ or {}) + self.assertIsNone(event.deleted_at) + self.assertIsNotNone(self.calendar.deleted_at) + self.assertIsNotNone(self.source.deleted_at) + self.session.commit() + + result = dispatch_calendar_outbox( + self.session, + tenant_id="tenant-1", + client_factory=lambda _session, _source: client, + ) + self.assertEqual(result["processed"], 0) + self.assertEqual(client.deletes, []) + self.assertIn(delivered_href, client.resources) + + def test_detach_move_is_rejected_while_source_has_a_live_delivery_lease(self) -> None: + event = self.create_local_event(uid="leased@example.test") + local_calendar = create_calendar( + self.session, + tenant_id="tenant-1", + user_id=None, + payload=CalendarCollectionCreateRequest(name="Local"), + ) + self.session.commit() + operation = claim_next_calendar_outbox_operation(self.session, tenant_id="tenant-1") + assert operation is not None + self.session.commit() + + with self.assertRaisesRegex(CalendarError, "active lease"): + delete_calendar( + self.session, + tenant_id="tenant-1", + calendar_id=self.calendar.id, + payload=CalendarCollectionDeleteRequest( + event_action="move", + target_calendar_id=local_calendar.id, + external_action="detach_keep_remote", + ), + ) + self.session.rollback() + + self.assertEqual(event.calendar_id, self.calendar.id) + self.assertEqual(operation.status, "in_progress") + self.assertIsNone(self.calendar.deleted_at) + self.assertIsNone(self.source.deleted_at) + + def test_copy_move_queues_destination_put_and_keeps_failure_retryable(self) -> None: + local_calendar = create_calendar( + self.session, + tenant_id="tenant-1", + user_id=None, + payload=CalendarCollectionCreateRequest(name="Local"), + ) + event = create_event( + self.session, + tenant_id="tenant-1", + user_id=None, + payload=CalendarEventCreateRequest( + calendar_id=local_calendar.id, + uid="copy@example.test", + summary="Copy through durable outbox", + start_at=datetime(2026, 7, 9, 9, 0, tzinfo=timezone.utc), + ), + ) + self.session.commit() + + delete_calendar( + self.session, + tenant_id="tenant-1", + calendar_id=local_calendar.id, + payload=CalendarCollectionDeleteRequest( + event_action="move", + target_calendar_id=self.calendar.id, + external_action="copy_to_remote", + ), + ) + + operation_id = (event.metadata_ or {})["caldav"]["outbox_operation_id"] + operation = self.session.get(CalendarOutboxOperation, operation_id) + assert operation is not None + self.assertEqual(operation.operation_kind, "put") + self.assertEqual(operation.status, "pending") + self.assertEqual(event.calendar_id, self.calendar.id) + self.assertEqual(event.source_kind, "caldav") + self.assertIsNone(event.deleted_at) + self.assertIsNotNone(local_calendar.deleted_at) + move_change = ( + self.session.query(ChangeSequenceEntry) + .filter_by(resource_id=event.id) + .order_by(ChangeSequenceEntry.id.desc()) + .first() + ) + assert move_change is not None + self.assertEqual(move_change.payload["previous_calendar_id"], local_calendar.id) + self.assertEqual(move_change.payload["calendar_id"], self.calendar.id) + self.assertEqual(move_change.payload["external_state"], "queued") + self.session.commit() + + client = StatefulCalDAVClient() + client.fail_before_write = True + result = dispatch_calendar_outbox( + self.session, + tenant_id="tenant-1", + client_factory=lambda _session, _source: client, + ) + self.assertEqual(result["retrying"], 1) + self.assertEqual(operation.status, "retry") + self.assertEqual((event.metadata_ or {})["caldav"]["external_state"], "retry") + self.assertEqual(client.deletes, []) + + retry_calendar_outbox_operation( + self.session, + tenant_id="tenant-1", + operation_id=operation.id, + ) + self.session.commit() + client.fail_before_write = False + result = dispatch_calendar_outbox( + self.session, + tenant_id="tenant-1", + client_factory=lambda _session, _source: client, + ) + self.assertEqual(result["succeeded"], 1) + self.assertEqual(operation.status, "succeeded") + self.assertIn(event.source_href or "", client.resources) + self.assertEqual(client.deletes, []) + + def test_copy_move_rejects_an_inbound_only_target(self) -> None: + local_calendar = create_calendar( + self.session, + tenant_id="tenant-1", + user_id=None, + payload=CalendarCollectionCreateRequest(name="Local"), + ) + self.source.sync_direction = "inbound" + self.session.commit() + + with self.assertRaisesRegex(CalendarError, "active two-way CalDAV target"): + delete_calendar( + self.session, + tenant_id="tenant-1", + calendar_id=local_calendar.id, + payload=CalendarCollectionDeleteRequest( + event_action="move", + target_calendar_id=self.calendar.id, + external_action="copy_to_remote", + ), + ) + self.session.rollback() + self.assertIsNone(local_calendar.deleted_at) + + def test_local_to_local_move_remains_compatible_without_external_action(self) -> None: + source_calendar = create_calendar( + self.session, + tenant_id="tenant-1", + user_id=None, + payload=CalendarCollectionCreateRequest(name="Local source"), + ) + target_calendar = create_calendar( + self.session, + tenant_id="tenant-1", + user_id=None, + payload=CalendarCollectionCreateRequest(name="Local target"), + ) + event = create_event( + self.session, + tenant_id="tenant-1", + user_id=None, + payload=CalendarEventCreateRequest( + calendar_id=source_calendar.id, + uid="local-move@example.test", + summary="Local move", + start_at=datetime(2026, 7, 9, 9, 0, tzinfo=timezone.utc), + ), + ) + self.session.commit() + + delete_calendar( + self.session, + tenant_id="tenant-1", + calendar_id=source_calendar.id, + payload=CalendarCollectionDeleteRequest( + event_action="move", + target_calendar_id=target_calendar.id, + ), + ) + + self.assertEqual(event.calendar_id, target_calendar.id) + self.assertEqual(event.source_kind, "local") + self.assertIsNone(event.deleted_at) + self.assertIsNotNone(source_calendar.deleted_at) + self.assertEqual(self.session.query(CalendarOutboxOperation).count(), 0) + + def test_move_between_synchronized_calendars_remains_rejected(self) -> None: + other_calendar = create_calendar( + self.session, + tenant_id="tenant-1", + user_id=None, + payload=CalendarCollectionCreateRequest(name="Other remote"), + ) + create_caldav_source( + self.session, + tenant_id="tenant-1", + user_id=None, + payload=CalendarCalDavSourceCreateRequest( + calendar_id=other_calendar.id, + collection_url="https://dav.example.test/other", + ), + ) + self.session.commit() + + with self.assertRaisesRegex(CalendarError, "between synchronized calendars"): + delete_calendar( + self.session, + tenant_id="tenant-1", + calendar_id=self.calendar.id, + payload=CalendarCollectionDeleteRequest( + event_action="move", + target_calendar_id=other_calendar.id, + external_action="detach_keep_remote", + ), + ) + self.session.rollback() + self.assertIsNone(self.calendar.deleted_at) + self.assertIsNone(other_calendar.deleted_at) + + def test_deleting_synced_collection_is_local_unlink_and_preserves_remote(self) -> None: + event = self.create_local_event(uid="preserve-on-unlink@example.test") + self.session.commit() + client = StatefulCalDAVClient() + dispatch_calendar_outbox( + self.session, + tenant_id="tenant-1", + client_factory=lambda _session, _source: client, + ) + href = event.source_href or "" + self.assertIn(href, client.resources) + + delete_calendar( + self.session, + tenant_id="tenant-1", + calendar_id=self.calendar.id, + ) + self.session.commit() + result = dispatch_calendar_outbox( + self.session, + tenant_id="tenant-1", + client_factory=lambda _session, _source: client, + ) + + self.assertEqual(result["processed"], 0) + self.assertEqual(client.deletes, []) + self.assertIn(href, client.resources) + self.assertIsNotNone(event.deleted_at) + self.assertIsNotNone(self.source.deleted_at) + + def test_material_source_change_is_rejected_while_delivery_lease_is_live(self) -> None: + self.create_local_event() + self.session.commit() + operation = claim_next_calendar_outbox_operation(self.session, tenant_id="tenant-1") + assert operation is not None + self.session.commit() + + with self.assertRaisesRegex(CalendarError, "active lease"): + update_caldav_source( + self.session, + tenant_id="tenant-1", + source_id=self.source.id, + payload=CalendarCalDavSourceUpdateRequest(sync_direction="inbound"), + ) + self.session.rollback() + + self.assertEqual(self.source.sync_direction, "two_way") + self.assertEqual(operation.status, "in_progress") + + def test_source_and_calendar_retirement_are_rejected_while_lease_is_live(self) -> None: + self.create_local_event() + self.session.commit() + operation = claim_next_calendar_outbox_operation(self.session, tenant_id="tenant-1") + assert operation is not None + self.session.commit() + + with self.assertRaisesRegex(CalendarError, "cannot be retired.*active lease"): + delete_caldav_source( + self.session, + tenant_id="tenant-1", + source_id=self.source.id, + ) + self.session.rollback() + with self.assertRaisesRegex(CalendarError, "cannot be retired.*active lease"): + delete_calendar( + self.session, + tenant_id="tenant-1", + calendar_id=self.calendar.id, + ) + self.session.rollback() + + self.assertIsNone(self.source.deleted_at) + self.assertIsNone(self.calendar.deleted_at) + self.assertEqual(operation.status, "in_progress") + + def test_tenant_dispatch_does_not_reclaim_another_tenants_expired_lease(self) -> None: + self.create_local_event() + self.session.add(Tenant(id="tenant-2", slug="tenant-2", name="Tenant 2")) + calendar_2 = create_calendar( + self.session, + tenant_id="tenant-2", + user_id=None, + payload=CalendarCollectionCreateRequest(name="Remote 2"), + ) + create_caldav_source( + self.session, + tenant_id="tenant-2", + user_id=None, + payload=CalendarCalDavSourceCreateRequest( + calendar_id=calendar_2.id, + collection_url="https://dav.example.test/tenant-2", + ), + ) + create_event( + self.session, + tenant_id="tenant-2", + user_id=None, + payload=CalendarEventCreateRequest( + calendar_id=calendar_2.id, + uid="event-2@example.test", + summary="Tenant 2", + start_at=datetime(2026, 7, 8, 11, 0, tzinfo=timezone.utc), + ), + ) + self.session.commit() + tenant_2_operation = claim_next_calendar_outbox_operation( + self.session, + tenant_id="tenant-2", + ) + assert tenant_2_operation is not None + tenant_2_operation.lease_expires_at = utcnow() - timedelta(seconds=1) + self.session.commit() + + tenant_1_operation = claim_next_calendar_outbox_operation( + self.session, + tenant_id="tenant-1", + ) + + self.assertIsNotNone(tenant_1_operation) + self.assertEqual(tenant_2_operation.status, "in_progress") + + def test_corrupt_cross_tenant_source_reference_is_quarantined_without_network(self) -> None: + self.session.add(Tenant(id="tenant-2", slug="tenant-2", name="Tenant 2")) + calendar_2 = create_calendar( + self.session, + tenant_id="tenant-2", + user_id=None, + payload=CalendarCollectionCreateRequest(name="Remote 2"), + ) + source_2 = create_caldav_source( + self.session, + tenant_id="tenant-2", + user_id=None, + payload=CalendarCalDavSourceCreateRequest( + calendar_id=calendar_2.id, + collection_url="https://dav.example.test/tenant-2", + ), + ) + create_event( + self.session, + tenant_id="tenant-2", + user_id=None, + payload=CalendarEventCreateRequest( + calendar_id=calendar_2.id, + uid="event-2@example.test", + summary="Tenant 2", + start_at=datetime(2026, 7, 8, 11, 0, tzinfo=timezone.utc), + ), + ) + self.session.commit() + operation = claim_next_calendar_outbox_operation(self.session, tenant_id="tenant-2") + assert operation is not None + lease_token = operation.lease_token or "" + operation.source_id = self.source.id + self.session.commit() + client = StatefulCalDAVClient() + + execute_calendar_outbox_operation( + self.session, + operation_id=operation.id, + lease_token=lease_token, + client_factory=lambda _session, _source: client, + ) + self.session.commit() + + self.assertEqual(operation.status, "cancelled") + self.assertIn("tenant does not match", operation.last_error or "") + self.assertEqual(client.puts, []) + self.assertNotEqual(source_2.id, self.source.id) + + def test_manual_reconcile_respects_live_lease_and_marks_mismatch_conflict(self) -> None: + event = self.create_local_event() + self.session.commit() + operation = claim_next_calendar_outbox_operation(self.session, tenant_id="tenant-1") + assert operation is not None + self.session.commit() + client = StatefulCalDAVClient() + + with self.assertRaisesRegex(ValueError, "actively leased"): + reconcile_calendar_outbox_operation( + self.session, + tenant_id="tenant-1", + operation_id=operation.id, + client_factory=lambda _session, _source: client, + ) + + operation.lease_expires_at = utcnow() - timedelta(seconds=1) + operation.status = "retry" + operation.lease_token = None + self.session.commit() + client.resources[operation.resource_href] = (different_event_ics(), '"remote"') + reconcile_calendar_outbox_operation( + self.session, + tenant_id="tenant-1", + operation_id=operation.id, + client_factory=lambda _session, _source: client, + ) + self.session.commit() + + self.assertEqual(operation.status, "conflict") + self.assertEqual(self.source.last_status, "outbound_error") + self.assertEqual((event.metadata_ or {})["caldav"]["external_state"], "failed") + with self.assertRaisesRegex(ValueError, "succeeded"): + operation.status = "succeeded" + self.session.flush() + retry_calendar_outbox_operation( + self.session, + tenant_id="tenant-1", + operation_id=operation.id, + ) + + def test_manual_reconcile_live_lease_is_timezone_safe_in_fresh_session(self) -> None: + self.create_local_event() + self.session.commit() + operation = claim_next_calendar_outbox_operation(self.session, tenant_id="tenant-1") + assert operation is not None + operation_id = operation.id + self.session.commit() + self.session.close() + self.session = self.Session() + + with self.assertRaisesRegex(ValueError, "actively leased"): + reconcile_calendar_outbox_operation( + self.session, + tenant_id="tenant-1", + operation_id=operation_id, + client_factory=lambda _session, _source: StatefulCalDAVClient(), + ) + + def test_delete_retry_is_projected_on_deleted_trigger_event(self) -> None: + event = self.create_local_event() + self.session.commit() + client = StatefulCalDAVClient() + dispatch_calendar_outbox( + self.session, + tenant_id="tenant-1", + client_factory=lambda _session, _source: client, + ) + delete_event(self.session, tenant_id="tenant-1", event_id=event.id) + self.session.commit() + client.fail_before_write = True + client.fetch_fails = True + + result = dispatch_calendar_outbox( + self.session, + tenant_id="tenant-1", + client_factory=lambda _session, _source: client, + ) + + self.assertEqual(result["retrying"], 1) + self.assertIsNotNone(event.deleted_at) + self.assertEqual((event.metadata_ or {})["caldav"]["external_state"], "retry") + + def test_old_source_delete_does_not_rebind_event_moved_to_local_calendar(self) -> None: + event = self.create_local_event() + local_calendar = create_calendar( + self.session, + tenant_id="tenant-1", + user_id=None, + payload=CalendarCollectionCreateRequest(name="Local"), + ) + self.session.commit() + client = StatefulCalDAVClient() + dispatch_calendar_outbox( + self.session, + tenant_id="tenant-1", + client_factory=lambda _session, _source: client, + ) + + update_event( + self.session, + tenant_id="tenant-1", + user_id=None, + event_id=event.id, + payload=CalendarEventUpdateRequest(calendar_id=local_calendar.id), + ) + self.session.commit() + dispatch_calendar_outbox( + self.session, + tenant_id="tenant-1", + client_factory=lambda _session, _source: client, + ) + + self.assertEqual(event.calendar_id, local_calendar.id) + self.assertEqual(event.source_kind, "local") + self.assertIsNone(event.source_href) + self.assertIsNone(event.etag) + + def test_resource_href_must_remain_in_configured_collection(self) -> None: + for href in ( + "https://evil.example.test/steal.ics", + "https://dav.example.test/other/steal.ics", + ): + with self.subTest(href=href), self.assertRaisesRegex( + CalendarError, + "origin|collection path", + ): + normalize_caldav_href(self.source.collection_url, href) + + self.assertEqual( + normalize_caldav_href(self.source.collection_url, "/cal/legit.ics"), + "https://dav.example.test/cal/legit.ics", + ) + with self.assertRaisesRegex(CalendarError, "sync-owned"): + self.create_local_event(source_href="/cal/legit.ics") + self.session.rollback() + self.assertEqual(self.session.query(CalendarOutboxOperation).count(), 0) + + def test_public_update_cannot_retarget_sync_owned_resource_fields(self) -> None: + event = self.create_local_event() + self.session.commit() + + with self.assertRaisesRegex(CalendarError, "sync-owned"): + update_event( + self.session, + tenant_id="tenant-1", + user_id=None, + event_id=event.id, + payload=CalendarEventUpdateRequest( + source_href="https://dav.example.test/cal/other.ics", + ), + ) + self.session.rollback() + self.assertEqual( + self.session.query(CalendarOutboxOperation).filter_by(status="pending").count(), + 1, + ) + + def test_public_ics_import_create_and_upsert_are_local_desired_mutations(self) -> None: + created = import_ics_event( + self.session, + tenant_id="tenant-1", + user_id=None, + calendar_id=self.calendar.id, + ics=imported_event_ics("Imported first"), + upsert=True, + source_kind="ical", + source_href="https://untrusted.example.test/chosen.ics", + etag='"untrusted"', + ) + self.session.commit() + first_operation = self.session.query(CalendarOutboxOperation).one() + + self.assertEqual(created.source_kind, "caldav") + self.assertTrue((created.source_href or "").startswith(self.source.collection_url)) + self.assertNotEqual( + created.source_href, + "https://untrusted.example.test/chosen.ics", + ) + self.assertIsNone(created.etag) + self.assertEqual(first_operation.status, "pending") + self.assertIn("SUMMARY:Imported first", first_operation.payload_ics or "") + + updated = import_ics_event( + self.session, + tenant_id="tenant-1", + user_id=None, + calendar_id=self.calendar.id, + ics=imported_event_ics("Imported updated"), + upsert=True, + source_kind="ical", + source_href="https://untrusted.example.test/other.ics", + etag='"other-untrusted"', + ) + self.session.commit() + latest_operation = ( + self.session.query(CalendarOutboxOperation) + .order_by(CalendarOutboxOperation.created_at.desc()) + .first() + ) + + assert latest_operation is not None + self.assertEqual(updated.id, created.id) + self.assertEqual(updated.summary, "Imported updated") + self.assertEqual(updated.source_href, created.source_href) + self.assertEqual(first_operation.status, "superseded") + self.assertEqual(latest_operation.status, "pending") + self.assertIn("SUMMARY:Imported updated", latest_operation.payload_ics or "") + + def test_outbox_projection_emits_followup_calendar_delta(self) -> None: + event = self.create_local_event() + self.session.commit() + initial_change = ( + self.session.query(ChangeSequenceEntry) + .filter_by(resource_id=event.id) + .order_by(ChangeSequenceEntry.id.desc()) + .first() + ) + assert initial_change is not None + self.assertEqual(initial_change.payload["external_state"], "queued") + client = StatefulCalDAVClient() + + dispatch_calendar_outbox( + self.session, + tenant_id="tenant-1", + client_factory=lambda _session, _source: client, + ) + projected_change = ( + self.session.query(ChangeSequenceEntry) + .filter( + ChangeSequenceEntry.resource_id == event.id, + ChangeSequenceEntry.id > initial_change.id, + ) + .order_by(ChangeSequenceEntry.id.desc()) + .first() + ) + + assert projected_change is not None + self.assertEqual(projected_change.operation, "updated") + self.assertEqual(projected_change.actor_type, "system") + self.assertEqual(projected_change.payload["previous_external_state"], "queued") + self.assertEqual(projected_change.payload["external_state"], "synced") + + def test_inbound_sync_does_not_overwrite_active_local_desired_state(self) -> None: + event = self.create_local_event(summary="Local desired") + self.session.commit() + report = CalDAVReportResult( + objects=[ + CalDAVObject( + href=event.source_href or "", + etag='"remote"', + calendar_data=different_event_ics(), + ) + ] + ) + stats = type("Stats", (), {"created": 0, "updated": 0, "deleted": 0, "unchanged": 0, "fetched": 0, "full_sync": False})() + apply_caldav_report( + self.session, + source=self.source, + client=StatefulCalDAVClient(), + report=report, + stats=stats, + user_id=None, + ) + + self.assertEqual(event.summary, "Local desired") + self.assertEqual(stats.unchanged, 1) + + def test_conflict_blocks_inbound_until_admin_discards_local_desired_state(self) -> None: + event = self.create_local_event(summary="Local desired") + self.session.commit() + client = StatefulCalDAVClient() + client.resources[event.source_href or ""] = (different_event_ics(), '"remote"') + dispatch_calendar_outbox( + self.session, + tenant_id="tenant-1", + client_factory=lambda _session, _source: client, + ) + operation = self.session.query(CalendarOutboxOperation).one() + self.assertEqual(operation.status, "conflict") + report = CalDAVReportResult( + objects=[ + CalDAVObject( + href=event.source_href or "", + etag='"remote"', + calendar_data=different_event_ics(), + ) + ] + ) + stats = type( + "Stats", + (), + { + "created": 0, + "updated": 0, + "deleted": 0, + "unchanged": 0, + "fetched": 0, + "full_sync": False, + }, + )() + apply_caldav_report( + self.session, + source=self.source, + client=client, + report=report, + stats=stats, + user_id=None, + ) + self.assertEqual(event.summary, "Local desired") + self.assertEqual(stats.unchanged, 1) + + self.source.sync_token = "incremental-token" + discard_calendar_outbox_operation( + self.session, + tenant_id="tenant-1", + operation_id=operation.id, + ) + self.session.commit() + self.assertEqual(operation.status, "cancelled") + self.assertEqual((event.metadata_ or {})["caldav"]["external_state"], "discarded") + self.assertIsNone(self.source.sync_token) + self.assertIsNotNone(self.source.next_sync_at) + + stats.unchanged = 0 + apply_caldav_report( + self.session, + source=self.source, + client=client, + report=report, + stats=stats, + user_id=None, + ) + self.assertEqual(event.summary, "Remote different") + + def test_newer_success_supersedes_old_conflict_and_unblocks_inbound(self) -> None: + event = self.create_local_event(summary="First local") + self.session.commit() + client = StatefulCalDAVClient() + client.resources[event.source_href or ""] = (different_event_ics(), '"remote"') + dispatch_calendar_outbox( + self.session, + tenant_id="tenant-1", + client_factory=lambda _session, _source: client, + ) + first = self.session.query(CalendarOutboxOperation).one() + self.assertEqual(first.status, "conflict") + + update_event( + self.session, + tenant_id="tenant-1", + user_id=None, + event_id=event.id, + payload=CalendarEventUpdateRequest(summary="Second local"), + ) + self.session.commit() + self.assertEqual(first.status, "superseded") + client.resources.clear() + dispatch_calendar_outbox( + self.session, + tenant_id="tenant-1", + client_factory=lambda _session, _source: client, + ) + latest = ( + self.session.query(CalendarOutboxOperation) + .order_by(CalendarOutboxOperation.created_at.desc()) + .first() + ) + assert latest is not None + self.assertEqual(latest.status, "succeeded") + + report = CalDAVReportResult( + objects=[ + CalDAVObject( + href=event.source_href or "", + etag='"remote-new"', + calendar_data=different_event_ics(), + ) + ] + ) + stats = type( + "Stats", + (), + { + "created": 0, + "updated": 0, + "deleted": 0, + "unchanged": 0, + "fetched": 0, + "full_sync": False, + }, + )() + apply_caldav_report( + self.session, + source=self.source, + client=client, + report=report, + stats=stats, + user_id=None, + ) + self.assertEqual(event.summary, "Remote different") + + +def different_event_ics() -> str: + return """BEGIN:VCALENDAR +VERSION:2.0 +BEGIN:VEVENT +UID:event-1@example.test +DTSTART:20260708T090000Z +DTEND:20260708T100000Z +SUMMARY:Remote different +END:VEVENT +END:VCALENDAR +""" + + +def imported_event_ics(summary: str) -> str: + return f"""BEGIN:VCALENDAR +VERSION:2.0 +BEGIN:VEVENT +UID:imported-event@example.test +DTSTART:20260710T090000Z +DTEND:20260710T100000Z +SUMMARY:{summary} +END:VEVENT +END:VCALENDAR +""" + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_service.py b/tests/test_service.py index 79684a3..9c3286e 100644 --- a/tests/test_service.py +++ b/tests/test_service.py @@ -6,7 +6,7 @@ from types import SimpleNamespace from unittest.mock import patch from govoplan_calendar.backend.schemas import CalendarCollectionDeleteRequest, CalendarEventResponse -from govoplan_calendar.backend.service import delete_calendar, event_response +from govoplan_calendar.backend.service import calendar_is_visible_to_principal, delete_calendar, event_response class FakeSession: @@ -65,6 +65,67 @@ class CalendarServiceResponseTests(unittest.TestCase): self.assertEqual(response.start_at.tzinfo, timezone.utc) +class CalendarVisibilityTests(unittest.TestCase): + @staticmethod + def calendar(**overrides): + values = { + "visibility": "private", + "owner_type": "user", + "owner_id": "owner-1", + "created_by_user_id": "creator-1", + } + values.update(overrides) + return SimpleNamespace(**values) + + def test_tenant_shared_and_public_calendars_are_visible_to_readers(self) -> None: + for visibility in ("tenant", "shared", "public"): + with self.subTest(visibility=visibility): + self.assertTrue( + calendar_is_visible_to_principal( + self.calendar(visibility=visibility), + user_id="reader-1", + ) + ) + + def test_private_calendar_is_visible_to_user_owner_creator_or_group_member(self) -> None: + self.assertTrue( + calendar_is_visible_to_principal( + self.calendar(), + user_id="owner-1", + ) + ) + self.assertTrue( + calendar_is_visible_to_principal( + self.calendar(), + user_id="creator-1", + ) + ) + self.assertTrue( + calendar_is_visible_to_principal( + self.calendar(owner_type="group", owner_id="group-1"), + user_id="member-1", + group_ids=("group-1",), + ) + ) + + def test_private_calendar_is_hidden_from_other_readers_but_visible_to_admin(self) -> None: + calendar = self.calendar() + self.assertFalse( + calendar_is_visible_to_principal( + calendar, + user_id="other-1", + group_ids=("group-2",), + ) + ) + self.assertTrue( + calendar_is_visible_to_principal( + calendar, + user_id="other-1", + can_admin=True, + ) + ) + + class CalendarServiceDeleteTests(unittest.TestCase): def test_delete_default_calendar_soft_deletes_calendar_and_events(self) -> None: session = FakeSession()