From 19e9096572ac386d7792dcc9a34cb78dde993780 Mon Sep 17 00:00:00 2001 From: Albrecht Degering Date: Sun, 2 Aug 2026 07:03:27 +0200 Subject: [PATCH] Implement address quality and reversible contact merges --- README.md | 30 +- docs/ADDRESS_MODULE_ARCHITECTURE.md | 30 +- docs/IMPLEMENTATION_PLAN.md | 26 +- docs/QUALITY_AND_MERGE.md | 57 + .../backend/capabilities.py | 115 +- src/govoplan_addresses/backend/db/models.py | 163 ++ src/govoplan_addresses/backend/manifest.py | 31 + ...b4c6d7e8f9a0_contact_quality_and_merges.py | 281 +++ src/govoplan_addresses/backend/router.py | 568 ++++++- src/govoplan_addresses/backend/schemas.py | 185 ++ src/govoplan_addresses/backend/service.py | 1503 ++++++++++++++++- tests/test_addresses_service.py | 429 ++++- webui/src/api/addresses.ts | 232 +++ .../features/addressbook/AddressBookPage.tsx | 626 ++++++- webui/src/styles/addresses.css | 124 ++ 15 files changed, 4369 insertions(+), 31 deletions(-) create mode 100644 docs/QUALITY_AND_MERGE.md create mode 100644 src/govoplan_addresses/backend/migrations/versions/b4c6d7e8f9a0_contact_quality_and_merges.py diff --git a/README.md b/README.md index 6e1eccd..1e82745 100644 --- a/README.md +++ b/README.md @@ -42,6 +42,16 @@ inspection UI are implemented. The conflict review UI compares stored local and remote field payloads, can apply a stored remote vCard payload, and supports manual per-field local/remote merge choices. +Address quality and duplicate handling are implemented as an operator workflow. +Contact points retain both their original and normalized values, field-level +provenance is append-only, and current quality states can mark a point valid, +invalid, returned, stale, or undeliverable. Those states flow into recipient +resolution with stable reason codes. The quality dialog shows bounded, +explainable duplicate suggestions and a correction queue. Merges record explicit +survivorship decisions, repair address-list memberships, preserve redirects for +stored contact references, and can be undone or split while the post-merge +evidence hash still matches. + API-managed CardDAV credentials are encrypted inside the source record. Source deletion physically removes that credential material and records a non-secret audit event in the same database transaction; destructive module retirement @@ -72,9 +82,9 @@ It must not own: effective function assignments - operational distribution lists/`Verteiler` with mixed recipient types -## First Capabilities +## Capabilities -The module exposes four core-mediated capabilities: +The module exposes core-mediated capabilities for: - `addresses.lookup`: read-only contact/recipient lookup for autocomplete. - `addresses.recipient_source`: immutable recipient snapshots for campaign, @@ -84,6 +94,10 @@ The module exposes four core-mediated capabilities: - `addresses.contact_point_resolution`: purpose-aware, channel-neutral resolution and immutable snapshots for email, postal, internal-mail, and portal targets. +- `addresses.people_search`: privacy-aware contact candidates for shared people + pickers. +- `distribution.recipient_channel_facts`: current channel, governance, and + quality facts for distribution and Policy consumers. `addresses.recipient_source` returns: @@ -131,8 +145,20 @@ The corresponding HTTP API is available below `/api/v1/addresses`: - `POST /contact-point-snapshots` - `GET /contact-point-snapshots/{snapshot_id}` +Quality, provenance, and reversible merge operations are available through: + +- `GET /address-books/{book_id}/quality-summary` +- `GET /address-books/{book_id}/duplicate-suggestions` +- `GET|POST /contacts/{contact_id}/quality-decisions` +- `GET /contacts/{contact_id}/provenance` +- `GET /contacts/{contact_id}/redirect` +- `GET|POST /contact-merges` +- `POST /contact-merges/{merge_id}/undo` +- `POST /contact-merges/{merge_id}/split` + ## Design Documents - [Address module architecture](docs/ADDRESS_MODULE_ARCHITECTURE.md) - [Implementation plan](docs/IMPLEMENTATION_PLAN.md) +- [Address quality and reversible merges](docs/QUALITY_AND_MERGE.md) - [AdreMa capability assessment and Distribution Lists roadmap](https://git.add-ideas.de/GovOPlaN/govoplan-dist-lists/src/branch/main/docs/ADREMA_CAPABILITY_ASSESSMENT.md) diff --git a/docs/ADDRESS_MODULE_ARCHITECTURE.md b/docs/ADDRESS_MODULE_ARCHITECTURE.md index 8195bbe..1d63664 100644 --- a/docs/ADDRESS_MODULE_ARCHITECTURE.md +++ b/docs/ADDRESS_MODULE_ARCHITECTURE.md @@ -70,7 +70,8 @@ representation for import/export and conflict handling. The local baseline implements scoped address books, contacts, normalized email/phone/postal-address tables, tags, source kind/reference fields, -first-class source payload/revision fields, and provenance JSON. Imported +first-class source payload/revision fields, preserved original contact-point +values, and append-only field provenance. Imported vCards preserve raw source payload and revision metadata for audit/debugging. Sync sources, attempt state, tombstones, conflicts, and diagnostics are now first-class backend tables and API resources. Connector-specific diffing, @@ -175,6 +176,32 @@ module retirement audits all remaining owned credential material before table removal. An unowned legacy reference is detached rather than passed to an external secret provider. +## Quality, Deduplication, And Recovery + +Quality is evidence about a concrete contact point, separate from communication +consent or Policy. Effective decisions use one of `valid`, `invalid`, +`returned`, `stale`, or `undeliverable`, retain reason/evidence references, and +end an overlapping prior decision rather than rewriting history. Recipient +capabilities project the current decision into a stable status and reason code; +consumers can exclude invalid points or explicitly handle stale points without +copying Addresses rules. + +Duplicate suggestions are bounded to 500 scanned contacts and 100 returned +pairs. Every score is composed from visible exact-match features such as a +normalized email, phone, postal address, or name/organization combination. A +suggestion does not mutate data. + +A merge is an explicit, transactional decision. The caller selects a surviving +contact, scalar-field sources, source precedence, and either union or +survivor-only contact-point handling. The merge records before/after evidence +and hashes, field/contact-point decisions, copied quality/governance evidence, +and stable loser-to-winner redirects. Address-list entries are repointed in the +same transaction. Undo and split restore the recorded contacts and memberships +only when the current evidence still matches the post-merge hash; later edits +must be reconciled first. Core change-sequence evidence is always written. Core +audit entries are written by HTTP mutation routes without requiring the +optional Audit module. + ## Connector Direction Implement connectors in this order: @@ -218,7 +245,6 @@ those provider-owned facts. The following are valuable but not required for the first functional milestone: -- automatic deduplication and merge suggestions - two-way sync conflict UI - Microsoft/Google connectors - richer vCard `KIND`/`RELATED` round-trip and provider-reference linking diff --git a/docs/IMPLEMENTATION_PLAN.md b/docs/IMPLEMENTATION_PLAN.md index 7f3550a..756bf47 100644 --- a/docs/IMPLEMENTATION_PLAN.md +++ b/docs/IMPLEMENTATION_PLAN.md @@ -219,17 +219,19 @@ Primary issues: `govoplan-addresses#8`, `govoplan-addresses#9`, Tasks: -- LDAP/Active Directory read-only directory connector -- Exchange/Microsoft 365 contacts connector -- Google Contacts connector -- CSV/XLSX/LDIF import mapping profiles -- classical address-list UI; reusable static/dynamic operational segments move +- [ ] LDAP/Active Directory read-only directory connector +- [ ] Exchange/Microsoft 365 contacts connector +- [ ] Google Contacts connector +- [ ] CSV/XLSX/LDIF import mapping profiles +- [x] classical address-list UI; reusable static/dynamic operational segments move to `govoplan-dist-lists` -- operational distribution lists move to `govoplan-dist-lists` -- consent, legal-basis, suppression, and communication preferences -- deduplication and merge workflow -- address quality checks and normalization -- richer vCard `KIND`/`RELATED` round-trip and stable links to IDM/Organizations +- [x] operational distribution lists move to `govoplan-dist-lists` +- [x] consent, legal-basis, suppression, and communication preferences +- [x] bounded, explainable deduplication and reversible merge/split workflow +- [x] contact-point quality states, normalization, original-value preservation, + field provenance, and correction dashboard +- [x] stable redirect resolution for merged contact references +- [ ] richer vCard `KIND`/`RELATED` round-trip and stable links to IDM/Organizations Exit criteria: @@ -237,6 +239,10 @@ Exit criteria: - users can understand where data came from and whether they may edit it - downstream modules can safely use contacts without owning them +Issues #9 and #10 are implemented. Issue #8 tracks the connector portfolio and +is split into independently deliverable connector/import follow-ups rather than +keeping one cross-protocol implementation ticket open. + ## First Implementation Recommendation Start with Milestone 1 and enough of Milestone 2 to define the data model diff --git a/docs/QUALITY_AND_MERGE.md b/docs/QUALITY_AND_MERGE.md new file mode 100644 index 0000000..09f0f23 --- /dev/null +++ b/docs/QUALITY_AND_MERGE.md @@ -0,0 +1,57 @@ +# Address Quality And Reversible Merges + +## Operator Workflow + +Open the shield action for a selected address book to review its quality. The +dialog shows: + +- the number of contacts and contact points in the bounded scan +- current invalid, returned, stale, and undeliverable contact points +- explainable duplicate suggestions with their score inputs +- active and recovered merge records + +Each contact point also has a `Quality` action in the contact detail. Recording +a new state ends an overlapping current state and retains both entries in +history. Use a stable reason code and an evidence reference when the state came +from delivery, import, or correction evidence. + +`valid` makes the point normally usable. `invalid`, `returned`, and +`undeliverable` make it invalid for recipient resolution. `stale` remains a +distinct status so a downstream workflow can warn, request confirmation, or +block according to Policy. A later `valid` decision is a correction; it does +not delete the earlier evidence. + +## Duplicate Review + +Suggestions do not merge automatically. The score is the bounded sum of named +exact-match features. The operator chooses the surviving contact and whether to +combine unique contact points or retain only the survivor's points. The API can +additionally select the source contact for each scalar field and rank source +kinds. + +A successful merge: + +- archives each duplicate and redirects its stable contact ID to the survivor +- records scalar and contact-point survivorship decisions +- carries field and contact-point source provenance forward +- copies applicable quality and communication-governance evidence +- repoints address-list entries to the survivor and mapped contact point +- stores deterministic before/after evidence hashes +- emits core change-sequence and audit evidence + +The merge history offers `Undo` and `Split`. Both restore the exact recorded +pre-merge contacts and list memberships. Recovery is deliberately rejected when +the contact or membership evidence changed after the merge. Reconcile those +later edits before retrying; the system does not silently discard them. + +## Consumer Contract + +Consumers resolve live contacts through `addresses.contact_point_resolution` or +`distribution.recipient_channel_facts`. They receive quality status, stable +reason codes, evidence provenance, and the current source revision. Consumers +must not read Addresses tables or recreate quality rules. A workflow requiring +historical proof freezes a contact-point snapshot before delivery. + +The duplicate and quality endpoints are bounded. `truncated=true` means the +operator should narrow the source or run a staged API review; it does not mean +that the unreturned contacts were found clean. diff --git a/src/govoplan_addresses/backend/capabilities.py b/src/govoplan_addresses/backend/capabilities.py index f06bffd..b61a835 100644 --- a/src/govoplan_addresses/backend/capabilities.py +++ b/src/govoplan_addresses/backend/capabilities.py @@ -41,6 +41,7 @@ from govoplan_addresses.backend.db.models import ( ContactChannelRule, ContactEmail, ContactPhone, + ContactPointQualityDecision, ContactPointSnapshot, ContactPostalAddress, ) @@ -48,6 +49,7 @@ from govoplan_addresses.backend.schemas import ContactCreateRequest from govoplan_addresses.backend.service import ( AddressBookError, create_contact, + current_contact_quality, get_visible_address_book, get_visible_address_list, get_visible_contact, @@ -55,6 +57,7 @@ from govoplan_addresses.backend.service import ( list_address_list_entries, list_address_lists, list_contacts, + resolve_contact_redirect, ) @@ -488,6 +491,7 @@ class AddressesChannelFactsCapability: and _aware_datetime(rule.effective_until) <= effective_at ] revision, fingerprint = _contact_channel_revision(contact) + quality = current_contact_quality(contact, effective_at=effective_at) source = DistributionSourceReference( provider="addresses", resource_type="contact", @@ -529,6 +533,20 @@ class AddressesChannelFactsCapability: else f"The {channel.replace('_', ' ')} contact point is incomplete or invalid." ) ) + quality_decision = quality.get((channel, point_id)) or quality.get( + (channel, None) + ) + if quality_decision is not None and quality_decision.state != "valid": + status = ( + "stale" + if quality_decision.state == "stale" + else "invalid" + ) + reason_code = quality_decision.reason_code + explanation = quality_decision.reason or ( + f"This {channel.replace('_', ' ')} contact point is marked " + f"{quality_decision.state}." + ) candidates.append( DistributionChannelCandidate( channel=channel, @@ -551,6 +569,17 @@ class AddressesChannelFactsCapability: "legal_basis": selected.legal_basis if selected is not None else None, "evidence_ref": selected.evidence_ref if selected is not None else None, "preference_rank": selected.preference_rank if selected is not None else None, + "quality_decision_id": ( + quality_decision.id if quality_decision is not None else None + ), + "quality_state": ( + quality_decision.state if quality_decision is not None else "valid" + ), + "quality_evidence_ref": ( + quality_decision.evidence_ref + if quality_decision is not None + else None + ), }, ) ) @@ -1178,7 +1207,28 @@ def _address_book_contact_updated_at(session: Any, address_book_ids: list[str]) .group_by(Contact.address_book_id) .all() ) - for address_book_id, updated_at in [*contact_rows, *deletion_rows, *email_rows, *phone_rows, *postal_rows, *rule_rows]: + quality_rows = ( + session.query( + Contact.address_book_id, + func.max(ContactPointQualityDecision.updated_at), + ) + .join(Contact, ContactPointQualityDecision.contact_id == Contact.id) + .filter( + Contact.address_book_id.in_(address_book_ids), + Contact.deleted_at.is_(None), + ) + .group_by(Contact.address_book_id) + .all() + ) + for address_book_id, updated_at in [ + *contact_rows, + *deletion_rows, + *email_rows, + *phone_rows, + *postal_rows, + *rule_rows, + *quality_rows, + ]: if updated_at is None: continue key = str(address_book_id) @@ -1221,6 +1271,16 @@ def _address_list_updated_at(session: Any, address_list_ids: list[str]) -> dict[ .group_by(AddressListEntry.address_list_id) .all() ) + quality_rows = ( + session.query( + AddressListEntry.address_list_id, + func.max(ContactPointQualityDecision.updated_at), + ) + .join(ContactPointQualityDecision, AddressListEntry.contact_id == ContactPointQualityDecision.contact_id) + .filter(AddressListEntry.address_list_id.in_(address_list_ids)) + .group_by(AddressListEntry.address_list_id) + .all() + ) postal_rows = ( session.query( AddressListEntry.address_list_id, @@ -1241,6 +1301,7 @@ def _address_list_updated_at(session: Any, address_list_ids: list[str]) -> dict[ *email_rows, *postal_rows, *rule_rows, + *quality_rows, ]: if updated_at is None: continue @@ -1359,7 +1420,15 @@ def _channel_rule_explanation(rule: ContactChannelRule) -> str: def _contact_channel_revision(contact: Contact) -> tuple[str, str]: stamps = [contact.updated_at] - stamps.extend(item.updated_at for item in (*contact.emails, *contact.postal_addresses, *contact.channel_rules)) + stamps.extend( + item.updated_at + for item in ( + *contact.emails, + *contact.postal_addresses, + *contact.channel_rules, + *contact.quality_decisions, + ) + ) revision = max(_aware_datetime(item) for item in stamps).isoformat() payload = { "contact_id": contact.id, @@ -1397,6 +1466,20 @@ def _contact_channel_revision(contact: Contact) -> tuple[str, str]: } for rule in sorted(contact.channel_rules, key=lambda item: item.id) ], + "quality": [ + { + "id": item.id, + "channel": item.channel, + "point": item.contact_point_id, + "state": item.state, + "reason_code": item.reason_code, + "evidence": item.evidence_ref, + "from": item.effective_from.isoformat(), + "until": item.effective_until.isoformat() if item.effective_until else None, + "updated": item.updated_at.isoformat(), + } + for item in sorted(contact.quality_decisions, key=lambda row: row.id) + ], } fingerprint = hashlib.sha256( json.dumps(payload, sort_keys=True, separators=(",", ":")).encode("utf-8") @@ -1426,7 +1509,21 @@ def _contacts_for_subject( try: return [get_visible_contact(session, principal, str(direct_id))] except AddressBookError: - return [] + try: + resolution = resolve_contact_redirect( + session, + principal, + str(direct_id), + ) + return [ + get_visible_contact( + session, + principal, + resolution.resolved_contact_id, + ) + ] + except AddressBookError: + return [] explicit_ref = subject.metadata.get("source_ref") source_refs = { @@ -1445,16 +1542,24 @@ def _contacts_for_subject( .filter( Contact.source_ref.in_(source_refs), or_(Contact.tenant_id == principal.tenant_id, Contact.tenant_id.is_(None)), - Contact.deleted_at.is_(None), ) .order_by(Contact.id.asc()) .limit(3) .all() ) visible: list[Contact] = [] + visible_ids: set[str] = set() for row in rows: try: - visible.append(get_visible_contact(session, principal, row.id)) + resolution = resolve_contact_redirect(session, principal, row.id) + contact = get_visible_contact( + session, + principal, + resolution.resolved_contact_id, + ) + if contact.id not in visible_ids: + visible.append(contact) + visible_ids.add(contact.id) except AddressBookError: continue return visible diff --git a/src/govoplan_addresses/backend/db/models.py b/src/govoplan_addresses/backend/db/models.py index 30b53cb..95ba733 100644 --- a/src/govoplan_addresses/backend/db/models.py +++ b/src/govoplan_addresses/backend/db/models.py @@ -102,6 +102,16 @@ class Contact(Base, TimestampMixin): cascade="all, delete-orphan", order_by="ContactChannelRule.created_at", ) + quality_decisions: Mapped[list["ContactPointQualityDecision"]] = relationship( + back_populates="contact", + cascade="all, delete-orphan", + order_by="ContactPointQualityDecision.created_at", + ) + field_provenance: Mapped[list["ContactFieldProvenance"]] = relationship( + back_populates="contact", + cascade="all, delete-orphan", + order_by="ContactFieldProvenance.created_at", + ) class ContactEmail(Base, TimestampMixin): @@ -115,6 +125,9 @@ class ContactEmail(Base, TimestampMixin): contact_id: Mapped[str] = mapped_column(ForeignKey("addresses_contacts.id", ondelete="CASCADE"), nullable=False, index=True) label: Mapped[str | None] = mapped_column(String(80)) email: Mapped[str] = mapped_column(String(320), nullable=False, index=True) + original_email: Mapped[str] = mapped_column(String(320), nullable=False, default="") + normalized_email: Mapped[str] = mapped_column(String(320), nullable=False, default="", index=True) + provenance: Mapped[dict[str, Any]] = mapped_column(JSON, default=dict, nullable=False) is_primary: Mapped[bool] = mapped_column(Boolean, default=False, nullable=False) order_index: Mapped[int] = mapped_column(Integer, default=0, nullable=False) @@ -130,6 +143,9 @@ class ContactPhone(Base, TimestampMixin): contact_id: Mapped[str] = mapped_column(ForeignKey("addresses_contacts.id", ondelete="CASCADE"), nullable=False, index=True) label: Mapped[str | None] = mapped_column(String(80)) phone: Mapped[str] = mapped_column(String(100), nullable=False) + original_phone: Mapped[str] = mapped_column(String(100), nullable=False, default="") + normalized_phone: Mapped[str] = mapped_column(String(100), nullable=False, default="", index=True) + provenance: Mapped[dict[str, Any]] = mapped_column(JSON, default=dict, nullable=False) is_primary: Mapped[bool] = mapped_column(Boolean, default=False, nullable=False) order_index: Mapped[int] = mapped_column(Integer, default=0, nullable=False) @@ -148,6 +164,9 @@ class ContactPostalAddress(Base, TimestampMixin): locality: Mapped[str | None] = mapped_column(String(255)) region: Mapped[str | None] = mapped_column(String(255)) country: Mapped[str | None] = mapped_column(String(255)) + original_value: Mapped[dict[str, Any]] = mapped_column(JSON, default=dict, nullable=False) + normalized_value: Mapped[dict[str, Any]] = mapped_column(JSON, default=dict, nullable=False) + provenance: Mapped[dict[str, Any]] = mapped_column(JSON, default=dict, nullable=False) is_primary: Mapped[bool] = mapped_column(Boolean, default=False, nullable=False) order_index: Mapped[int] = mapped_column(Integer, default=0, nullable=False) @@ -221,6 +240,146 @@ class ContactPointSnapshot(Base, TimestampMixin): provenance: Mapped[dict[str, Any]] = mapped_column(JSON, default=dict, nullable=False) +class ContactPointQualityDecision(Base, TimestampMixin): + __tablename__ = "addresses_contact_point_quality_decisions" + __table_args__ = ( + Index( + "ix_addresses_quality_current", + "tenant_id", + "contact_id", + "channel", + "contact_point_id", + "effective_until", + ), + Index("ix_addresses_quality_state", "tenant_id", "state", "effective_until"), + ) + + id: Mapped[str] = mapped_column(String(36), primary_key=True, default=new_uuid) + tenant_id: Mapped[str | None] = mapped_column(String(36), nullable=True, index=True) + contact_id: Mapped[str] = mapped_column( + ForeignKey("addresses_contacts.id", ondelete="CASCADE"), + nullable=False, + index=True, + ) + channel: Mapped[str] = mapped_column(String(30), nullable=False, index=True) + contact_point_id: Mapped[str | None] = mapped_column(String(36), nullable=True, index=True) + state: Mapped[str] = mapped_column(String(30), nullable=False, index=True) + reason_code: Mapped[str] = mapped_column(String(120), nullable=False) + reason: Mapped[str | None] = mapped_column(Text, nullable=True) + evidence_ref: Mapped[str | None] = mapped_column(String(1000), nullable=True) + effective_from: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False, index=True) + effective_until: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True, index=True) + created_by_account_id: Mapped[str | None] = mapped_column(String(36), nullable=True, index=True) + metadata_: Mapped[dict[str, Any]] = mapped_column("metadata", JSON, default=dict, nullable=False) + + contact: Mapped[Contact] = relationship(back_populates="quality_decisions") + + +class ContactMergeRecord(Base, TimestampMixin): + __tablename__ = "addresses_contact_merge_records" + __table_args__ = ( + Index("ix_addresses_merge_winner", "tenant_id", "winner_contact_id", "created_at"), + Index("ix_addresses_merge_status", "tenant_id", "status", "created_at"), + ) + + id: Mapped[str] = mapped_column(String(36), primary_key=True, default=new_uuid) + tenant_id: Mapped[str | None] = mapped_column(String(36), nullable=True, index=True) + address_book_id: Mapped[str] = mapped_column( + ForeignKey("addresses_address_books.id", ondelete="CASCADE"), + nullable=False, + index=True, + ) + winner_contact_id: Mapped[str] = mapped_column( + ForeignKey("addresses_contacts.id", ondelete="RESTRICT"), + nullable=False, + index=True, + ) + loser_contact_ids: Mapped[list[str]] = mapped_column(JSON, nullable=False) + status: Mapped[str] = mapped_column(String(30), nullable=False, default="active", index=True) + reason: Mapped[str] = mapped_column(Text, nullable=False) + survivorship: Mapped[dict[str, Any]] = mapped_column(JSON, default=dict, nullable=False) + decisions: Mapped[list[dict[str, Any]]] = mapped_column(JSON, default=list, nullable=False) + before_payload: Mapped[dict[str, Any]] = mapped_column(JSON, nullable=False) + after_payload: Mapped[dict[str, Any]] = mapped_column(JSON, nullable=False) + before_hash: Mapped[str] = mapped_column(String(64), nullable=False) + after_hash: Mapped[str] = mapped_column(String(64), nullable=False) + created_by_account_id: Mapped[str | None] = mapped_column(String(36), nullable=True, index=True) + recovered_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True) + recovered_by_account_id: Mapped[str | None] = mapped_column(String(36), nullable=True) + recovery_action: Mapped[str | None] = mapped_column(String(30), nullable=True) + recovery_reason: Mapped[str | None] = mapped_column(Text, nullable=True) + provenance: Mapped[dict[str, Any]] = mapped_column(JSON, default=dict, nullable=False) + + +class ContactRedirect(Base, TimestampMixin): + __tablename__ = "addresses_contact_redirects" + __table_args__ = ( + Index( + "uq_addresses_contact_redirects_active_source", + "tenant_id", + "source_contact_id", + unique=True, + sqlite_where=text("ended_at IS NULL"), + postgresql_where=text("ended_at IS NULL"), + ), + Index("ix_addresses_contact_redirects_target", "tenant_id", "target_contact_id", "ended_at"), + ) + + id: Mapped[str] = mapped_column(String(36), primary_key=True, default=new_uuid) + tenant_id: Mapped[str | None] = mapped_column(String(36), nullable=True, index=True) + source_contact_id: Mapped[str] = mapped_column( + ForeignKey("addresses_contacts.id", ondelete="CASCADE"), + nullable=False, + index=True, + ) + target_contact_id: Mapped[str] = mapped_column( + ForeignKey("addresses_contacts.id", ondelete="RESTRICT"), + nullable=False, + index=True, + ) + merge_record_id: Mapped[str] = mapped_column( + ForeignKey("addresses_contact_merge_records.id", ondelete="CASCADE"), + nullable=False, + index=True, + ) + ended_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True, index=True) + + +class ContactFieldProvenance(Base, TimestampMixin): + __tablename__ = "addresses_contact_field_provenance" + __table_args__ = ( + Index("ix_addresses_field_provenance_contact", "contact_id", "field_path", "created_at"), + Index("ix_addresses_field_provenance_selected", "tenant_id", "contact_id", "selected"), + ) + + id: Mapped[str] = mapped_column(String(36), primary_key=True, default=new_uuid) + tenant_id: Mapped[str | None] = mapped_column(String(36), nullable=True, index=True) + contact_id: Mapped[str] = mapped_column( + ForeignKey("addresses_contacts.id", ondelete="CASCADE"), + nullable=False, + index=True, + ) + field_path: Mapped[str] = mapped_column(String(255), nullable=False, index=True) + value: Mapped[Any] = mapped_column(JSON, nullable=True) + source_kind: Mapped[str] = mapped_column(String(40), nullable=False) + source_ref: Mapped[str | None] = mapped_column(String(1000), nullable=True) + source_revision: Mapped[str | None] = mapped_column(String(255), nullable=True) + precedence: Mapped[int] = mapped_column(Integer, nullable=False, default=0) + selected: Mapped[bool] = mapped_column(Boolean, nullable=False, default=True, index=True) + reason_code: Mapped[str] = mapped_column(String(120), nullable=False) + explanation: Mapped[str | None] = mapped_column(Text, nullable=True) + visibility: Mapped[str] = mapped_column(String(30), nullable=False, default="inherit") + merge_record_id: Mapped[str | None] = mapped_column( + ForeignKey("addresses_contact_merge_records.id", ondelete="SET NULL"), + nullable=True, + index=True, + ) + created_by_account_id: Mapped[str | None] = mapped_column(String(36), nullable=True, index=True) + metadata_: Mapped[dict[str, Any]] = mapped_column("metadata", JSON, default=dict, nullable=False) + + contact: Mapped[Contact] = relationship(back_populates="field_provenance") + + class AddressList(Base, TimestampMixin): __tablename__ = "addresses_address_lists" __table_args__ = ( @@ -401,7 +560,11 @@ __all__ = [ "Contact", "ContactEmail", "ContactPhone", + "ContactFieldProvenance", + "ContactMergeRecord", + "ContactPointQualityDecision", "ContactPointSnapshot", "ContactPostalAddress", + "ContactRedirect", "new_uuid", ] diff --git a/src/govoplan_addresses/backend/manifest.py b/src/govoplan_addresses/backend/manifest.py index e7b3c53..3ecdd8d 100644 --- a/src/govoplan_addresses/backend/manifest.py +++ b/src/govoplan_addresses/backend/manifest.py @@ -43,6 +43,10 @@ from govoplan_addresses.backend.provider_state import ( _addresses_table_retirement_provider = drop_table_retirement_provider( + addresses_models.ContactFieldProvenance, + addresses_models.ContactRedirect, + addresses_models.ContactMergeRecord, + addresses_models.ContactPointQualityDecision, addresses_models.ContactPointSnapshot, addresses_models.AddressSyncDiagnostic, addresses_models.AddressSyncConflict, @@ -154,6 +158,8 @@ def _tenant_summary(session, tenant_id: str) -> dict[str, int]: AddressList, AddressSyncSource, Contact, + ContactMergeRecord, + ContactPointQualityDecision, ContactPointSnapshot, ) @@ -161,6 +167,8 @@ def _tenant_summary(session, tenant_id: str) -> dict[str, int]: "address_books": session.query(AddressBook).filter(AddressBook.tenant_id == tenant_id, AddressBook.deleted_at.is_(None)).count(), "address_lists": session.query(AddressList).filter(AddressList.tenant_id == tenant_id, AddressList.deleted_at.is_(None)).count(), "contacts": session.query(Contact).filter(Contact.tenant_id == tenant_id, Contact.deleted_at.is_(None)).count(), + "active_contact_merges": session.query(ContactMergeRecord).filter(ContactMergeRecord.tenant_id == tenant_id, ContactMergeRecord.status == "active").count(), + "contact_quality_decisions": session.query(ContactPointQualityDecision).filter(ContactPointQualityDecision.tenant_id == tenant_id).count(), "contact_point_snapshots": session.query(ContactPointSnapshot).filter(ContactPointSnapshot.tenant_id == tenant_id).count(), "sync_sources": session.query(AddressSyncSource).filter(AddressSyncSource.tenant_id == tenant_id, AddressSyncSource.enabled.is_(True)).count(), } @@ -286,6 +294,10 @@ manifest = ModuleManifest( addresses_models.AddressSyncSource, addresses_models.AddressListEntry, addresses_models.AddressList, + addresses_models.ContactFieldProvenance, + addresses_models.ContactRedirect, + addresses_models.ContactMergeRecord, + addresses_models.ContactPointQualityDecision, addresses_models.ContactPointSnapshot, addresses_models.AddressBook, addresses_models.Contact, @@ -328,6 +340,25 @@ manifest = ModuleManifest( related_modules=("dist_lists", "campaigns", "policy", "templates"), order=31, ), + DocumentationTopic( + id="addresses.quality-and-merge", + title="Contact quality, duplicates, and reversible merges", + summary="Review address quality and duplicate suggestions without losing source evidence.", + body=( + "Addresses preserves original and normalized contact-point values, records field-level provenance, " + "and projects invalid, returned, stale, or undeliverable states into recipient resolution with stable " + "reason codes. Duplicate suggestions are bounded and explain their matching features. An operator can " + "choose the surviving values, merge contact points, and later undo or split the merge while the recorded " + "post-merge evidence still matches. Contact redirects keep stored references resolvable, and address-list " + "memberships are repaired transactionally. Audit remains an optional integration; the Addresses change " + "sequence and merge evidence are always retained." + ), + layer="configured", + documentation_types=("admin", "user"), + audience=("tenant_admin", "operator", "module_admin"), + related_modules=("campaigns", "dist_lists", "policy", "audit"), + order=32, + ), ), external_providers=(CARDDAV_PROVIDER,), external_provider_state_providers=( diff --git a/src/govoplan_addresses/backend/migrations/versions/b4c6d7e8f9a0_contact_quality_and_merges.py b/src/govoplan_addresses/backend/migrations/versions/b4c6d7e8f9a0_contact_quality_and_merges.py new file mode 100644 index 0000000..19d1eb6 --- /dev/null +++ b/src/govoplan_addresses/backend/migrations/versions/b4c6d7e8f9a0_contact_quality_and_merges.py @@ -0,0 +1,281 @@ +"""Add address quality, provenance, merge evidence, and redirects. + +Revision ID: b4c6d7e8f9a0 +Revises: a3b5c6d7e8f9 +""" + +from __future__ import annotations + +import re + +from alembic import op +import sqlalchemy as sa + + +revision = "b4c6d7e8f9a0" +down_revision = "a3b5c6d7e8f9" +branch_labels = None +depends_on = None + +_JSON_OBJECT = sa.text("'{}'") + + +def upgrade() -> None: + with op.batch_alter_table("addresses_contact_emails") as batch: + batch.add_column(sa.Column("original_email", sa.String(length=320), nullable=False, server_default="")) + batch.add_column(sa.Column("normalized_email", sa.String(length=320), nullable=False, server_default="")) + batch.add_column(sa.Column("provenance", sa.JSON(), nullable=False, server_default=_JSON_OBJECT)) + with op.batch_alter_table("addresses_contact_phones") as batch: + batch.add_column(sa.Column("original_phone", sa.String(length=100), nullable=False, server_default="")) + batch.add_column(sa.Column("normalized_phone", sa.String(length=100), nullable=False, server_default="")) + batch.add_column(sa.Column("provenance", sa.JSON(), nullable=False, server_default=_JSON_OBJECT)) + with op.batch_alter_table("addresses_contact_postal_addresses") as batch: + batch.add_column(sa.Column("original_value", sa.JSON(), nullable=False, server_default=_JSON_OBJECT)) + batch.add_column(sa.Column("normalized_value", sa.JSON(), nullable=False, server_default=_JSON_OBJECT)) + batch.add_column(sa.Column("provenance", sa.JSON(), nullable=False, server_default=_JSON_OBJECT)) + + bind = op.get_bind() + bind.execute( + sa.text( + "UPDATE addresses_contact_emails " + "SET original_email = email, normalized_email = lower(trim(email))" + ) + ) + phone_rows = bind.execute( + sa.text("SELECT id, phone FROM addresses_contact_phones") + ).mappings().all() + for row in phone_rows: + bind.execute( + sa.text( + "UPDATE addresses_contact_phones " + "SET original_phone = :original, normalized_phone = :normalized " + "WHERE id = :id" + ), + { + "id": row["id"], + "original": row["phone"], + "normalized": _normalized_phone(str(row["phone"] or "")), + }, + ) + postal = sa.table( + "addresses_contact_postal_addresses", + sa.column("id", sa.String()), + sa.column("label", sa.String()), + sa.column("street", sa.String()), + sa.column("postal_code", sa.String()), + sa.column("locality", sa.String()), + sa.column("region", sa.String()), + sa.column("country", sa.String()), + sa.column("original_value", sa.JSON()), + sa.column("normalized_value", sa.JSON()), + ) + postal_rows = bind.execute( + sa.select( + postal.c.id, + postal.c.label, + postal.c.street, + postal.c.postal_code, + postal.c.locality, + postal.c.region, + postal.c.country, + ) + ).mappings().all() + for row in postal_rows: + original = { + key: row[key] + for key in ("label", "street", "postal_code", "locality", "region", "country") + } + normalized = { + key: _normalized_text(row[key]) + for key in ("label", "street", "postal_code", "locality", "region", "country") + } + bind.execute( + postal.update() + .where(postal.c.id == row["id"]) + .values(original_value=original, normalized_value=normalized) + ) + + op.create_index( + "ix_addresses_contact_emails_normalized_email", + "addresses_contact_emails", + ["normalized_email"], + ) + op.create_index( + "ix_addresses_contact_phones_normalized_phone", + "addresses_contact_phones", + ["normalized_phone"], + ) + + op.create_table( + "addresses_contact_point_quality_decisions", + sa.Column("id", sa.String(length=36), nullable=False), + sa.Column("tenant_id", sa.String(length=36), nullable=True), + sa.Column("contact_id", sa.String(length=36), nullable=False), + sa.Column("channel", sa.String(length=30), nullable=False), + sa.Column("contact_point_id", sa.String(length=36), nullable=True), + sa.Column("state", sa.String(length=30), nullable=False), + sa.Column("reason_code", sa.String(length=120), nullable=False), + sa.Column("reason", sa.Text(), nullable=True), + sa.Column("evidence_ref", sa.String(length=1000), nullable=True), + sa.Column("effective_from", sa.DateTime(timezone=True), nullable=False), + sa.Column("effective_until", sa.DateTime(timezone=True), nullable=True), + sa.Column("created_by_account_id", sa.String(length=36), nullable=True), + sa.Column("metadata", sa.JSON(), nullable=False, server_default=_JSON_OBJECT), + sa.Column("created_at", sa.DateTime(timezone=True), nullable=False), + sa.Column("updated_at", sa.DateTime(timezone=True), nullable=False), + sa.ForeignKeyConstraint(["contact_id"], ["addresses_contacts.id"], ondelete="CASCADE"), + sa.PrimaryKeyConstraint("id"), + ) + for name, columns in ( + ("ix_addresses_contact_point_quality_decisions_tenant_id", ["tenant_id"]), + ("ix_addresses_contact_point_quality_decisions_contact_id", ["contact_id"]), + ("ix_addresses_contact_point_quality_decisions_channel", ["channel"]), + ("ix_addresses_contact_point_quality_decisions_contact_point_id", ["contact_point_id"]), + ("ix_addresses_contact_point_quality_decisions_state", ["state"]), + ("ix_addresses_contact_point_quality_decisions_effective_from", ["effective_from"]), + ("ix_addresses_contact_point_quality_decisions_effective_until", ["effective_until"]), + ("ix_addresses_contact_point_quality_decisions_created_by_account_id", ["created_by_account_id"]), + ("ix_addresses_quality_current", ["tenant_id", "contact_id", "channel", "contact_point_id", "effective_until"]), + ("ix_addresses_quality_state", ["tenant_id", "state", "effective_until"]), + ): + op.create_index(name, "addresses_contact_point_quality_decisions", columns) + + op.create_table( + "addresses_contact_merge_records", + sa.Column("id", sa.String(length=36), nullable=False), + sa.Column("tenant_id", sa.String(length=36), nullable=True), + sa.Column("address_book_id", sa.String(length=36), nullable=False), + sa.Column("winner_contact_id", sa.String(length=36), nullable=False), + sa.Column("loser_contact_ids", sa.JSON(), nullable=False), + sa.Column("status", sa.String(length=30), nullable=False), + sa.Column("reason", sa.Text(), nullable=False), + sa.Column("survivorship", sa.JSON(), nullable=False, server_default=_JSON_OBJECT), + sa.Column("decisions", sa.JSON(), nullable=False, server_default="[]"), + sa.Column("before_payload", sa.JSON(), nullable=False), + sa.Column("after_payload", sa.JSON(), nullable=False), + sa.Column("before_hash", sa.String(length=64), nullable=False), + sa.Column("after_hash", sa.String(length=64), nullable=False), + sa.Column("created_by_account_id", sa.String(length=36), nullable=True), + sa.Column("recovered_at", sa.DateTime(timezone=True), nullable=True), + sa.Column("recovered_by_account_id", sa.String(length=36), nullable=True), + sa.Column("recovery_action", sa.String(length=30), nullable=True), + sa.Column("recovery_reason", sa.Text(), nullable=True), + sa.Column("provenance", sa.JSON(), nullable=False, server_default=_JSON_OBJECT), + sa.Column("created_at", sa.DateTime(timezone=True), nullable=False), + sa.Column("updated_at", sa.DateTime(timezone=True), nullable=False), + sa.ForeignKeyConstraint(["address_book_id"], ["addresses_address_books.id"], ondelete="CASCADE"), + sa.ForeignKeyConstraint(["winner_contact_id"], ["addresses_contacts.id"], ondelete="RESTRICT"), + sa.PrimaryKeyConstraint("id"), + ) + for name, columns in ( + ("ix_addresses_contact_merge_records_tenant_id", ["tenant_id"]), + ("ix_addresses_contact_merge_records_address_book_id", ["address_book_id"]), + ("ix_addresses_contact_merge_records_winner_contact_id", ["winner_contact_id"]), + ("ix_addresses_contact_merge_records_status", ["status"]), + ("ix_addresses_contact_merge_records_created_by_account_id", ["created_by_account_id"]), + ("ix_addresses_merge_winner", ["tenant_id", "winner_contact_id", "created_at"]), + ("ix_addresses_merge_status", ["tenant_id", "status", "created_at"]), + ): + op.create_index(name, "addresses_contact_merge_records", columns) + + op.create_table( + "addresses_contact_redirects", + sa.Column("id", sa.String(length=36), nullable=False), + sa.Column("tenant_id", sa.String(length=36), nullable=True), + sa.Column("source_contact_id", sa.String(length=36), nullable=False), + sa.Column("target_contact_id", sa.String(length=36), nullable=False), + sa.Column("merge_record_id", sa.String(length=36), nullable=False), + sa.Column("ended_at", sa.DateTime(timezone=True), nullable=True), + sa.Column("created_at", sa.DateTime(timezone=True), nullable=False), + sa.Column("updated_at", sa.DateTime(timezone=True), nullable=False), + sa.ForeignKeyConstraint(["merge_record_id"], ["addresses_contact_merge_records.id"], ondelete="CASCADE"), + sa.ForeignKeyConstraint(["source_contact_id"], ["addresses_contacts.id"], ondelete="CASCADE"), + sa.ForeignKeyConstraint(["target_contact_id"], ["addresses_contacts.id"], ondelete="RESTRICT"), + sa.PrimaryKeyConstraint("id"), + ) + for name, columns in ( + ("ix_addresses_contact_redirects_tenant_id", ["tenant_id"]), + ("ix_addresses_contact_redirects_source_contact_id", ["source_contact_id"]), + ("ix_addresses_contact_redirects_target_contact_id", ["target_contact_id"]), + ("ix_addresses_contact_redirects_merge_record_id", ["merge_record_id"]), + ("ix_addresses_contact_redirects_ended_at", ["ended_at"]), + ("ix_addresses_contact_redirects_target", ["tenant_id", "target_contact_id", "ended_at"]), + ): + op.create_index(name, "addresses_contact_redirects", columns) + op.create_index( + "uq_addresses_contact_redirects_active_source", + "addresses_contact_redirects", + ["tenant_id", "source_contact_id"], + unique=True, + sqlite_where=sa.text("ended_at IS NULL"), + postgresql_where=sa.text("ended_at IS NULL"), + ) + + op.create_table( + "addresses_contact_field_provenance", + sa.Column("id", sa.String(length=36), nullable=False), + sa.Column("tenant_id", sa.String(length=36), nullable=True), + sa.Column("contact_id", sa.String(length=36), nullable=False), + sa.Column("field_path", sa.String(length=255), nullable=False), + sa.Column("value", sa.JSON(), nullable=True), + sa.Column("source_kind", sa.String(length=40), nullable=False), + sa.Column("source_ref", sa.String(length=1000), nullable=True), + sa.Column("source_revision", sa.String(length=255), nullable=True), + sa.Column("precedence", sa.Integer(), nullable=False), + sa.Column("selected", sa.Boolean(), nullable=False), + sa.Column("reason_code", sa.String(length=120), nullable=False), + sa.Column("explanation", sa.Text(), nullable=True), + sa.Column("visibility", sa.String(length=30), nullable=False), + sa.Column("merge_record_id", sa.String(length=36), nullable=True), + sa.Column("created_by_account_id", sa.String(length=36), nullable=True), + sa.Column("metadata", sa.JSON(), nullable=False, server_default=_JSON_OBJECT), + sa.Column("created_at", sa.DateTime(timezone=True), nullable=False), + sa.Column("updated_at", sa.DateTime(timezone=True), nullable=False), + sa.ForeignKeyConstraint(["contact_id"], ["addresses_contacts.id"], ondelete="CASCADE"), + sa.ForeignKeyConstraint(["merge_record_id"], ["addresses_contact_merge_records.id"], ondelete="SET NULL"), + sa.PrimaryKeyConstraint("id"), + ) + for name, columns in ( + ("ix_addresses_contact_field_provenance_tenant_id", ["tenant_id"]), + ("ix_addresses_contact_field_provenance_contact_id", ["contact_id"]), + ("ix_addresses_contact_field_provenance_field_path", ["field_path"]), + ("ix_addresses_contact_field_provenance_selected", ["selected"]), + ("ix_addresses_contact_field_provenance_merge_record_id", ["merge_record_id"]), + ("ix_addresses_contact_field_provenance_created_by_account_id", ["created_by_account_id"]), + ("ix_addresses_field_provenance_contact", ["contact_id", "field_path", "created_at"]), + ("ix_addresses_field_provenance_selected", ["tenant_id", "contact_id", "selected"]), + ): + op.create_index(name, "addresses_contact_field_provenance", columns) + + +def downgrade() -> None: + op.drop_table("addresses_contact_field_provenance") + op.drop_table("addresses_contact_redirects") + op.drop_table("addresses_contact_merge_records") + op.drop_table("addresses_contact_point_quality_decisions") + with op.batch_alter_table("addresses_contact_postal_addresses") as batch: + batch.drop_column("provenance") + batch.drop_column("normalized_value") + batch.drop_column("original_value") + with op.batch_alter_table("addresses_contact_phones") as batch: + batch.drop_index("ix_addresses_contact_phones_normalized_phone") + batch.drop_column("provenance") + batch.drop_column("normalized_phone") + batch.drop_column("original_phone") + with op.batch_alter_table("addresses_contact_emails") as batch: + batch.drop_index("ix_addresses_contact_emails_normalized_email") + batch.drop_column("provenance") + batch.drop_column("normalized_email") + batch.drop_column("original_email") + + +def _normalized_text(value: object) -> str | None: + if value is None: + return None + normalized = " ".join(str(value).strip().casefold().split()) + return normalized or None + + +def _normalized_phone(value: str) -> str: + prefix = "+" if value.strip().startswith("+") else "" + return prefix + re.sub(r"\D", "", value) diff --git a/src/govoplan_addresses/backend/router.py b/src/govoplan_addresses/backend/router.py index fe85996..07670fb 100644 --- a/src/govoplan_addresses/backend/router.py +++ b/src/govoplan_addresses/backend/router.py @@ -25,6 +25,8 @@ from govoplan_addresses.backend.db.models import ( AddressSyncTombstone, Contact, ContactChannelRule, + ContactMergeRecord, + ContactPointQualityDecision, ContactPostalAddress, ) from govoplan_addresses.backend.capabilities import ( @@ -74,6 +76,18 @@ from govoplan_addresses.backend.schemas import ( ContactChannelRuleCreateRequest, ContactChannelRuleListResponse, ContactChannelRuleResponse, + ContactDuplicateFeatureResponse, + ContactDuplicateSuggestionListResponse, + ContactDuplicateSuggestionResponse, + ContactFieldProvenanceResponse, + ContactMergeRecordListResponse, + ContactMergeRecordResponse, + ContactMergeRecoveryRequest, + ContactMergeRequest, + ContactPointQualityDecisionCreateRequest, + ContactPointQualityDecisionListResponse, + ContactPointQualityDecisionResponse, + ContactRedirectResponse, ContactPointResolveRequest, ContactPointResolutionResponse, ContactPointSnapshotResponse, @@ -82,6 +96,8 @@ from govoplan_addresses.backend.schemas import ( ContactListResponse, ContactResponse, ContactUpdateRequest, + AddressQualityCorrectionResponse, + AddressQualitySummaryResponse, VCardImportIssue, VCardImportRequest, VCardImportResponse, @@ -90,6 +106,7 @@ from govoplan_addresses.backend.service import ( AddressBookError, available_address_credentials, address_book_contact_counts, + address_quality_summary, address_list_entry_counts, create_address_book, create_address_list, @@ -97,6 +114,8 @@ from govoplan_addresses.backend.service import ( create_carddav_sync_source, create_contact, create_contact_channel_rule, + create_contact_quality_decision, + current_contact_quality, create_sync_source, count_contacts, delete_address_book, @@ -115,6 +134,9 @@ from govoplan_addresses.backend.service import ( list_address_books, list_contacts, list_contact_channel_rules, + list_contact_field_provenance, + list_contact_merges, + list_contact_quality_decisions, list_sync_conflicts, list_sync_diagnostics, list_sync_sources, @@ -122,9 +144,12 @@ from govoplan_addresses.backend.service import ( record_sync_conflict, record_sync_diagnostic, record_sync_tombstone, + merge_contacts, + recover_contact_merge, restore_address_book, restore_address_list, restore_contact, + resolve_contact_redirect, resolve_sync_conflict, preview_sync_source, public_address_sync_metadata, @@ -133,6 +158,7 @@ from govoplan_addresses.backend.service import ( update_address_book, update_address_list, update_contact, + suggest_duplicate_contacts, update_sync_source, ) @@ -172,14 +198,123 @@ def _book_response(book: AddressBook, *, contact_count: int = 0) -> AddressBookR ) -def _contact_response(contact: Contact) -> ContactResponse: - return ContactResponse.model_validate(contact) +def _contact_response( + contact: Contact, + *, + field_provenance: list | None = None, +) -> ContactResponse: + quality = current_contact_quality(contact) + + def quality_payload(channel: str, point_id: str) -> dict: + decision = quality.get((channel, point_id)) or quality.get((channel, None)) + return { + "quality_state": decision.state if decision is not None else "valid", + "quality_reason_code": ( + decision.reason_code if decision is not None else None + ), + } + + return ContactResponse.model_validate( + { + "id": contact.id, + "tenant_id": contact.tenant_id, + "address_book_id": contact.address_book_id, + "display_name": contact.display_name, + "given_name": contact.given_name, + "family_name": contact.family_name, + "organization": contact.organization, + "role_title": contact.role_title, + "note": contact.note, + "tags": list(contact.tags or []), + "source_kind": contact.source_kind, + "source_ref": contact.source_ref, + "source_payload_kind": contact.source_payload_kind, + "source_revision": contact.source_revision, + "provenance": dict(contact.provenance or {}), + "emails": [ + { + "id": item.id, + "label": item.label, + "email": item.email, + "original_email": item.original_email or item.email, + "normalized_email": item.normalized_email or item.email.casefold(), + "provenance": dict(item.provenance or {}), + "is_primary": item.is_primary, + **quality_payload("email", item.id), + } + for item in contact.emails + ], + "phones": [ + { + "id": item.id, + "label": item.label, + "phone": item.phone, + "original_phone": item.original_phone or item.phone, + "normalized_phone": item.normalized_phone or item.phone, + "provenance": dict(item.provenance or {}), + "is_primary": item.is_primary, + **quality_payload("phone", item.id), + } + for item in contact.phones + ], + "postal_addresses": [ + { + "id": item.id, + "label": item.label, + "street": item.street, + "postal_code": item.postal_code, + "locality": item.locality, + "region": item.region, + "country": item.country, + "original_value": dict(item.original_value or {}), + "normalized_value": dict(item.normalized_value or {}), + "provenance": dict(item.provenance or {}), + "is_primary": item.is_primary, + **quality_payload("postal", item.id), + } + for item in contact.postal_addresses + ], + "field_provenance": field_provenance or [], + "deleted_at": contact.deleted_at, + "created_at": contact.created_at, + "updated_at": contact.updated_at, + } + ) + + +def _contact_point_audit_details( + contact: Contact, + *, + prefix: str = "", +) -> dict[str, object]: + key_prefix = f"{prefix}_" if prefix else "" + point_ids = { + "email": [item.id for item in contact.emails], + "phone": [item.id for item in contact.phones], + "postal": [item.id for item in contact.postal_addresses], + } + return { + f"{key_prefix}contact_point_counts": { + channel: len(ids) for channel, ids in point_ids.items() + }, + f"{key_prefix}contact_point_ids": point_ids, + } def _channel_rule_response(rule: ContactChannelRule) -> ContactChannelRuleResponse: return ContactChannelRuleResponse.model_validate(rule) +def _quality_decision_response( + decision: ContactPointQualityDecision, +) -> ContactPointQualityDecisionResponse: + return ContactPointQualityDecisionResponse.model_validate(decision) + + +def _merge_response(record: ContactMergeRecord) -> ContactMergeRecordResponse: + return ContactMergeRecordResponse.model_validate(record) + + def _address_list_response(address_list: AddressList, *, entry_count: int = 0) -> AddressListResponse: return AddressListResponse.model_validate( { @@ -543,6 +678,339 @@ def api_lookup_addresses( return AddressLookupResponse(contacts=[_contact_response(contact) for contact in contacts]) +@router.get( + "/address-books/{book_id}/duplicate-suggestions", + response_model=ContactDuplicateSuggestionListResponse, +) +def api_suggest_duplicate_contacts( + book_id: str, + contact_id: str | None = Query(default=None), + minimum_score: int = Query(default=40, ge=1, le=100), + limit: int = Query(default=100, ge=1, le=100), + scan_limit: int = Query(default=500, ge=2, le=500), + principal: ApiPrincipal = Depends(get_api_principal), + session: Session = Depends(get_session), +): + _require_scope(principal, "addresses:contact:read") + try: + scan = suggest_duplicate_contacts( + session, + principal, + address_book_id=book_id, + contact_id=contact_id, + minimum_score=minimum_score, + limit=limit, + scan_limit=scan_limit, + ) + return ContactDuplicateSuggestionListResponse( + suggestions=[ + ContactDuplicateSuggestionResponse( + left=_contact_response(item.left), + right=_contact_response(item.right), + score=item.score, + confidence=item.confidence, + features=[ + ContactDuplicateFeatureResponse(**asdict(feature)) + for feature in item.features + ], + ) + for item in scan.suggestions + ], + scanned_contacts=scan.scanned_contacts, + candidate_pairs=scan.candidate_pairs, + truncated=scan.truncated, + ) + except AddressBookError as exc: + raise _error(exc) from exc + + +@router.get( + "/address-books/{book_id}/quality-summary", + response_model=AddressQualitySummaryResponse, +) +def api_address_quality_summary( + book_id: str, + correction_limit: int = Query(default=100, ge=1, le=500), + principal: ApiPrincipal = Depends(get_api_principal), + session: Session = Depends(get_session), +): + _require_scope(principal, "addresses:contact:read") + _require_scope(principal, "addresses:governance:read") + try: + summary = address_quality_summary( + session, + principal, + address_book_id=book_id, + correction_limit=correction_limit, + ) + return AddressQualitySummaryResponse( + contact_count=summary.contact_count, + contact_point_count=summary.contact_point_count, + quality_counts=summary.quality_counts, + duplicate_suggestion_count=summary.duplicate_suggestion_count, + correction_count=summary.correction_count, + corrections=[ + AddressQualityCorrectionResponse(**asdict(item)) + for item in summary.corrections + ], + truncated=summary.truncated, + ) + except AddressBookError as exc: + raise _error(exc) from exc + + +@router.get( + "/contacts/{contact_id}/quality-decisions", + response_model=ContactPointQualityDecisionListResponse, +) +def api_list_contact_quality_decisions( + contact_id: str, + include_ended: bool = Query(default=True), + principal: ApiPrincipal = Depends(get_api_principal), + session: Session = Depends(get_session), +): + _require_scope(principal, "addresses:governance:read") + try: + return ContactPointQualityDecisionListResponse( + decisions=[ + _quality_decision_response(item) + for item in list_contact_quality_decisions( + session, + principal, + contact_id, + include_ended=include_ended, + ) + ] + ) + except AddressBookError as exc: + raise _error(exc) from exc + + +@router.post( + "/contacts/{contact_id}/quality-decisions", + response_model=ContactPointQualityDecisionResponse, + status_code=status.HTTP_201_CREATED, +) +def api_create_contact_quality_decision( + contact_id: str, + payload: ContactPointQualityDecisionCreateRequest, + principal: ApiPrincipal = Depends(get_api_principal), + session: Session = Depends(get_session), +): + _require_scope(principal, "addresses:governance:write") + try: + decision = create_contact_quality_decision( + session, + principal, + contact_id, + payload, + ) + audit_from_principal( + session, + principal, + action="addresses.contact_quality_changed", + object_type="address_contact_quality_decision", + object_id=decision.id, + details={ + "contact_id": contact_id, + "channel": decision.channel, + "contact_point_id": decision.contact_point_id, + "state": decision.state, + "reason_code": decision.reason_code, + "evidence_ref": decision.evidence_ref, + }, + ) + session.commit() + session.refresh(decision) + return _quality_decision_response(decision) + except AddressBookError as exc: + session.rollback() + raise _error(exc) from exc + + +@router.get( + "/contacts/{contact_id}/provenance", + response_model=list[ContactFieldProvenanceResponse], +) +def api_list_contact_provenance( + contact_id: str, + current_only: bool = Query(default=False), + limit: int = Query(default=500, ge=1, le=2000), + principal: ApiPrincipal = Depends(get_api_principal), + session: Session = Depends(get_session), +): + _require_scope(principal, "addresses:contact:read") + try: + return [ + ContactFieldProvenanceResponse.model_validate(item) + for item in list_contact_field_provenance( + session, + principal, + contact_id, + current_only=current_only, + limit=limit, + ) + ] + except AddressBookError as exc: + raise _error(exc) from exc + + +@router.get( + "/contacts/{contact_id}/redirect", + response_model=ContactRedirectResponse, +) +def api_resolve_contact_redirect( + contact_id: str, + principal: ApiPrincipal = Depends(get_api_principal), + session: Session = Depends(get_session), +): + _require_scope(principal, "addresses:contact:read") + try: + return ContactRedirectResponse.model_validate( + asdict(resolve_contact_redirect(session, principal, contact_id)) + ) + except AddressBookError as exc: + raise _error(exc) from exc + + +@router.get("/contact-merges", response_model=ContactMergeRecordListResponse) +def api_list_contact_merges( + address_book_id: str | None = Query(default=None), + contact_id: str | None = Query(default=None), + limit: int = Query(default=100, ge=1, le=500), + principal: ApiPrincipal = Depends(get_api_principal), + session: Session = Depends(get_session), +): + _require_scope(principal, "addresses:contact:read") + try: + return ContactMergeRecordListResponse( + merges=[ + _merge_response(item) + for item in list_contact_merges( + session, + principal, + address_book_id=address_book_id, + contact_id=contact_id, + limit=limit, + ) + ] + ) + except AddressBookError as exc: + raise _error(exc) from exc + + +@router.post( + "/contact-merges", + response_model=ContactMergeRecordResponse, + status_code=status.HTTP_201_CREATED, +) +def api_merge_contacts( + payload: ContactMergeRequest, + principal: ApiPrincipal = Depends(get_api_principal), + session: Session = Depends(get_session), +): + _require_scope(principal, "addresses:contact:write") + _require_scope(principal, "addresses:contact:delete") + try: + record = merge_contacts(session, principal, payload) + audit_from_principal( + session, + principal, + action="addresses.contacts_merged", + object_type="address_contact_merge", + object_id=record.id, + details={ + "winner_contact_id": record.winner_contact_id, + "loser_contact_ids": list(record.loser_contact_ids), + "before_hash": record.before_hash, + "after_hash": record.after_hash, + "reason": record.reason, + }, + ) + session.commit() + session.refresh(record) + return _merge_response(record) + except AddressBookError as exc: + session.rollback() + raise _error(exc) from exc + + +def _recover_contact_merge_api( + merge_id: str, + payload: ContactMergeRecoveryRequest, + principal: ApiPrincipal, + session: Session, + *, + action: str, +) -> ContactMergeRecordResponse: + _require_scope(principal, "addresses:contact:write") + try: + record = recover_contact_merge( + session, + principal, + merge_id, + payload, + action=action, + ) + audit_from_principal( + session, + principal, + action=f"addresses.contact_merge_{action}", + object_type="address_contact_merge", + object_id=record.id, + details={ + "winner_contact_id": record.winner_contact_id, + "loser_contact_ids": list(record.loser_contact_ids), + "expected_after_hash": payload.expected_after_hash, + "reason": payload.reason, + }, + ) + session.commit() + session.refresh(record) + return _merge_response(record) + except AddressBookError as exc: + session.rollback() + raise _error(exc) from exc + + +@router.post( + "/contact-merges/{merge_id}/undo", + response_model=ContactMergeRecordResponse, +) +def api_undo_contact_merge( + merge_id: str, + payload: ContactMergeRecoveryRequest, + principal: ApiPrincipal = Depends(get_api_principal), + session: Session = Depends(get_session), +): + return _recover_contact_merge_api( + merge_id, + payload, + principal, + session, + action="undo", + ) + + +@router.post( + "/contact-merges/{merge_id}/split", + response_model=ContactMergeRecordResponse, +) +def api_split_contact_merge( + merge_id: str, + payload: ContactMergeRecoveryRequest, + principal: ApiPrincipal = Depends(get_api_principal), + session: Session = Depends(get_session), +): + return _recover_contact_merge_api( + merge_id, + payload, + principal, + session, + action="split", + ) + + @router.post("/contact-points/resolve", response_model=ContactPointResolutionResponse) def api_resolve_contact_points( payload: ContactPointResolveRequest, @@ -774,6 +1242,19 @@ def api_create_address_list_entry( _require_scope(principal, "addresses:address_list:write") try: entry = create_address_list_entry(session, principal, address_list_id, payload) + session.flush() + audit_from_principal( + session, + principal, + action="addresses.address_list_entry_created", + object_type="address_list_entry", + object_id=entry.id, + details={ + "address_list_id": entry.address_list_id, + "contact_id": entry.contact_id, + "target_kind": entry.target_kind, + }, + ) session.commit() session.refresh(entry) return _address_list_entry_response(entry) @@ -790,7 +1271,20 @@ def api_delete_address_list_entry( ): _require_scope(principal, "addresses:address_list:write") try: + entry = session.get(AddressListEntry, entry_id) delete_address_list_entry(session, principal, entry_id) + audit_from_principal( + session, + principal, + action="addresses.address_list_entry_deleted", + object_type="address_list_entry", + object_id=entry_id, + details={ + "address_list_id": entry.address_list_id if entry is not None else None, + "contact_id": entry.contact_id if entry is not None else None, + "target_kind": entry.target_kind if entry is not None else None, + }, + ) session.commit() return Response(status_code=status.HTTP_204_NO_CONTENT) except AddressBookError as exc: @@ -1234,6 +1728,19 @@ def api_create_contact( _require_scope(principal, "addresses:contact:write") try: contact = create_contact(session, principal, book_id, payload) + audit_from_principal( + session, + principal, + action="addresses.contact_created", + object_type="address_contact", + object_id=contact.id, + details={ + "address_book_id": contact.address_book_id, + "source_kind": contact.source_kind, + "field_names": sorted(payload.model_fields_set), + **_contact_point_audit_details(contact), + }, + ) session.commit() session.refresh(contact) return _contact_response(contact) @@ -1251,7 +1758,27 @@ def api_update_contact( ): _require_scope(principal, "addresses:contact:write") try: + previous_contact = session.get(Contact, contact_id) + previous_point_details = ( + _contact_point_audit_details(previous_contact, prefix="previous") + if previous_contact is not None + else {} + ) contact = update_contact(session, principal, contact_id, payload) + audit_from_principal( + session, + principal, + action="addresses.contact_updated", + object_type="address_contact", + object_id=contact.id, + details={ + "address_book_id": contact.address_book_id, + "source_kind": contact.source_kind, + "field_names": sorted(payload.model_fields_set), + **previous_point_details, + **_contact_point_audit_details(contact), + }, + ) session.commit() session.refresh(contact) return _contact_response(contact) @@ -1360,6 +1887,18 @@ def api_delete_contact( _require_scope(principal, "addresses:contact:delete") try: delete_contact(session, principal, contact_id) + contact = session.get(Contact, contact_id) + audit_from_principal( + session, + principal, + action="addresses.contact_deleted", + object_type="address_contact", + object_id=contact_id, + details={ + "address_book_id": contact.address_book_id if contact is not None else None, + "source_kind": contact.source_kind if contact is not None else None, + }, + ) session.commit() return Response(status_code=status.HTTP_204_NO_CONTENT) except AddressBookError as exc: @@ -1376,6 +1915,17 @@ def api_restore_contact( _require_scope(principal, "addresses:contact:write") try: contact = restore_contact(session, principal, contact_id) + audit_from_principal( + session, + principal, + action="addresses.contact_restored", + object_type="address_contact", + object_id=contact.id, + details={ + "address_book_id": contact.address_book_id, + "source_kind": contact.source_kind, + }, + ) session.commit() session.refresh(contact) return _contact_response(contact) @@ -1394,6 +1944,20 @@ def api_import_address_book_vcards( _require_scope(principal, "addresses:contact:write") try: result = import_vcards(session, principal, book_id, payload.content) + for contact in result.contacts: + audit_from_principal( + session, + principal, + action="addresses.contact_imported", + object_type="address_contact", + object_id=contact.id, + details={ + "address_book_id": book_id, + "source_kind": contact.source_kind, + "source_revision": contact.source_revision, + **_contact_point_audit_details(contact), + }, + ) session.commit() for contact in result.contacts: session.refresh(contact) diff --git a/src/govoplan_addresses/backend/schemas.py b/src/govoplan_addresses/backend/schemas.py index a137c2a..6e7b537 100644 --- a/src/govoplan_addresses/backend/schemas.py +++ b/src/govoplan_addresses/backend/schemas.py @@ -15,6 +15,13 @@ AddressSyncConflictResolution = Literal["keep_local", "use_remote", "merge", "ma AddressCardDavAuthType = Literal["none", "basic", "bearer"] AddressSyncPlanAction = Literal["create", "update", "delete", "remote_create", "remote_update", "remote_delete", "conflict", "unchanged", "error"] AddressDistributionChannel = Literal["email", "postal", "internal_mail", "portal"] +AddressContactPointChannel = Literal[ + "email", + "phone", + "postal", + "internal_mail", + "portal", +] AddressChannelDecision = Literal[ "allowed", "opted_in", @@ -38,6 +45,13 @@ AddressDistributionOutcome = Literal[ ] AddressContactPointFallbackRule = Literal["none", "primary", "any"] AddressPostalFormat = Literal["domestic", "international"] +ContactPointQualityState = Literal[ + "valid", + "invalid", + "returned", + "stale", + "undeliverable", +] class ContactEmailPayload(BaseModel): @@ -186,12 +200,75 @@ class ContactUpdateRequest(BaseModel): provenance: dict[str, Any] | None = None +class ContactFieldProvenanceResponse(BaseModel): + model_config = ConfigDict(from_attributes=True) + + id: str + contact_id: str + field_path: str + value: Any = None + source_kind: str + source_ref: str | None = None + source_revision: str | None = None + precedence: int + selected: bool + reason_code: str + explanation: str | None = None + visibility: str + merge_record_id: str | None = None + created_by_account_id: str | None = None + metadata: dict[str, Any] = Field(default_factory=dict, validation_alias="metadata_") + created_at: datetime + + +class ContactPointQualityDecisionCreateRequest(BaseModel): + model_config = ConfigDict(extra="forbid") + + channel: AddressContactPointChannel + contact_point_id: str | None = Field(default=None, max_length=36) + state: ContactPointQualityState + reason_code: str | None = Field(default=None, max_length=120) + reason: str | None = None + evidence_ref: str | None = Field(default=None, max_length=1000) + effective_from: datetime | None = None + metadata: dict[str, Any] = Field(default_factory=dict) + + +class ContactPointQualityDecisionResponse(BaseModel): + model_config = ConfigDict(from_attributes=True) + + id: str + tenant_id: str | None = None + contact_id: str + channel: AddressContactPointChannel + contact_point_id: str | None = None + state: ContactPointQualityState + reason_code: str + reason: str | None = None + evidence_ref: str | None = None + effective_from: datetime + effective_until: datetime | None = None + created_by_account_id: str | None = None + metadata: dict[str, Any] = Field(default_factory=dict, validation_alias="metadata_") + created_at: datetime + updated_at: datetime + + +class ContactPointQualityDecisionListResponse(BaseModel): + decisions: list[ContactPointQualityDecisionResponse] = Field(default_factory=list) + + class ContactEmailResponse(BaseModel): model_config = ConfigDict(from_attributes=True) id: str label: str | None = None email: str + original_email: str = "" + normalized_email: str = "" + provenance: dict[str, Any] = Field(default_factory=dict) + quality_state: ContactPointQualityState = "valid" + quality_reason_code: str | None = None is_primary: bool @@ -201,6 +278,11 @@ class ContactPhoneResponse(BaseModel): id: str label: str | None = None phone: str + original_phone: str = "" + normalized_phone: str = "" + provenance: dict[str, Any] = Field(default_factory=dict) + quality_state: ContactPointQualityState = "valid" + quality_reason_code: str | None = None is_primary: bool @@ -214,6 +296,11 @@ class ContactPostalAddressResponse(BaseModel): locality: str | None = None region: str | None = None country: str | None = None + original_value: dict[str, Any] = Field(default_factory=dict) + normalized_value: dict[str, Any] = Field(default_factory=dict) + provenance: dict[str, Any] = Field(default_factory=dict) + quality_state: ContactPointQualityState = "valid" + quality_reason_code: str | None = None is_primary: bool @@ -238,6 +325,7 @@ class ContactResponse(BaseModel): emails: list[ContactEmailResponse] phones: list[ContactPhoneResponse] postal_addresses: list[ContactPostalAddressResponse] + field_provenance: list[ContactFieldProvenanceResponse] = Field(default_factory=list) deleted_at: datetime | None = None created_at: datetime updated_at: datetime @@ -251,6 +339,103 @@ class ContactListResponse(BaseModel): has_more: bool +class ContactDuplicateFeatureResponse(BaseModel): + code: str + label: str + weight: int + value: str + + +class ContactDuplicateSuggestionResponse(BaseModel): + left: ContactResponse + right: ContactResponse + score: int + confidence: Literal["possible", "likely", "strong"] + features: list[ContactDuplicateFeatureResponse] + + +class ContactDuplicateSuggestionListResponse(BaseModel): + suggestions: list[ContactDuplicateSuggestionResponse] = Field(default_factory=list) + scanned_contacts: int + candidate_pairs: int + truncated: bool + + +class ContactMergeRequest(BaseModel): + model_config = ConfigDict(extra="forbid") + + winner_contact_id: str = Field(max_length=36) + duplicate_contact_ids: list[str] = Field(min_length=1, max_length=20) + reason: str = Field(min_length=3) + field_sources: dict[str, str] = Field(default_factory=dict) + contact_point_strategy: Literal["union", "winner_only"] = "union" + source_precedence: list[str] = Field(default_factory=list, max_length=20) + + +class ContactMergeRecoveryRequest(BaseModel): + model_config = ConfigDict(extra="forbid") + + reason: str = Field(min_length=3) + expected_after_hash: str = Field(min_length=64, max_length=64) + + +class ContactMergeRecordResponse(BaseModel): + model_config = ConfigDict(from_attributes=True) + + id: str + tenant_id: str | None = None + address_book_id: str + winner_contact_id: str + loser_contact_ids: list[str] + status: str + reason: str + survivorship: dict[str, Any] + decisions: list[dict[str, Any]] + before_hash: str + after_hash: str + created_by_account_id: str | None = None + recovered_at: datetime | None = None + recovered_by_account_id: str | None = None + recovery_action: str | None = None + recovery_reason: str | None = None + provenance: dict[str, Any] + created_at: datetime + updated_at: datetime + + +class ContactMergeRecordListResponse(BaseModel): + merges: list[ContactMergeRecordResponse] = Field(default_factory=list) + + +class ContactRedirectResponse(BaseModel): + requested_contact_id: str + resolved_contact_id: str + redirected: bool + redirect_chain: list[str] = Field(default_factory=list) + merge_record_ids: list[str] = Field(default_factory=list) + + +class AddressQualityCorrectionResponse(BaseModel): + contact_id: str + display_name: str + channel: AddressContactPointChannel + contact_point_id: str | None = None + state: ContactPointQualityState + reason_code: str + reason: str | None = None + effective_from: datetime + + +class AddressQualitySummaryResponse(BaseModel): + contact_count: int + contact_point_count: int + quality_counts: dict[str, int] = Field(default_factory=dict) + duplicate_suggestion_count: int + correction_count: int + corrections: list[AddressQualityCorrectionResponse] = Field(default_factory=list) + truncated: bool = False + + class ContactChannelRuleCreateRequest(BaseModel): model_config = ConfigDict(extra="forbid") diff --git a/src/govoplan_addresses/backend/service.py b/src/govoplan_addresses/backend/service.py index 7d942d8..5e5bfa3 100644 --- a/src/govoplan_addresses/backend/service.py +++ b/src/govoplan_addresses/backend/service.py @@ -3,13 +3,17 @@ from __future__ import annotations import copy from collections.abc import Iterable from dataclasses import dataclass, field +from datetime import UTC, datetime +import hashlib +import json import os import re +import unicodedata import urllib.parse from typing import Any from sqlalchemy import and_, false, func, or_ -from sqlalchemy.orm import Session +from sqlalchemy.orm import Session, selectinload from govoplan_core.auth import ApiPrincipal from govoplan_core.audit.logging import audit_event @@ -45,8 +49,13 @@ from govoplan_addresses.backend.db.models import ( Contact, ContactChannelRule, ContactEmail, + ContactFieldProvenance, + ContactMergeRecord, ContactPhone, + ContactPointQualityDecision, ContactPostalAddress, + ContactRedirect, + new_uuid, ) from govoplan_addresses.backend.schemas import ( AddressBookCreateRequest, @@ -70,6 +79,9 @@ from govoplan_addresses.backend.schemas import ( ContactPostalAddressPayload, ContactUpdateRequest, ContactChannelRuleCreateRequest, + ContactMergeRequest, + ContactMergeRecoveryRequest, + ContactPointQualityDecisionCreateRequest, ) from govoplan_addresses.backend.vcard import ParsedVCard, ParsedVCardIssue, contacts_to_vcard, parse_vcards_with_issues @@ -139,6 +151,63 @@ class AddressSyncPlan: items: list[AddressSyncPlanItem] = field(default_factory=list) +@dataclass(frozen=True, slots=True) +class ContactDuplicateFeature: + code: str + label: str + weight: int + value: str + + +@dataclass(frozen=True, slots=True) +class ContactDuplicateSuggestion: + left: Contact + right: Contact + score: int + confidence: str + features: tuple[ContactDuplicateFeature, ...] + + +@dataclass(frozen=True, slots=True) +class ContactDuplicateScan: + suggestions: tuple[ContactDuplicateSuggestion, ...] + scanned_contacts: int + candidate_pairs: int + truncated: bool + + +@dataclass(frozen=True, slots=True) +class ContactRedirectResolution: + requested_contact_id: str + resolved_contact_id: str + redirected: bool + redirect_chain: tuple[str, ...] + merge_record_ids: tuple[str, ...] + + +@dataclass(frozen=True, slots=True) +class AddressQualityCorrection: + contact_id: str + display_name: str + channel: str + contact_point_id: str | None + state: str + reason_code: str + reason: str | None + effective_from: datetime + + +@dataclass(frozen=True, slots=True) +class AddressQualitySummary: + contact_count: int + contact_point_count: int + quality_counts: dict[str, int] + duplicate_suggestion_count: int + correction_count: int + corrections: tuple[AddressQualityCorrection, ...] + truncated: bool + + READ_ONLY_SYNC_DIRECTIONS = {"read_only", "import"} OUTBOUND_SYNC_DIRECTIONS = {"export", "two_way"} INBOUND_SYNC_DIRECTIONS = {"read_only", "import", "two_way"} @@ -1986,7 +2055,9 @@ def _record_address_contact_change( contact: Contact, operation: str, previous: dict[str, Any] | None = None, + merge_record_id: str | None = None, ) -> None: + session.flush() payload = _contact_change_payload(contact) if previous: payload.update(previous) @@ -2002,6 +2073,140 @@ def _record_address_contact_change( actor_id=_account_id(principal), payload=payload, ) + _record_contact_field_provenance( + session, + principal, + contact=contact, + reason_code=f"addresses.contact.{operation}", + explanation=f"The retained value was selected during contact {operation}.", + merge_record_id=merge_record_id, + ) + + +def _record_contact_field_provenance( + session: Session, + principal: ApiPrincipal, + *, + contact: Contact, + reason_code: str, + explanation: str, + merge_record_id: str | None = None, +) -> None: + session.flush() + session.query(ContactFieldProvenance).filter( + ContactFieldProvenance.contact_id == contact.id, + ContactFieldProvenance.selected.is_(True), + ).update({ContactFieldProvenance.selected: False}, synchronize_session=False) + contact_provenance = contact.provenance if isinstance(contact.provenance, dict) else {} + raw_visibility = contact_provenance.get("field_visibility") + visibility_by_path = dict(raw_visibility) if isinstance(raw_visibility, dict) else {} + raw_field_sources = contact_provenance.get("field_sources") + field_sources = dict(raw_field_sources) if isinstance(raw_field_sources, dict) else {} + for path, value, provenance in _retained_contact_fields(contact): + field_source = field_sources.get(path) + if isinstance(field_source, dict): + provenance = {**provenance, **field_source} + source_kind = str(provenance.get("source_kind") or contact.source_kind or "local") + source_ref = provenance.get("source_ref") or contact.source_ref + source_revision = provenance.get("source_revision") or contact.source_revision + visibility = str(visibility_by_path.get(path) or "inherit") + if visibility not in {"inherit", "private", "restricted", "public"}: + visibility = "inherit" + session.add( + ContactFieldProvenance( + tenant_id=contact.tenant_id, + contact_id=contact.id, + field_path=path, + value=value, + source_kind=source_kind, + source_ref=str(source_ref) if source_ref else None, + source_revision=str(source_revision) if source_revision else None, + precedence=_source_precedence(source_kind), + selected=True, + reason_code=reason_code, + explanation=explanation, + visibility=visibility, + merge_record_id=merge_record_id, + created_by_account_id=_account_id(principal), + metadata_={ + "operation": reason_code.rsplit(".", 1)[-1], + **{ + key: value + for key, value in provenance.items() + if key not in {"source_kind", "source_ref", "source_revision"} + }, + }, + ) + ) + + +def _retained_contact_fields( + contact: Contact, +) -> list[tuple[str, Any, dict[str, Any]]]: + contact_source = { + "source_kind": contact.source_kind, + "source_ref": contact.source_ref, + "source_revision": contact.source_revision, + } + rows: list[tuple[str, Any, dict[str, Any]]] = [ + (field_name, getattr(contact, field_name), contact_source) + for field_name in ( + "display_name", + "given_name", + "family_name", + "organization", + "role_title", + "note", + "tags", + ) + ] + for point in contact.emails: + provenance = dict(point.provenance or contact_source) + base = f"emails.{point.id}" + rows.extend( + ( + (f"{base}.label", point.label, provenance), + (f"{base}.email", point.email, provenance), + (f"{base}.is_primary", point.is_primary, provenance), + ) + ) + for point in contact.phones: + provenance = dict(point.provenance or contact_source) + base = f"phones.{point.id}" + rows.extend( + ( + (f"{base}.label", point.label, provenance), + (f"{base}.phone", point.phone, provenance), + (f"{base}.is_primary", point.is_primary, provenance), + ) + ) + for point in contact.postal_addresses: + provenance = dict(point.provenance or contact_source) + base = f"postal_addresses.{point.id}" + for field_name in ( + "label", + "street", + "postal_code", + "locality", + "region", + "country", + "is_primary", + ): + rows.append((f"{base}.{field_name}", getattr(point, field_name), provenance)) + return rows + + +def _source_precedence(source_kind: str) -> int: + return { + "manual": 100, + "local": 90, + "carddav": 80, + "microsoft_graph": 75, + "google": 75, + "ldap": 70, + "vcard": 60, + "csv": 50, + }.get(source_kind, 40) def _read_only_from_sync_direction(sync_direction: str, explicit_read_only: bool | None) -> bool: @@ -2311,6 +2516,59 @@ def _normalize_tags(tags: list[str] | None) -> list[str]: return normalized +def _normalize_match_text(value: str | None) -> str: + if value is None: + return "" + return " ".join(unicodedata.normalize("NFKC", value).strip().casefold().split()) + + +def _normalize_email(value: str | None) -> str: + return _normalize_match_text(value) + + +def _normalize_phone(value: str | None) -> str: + if value is None: + return "" + normalized = unicodedata.normalize("NFKC", value).strip() + prefix = "+" if normalized.startswith("+") else "" + return prefix + re.sub(r"\D", "", normalized) + + +def _normalized_postal_value(payload: ContactPostalAddressPayload) -> dict[str, str | None]: + return { + key: _normalize_match_text(getattr(payload, key)) or None + for key in ("label", "street", "postal_code", "locality", "region", "country") + } + + +def _original_postal_value(payload: ContactPostalAddressPayload) -> dict[str, str | None]: + return { + key: getattr(payload, key) + for key in ("label", "street", "postal_code", "locality", "region", "country") + } + + +def _contact_point_provenance(contact: Contact, *, point_kind: str) -> dict[str, Any]: + return { + "module": "addresses", + "point_kind": point_kind, + "source_kind": contact.source_kind, + "source_ref": contact.source_ref, + "source_revision": contact.source_revision, + "supplied_by_account_id": contact.updated_by_account_id or contact.created_by_account_id, + "original_value_preserved": True, + } + + +def _restamp_contact_point_provenance(contact: Contact) -> None: + for point in contact.emails: + point.provenance = _contact_point_provenance(contact, point_kind="email") + for point in contact.phones: + point.provenance = _contact_point_provenance(contact, point_kind="phone") + for point in contact.postal_addresses: + point.provenance = _contact_point_provenance(contact, point_kind="postal") + + def _replace_emails(contact: Contact, payloads: list[ContactEmailPayload]) -> None: contact.emails.clear() normalized = [_trim(item.email) for item in payloads] @@ -2321,6 +2579,9 @@ def _replace_emails(contact: Contact, payloads: list[ContactEmailPayload]) -> No ContactEmail( label=_trim(item.label), email=email or "", + original_email=item.email, + normalized_email=_normalize_email(item.email), + provenance=_contact_point_provenance(contact, point_kind="email"), is_primary=index == primary_index, order_index=index, ) @@ -2337,6 +2598,9 @@ def _replace_phones(contact: Contact, payloads: list[ContactPhonePayload]) -> No ContactPhone( label=_trim(item.label), phone=phone or "", + original_phone=item.phone, + normalized_phone=_normalize_phone(item.phone), + provenance=_contact_point_provenance(contact, point_kind="phone"), is_primary=index == primary_index, order_index=index, ) @@ -2360,6 +2624,9 @@ def _replace_postal_addresses(contact: Contact, payloads: list[ContactPostalAddr locality=_trim(item.locality), region=_trim(item.region), country=_trim(item.country), + original_value=_original_postal_value(item), + normalized_value=_normalized_postal_value(item), + provenance=_contact_point_provenance(contact, point_kind="postal"), is_primary=index == primary_index, order_index=index, ) @@ -2389,6 +2656,12 @@ def create_contact(session: Session, principal: ApiPrincipal, address_book_id: s _replace_phones(contact, payload.phones) _replace_postal_addresses(contact, payload.postal_addresses) session.add(contact) + _record_address_contact_change( + session, + principal, + contact=contact, + operation="created", + ) return contact @@ -2404,6 +2677,13 @@ def import_vcards(session: Session, principal: ApiPrincipal, address_book_id: st contact.source_payload_kind = "vcard" contact.source_payload_raw = parsed.raw contact.source_revision = _trim(parsed.source_revision) + _restamp_contact_point_provenance(contact) + _record_address_contact_change( + session, + principal, + contact=contact, + operation="imported", + ) contacts.append(contact) return VCardImportResult(contacts=contacts, issues=result.issues, skipped=result.skipped) @@ -2412,7 +2692,16 @@ def _visible_contact_query(session: Session, principal: ApiPrincipal, *, include book_ids = [book.id for book in list_address_books(session, principal, include_deleted=include_deleted_books)] if not book_ids: return session.query(Contact).filter(false()) - query = session.query(Contact).filter(Contact.address_book_id.in_(book_ids)) + query = ( + session.query(Contact) + .options( + selectinload(Contact.emails), + selectinload(Contact.phones), + selectinload(Contact.postal_addresses), + selectinload(Contact.quality_decisions), + ) + .filter(Contact.address_book_id.in_(book_ids)) + ) if not include_deleted: query = query.filter(Contact.deleted_at.is_(None)) return query @@ -2593,6 +2882,7 @@ def end_contact_channel_rule( def update_contact(session: Session, principal: ApiPrincipal, contact_id: str, payload: ContactUpdateRequest) -> Contact: contact = get_visible_contact(session, principal, contact_id) _require_mutable_book(contact.address_book) + previous = _contact_change_payload(contact, prefix="previous_") fallback_name = contact.display_name if any(field in payload.model_fields_set for field in ("display_name", "given_name", "family_name", "emails")): contact.display_name = _display_name_from_payload(payload, fallback=fallback_name) @@ -2616,24 +2906,1233 @@ def update_contact(session: Session, principal: ApiPrincipal, contact_id: str, p if "postal_addresses" in payload.model_fields_set: _replace_postal_addresses(contact, payload.postal_addresses or []) contact.updated_by_account_id = _account_id(principal) + _record_address_contact_change( + session, + principal, + contact=contact, + operation="updated", + previous=previous, + ) return contact def delete_contact(session: Session, principal: ApiPrincipal, contact_id: str) -> None: contact = get_visible_contact(session, principal, contact_id) _require_mutable_book(contact.address_book) + previous = _contact_change_payload(contact, prefix="previous_") contact.deleted_at = utcnow() contact.updated_by_account_id = _account_id(principal) + _record_address_contact_change( + session, + principal, + contact=contact, + operation="deleted", + previous=previous, + ) def restore_contact(session: Session, principal: ApiPrincipal, contact_id: str) -> Contact: contact = get_visible_contact(session, principal, contact_id, include_deleted=True) _require_mutable_book(contact.address_book) + previous = _contact_change_payload(contact, prefix="previous_") contact.deleted_at = None contact.updated_by_account_id = _account_id(principal) + _record_address_contact_change( + session, + principal, + contact=contact, + operation="restored", + previous=previous, + ) return contact +def list_contact_quality_decisions( + session: Session, + principal: ApiPrincipal, + contact_id: str, + *, + include_ended: bool = True, +) -> list[ContactPointQualityDecision]: + contact = get_visible_contact(session, principal, contact_id, include_deleted=True) + query = session.query(ContactPointQualityDecision).filter( + ContactPointQualityDecision.contact_id == contact.id + ) + if not include_ended: + now = utcnow() + query = query.filter( + or_( + ContactPointQualityDecision.effective_until.is_(None), + ContactPointQualityDecision.effective_until > now, + ) + ) + return query.order_by( + ContactPointQualityDecision.effective_from.desc(), + ContactPointQualityDecision.created_at.desc(), + ContactPointQualityDecision.id.asc(), + ).all() + + +def create_contact_quality_decision( + session: Session, + principal: ApiPrincipal, + contact_id: str, + payload: ContactPointQualityDecisionCreateRequest, +) -> ContactPointQualityDecision: + contact = get_visible_contact(session, principal, contact_id) + _require_mutable_book(contact.address_book) + _validate_quality_contact_point(contact, payload.channel, payload.contact_point_id) + effective_from = payload.effective_from or utcnow() + overlapping = ( + session.query(ContactPointQualityDecision) + .filter( + ContactPointQualityDecision.contact_id == contact.id, + ContactPointQualityDecision.channel == payload.channel, + ContactPointQualityDecision.contact_point_id == payload.contact_point_id, + ContactPointQualityDecision.effective_from <= effective_from, + or_( + ContactPointQualityDecision.effective_until.is_(None), + ContactPointQualityDecision.effective_until > effective_from, + ), + ) + .all() + ) + for decision in overlapping: + decision.effective_until = effective_from + decision = ContactPointQualityDecision( + tenant_id=contact.tenant_id, + contact_id=contact.id, + channel=payload.channel, + contact_point_id=payload.contact_point_id, + state=payload.state, + reason_code=payload.reason_code or f"addresses.quality.{payload.state}", + reason=_trim(payload.reason), + evidence_ref=_trim(payload.evidence_ref), + effective_from=effective_from, + created_by_account_id=_account_id(principal), + metadata_=dict(payload.metadata or {}), + ) + session.add(decision) + session.flush() + _record_address_contact_change( + session, + principal, + contact=contact, + operation="quality_updated", + ) + return decision + + +def current_contact_quality( + contact: Contact, + *, + effective_at: datetime | None = None, +) -> dict[tuple[str, str | None], ContactPointQualityDecision]: + at = _aware_time(effective_at or utcnow()) + current: dict[tuple[str, str | None], ContactPointQualityDecision] = {} + for decision in contact.quality_decisions: + decision_from = _aware_time(decision.effective_from) + decision_until = ( + _aware_time(decision.effective_until) + if decision.effective_until is not None + else None + ) + if decision_from > at: + continue + if decision_until is not None and decision_until <= at: + continue + key = (decision.channel, decision.contact_point_id) + existing = current.get(key) + if existing is None or ( + decision_from, + _aware_time(decision.created_at), + decision.id, + ) > ( + _aware_time(existing.effective_from), + _aware_time(existing.created_at), + existing.id, + ): + current[key] = decision + return current + + +def list_contact_field_provenance( + session: Session, + principal: ApiPrincipal, + contact_id: str, + *, + current_only: bool = False, + limit: int = 500, +) -> list[ContactFieldProvenance]: + contact = get_visible_contact(session, principal, contact_id, include_deleted=True) + query = session.query(ContactFieldProvenance).filter( + ContactFieldProvenance.contact_id == contact.id + ) + if current_only: + query = query.filter(ContactFieldProvenance.selected.is_(True)) + return query.order_by( + ContactFieldProvenance.selected.desc(), + ContactFieldProvenance.field_path.asc(), + ContactFieldProvenance.created_at.desc(), + ).limit(max(1, min(limit, 2_000))).all() + + +def suggest_duplicate_contacts( + session: Session, + principal: ApiPrincipal, + *, + address_book_id: str, + contact_id: str | None = None, + minimum_score: int = 40, + limit: int = 100, + scan_limit: int = 500, +) -> ContactDuplicateScan: + get_visible_address_book(session, principal, address_book_id) + bounded_scan = max(2, min(scan_limit, 500)) + total = count_contacts( + session, + principal, + address_book_id=address_book_id, + ) + contacts = list_contacts( + session, + principal, + address_book_id=address_book_id, + limit=bounded_scan, + ) + if contact_id and not any(item.id == contact_id for item in contacts): + selected = get_visible_contact(session, principal, contact_id) + if selected.address_book_id != address_book_id: + raise AddressBookError("Duplicate scan contact must belong to the selected address book.") + contacts = [selected, *contacts[:-1]] if contacts else [selected] + + feature_index: dict[tuple[str, str], list[Contact]] = {} + feature_details: dict[tuple[str, str], ContactDuplicateFeature] = {} + for contact in contacts: + for key, feature in _duplicate_features(contact): + feature_index.setdefault(key, []).append(contact) + feature_details[key] = feature + + pair_features: dict[tuple[str, str], dict[str, ContactDuplicateFeature]] = {} + contacts_by_id = {item.id: item for item in contacts} + for key, matched in feature_index.items(): + if len(matched) < 2: + continue + for left_index, left in enumerate(matched[:-1]): + for right in matched[left_index + 1 :]: + if contact_id and contact_id not in {left.id, right.id}: + continue + pair = tuple(sorted((left.id, right.id))) + feature = feature_details[key] + pair_features.setdefault(pair, {})[feature.code] = feature + + suggestions: list[ContactDuplicateSuggestion] = [] + for pair, features_by_code in pair_features.items(): + features = tuple( + sorted( + features_by_code.values(), + key=lambda item: (-item.weight, item.code), + ) + ) + score = min(100, sum(item.weight for item in features)) + if score < max(1, min(minimum_score, 100)): + continue + suggestions.append( + ContactDuplicateSuggestion( + left=contacts_by_id[pair[0]], + right=contacts_by_id[pair[1]], + score=score, + confidence="strong" if score >= 85 else "likely" if score >= 60 else "possible", + features=features, + ) + ) + suggestions.sort( + key=lambda item: ( + -item.score, + item.left.display_name.casefold(), + item.right.display_name.casefold(), + item.left.id, + item.right.id, + ) + ) + bounded_limit = max(1, min(limit, 100)) + return ContactDuplicateScan( + suggestions=tuple(suggestions[:bounded_limit]), + scanned_contacts=len(contacts), + candidate_pairs=len(suggestions), + truncated=total > len(contacts) or len(suggestions) > bounded_limit, + ) + + +def address_quality_summary( + session: Session, + principal: ApiPrincipal, + *, + address_book_id: str, + correction_limit: int = 100, +) -> AddressQualitySummary: + get_visible_address_book(session, principal, address_book_id) + contacts = list_contacts( + session, + principal, + address_book_id=address_book_id, + limit=500, + ) + total = count_contacts(session, principal, address_book_id=address_book_id) + points = sum( + len(contact.emails) + len(contact.phones) + len(contact.postal_addresses) + for contact in contacts + ) + quality_counts = { + "valid": points, + "invalid": 0, + "returned": 0, + "stale": 0, + "undeliverable": 0, + } + corrections: list[AddressQualityCorrection] = [] + for contact in contacts: + for (channel, point_id), decision in current_contact_quality(contact).items(): + quality_counts[decision.state] = quality_counts.get(decision.state, 0) + 1 + if decision.state != "valid": + quality_counts["valid"] = max(0, quality_counts["valid"] - 1) + corrections.append( + AddressQualityCorrection( + contact_id=contact.id, + display_name=contact.display_name, + channel=channel, + contact_point_id=point_id, + state=decision.state, + reason_code=decision.reason_code, + reason=decision.reason, + effective_from=decision.effective_from, + ) + ) + corrections.sort( + key=lambda item: ( + item.state, + item.display_name.casefold(), + item.contact_point_id or "", + ) + ) + duplicates = suggest_duplicate_contacts( + session, + principal, + address_book_id=address_book_id, + limit=100, + ) + bounded_limit = max(1, min(correction_limit, 500)) + return AddressQualitySummary( + contact_count=total, + contact_point_count=points, + quality_counts=quality_counts, + duplicate_suggestion_count=duplicates.candidate_pairs, + correction_count=len(corrections), + corrections=tuple(corrections[:bounded_limit]), + truncated=( + total > len(contacts) + or len(corrections) > bounded_limit + or duplicates.truncated + ), + ) + + +def merge_contacts( + session: Session, + principal: ApiPrincipal, + payload: ContactMergeRequest, +) -> ContactMergeRecord: + contact_ids = list(dict.fromkeys([payload.winner_contact_id, *payload.duplicate_contact_ids])) + if len(contact_ids) < 2: + raise AddressBookError("A merge requires one winner and at least one distinct duplicate.") + contacts = [get_visible_contact(session, principal, item) for item in contact_ids] + winner = contacts[0] + losers = contacts[1:] + if any(item.address_book_id != winner.address_book_id for item in losers): + raise AddressBookError("Contacts can only be merged inside one address book.") + _require_mutable_book(winner.address_book) + if session.query(ContactRedirect).filter( + ContactRedirect.source_contact_id.in_(contact_ids), + ContactRedirect.ended_at.is_(None), + ).first() is not None: + raise AddressBookError("A contact in this merge already has an active redirect.") + allowed_sources = set(contact_ids) + unsupported_fields = set(payload.field_sources) - set(_MERGE_SCALAR_FIELDS) + if unsupported_fields: + raise AddressBookError( + f"Unsupported merge field source: {sorted(unsupported_fields)[0]}." + ) + for field_name, source_id in payload.field_sources.items(): + if source_id not in allowed_sources: + raise AddressBookError( + f"Merge source for {field_name} must be one of the merged contacts." + ) + + before_payload = _merge_state_payload(session, contacts) + record = ContactMergeRecord( + id=new_uuid(), + tenant_id=winner.tenant_id, + address_book_id=winner.address_book_id, + winner_contact_id=winner.id, + loser_contact_ids=[item.id for item in losers], + status="active", + reason=payload.reason.strip(), + survivorship={ + "field_sources": dict(payload.field_sources), + "contact_point_strategy": payload.contact_point_strategy, + "source_precedence": list(payload.source_precedence), + }, + decisions=[], + before_payload=before_payload, + after_payload={}, + before_hash=_evidence_hash(before_payload), + after_hash="0" * 64, + created_by_account_id=_account_id(principal), + provenance={"module": "addresses", "reversible": True}, + ) + session.add(record) + session.flush() + + decisions: list[dict[str, Any]] = [] + winner_provenance = copy.deepcopy( + winner.provenance if isinstance(winner.provenance, dict) else {} + ) + raw_field_sources = winner_provenance.get("field_sources") + retained_field_sources = ( + copy.deepcopy(raw_field_sources) if isinstance(raw_field_sources, dict) else {} + ) + for field_name in _MERGE_SCALAR_FIELDS: + selected = _merge_field_source( + field_name, + contacts, + explicit_source_id=payload.field_sources.get(field_name), + source_precedence=payload.source_precedence, + ) + value = getattr(selected, field_name) + if field_name == "display_name" and not _has_merge_value(value): + selected = winner + value = winner.display_name + setattr(winner, field_name, copy.deepcopy(value)) + retained_field_sources[field_name] = { + "source_kind": selected.source_kind, + "source_ref": selected.source_ref or f"addresses:contact:{selected.id}", + "source_revision": selected.source_revision, + "source_contact_id": selected.id, + } + decisions.append( + { + "field_path": field_name, + "source_contact_id": selected.id, + "source_kind": selected.source_kind, + "reason_code": ( + "addresses.merge.explicit_survivor" + if field_name in payload.field_sources + else "addresses.merge.precedence_survivor" + if payload.source_precedence + else "addresses.merge.non_empty_survivor" + ), + } + ) + winner_provenance["field_sources"] = retained_field_sources + winner.provenance = winner_provenance + if payload.contact_point_strategy == "union": + winner.tags = _merge_tags(contacts) + decisions.append( + { + "field_path": "tags", + "source_contact_ids": contact_ids, + "reason_code": "addresses.merge.union", + } + ) + point_map = _merge_contact_points( + winner, + losers, + strategy=payload.contact_point_strategy, + merge_record_id=record.id, + decisions=decisions, + ) + session.flush() + _copy_merge_governance( + session, + winner=winner, + losers=losers, + point_map=point_map, + merge_record_id=record.id, + principal=principal, + ) + _redirect_address_list_entries( + session, + winner=winner, + losers=losers, + point_map=point_map, + ) + merged_at = utcnow() + for loser in losers: + loser.deleted_at = merged_at + loser.updated_by_account_id = _account_id(principal) + session.add( + ContactRedirect( + tenant_id=winner.tenant_id, + source_contact_id=loser.id, + target_contact_id=winner.id, + merge_record_id=record.id, + ) + ) + _record_address_contact_change( + session, + principal, + contact=loser, + operation="merged_redirect", + merge_record_id=record.id, + ) + winner.updated_by_account_id = _account_id(principal) + _record_address_contact_change( + session, + principal, + contact=winner, + operation="merged", + merge_record_id=record.id, + ) + session.flush() + after_payload = _merge_state_payload(session, contacts) + record.decisions = decisions + record.after_payload = after_payload + record.after_hash = _evidence_hash(after_payload) + return record + + +def recover_contact_merge( + session: Session, + principal: ApiPrincipal, + merge_id: str, + payload: ContactMergeRecoveryRequest, + *, + action: str, +) -> ContactMergeRecord: + if action not in {"undo", "split"}: + raise AddressBookError("Unsupported merge recovery action.") + record = _visible_merge_record(session, principal, merge_id) + if record.status != "active": + raise AddressBookError("This merge has already been recovered.") + if payload.expected_after_hash != record.after_hash: + raise AddressBookError("Merge recovery evidence does not match the recorded post-merge state.") + ids = [record.winner_contact_id, *record.loser_contact_ids] + contacts = [get_visible_contact(session, principal, item, include_deleted=True) for item in ids] + current_payload = _merge_state_payload(session, contacts) + if _evidence_hash(current_payload) != record.after_hash: + raise AddressBookError( + "Contacts or list memberships changed after this merge; reconcile those edits before recovery." + ) + _require_mutable_book(contacts[0].address_book) + + snapshots = { + str(item["id"]): item + for item in record.before_payload.get("contacts", []) + if isinstance(item, dict) and item.get("id") + } + for contact in contacts: + snapshot = snapshots.get(contact.id) + if snapshot is None: + raise AddressBookError("Merge recovery evidence is incomplete.") + _restore_contact_evidence(session, contact, snapshot, merge_record_id=record.id) + session.flush() + for item in record.before_payload.get("address_list_entries", []): + if not isinstance(item, dict) or not item.get("id"): + continue + entry = session.get(AddressListEntry, str(item["id"])) + if entry is None: + raise AddressBookError("An address-list membership needed for recovery no longer exists.") + entry.contact_id = str(item["contact_id"]) + entry.contact_email_id = str(item["contact_email_id"]) if item.get("contact_email_id") else None + entry.contact_postal_address_id = ( + str(item["contact_postal_address_id"]) + if item.get("contact_postal_address_id") + else None + ) + entry.target_kind = str(item["target_kind"]) + entry.label = str(item["label"]) if item.get("label") else None + entry.order_index = int(item.get("order_index") or 0) + recovered_at = utcnow() + redirects = session.query(ContactRedirect).filter( + ContactRedirect.merge_record_id == record.id, + ContactRedirect.ended_at.is_(None), + ).all() + for redirect in redirects: + redirect.ended_at = recovered_at + record.status = "split" if action == "split" else "undone" + record.recovered_at = recovered_at + record.recovered_by_account_id = _account_id(principal) + record.recovery_action = action + record.recovery_reason = payload.reason.strip() + for contact in contacts: + _record_address_contact_change( + session, + principal, + contact=contact, + operation=f"merge_{action}", + merge_record_id=record.id, + ) + return record + + +def list_contact_merges( + session: Session, + principal: ApiPrincipal, + *, + address_book_id: str | None = None, + contact_id: str | None = None, + limit: int = 100, +) -> list[ContactMergeRecord]: + visible_book_ids = [item.id for item in list_address_books(session, principal, include_deleted=True)] + if not visible_book_ids: + return [] + query = session.query(ContactMergeRecord).filter( + ContactMergeRecord.address_book_id.in_(visible_book_ids) + ) + if address_book_id: + get_visible_address_book(session, principal, address_book_id, include_deleted=True) + query = query.filter(ContactMergeRecord.address_book_id == address_book_id) + if contact_id: + get_visible_contact(session, principal, contact_id, include_deleted=True) + rows = query.order_by(ContactMergeRecord.created_at.desc()).limit(500).all() + if contact_id: + rows = [ + item + for item in rows + if item.winner_contact_id == contact_id + or contact_id in (item.loser_contact_ids or []) + ] + return rows[: max(1, min(limit, 500))] + + +def resolve_contact_redirect( + session: Session, + principal: ApiPrincipal, + contact_id: str, +) -> ContactRedirectResolution: + get_visible_contact(session, principal, contact_id, include_deleted=True) + current_id = contact_id + chain: list[str] = [] + merge_ids: list[str] = [] + seen = {contact_id} + for _ in range(20): + redirect = ( + session.query(ContactRedirect) + .filter( + ContactRedirect.source_contact_id == current_id, + ContactRedirect.ended_at.is_(None), + ) + .order_by(ContactRedirect.created_at.desc()) + .first() + ) + if redirect is None: + break + if redirect.target_contact_id in seen: + raise AddressBookError("Contact redirect cycle detected.") + current_id = redirect.target_contact_id + seen.add(current_id) + chain.append(current_id) + merge_ids.append(redirect.merge_record_id) + get_visible_contact(session, principal, current_id, include_deleted=True) + return ContactRedirectResolution( + requested_contact_id=contact_id, + resolved_contact_id=current_id, + redirected=current_id != contact_id, + redirect_chain=tuple(chain), + merge_record_ids=tuple(merge_ids), + ) + + +_MERGE_SCALAR_FIELDS = ( + "display_name", + "given_name", + "family_name", + "organization", + "role_title", + "note", +) + + +def _validate_quality_contact_point( + contact: Contact, + channel: str, + point_id: str | None, +) -> None: + if point_id is None: + return + point_ids: set[str] + if channel == "email": + point_ids = {item.id for item in contact.emails} + elif channel == "postal": + point_ids = {item.id for item in contact.postal_addresses} + elif channel in {"internal_mail", "portal"}: + point_ids = set() + else: + point_ids = {item.id for item in contact.phones} + if point_id not in point_ids: + raise AddressBookError("The selected quality contact point does not belong to this contact and channel.") + + +def _duplicate_features( + contact: Contact, +) -> list[tuple[tuple[str, str], ContactDuplicateFeature]]: + rows: list[tuple[tuple[str, str], ContactDuplicateFeature]] = [] + for item in contact.emails: + value = item.normalized_email or _normalize_email(item.email) + if value: + rows.append( + (("email", value), ContactDuplicateFeature("email_exact", "Same email address", 90, item.email)) + ) + for item in contact.phones: + value = item.normalized_phone or _normalize_phone(item.phone) + if len(value.lstrip("+")) >= 6: + rows.append( + (("phone", value), ContactDuplicateFeature("phone_exact", "Same phone number", 80, item.phone)) + ) + for item in contact.postal_addresses: + value = _postal_match_key(item) + if value: + rows.append( + (("postal", value), ContactDuplicateFeature("postal_exact", "Same postal address", 65, _postal_display_value(item))) + ) + name = _normalize_match_text(contact.display_name) + organization = _normalize_match_text(contact.organization) + if name and organization: + rows.append( + (("name_org", f"{name}|{organization}"), ContactDuplicateFeature("name_organization_exact", "Same name and organization", 55, f"{contact.display_name} · {contact.organization}")) + ) + elif name: + rows.append( + (("name", name), ContactDuplicateFeature("name_exact", "Same display name", 35, contact.display_name)) + ) + return rows + + +def _postal_match_key(item: ContactPostalAddress) -> str: + normalized = dict(item.normalized_value or {}) + values = [ + str(normalized.get(key) or _normalize_match_text(getattr(item, key))) + for key in ("street", "postal_code", "locality", "region", "country") + ] + return "|".join(values) if any(values) else "" + + +def _postal_display_value(item: ContactPostalAddress) -> str: + return ", ".join( + value + for value in ( + item.street, + " ".join(part for part in (item.postal_code, item.locality) if part), + item.region, + item.country, + ) + if value + ) + + +def _has_merge_value(value: Any) -> bool: + return value not in (None, "", [], {}) + + +def _merge_field_source( + field_name: str, + contacts: list[Contact], + *, + explicit_source_id: str | None, + source_precedence: list[str], +) -> Contact: + if explicit_source_id: + return next(item for item in contacts if item.id == explicit_source_id) + available = [item for item in contacts if _has_merge_value(getattr(item, field_name))] + if not available: + return contacts[0] + if not source_precedence: + return available[0] + ranks = {kind: len(source_precedence) - index for index, kind in enumerate(source_precedence)} + return max( + available, + key=lambda item: ( + ranks.get(item.source_kind, 0), + _source_precedence(item.source_kind), + item.id == contacts[0].id, + ), + ) + + +def _merge_tags(contacts: list[Contact]) -> list[str]: + seen: set[str] = set() + result: list[str] = [] + for contact in contacts: + for value in contact.tags or []: + key = value.casefold() + if key not in seen: + seen.add(key) + result.append(value) + return result + + +def _merge_contact_points( + winner: Contact, + losers: list[Contact], + *, + strategy: str, + merge_record_id: str, + decisions: list[dict[str, Any]], +) -> dict[str, str]: + point_map: dict[str, str] = {} + if strategy == "winner_only": + return point_map + email_keys = { + item.normalized_email or _normalize_email(item.email): item + for item in winner.emails + if item.normalized_email or _normalize_email(item.email) + } + phone_keys = { + item.normalized_phone or _normalize_phone(item.phone): item + for item in winner.phones + if item.normalized_phone or _normalize_phone(item.phone) + } + postal_keys = { + _postal_match_key(item): item + for item in winner.postal_addresses + if _postal_match_key(item) + } + for loser in losers: + for item in loser.emails: + key = item.normalized_email or _normalize_email(item.email) + existing = email_keys.get(key) + created = existing is None + if existing is None: + existing = ContactEmail( + id=new_uuid(), + label=item.label, + email=item.email, + original_email=item.original_email or item.email, + normalized_email=key, + provenance={ + **dict(item.provenance or {}), + "merge_record_id": merge_record_id, + "copied_from_contact_id": loser.id, + "copied_from_contact_point_id": item.id, + }, + is_primary=item.is_primary and not any(row.is_primary for row in winner.emails), + order_index=len(winner.emails), + ) + winner.emails.append(existing) + email_keys[key] = existing + point_map[item.id] = existing.id + decisions.append(_contact_point_merge_decision("email", item.id, existing.id, loser.id, created)) + for item in loser.phones: + key = item.normalized_phone or _normalize_phone(item.phone) + existing = phone_keys.get(key) + created = existing is None + if existing is None: + existing = ContactPhone( + id=new_uuid(), + label=item.label, + phone=item.phone, + original_phone=item.original_phone or item.phone, + normalized_phone=key, + provenance={ + **dict(item.provenance or {}), + "merge_record_id": merge_record_id, + "copied_from_contact_id": loser.id, + "copied_from_contact_point_id": item.id, + }, + is_primary=item.is_primary and not any(row.is_primary for row in winner.phones), + order_index=len(winner.phones), + ) + winner.phones.append(existing) + phone_keys[key] = existing + point_map[item.id] = existing.id + decisions.append(_contact_point_merge_decision("phone", item.id, existing.id, loser.id, created)) + for item in loser.postal_addresses: + key = _postal_match_key(item) + existing = postal_keys.get(key) + created = existing is None + if existing is None: + existing = ContactPostalAddress( + id=new_uuid(), + label=item.label, + street=item.street, + postal_code=item.postal_code, + locality=item.locality, + region=item.region, + country=item.country, + original_value=copy.deepcopy(item.original_value or {}), + normalized_value=copy.deepcopy(item.normalized_value or {}), + provenance={ + **dict(item.provenance or {}), + "merge_record_id": merge_record_id, + "copied_from_contact_id": loser.id, + "copied_from_contact_point_id": item.id, + }, + is_primary=item.is_primary and not any(row.is_primary for row in winner.postal_addresses), + order_index=len(winner.postal_addresses), + ) + winner.postal_addresses.append(existing) + postal_keys[key] = existing + point_map[item.id] = existing.id + decisions.append(_contact_point_merge_decision("postal", item.id, existing.id, loser.id, created)) + return point_map + + +def _contact_point_merge_decision( + channel: str, + source_id: str, + selected_id: str, + source_contact_id: str, + created: bool, +) -> dict[str, Any]: + return { + "field_path": f"{channel}.{selected_id}", + "source_contact_id": source_contact_id, + "source_contact_point_id": source_id, + "selected_contact_point_id": selected_id, + "reason_code": "addresses.merge.union" if created else "addresses.merge.normalized_duplicate", + } + + +def _copy_merge_governance( + session: Session, + *, + winner: Contact, + losers: list[Contact], + point_map: dict[str, str], + merge_record_id: str, + principal: ApiPrincipal, +) -> None: + now = _aware_time(utcnow()) + for loser in losers: + for rule in loser.channel_rules: + if rule.contact_point_id is not None and rule.contact_point_id not in point_map: + continue + winner.channel_rules.append( + ContactChannelRule( + tenant_id=winner.tenant_id, + channel=rule.channel, + purpose=rule.purpose, + contact_point_id=point_map.get(rule.contact_point_id or ""), + decision=rule.decision, + legal_basis=rule.legal_basis, + evidence_ref=rule.evidence_ref, + reason=rule.reason, + preference_rank=rule.preference_rank, + locale=rule.locale, + effective_from=rule.effective_from, + effective_until=rule.effective_until, + created_by_account_id=_account_id(principal), + metadata_={ + **dict(rule.metadata_ or {}), + "merge_record_id": merge_record_id, + "copied_from_contact_id": loser.id, + "copied_from_rule_id": rule.id, + }, + ) + ) + for decision in loser.quality_decisions: + if ( + decision.contact_point_id is not None + and decision.contact_point_id not in point_map + ): + continue + if _aware_time(decision.effective_from) > now or ( + decision.effective_until is not None + and _aware_time(decision.effective_until) <= now + ): + continue + winner.quality_decisions.append( + ContactPointQualityDecision( + tenant_id=winner.tenant_id, + channel=decision.channel, + contact_point_id=point_map.get(decision.contact_point_id or ""), + state=decision.state, + reason_code=decision.reason_code, + reason=decision.reason, + evidence_ref=decision.evidence_ref, + effective_from=decision.effective_from, + effective_until=decision.effective_until, + created_by_account_id=_account_id(principal), + metadata_={ + **dict(decision.metadata_ or {}), + "merge_record_id": merge_record_id, + "copied_from_contact_id": loser.id, + "copied_from_quality_decision_id": decision.id, + }, + ) + ) + + +def _redirect_address_list_entries( + session: Session, + *, + winner: Contact, + losers: list[Contact], + point_map: dict[str, str], +) -> None: + loser_ids = [item.id for item in losers] + entries = session.query(AddressListEntry).filter( + AddressListEntry.contact_id.in_(loser_ids) + ).all() + for entry in entries: + entry.contact_id = winner.id + if entry.contact_email_id: + entry.contact_email_id = point_map.get(entry.contact_email_id) + if entry.contact_postal_address_id: + entry.contact_postal_address_id = point_map.get(entry.contact_postal_address_id) + entry.target_kind = _address_list_entry_kind( + next((item for item in winner.emails if item.id == entry.contact_email_id), None), + next((item for item in winner.postal_addresses if item.id == entry.contact_postal_address_id), None), + ) + + +def _contact_evidence_payload(contact: Contact) -> dict[str, Any]: + return { + "id": contact.id, + "tenant_id": contact.tenant_id, + "address_book_id": contact.address_book_id, + "display_name": contact.display_name, + "given_name": contact.given_name, + "family_name": contact.family_name, + "organization": contact.organization, + "role_title": contact.role_title, + "note": contact.note, + "tags": list(contact.tags or []), + "source_kind": contact.source_kind, + "source_ref": contact.source_ref, + "source_payload_kind": contact.source_payload_kind, + "source_payload_raw": contact.source_payload_raw, + "source_revision": contact.source_revision, + "provenance": copy.deepcopy(contact.provenance or {}), + "metadata": copy.deepcopy(contact.metadata_ or {}), + "created_by_account_id": contact.created_by_account_id, + "updated_by_account_id": contact.updated_by_account_id, + "deleted_at": _evidence_datetime(contact.deleted_at), + "emails": [ + { + "id": item.id, + "label": item.label, + "email": item.email, + "original_email": item.original_email, + "normalized_email": item.normalized_email, + "provenance": copy.deepcopy(item.provenance or {}), + "is_primary": item.is_primary, + "order_index": item.order_index, + } + for item in contact.emails + ], + "phones": [ + { + "id": item.id, + "label": item.label, + "phone": item.phone, + "original_phone": item.original_phone, + "normalized_phone": item.normalized_phone, + "provenance": copy.deepcopy(item.provenance or {}), + "is_primary": item.is_primary, + "order_index": item.order_index, + } + for item in contact.phones + ], + "postal_addresses": [ + { + "id": item.id, + "label": item.label, + "street": item.street, + "postal_code": item.postal_code, + "locality": item.locality, + "region": item.region, + "country": item.country, + "original_value": copy.deepcopy(item.original_value or {}), + "normalized_value": copy.deepcopy(item.normalized_value or {}), + "provenance": copy.deepcopy(item.provenance or {}), + "is_primary": item.is_primary, + "order_index": item.order_index, + } + for item in contact.postal_addresses + ], + "channel_rules": [ + { + "id": item.id, + "channel": item.channel, + "purpose": item.purpose, + "contact_point_id": item.contact_point_id, + "decision": item.decision, + "legal_basis": item.legal_basis, + "evidence_ref": item.evidence_ref, + "reason": item.reason, + "preference_rank": item.preference_rank, + "locale": item.locale, + "effective_from": _evidence_datetime(item.effective_from), + "effective_until": _evidence_datetime(item.effective_until), + "metadata": copy.deepcopy(item.metadata_ or {}), + } + for item in contact.channel_rules + ], + "quality_decisions": [ + { + "id": item.id, + "channel": item.channel, + "contact_point_id": item.contact_point_id, + "state": item.state, + "reason_code": item.reason_code, + "reason": item.reason, + "evidence_ref": item.evidence_ref, + "effective_from": _evidence_datetime(item.effective_from), + "effective_until": _evidence_datetime(item.effective_until), + "metadata": copy.deepcopy(item.metadata_ or {}), + } + for item in contact.quality_decisions + ], + } + + +def _address_list_entry_evidence(entry: AddressListEntry) -> dict[str, Any]: + return { + "id": entry.id, + "address_list_id": entry.address_list_id, + "contact_id": entry.contact_id, + "contact_email_id": entry.contact_email_id, + "contact_postal_address_id": entry.contact_postal_address_id, + "target_kind": entry.target_kind, + "label": entry.label, + "order_index": entry.order_index, + } + + +def _merge_state_payload(session: Session, contacts: list[Contact]) -> dict[str, Any]: + ids = [item.id for item in contacts] + session.flush() + entries = session.query(AddressListEntry).filter(AddressListEntry.contact_id.in_(ids)).all() + return { + "contacts": [ + _contact_evidence_payload(item) + for item in sorted(contacts, key=lambda row: row.id) + ], + "address_list_entries": [ + _address_list_entry_evidence(item) + for item in sorted(entries, key=lambda row: row.id) + ], + } + + +def _evidence_hash(payload: dict[str, Any]) -> str: + return hashlib.sha256( + json.dumps(payload, sort_keys=True, separators=(",", ":"), ensure_ascii=True).encode("utf-8") + ).hexdigest() + + +def _evidence_datetime(value: datetime | None) -> str | None: + if value is None: + return None + return _aware_time(value).astimezone(UTC).isoformat() + + +def _visible_merge_record( + session: Session, + principal: ApiPrincipal, + merge_id: str, +) -> ContactMergeRecord: + record = session.get(ContactMergeRecord, merge_id) + if record is None: + raise AddressBookError("Contact merge record not found.") + get_visible_address_book(session, principal, record.address_book_id, include_deleted=True) + return record + + +def _restore_contact_evidence( + session: Session, + contact: Contact, + snapshot: dict[str, Any], + *, + merge_record_id: str, +) -> None: + for field_name in ( + "display_name", + "given_name", + "family_name", + "organization", + "role_title", + "note", + "source_kind", + "source_ref", + "source_payload_kind", + "source_payload_raw", + "source_revision", + "created_by_account_id", + "updated_by_account_id", + ): + setattr(contact, field_name, snapshot.get(field_name)) + contact.tags = list(snapshot.get("tags") or []) + contact.provenance = copy.deepcopy(snapshot.get("provenance") or {}) + contact.metadata_ = copy.deepcopy(snapshot.get("metadata") or {}) + contact.deleted_at = ( + datetime.fromisoformat(str(snapshot["deleted_at"])) + if snapshot.get("deleted_at") + else None + ) + for rule in list(contact.channel_rules): + if (rule.metadata_ or {}).get("merge_record_id") == merge_record_id: + session.delete(rule) + for decision in list(contact.quality_decisions): + if (decision.metadata_ or {}).get("merge_record_id") == merge_record_id: + session.delete(decision) + contact.emails.clear() + contact.phones.clear() + contact.postal_addresses.clear() + session.flush() + contact.emails.extend( + ContactEmail( + id=str(item["id"]), + label=item.get("label"), + email=str(item.get("email") or ""), + original_email=str(item.get("original_email") or item.get("email") or ""), + normalized_email=str(item.get("normalized_email") or ""), + provenance=copy.deepcopy(item.get("provenance") or {}), + is_primary=bool(item.get("is_primary")), + order_index=int(item.get("order_index") or 0), + ) + for item in snapshot.get("emails", []) + if isinstance(item, dict) and item.get("id") + ) + contact.phones.extend( + ContactPhone( + id=str(item["id"]), + label=item.get("label"), + phone=str(item.get("phone") or ""), + original_phone=str(item.get("original_phone") or item.get("phone") or ""), + normalized_phone=str(item.get("normalized_phone") or ""), + provenance=copy.deepcopy(item.get("provenance") or {}), + is_primary=bool(item.get("is_primary")), + order_index=int(item.get("order_index") or 0), + ) + for item in snapshot.get("phones", []) + if isinstance(item, dict) and item.get("id") + ) + contact.postal_addresses.extend( + ContactPostalAddress( + id=str(item["id"]), + label=item.get("label"), + street=item.get("street"), + postal_code=item.get("postal_code"), + locality=item.get("locality"), + region=item.get("region"), + country=item.get("country"), + original_value=copy.deepcopy(item.get("original_value") or {}), + normalized_value=copy.deepcopy(item.get("normalized_value") or {}), + provenance=copy.deepcopy(item.get("provenance") or {}), + is_primary=bool(item.get("is_primary")), + order_index=int(item.get("order_index") or 0), + ) + for item in snapshot.get("postal_addresses", []) + if isinstance(item, dict) and item.get("id") + ) + + +def _aware_time(value: datetime) -> datetime: + return value if value.tzinfo is not None else value.replace(tzinfo=UTC) + + def export_address_book_vcard(session: Session, principal: ApiPrincipal, address_book_id: str) -> tuple[AddressBook, str]: book = get_visible_address_book(session, principal, address_book_id) contacts = ( diff --git a/tests/test_addresses_service.py b/tests/test_addresses_service.py index 241bdf1..9d144b2 100644 --- a/tests/test_addresses_service.py +++ b/tests/test_addresses_service.py @@ -51,9 +51,13 @@ from govoplan_addresses.backend.db.models import ( Contact, ContactChannelRule, ContactEmail, + ContactFieldProvenance, + ContactMergeRecord, ContactPhone, ContactPointSnapshot, + ContactPointQualityDecision, ContactPostalAddress, + ContactRedirect, ) from govoplan_addresses.backend.schemas import ( AddressBookCreateRequest, @@ -71,13 +75,27 @@ from govoplan_addresses.backend.schemas import ( ContactCreateRequest, ContactChannelRuleCreateRequest, ContactEmailPayload, + ContactMergeRecoveryRequest, + ContactMergeRequest, + ContactPhonePayload, + ContactPointQualityDecisionCreateRequest, ContactPostalAddressPayload, ContactPointSnapshotResponse, + ContactUpdateRequest, ) from govoplan_addresses.backend.manifest import manifest -from govoplan_addresses.backend.router import _sync_source_response +from govoplan_addresses.backend.router import ( + _sync_source_response, + api_create_address_list_entry, + api_create_contact, + api_delete_address_list_entry, + api_delete_contact, + api_restore_contact, + api_update_contact, +) from govoplan_addresses.backend.service import ( AddressBookError, + address_quality_summary, address_book_contact_counts, address_list_entry_counts, create_address_book, @@ -86,6 +104,7 @@ from govoplan_addresses.backend.service import ( create_carddav_sync_source, create_contact, create_contact_channel_rule, + create_contact_quality_decision, create_sync_source, count_contacts, delete_address_list_entry, @@ -100,6 +119,8 @@ from govoplan_addresses.backend.service import ( list_address_books, list_contacts, list_contact_channel_rules, + list_contact_field_provenance, + list_contact_merges, list_sync_conflicts, list_sync_diagnostics, list_sync_sources, @@ -109,11 +130,16 @@ from govoplan_addresses.backend.service import ( record_sync_tombstone, run_sync_source, preview_sync_source, + merge_contacts, + recover_contact_merge, restore_contact, + resolve_contact_redirect, resolve_sync_conflict, start_sync_attempt, finish_sync_attempt, update_sync_source, + update_contact, + suggest_duplicate_contacts, resolve_trusted_deployment_carddav_credential_ref, _carddav_client_for_source, ) @@ -207,6 +233,10 @@ class AddressServiceTest(unittest.TestCase): ContactPostalAddress.__table__, ContactChannelRule.__table__, ContactPointSnapshot.__table__, + ContactPointQualityDecision.__table__, + ContactMergeRecord.__table__, + ContactRedirect.__table__, + ContactFieldProvenance.__table__, AddressListEntry.__table__, AddressSyncSource.__table__, AddressSyncTombstone.__table__, @@ -279,6 +309,75 @@ class AddressServiceTest(unittest.TestCase): self.session.commit() self.assertEqual([item.id for item in list_contacts(self.session, self.principal, address_book_id=book.id)], [contact.id]) + def test_contact_and_relationship_routes_emit_value_free_audit_evidence(self) -> None: + book = create_address_book( + self.session, + self.principal, + AddressBookCreateRequest(scope_type="user", name="Audited"), + ) + self.session.commit() + with patch("govoplan_addresses.backend.router.audit_from_principal") as audit: + created = api_create_contact( + book.id, + ContactCreateRequest( + display_name="Ada Lovelace", + emails=[ContactEmailPayload(email="ada@example.local")], + ), + self.principal, + self.session, + ) + create_details = audit.call_args.kwargs["details"] + self.assertEqual(audit.call_args.kwargs["action"], "addresses.contact_created") + self.assertEqual(create_details["contact_point_counts"]["email"], 1) + self.assertNotIn("ada@example.local", repr(create_details)) + original_email_id = create_details["contact_point_ids"]["email"][0] + + updated = api_update_contact( + created.id, + ContactUpdateRequest( + emails=[ContactEmailPayload(email="ada.new@example.local")] + ), + self.principal, + self.session, + ) + update_details = audit.call_args.kwargs["details"] + self.assertEqual(audit.call_args.kwargs["action"], "addresses.contact_updated") + self.assertEqual(update_details["previous_contact_point_ids"]["email"], [original_email_id]) + self.assertNotEqual(update_details["contact_point_ids"]["email"], [original_email_id]) + self.assertNotIn("ada.new@example.local", repr(update_details)) + + api_delete_contact(created.id, self.principal, self.session) + self.assertEqual(audit.call_args.kwargs["action"], "addresses.contact_deleted") + api_restore_contact(created.id, self.principal, self.session) + self.assertEqual(audit.call_args.kwargs["action"], "addresses.contact_restored") + + address_list = create_address_list( + self.session, + self.principal, + book.id, + AddressListCreateRequest(name="Audited list"), + ) + self.session.commit() + entry = api_create_address_list_entry( + address_list.id, + AddressListEntryCreateRequest( + contact_id=updated.id, + contact_email_id=updated.emails[0].id, + ), + self.principal, + self.session, + ) + self.assertEqual( + audit.call_args.kwargs["action"], + "addresses.address_list_entry_created", + ) + self.assertEqual(audit.call_args.kwargs["details"]["contact_id"], updated.id) + api_delete_address_list_entry(entry.id, self.principal, self.session) + self.assertEqual( + audit.call_args.kwargs["action"], + "addresses.address_list_entry_deleted", + ) + def test_contact_windows_report_exact_totals(self) -> None: book = create_address_book( self.session, @@ -605,6 +704,334 @@ END:VCARD self.assertEqual(expired.explanations[0].code, "addresses.channel_fact.expired") self.assertEqual(list_contact_channel_rules(self.session, self.principal, contact.id)[0].id, rule.id) + def test_contact_quality_preserves_originals_provenance_and_excludes_invalid_targets(self) -> None: + book = create_address_book( + self.session, + self.principal, + AddressBookCreateRequest(scope_type="user", name="Quality review"), + ) + self.session.flush() + contact = create_contact( + self.session, + self.principal, + book.id, + ContactCreateRequest( + display_name="Ada Lovelace", + emails=[ContactEmailPayload(email=" Ada@Example.LOCAL ")], + phones=[ContactPhonePayload(phone="+49 (30) 123 45")], + postal_addresses=[ + ContactPostalAddressPayload( + street=" Main Street 1 ", + postal_code=" 10115 ", + locality=" Berlin ", + country=" Germany ", + ) + ], + provenance={ + "field_visibility": { + "organization": "restricted", + } + }, + ), + ) + self.session.commit() + self.session.refresh(contact) + + self.assertEqual(contact.emails[0].email, "Ada@Example.LOCAL") + self.assertEqual(contact.emails[0].original_email, " Ada@Example.LOCAL ") + self.assertEqual(contact.emails[0].normalized_email, "ada@example.local") + self.assertEqual(contact.phones[0].original_phone, "+49 (30) 123 45") + self.assertEqual(contact.phones[0].normalized_phone, "+493012345") + self.assertEqual(contact.postal_addresses[0].original_value["street"], " Main Street 1 ") + self.assertEqual(contact.postal_addresses[0].normalized_value["street"], "main street 1") + + initial_provenance = list_contact_field_provenance( + self.session, + self.principal, + contact.id, + current_only=True, + ) + self.assertTrue(any(item.field_path == "display_name" for item in initial_provenance)) + self.assertTrue(any(item.field_path.endswith(".email") for item in initial_provenance)) + self.assertEqual( + next(item for item in initial_provenance if item.field_path == "organization").visibility, + "restricted", + ) + + update_contact( + self.session, + self.principal, + contact.id, + ContactUpdateRequest(organization="Analytical Engine Office"), + ) + quality = create_contact_quality_decision( + self.session, + self.principal, + contact.id, + ContactPointQualityDecisionCreateRequest( + channel="email", + contact_point_id=contact.emails[0].id, + state="undeliverable", + reason_code="addresses.quality.smtp_hard_bounce", + reason="The remote server rejected this address permanently.", + evidence_ref="mail:delivery:42", + ), + ) + self.session.commit() + + history = list_contact_field_provenance( + self.session, + self.principal, + contact.id, + ) + self.assertTrue(any(not item.selected for item in history)) + current_organization = next( + item + for item in history + if item.field_path == "organization" and item.selected + ) + self.assertEqual(current_organization.value, "Analytical Engine Office") + self.assertEqual(current_organization.reason_code, "addresses.contact.quality_updated") + + facts = AddressesChannelFactsCapability().resolve_channel_facts( + self.session, + self.principal, + request=RecipientChannelFactsRequest( + tenant_id=self.principal.tenant_id, + source=DistributionSourceReference( + provider="addresses", + resource_type="contact", + resource_id=contact.id, + ), + recipient_key=f"contact:{contact.id}", + effective_at=utcnow() + timedelta(seconds=1), + purpose="campaign_delivery", + requested_channels=("email",), + ), + ) + self.assertEqual(facts.candidates[0].status, "invalid") + self.assertEqual(facts.candidates[0].reason_code, "addresses.quality.smtp_hard_bounce") + self.assertEqual(facts.candidates[0].decision_provenance["quality_decision_id"], quality.id) + + snapshot = AddressesRecipientSourceCapability().snapshot_address_book( + self.session, + self.principal, + address_book_id=book.id, + purpose="campaign_delivery", + ) + self.assertEqual(snapshot.recipients, ()) + self.assertEqual(snapshot.excluded[0].reason_code, "addresses.quality.smtp_hard_bounce") + + summary = address_quality_summary( + self.session, + self.principal, + address_book_id=book.id, + ) + self.assertEqual(summary.contact_count, 1) + self.assertEqual(summary.contact_point_count, 3) + self.assertEqual(summary.quality_counts["undeliverable"], 1) + self.assertEqual(summary.correction_count, 1) + self.assertEqual(summary.corrections[0].contact_id, contact.id) + + def test_duplicate_merge_recovery_preserves_references_and_rejects_tampering(self) -> None: + book = create_address_book( + self.session, + self.principal, + AddressBookCreateRequest(scope_type="user", name="Duplicate review"), + ) + self.session.flush() + winner = create_contact( + self.session, + self.principal, + book.id, + ContactCreateRequest( + display_name="Ada Lovelace", + organization="Analytical Engine Office", + emails=[ContactEmailPayload(email="ada@example.local")], + ), + ) + loser = create_contact( + self.session, + self.principal, + book.id, + ContactCreateRequest( + display_name="Ada Lovelace", + organization="Analytical Engine Office", + role_title="Mathematician", + emails=[ + ContactEmailPayload(email="ADA@example.local"), + ContactEmailPayload(email="ada.private@example.local"), + ], + ), + ) + address_list = create_address_list( + self.session, + self.principal, + book.id, + AddressListCreateRequest(name="Recipients"), + ) + self.session.flush() + original_loser_email_id = loser.emails[1].id + entry = create_address_list_entry( + self.session, + self.principal, + address_list.id, + AddressListEntryCreateRequest( + contact_id=loser.id, + contact_email_id=original_loser_email_id, + ), + ) + create_contact_quality_decision( + self.session, + self.principal, + loser.id, + ContactPointQualityDecisionCreateRequest( + channel="email", + contact_point_id=original_loser_email_id, + state="stale", + reason="This private address needs confirmation.", + ), + ) + self.session.commit() + + scan = suggest_duplicate_contacts( + self.session, + self.principal, + address_book_id=book.id, + ) + self.assertEqual(scan.scanned_contacts, 2) + self.assertEqual(scan.candidate_pairs, 1) + self.assertEqual(scan.suggestions[0].score, 100) + self.assertEqual(scan.suggestions[0].confidence, "strong") + self.assertEqual( + {feature.code for feature in scan.suggestions[0].features}, + {"email_exact", "name_organization_exact"}, + ) + + merge = merge_contacts( + self.session, + self.principal, + ContactMergeRequest( + winner_contact_id=winner.id, + duplicate_contact_ids=[loser.id], + reason="Confirmed duplicate record.", + field_sources={"role_title": loser.id}, + contact_point_strategy="union", + ), + ) + merge_id = merge.id + after_hash = merge.after_hash + winner_id = winner.id + loser_id = loser.id + entry_id = entry.id + self.session.commit() + self.session.expire_all() + + resolved = resolve_contact_redirect(self.session, self.principal, loser_id) + self.assertTrue(resolved.redirected) + self.assertEqual(resolved.resolved_contact_id, winner_id) + merged_winner = self.session.get(Contact, winner_id) + merged_loser = self.session.get(Contact, loser_id) + assert merged_winner is not None + assert merged_loser is not None + self.assertEqual(merged_winner.role_title, "Mathematician") + self.assertEqual( + {item.normalized_email for item in merged_winner.emails}, + {"ada@example.local", "ada.private@example.local"}, + ) + self.assertIsNotNone(merged_loser.deleted_at) + merged_entry = self.session.get(AddressListEntry, entry_id) + assert merged_entry is not None + self.assertEqual(merged_entry.contact_id, winner_id) + self.assertNotEqual(merged_entry.contact_email_id, original_loser_email_id) + self.assertTrue( + any( + item.state == "stale" + and item.contact_point_id == merged_entry.contact_email_id + for item in merged_winner.quality_decisions + ) + ) + retained_role_title = next( + item + for item in list_contact_field_provenance( + self.session, + self.principal, + winner_id, + current_only=True, + ) + if item.field_path == "role_title" + ) + self.assertEqual(retained_role_title.source_ref, f"addresses:contact:{loser_id}") + self.assertEqual(retained_role_title.metadata_["source_contact_id"], loser_id) + self.assertEqual(list_contact_merges(self.session, self.principal)[0].id, merge_id) + + merged_winner.note = "Changed after merge" + self.session.commit() + with self.assertRaisesRegex(AddressBookError, "changed after this merge"): + recover_contact_merge( + self.session, + self.principal, + merge_id, + ContactMergeRecoveryRequest( + reason="Correct the duplicate decision.", + expected_after_hash=after_hash, + ), + action="undo", + ) + self.session.rollback() + merged_winner = self.session.get(Contact, winner_id) + assert merged_winner is not None + merged_winner.note = None + self.session.commit() + + recovered = recover_contact_merge( + self.session, + self.principal, + merge_id, + ContactMergeRecoveryRequest( + reason="Correct the duplicate decision.", + expected_after_hash=after_hash, + ), + action="undo", + ) + self.session.commit() + self.assertEqual(recovered.status, "undone") + self.session.expire_all() + restored_winner = self.session.get(Contact, winner_id) + restored_loser = self.session.get(Contact, loser_id) + restored_entry = self.session.get(AddressListEntry, entry_id) + assert restored_winner is not None + assert restored_loser is not None + assert restored_entry is not None + self.assertIsNone(restored_winner.role_title) + self.assertIsNone(restored_loser.deleted_at) + self.assertEqual(restored_entry.contact_id, loser_id) + self.assertEqual(restored_entry.contact_email_id, original_loser_email_id) + self.assertFalse(resolve_contact_redirect(self.session, self.principal, loser_id).redirected) + + second_merge = merge_contacts( + self.session, + self.principal, + ContactMergeRequest( + winner_contact_id=winner_id, + duplicate_contact_ids=[loser_id], + reason="Re-run duplicate decision.", + ), + ) + self.session.commit() + split = recover_contact_merge( + self.session, + self.principal, + second_merge.id, + ContactMergeRecoveryRequest( + reason="Split records after review.", + expected_after_hash=second_merge.after_hash, + ), + action="split", + ) + self.session.commit() + self.assertEqual(split.status, "split") + def test_address_lists_group_contacts_and_expose_recipient_sources(self) -> None: book = create_address_book(self.session, self.principal, AddressBookCreateRequest(scope_type="user", name="Personal")) other_book = create_address_book(self.session, self.principal, AddressBookCreateRequest(scope_type="user", name="Other")) diff --git a/webui/src/api/addresses.ts b/webui/src/api/addresses.ts index c318aba..9d3803e 100644 --- a/webui/src/api/addresses.ts +++ b/webui/src/api/addresses.ts @@ -39,6 +39,11 @@ export type ContactEmail = { id?: string; label?: string | null; email: string; + original_email?: string; + normalized_email?: string; + provenance?: Record; + quality_state?: ContactPointQualityState; + quality_reason_code?: string | null; is_primary: boolean; }; @@ -46,6 +51,11 @@ export type ContactPhone = { id?: string; label?: string | null; phone: string; + original_phone?: string; + normalized_phone?: string; + provenance?: Record; + quality_state?: ContactPointQualityState; + quality_reason_code?: string | null; is_primary: boolean; }; @@ -57,9 +67,35 @@ export type ContactPostalAddress = { locality?: string | null; region?: string | null; country?: string | null; + original_value?: Record; + normalized_value?: Record; + provenance?: Record; + quality_state?: ContactPointQualityState; + quality_reason_code?: string | null; is_primary: boolean; }; +export type ContactPointQualityState = "valid" | "invalid" | "returned" | "stale" | "undeliverable"; + +export type ContactFieldProvenance = { + id: string; + contact_id: string; + field_path: string; + value?: unknown; + source_kind: string; + source_ref?: string | null; + source_revision?: string | null; + precedence: number; + selected: boolean; + reason_code: string; + explanation?: string | null; + visibility: "inherit" | "private" | "restricted" | "public"; + merge_record_id?: string | null; + created_by_account_id?: string | null; + metadata: Record; + created_at: string; +}; + export type Contact = { id: string; tenant_id?: string | null; @@ -79,11 +115,95 @@ export type Contact = { emails: ContactEmail[]; phones: ContactPhone[]; postal_addresses: ContactPostalAddress[]; + field_provenance?: ContactFieldProvenance[]; deleted_at?: string | null; created_at: string; updated_at: string; }; +export type ContactPointQualityDecision = { + id: string; + tenant_id?: string | null; + contact_id: string; + channel: "email" | "phone" | "postal" | "internal_mail" | "portal"; + contact_point_id?: string | null; + state: ContactPointQualityState; + reason_code: string; + reason?: string | null; + evidence_ref?: string | null; + effective_from: string; + effective_until?: string | null; + created_by_account_id?: string | null; + metadata: Record; + created_at: string; + updated_at: string; +}; + +export type ContactDuplicateFeature = { + code: string; + label: string; + weight: number; + value: string; +}; + +export type ContactDuplicateSuggestion = { + left: Contact; + right: Contact; + score: number; + confidence: "possible" | "likely" | "strong"; + features: ContactDuplicateFeature[]; +}; + +export type ContactDuplicateSuggestionList = { + suggestions: ContactDuplicateSuggestion[]; + scanned_contacts: number; + candidate_pairs: number; + truncated: boolean; +}; + +export type ContactMergeRecord = { + id: string; + tenant_id?: string | null; + address_book_id: string; + winner_contact_id: string; + loser_contact_ids: string[]; + status: string; + reason: string; + survivorship: Record; + decisions: Array>; + before_hash: string; + after_hash: string; + created_by_account_id?: string | null; + recovered_at?: string | null; + recovered_by_account_id?: string | null; + recovery_action?: string | null; + recovery_reason?: string | null; + provenance: Record; + created_at: string; + updated_at: string; +}; + +export type AddressQualityCorrection = { + contact_id: string; + display_name: string; + channel: "email" | "phone" | "postal" | "internal_mail" | "portal"; + contact_point_id?: string | null; + state: ContactPointQualityState; + reason_code: string; + reason?: string | null; + effective_from: string; +}; + +export type AddressQualitySummary = { + contact_count: number; + contact_point_count: number; + quality_counts: Record; + duplicate_suggestion_count: number; + correction_count: number; + corrections: AddressQualityCorrection[]; + truncated: boolean; +}; + export type AddressDistributionChannel = "email" | "postal" | "internal_mail" | "portal"; export type AddressChannelDecision = | "allowed" @@ -392,6 +512,14 @@ type ContactChannelRuleListResponse = { rules: ContactChannelRule[]; }; +type ContactPointQualityDecisionListResponse = { + decisions: ContactPointQualityDecision[]; +}; + +type ContactMergeRecordListResponse = { + merges: ContactMergeRecord[]; +}; + function queryString(params: Record): string { const search = new URLSearchParams(); for (const [key, value] of Object.entries(params)) { @@ -677,6 +805,110 @@ export function restoreContact(settings: ApiSettings, contactId: string): Promis return apiFetch(settings, `/api/v1/addresses/contacts/${contactId}/restore`, { method: "POST" }); } +export function getAddressQualitySummary(settings: ApiSettings, addressBookId: string): Promise { + return apiFetch(settings, `/api/v1/addresses/address-books/${addressBookId}/quality-summary`); +} + +export function listContactDuplicateSuggestions( + settings: ApiSettings, + addressBookId: string, + options: { contactId?: string | null; minimumScore?: number; limit?: number; scanLimit?: number } = {} +): Promise { + return apiFetch( + settings, + `/api/v1/addresses/address-books/${addressBookId}/duplicate-suggestions${queryString({ + contact_id: options.contactId, + minimum_score: options.minimumScore, + limit: options.limit, + scan_limit: options.scanLimit + })}` + ); +} + +export async function listContactQualityDecisions(settings: ApiSettings, contactId: string): Promise { + const response = await apiFetch( + settings, + `/api/v1/addresses/contacts/${contactId}/quality-decisions` + ); + return response.decisions; +} + +export function createContactQualityDecision( + settings: ApiSettings, + contactId: string, + payload: { + channel: ContactPointQualityDecision["channel"]; + contact_point_id?: string | null; + state: ContactPointQualityState; + reason_code?: string | null; + reason?: string | null; + evidence_ref?: string | null; + } +): Promise { + return apiFetch(settings, `/api/v1/addresses/contacts/${contactId}/quality-decisions`, { + method: "POST", + body: JSON.stringify(payload) + }); +} + +export function listContactProvenance( + settings: ApiSettings, + contactId: string, + options: { currentOnly?: boolean; limit?: number } = {} +): Promise { + return apiFetch( + settings, + `/api/v1/addresses/contacts/${contactId}/provenance${queryString({ + current_only: options.currentOnly ? "true" : null, + limit: options.limit + })}` + ); +} + +export async function listContactMerges( + settings: ApiSettings, + options: { addressBookId?: string | null; contactId?: string | null; limit?: number } = {} +): Promise { + const response = await apiFetch( + settings, + `/api/v1/addresses/contact-merges${queryString({ + address_book_id: options.addressBookId, + contact_id: options.contactId, + limit: options.limit + })}` + ); + return response.merges; +} + +export function mergeContacts( + settings: ApiSettings, + payload: { + winner_contact_id: string; + duplicate_contact_ids: string[]; + reason: string; + field_sources?: Record; + contact_point_strategy?: "union" | "winner_only"; + source_precedence?: string[]; + } +): Promise { + return apiFetch(settings, "/api/v1/addresses/contact-merges", { + method: "POST", + body: JSON.stringify(payload) + }); +} + +export function recoverContactMerge( + settings: ApiSettings, + merge: ContactMergeRecord, + action: "undo" | "split", + reason: string +): Promise { + return apiFetch(settings, `/api/v1/addresses/contact-merges/${merge.id}/${action}`, { + method: "POST", + body: JSON.stringify({ reason, expected_after_hash: merge.after_hash }) + }); +} + export async function listContactChannelRules(settings: ApiSettings, contactId: string): Promise { const response = await apiFetch( settings, diff --git a/webui/src/features/addressbook/AddressBookPage.tsx b/webui/src/features/addressbook/AddressBookPage.tsx index 956e698..708f260 100644 --- a/webui/src/features/addressbook/AddressBookPage.tsx +++ b/webui/src/features/addressbook/AddressBookPage.tsx @@ -1,4 +1,4 @@ -import { Download, Edit3, Link2, Plus, RefreshCw, RotateCcw, Save, Search, ShieldCheck, Trash2, Upload, UserPlus, X } from "lucide-react"; +import { Download, Edit3, GitMerge, History, Link2, Plus, RefreshCw, RotateCcw, Save, Search, ShieldCheck, Trash2, Upload, UserPlus, X } from "lucide-react"; import { useCallback, useEffect, useMemo, useState, type DragEvent as ReactDragEvent, type FormEvent } from "react"; import { ApiError, @@ -12,6 +12,7 @@ import { formatDateTime, FormField, LoadingFrame, + MetricCard, PasswordField, SegmentedControl, SelectionList, @@ -31,6 +32,7 @@ import { createCardDavSyncSource, createContact, createContactChannelRule, + createContactQualityDecision, deleteAddressBook, deleteAddressList, deleteAddressListEntry, @@ -41,6 +43,7 @@ import { exportAddressBookVcards, exportContactVcard, importAddressBookVcards, + getAddressQualitySummary, listAddressBooks, listAddressCredentials, listAddressListEntries, @@ -52,7 +55,12 @@ import { listContacts, listContactsPage, listContactChannelRules, + listContactDuplicateSuggestions, + listContactMerges, + listContactProvenance, previewAddressSyncSource, + mergeContacts, + recoverContactMerge, restoreAddressBook, restoreAddressList, restoreContact, @@ -75,9 +83,15 @@ import { type AddressSyncPlan, type AddressSyncSource, type AddressSyncTombstone, + type AddressQualitySummary, type Contact, type ContactChannelRule, - type ContactChannelRulePayload + type ContactChannelRulePayload, + type ContactDuplicateSuggestion, + type ContactFieldProvenance, + type ContactMergeRecord, + type ContactPointQualityDecision, + type ContactPointQualityState } from "../../api/addresses"; type Props = { @@ -210,6 +224,65 @@ type AddressTreeNode = { type ConflictMergeChoice = "local" | "remote"; +type QualityPointTarget = { + contact: Contact; + channel: ContactPointQualityDecision["channel"]; + contactPointId: string; + label: string; + currentState: ContactPointQualityState; +}; + +type QualityFormState = { + state: ContactPointQualityState; + reason_code: string; + reason: string; + evidence_ref: string; +}; + +type MergeDialogState = { + suggestion: ContactDuplicateSuggestion; + winnerId: string; + contactPointStrategy: "union" | "winner_only"; + fieldSources: Record; + reason: string; +}; + +type MergeRecoveryDialogState = { + merge: ContactMergeRecord; + action: "undo" | "split"; + reason: string; +}; + +const MERGE_SCALAR_FIELDS = [ + { id: "display_name", label: "Display name" }, + { id: "given_name", label: "Given name" }, + { id: "family_name", label: "Family name" }, + { id: "organization", label: "Organization" }, + { id: "role_title", label: "Role title" }, + { id: "note", label: "Note" } +] as const; + +type MergeScalarField = typeof MERGE_SCALAR_FIELDS[number]["id"]; + +function contactScalarValue(contact: Contact, field: MergeScalarField): string { + const value = contact[field]; + return typeof value === "string" && value.trim() ? value : "Not set"; +} + +function defaultMergeFieldSources( + suggestion: ContactDuplicateSuggestion, + winnerId: string +): Record { + const winner = suggestion.left.id === winnerId ? suggestion.left : suggestion.right; + const loser = suggestion.left.id === winnerId ? suggestion.right : suggestion.left; + return Object.fromEntries( + MERGE_SCALAR_FIELDS.map(({ id }) => [ + id, + contactScalarValue(winner, id) !== "Not set" ? winner.id : loser.id + ]) + ); +} + const EMPTY_BOOK_FORM: BookFormState = { scope_type: "user", group_id: "", @@ -247,6 +320,13 @@ const EMPTY_CHANNEL_RULE_FORM: ChannelRuleFormState = { effective_until: "" }; +const EMPTY_QUALITY_FORM: QualityFormState = { + state: "valid", + reason_code: "", + reason: "", + evidence_ref: "" +}; + const ADDRESS_CONTACT_DRAG_TYPE = "application/x-govoplan-address-contact-id"; const CONFLICT_PAYLOAD_FIELDS = ["display_name", "given_name", "family_name", "organization", "role_title", "emails", "phones", "postal_addresses", "tags", "note"] as const; @@ -322,6 +402,34 @@ function primaryPhone(contact: Contact): string { return contact.phones.find((phone) => phone.is_primary)?.phone ?? contact.phones[0]?.phone ?? ""; } +function contactPointSource(provenance?: Record): string { + if (!provenance) return ""; + const sourceKind = typeof provenance.source_kind === "string" ? provenance.source_kind : ""; + const sourceRef = typeof provenance.source_ref === "string" ? provenance.source_ref : ""; + return [sourceKind, sourceRef].filter(Boolean).join(" · "); +} + +function originalPostalSummary(address: Contact["postal_addresses"][number]): string { + const value = address.original_value; + if (!value) return ""; + return [ + value.street, + [value.postal_code, value.locality].filter(Boolean).join(" "), + value.region, + value.country + ].filter((item): item is string => typeof item === "string" && Boolean(item.trim())).join(", "); +} + +function provenanceValue(value: unknown): string { + if (value === null || value === undefined || value === "") return "Not set"; + if (typeof value === "string") return value; + try { + return JSON.stringify(value); + } catch { + return String(value); + } +} + function sourceGroupForBook(book: AddressBook): "local" | "linked" { return book.source_kind === "local" ? "local" : "linked"; } @@ -687,6 +795,19 @@ export default function AddressBookPage({ settings, auth, onAuthChange }: Props) const [governanceContact, setGovernanceContact] = useState(null); const [channelRules, setChannelRules] = useState([]); const [channelRuleForm, setChannelRuleForm] = useState(EMPTY_CHANNEL_RULE_FORM); + const [qualityOpen, setQualityOpen] = useState(false); + const [qualityLoading, setQualityLoading] = useState(false); + const [qualitySummary, setQualitySummary] = useState(null); + const [duplicateSuggestions, setDuplicateSuggestions] = useState([]); + const [contactMerges, setContactMerges] = useState([]); + const [provenanceContact, setProvenanceContact] = useState(null); + const [contactProvenance, setContactProvenance] = useState([]); + const [provenanceLoading, setProvenanceLoading] = useState(false); + const [showProvenanceHistory, setShowProvenanceHistory] = useState(false); + const [qualityPointTarget, setQualityPointTarget] = useState(null); + const [qualityForm, setQualityForm] = useState(EMPTY_QUALITY_FORM); + const [mergeDialog, setMergeDialog] = useState(null); + const [mergeRecoveryDialog, setMergeRecoveryDialog] = useState(null); const [memberDialogOpen, setMemberDialogOpen] = useState(false); const [memberCandidates, setMemberCandidates] = useState([]); const [memberQuery, setMemberQuery] = useState(""); @@ -853,6 +974,11 @@ export default function AddressBookPage({ settings, auth, onAuthChange }: Props) [!canWriteSync, "You need permission to run address sync."], [saving, savingReason] ); + const qualityDashboardReason = disabledReason( + [!selectedBook, "Select an address book before reviewing address quality."], + [!canReadGovernance, "You need permission to view address quality."], + [saving, savingReason] + ); const createContactReason = disabledReason( [!selectedBook, "Select an address book before adding a contact."], [!canWriteContacts, "You need permission to manage contacts."], @@ -1260,6 +1386,166 @@ export default function AddressBookPage({ settings, auth, onAuthChange }: Props) } } + async function refreshQualityReview(bookId: string) { + setQualityLoading(true); + try { + const [summary, duplicates, merges] = await Promise.all([ + getAddressQualitySummary(settings, bookId), + listContactDuplicateSuggestions(settings, bookId), + listContactMerges(settings, { addressBookId: bookId, limit: 100 }) + ]); + setQualitySummary(summary); + setDuplicateSuggestions(duplicates.suggestions); + setContactMerges(merges); + } finally { + setQualityLoading(false); + } + } + + async function openQualityReview() { + if (!selectedBook) return; + setQualityOpen(true); + setError(""); + try { + await refreshQualityReview(selectedBook.id); + } catch (err) { + setError(errorMessage(err)); + setQualityOpen(false); + } + } + + async function openProvenanceDialog(contact: Contact) { + setProvenanceContact(contact); + setContactProvenance([]); + setShowProvenanceHistory(false); + setProvenanceLoading(true); + setError(""); + try { + setContactProvenance(await listContactProvenance(settings, contact.id, { limit: 2000 })); + } catch (err) { + setError(errorMessage(err)); + setProvenanceContact(null); + } finally { + setProvenanceLoading(false); + } + } + + function openQualityPointEditor(target: QualityPointTarget) { + setQualityPointTarget(target); + setQualityForm({ + ...EMPTY_QUALITY_FORM, + state: target.currentState + }); + } + + async function submitQualityDecision(event: FormEvent) { + event.preventDefault(); + if (!qualityPointTarget || !canWriteGovernance) return; + setSaving(true); + setError(""); + setNotice(""); + try { + await createContactQualityDecision(settings, qualityPointTarget.contact.id, { + channel: qualityPointTarget.channel, + contact_point_id: qualityPointTarget.contactPointId, + state: qualityForm.state, + reason_code: qualityForm.reason_code.trim() || null, + reason: qualityForm.reason.trim() || null, + evidence_ref: qualityForm.evidence_ref.trim() || null + }); + setQualityPointTarget(null); + setNotice(`Quality state recorded for ${qualityPointTarget.label}.`); + await refreshContacts(selectedBookId, query); + if (qualityOpen && selectedBookId) await refreshQualityReview(selectedBookId); + } catch (err) { + setError(errorMessage(err)); + } finally { + setSaving(false); + } + } + + function openMergeDialog(suggestion: ContactDuplicateSuggestion, winnerId: string) { + setMergeDialog({ + suggestion, + winnerId, + contactPointStrategy: "union", + fieldSources: defaultMergeFieldSources(suggestion, winnerId), + reason: "Confirmed duplicate during address quality review." + }); + } + + function changeMergeWinner(winnerId: string) { + setMergeDialog((current) => current ? { + ...current, + winnerId, + fieldSources: defaultMergeFieldSources(current.suggestion, winnerId) + } : null); + } + + async function submitContactMerge(event: FormEvent) { + event.preventDefault(); + if (!mergeDialog || mergeDialog.reason.trim().length < 3) return; + const loser = mergeDialog.suggestion.left.id === mergeDialog.winnerId + ? mergeDialog.suggestion.right + : mergeDialog.suggestion.left; + setSaving(true); + setError(""); + setNotice(""); + try { + await mergeContacts(settings, { + winner_contact_id: mergeDialog.winnerId, + duplicate_contact_ids: [loser.id], + reason: mergeDialog.reason.trim(), + field_sources: mergeDialog.fieldSources, + contact_point_strategy: mergeDialog.contactPointStrategy + }); + const winner = mergeDialog.suggestion.left.id === mergeDialog.winnerId + ? mergeDialog.suggestion.left + : mergeDialog.suggestion.right; + setMergeDialog(null); + setSelectedContactId(winner.id); + setNotice(`Merged duplicate contact into "${winner.display_name}".`); + await refreshBooks(); + await refreshContacts(selectedBookId, query); + if (selectedListId) await refreshListEntries(selectedListId); + if (selectedBookId) await refreshQualityReview(selectedBookId); + } catch (err) { + setError(errorMessage(err)); + } finally { + setSaving(false); + } + } + + async function submitMergeRecovery(event: FormEvent) { + event.preventDefault(); + if (!mergeRecoveryDialog || mergeRecoveryDialog.reason.trim().length < 3) return; + setSaving(true); + setError(""); + setNotice(""); + try { + await recoverContactMerge( + settings, + mergeRecoveryDialog.merge, + mergeRecoveryDialog.action, + mergeRecoveryDialog.reason.trim() + ); + setNotice( + mergeRecoveryDialog.action === "undo" + ? "Contact merge undone." + : "Merged contacts split back into their recorded pre-merge state." + ); + setMergeRecoveryDialog(null); + await refreshBooks(); + await refreshContacts(selectedBookId, query); + if (selectedListId) await refreshListEntries(selectedListId); + if (selectedBookId) await refreshQualityReview(selectedBookId); + } catch (err) { + setError(errorMessage(err)); + } finally { + setSaving(false); + } + } + function updateEmailRow(rowId: string, patch: Partial>) { setContactForm((current) => ({ ...current, @@ -1439,8 +1725,7 @@ export default function AddressBookPage({ settings, auth, onAuthChange }: Props) tags: tagsFromForm(contactForm.tags), emails, phones, - postal_addresses, - provenance: {} + postal_addresses }; try { let savedContact: Contact; @@ -1973,6 +2258,7 @@ export default function AddressBookPage({ settings, auth, onAuthChange }: Props) return ( <> + @@ -2051,6 +2337,14 @@ export default function AddressBookPage({ settings, auth, onAuthChange }: Props) onClick={() => void openGovernanceDialog(selectedContact)}> Governance + @@ -2089,7 +2383,29 @@ export default function AddressBookPage({ settings, auth, onAuthChange }: Props) {selectedContact.emails.map((email, index) => (
{email.label || "email"}{email.is_primary ? " · primary" : ""}
-
{email.email}
+
+ {email.email} + + {email.id && } + {email.original_email && email.original_email !== email.email && Original: {email.original_email}} + {contactPointSource(email.provenance) && Source: {contactPointSource(email.provenance)}} +
))} @@ -2102,7 +2418,29 @@ export default function AddressBookPage({ settings, auth, onAuthChange }: Props) {selectedContact.phones.map((phone, index) => (
{phone.label || "phone"}{phone.is_primary ? " · primary" : ""}
-
{phone.phone}
+
+ {phone.phone} + + {phone.id && } + {phone.original_phone && phone.original_phone !== phone.phone && Original: {phone.original_phone}} + {contactPointSource(phone.provenance) && Source: {contactPointSource(phone.provenance)}} +
))} @@ -2115,7 +2453,29 @@ export default function AddressBookPage({ settings, auth, onAuthChange }: Props) {selectedContact.postal_addresses.map((address, index) => (
{address.label || "address"}{address.is_primary ? " · primary" : ""}
-
{formatPostalAddress(address) || "No formatted address."}
+
+ {formatPostalAddress(address) || "No formatted address."} + + {address.id && } + {originalPostalSummary(address) && originalPostalSummary(address) !== formatPostalAddress(address) && Original: {originalPostalSummary(address)}} + {contactPointSource(address.provenance) && Source: {contactPointSource(address.provenance)}} +
))} @@ -2507,6 +2867,258 @@ export default function AddressBookPage({ settings, auth, onAuthChange }: Props) + setProvenanceContact(null)} + closeDisabled={provenanceLoading} + className="address-quality-dialog" + footerClassName="button-row compact-actions" + footer={}> + +
+ setShowProvenanceHistory((current) => !current)} + help="Include superseded source decisions as well as the currently retained values." + /> +
+ {contactProvenance.filter((item) => showProvenanceHistory || item.selected).length === 0 ? +

No field provenance was recorded.

: + contactProvenance + .filter((item) => showProvenanceHistory || item.selected) + .map((item) => ( +
+
+ + {item.field_path} + + {item.visibility !== "inherit" && } + + {provenanceValue(item.value)} + + Source: {item.source_kind}{item.source_ref ? ` · ${item.source_ref}` : ""} + {` · ${item.reason_code}`} + {` · ${formatDateTime(item.created_at, ADDRESS_DATE_TIME_OPTIONS)}`} + + {item.explanation && {item.explanation}} +
+
+ )) + } +
+
+ + + + setQualityOpen(false)} + closeDisabled={saving} + className="address-quality-dialog" + footerClassName="button-row compact-actions" + footer={ + <> + + + + }> + +
+ {qualitySummary &&
+ + 0 ? "warning" : "good"} detail="Current non-valid states" /> + 0 ? "warning" : "good"} detail="Explainable suggestions" /> + 0 ? "danger" : "good"} detail="Returned or undeliverable" /> +
} + +
+
+
+ Duplicate suggestions +

Scores are bounded and show the exact matching features. Choose which contact survives.

+
+
+ {duplicateSuggestions.length === 0 ?

No duplicate suggestions above the current threshold.

: +
+ {duplicateSuggestions.map((suggestion) => ( +
+
+ + {suggestion.left.display_name} + and + {suggestion.right.display_name} + + + {suggestion.features.map((feature) => `${feature.label}: ${feature.value}`).join(" · ")} +
+
+ + +
+
+ ))} +
+ } +
+ +
+
+
+ Correction queue +

Current quality states are also applied to campaign and distribution-list recipient resolution.

+
+
+ {!qualitySummary || qualitySummary.corrections.length === 0 ?

No contact points need correction.

: +
+ {qualitySummary.corrections.map((correction) => ( + + ))} +
+ } +
+ +
+
+
+ Merge history +

Recovery is available only while the recorded post-merge state still matches.

+
+
+ {contactMerges.length === 0 ?

No contact merges recorded.

: +
+ {contactMerges.map((merge) => ( +
+
+ {merge.reason} + {merge.loser_contact_ids.length} merged contact{merge.loser_contact_ids.length === 1 ? "" : "s"} · {formatDateTime(merge.created_at, ADDRESS_DATE_TIME_OPTIONS)} +
+ {merge.status === "active" &&
+ + +
} +
+ ))} +
+ } +
+ {qualitySummary?.truncated && This bounded review is truncated. Narrow the address book or use the API for a complete staged review.} +
+ + + + setQualityPointTarget(null)} + closeDisabled={saving} + footerClassName="button-row compact-actions" + footer={ + <> + + + + }> +
void submitQualityDecision(event)}> + +
+ + + + setQualityForm((current) => ({ ...current, reason_code: event.target.value }))} /> +
+