Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
889cafaf20 | ||
|
|
be16f8218e | ||
|
|
33de5cac55 | ||
|
|
a54dae9536 | ||
|
|
65609665a2 | ||
|
|
a51d3781a9 | ||
|
|
d86894b321 | ||
|
|
b277e8d7ac | ||
|
|
79e2315e89 |
@@ -59,10 +59,30 @@ The source revision/cursor, redacted dry-run decision, canonical request digest,
|
|||||||
and distributed lease are durable before network I/O. Immutable snapshot rows
|
and distributed lease are durable before network I/O. Immutable snapshot rows
|
||||||
and the terminal recovery checkpoint commit atomically, and an
|
and the terminal recovery checkpoint commit atomically, and an
|
||||||
`Idempotency-Key` replays the committed result without contacting the provider.
|
`Idempotency-Key` replays the committed result without contacting the provider.
|
||||||
Current acquisition transports are read-only. The exported external-mutation
|
Most acquisition transports are read-only. The MediaWiki/BlueSpice knowledge
|
||||||
recovery contract requires stable idempotency, provider verification, and
|
adapter is the first governed publication path: it requires an expected remote
|
||||||
operator reconciliation, but no connector currently claims a production write
|
revision, a stable idempotency key, a scoped credential envelope, and durable
|
||||||
or delete path.
|
forward-recovery evidence. A timeout after dispatch becomes outcome-unknown and
|
||||||
|
blocks replay until the provider revision has been reconciled.
|
||||||
|
|
||||||
|
The knowledge adapter discovers MediaWiki or BlueSpice product/version and
|
||||||
|
capabilities, maps namespaces, pages, revisions, users, categories, links,
|
||||||
|
files, discussions, redirects, and permissions into identity-stable connector
|
||||||
|
snapshots, and consumes bounded full or recent-change deltas. Optional Search
|
||||||
|
integration indexes active pages and rechecks current profile state and ACLs on
|
||||||
|
every result. Migration into native Wiki is preview-only: conflicts, attachment
|
||||||
|
collisions, unsupported macros, truncation, and source fingerprints are
|
||||||
|
reported before a target-side write is considered.
|
||||||
|
|
||||||
|
The Znuny/OTRS-compatible service-desk adapter uses deployment-defined
|
||||||
|
GenericInterface REST routes. It supports identity-only links, bounded snapshot
|
||||||
|
imports, and ongoing synchronization with explicit authority, queue, ACL, and
|
||||||
|
dynamic-field mappings. Stable tickets, articles, and attachment references are
|
||||||
|
preserved with mapping-loss diagnostics; attachment bytes remain provider-side.
|
||||||
|
Optional Search integration rechecks the current profile, tenant, scope, and ACL
|
||||||
|
for every result. Revision-checked external updates are limited to governed-sync
|
||||||
|
profiles and use durable recovery evidence. Tickets, Helpdesk, and Cases remain
|
||||||
|
authoritative for their own business records and conversion workflows.
|
||||||
|
|
||||||
RSS and Atom emission is a bounded renderer, not an authority shortcut. Every
|
RSS and Atom emission is a bounded renderer, not an authority shortcut. Every
|
||||||
selected entry declares whether it came from a GovOPlaN event, publication,
|
selected entry declares whether it came from a GovOPlaN event, publication,
|
||||||
@@ -95,3 +115,23 @@ See:
|
|||||||
- [OpenProject connector concept](docs/OPENPROJECT_CONNECTOR.md)
|
- [OpenProject connector concept](docs/OPENPROJECT_CONNECTOR.md)
|
||||||
- [OpenDesk integration map](docs/OPENDESK_INTEGRATION_MAP.md)
|
- [OpenDesk integration map](docs/OPENDESK_INTEGRATION_MAP.md)
|
||||||
- [Governed connector configuration](docs/GOVERNED_CONNECTOR_CONFIGURATION.md)
|
- [Governed connector configuration](docs/GOVERNED_CONNECTOR_CONFIGURATION.md)
|
||||||
|
- [MediaWiki and BlueSpice connector](docs/MEDIAWIKI_BLUESPICE_CONNECTOR.md)
|
||||||
|
- [Znuny and OTRS-compatible service-desk connector](docs/ZNUNY_OTRS_CONNECTOR.md)
|
||||||
|
|
||||||
|
## Git-source WebUI package
|
||||||
|
|
||||||
|
The repository root exposes `@govoplan/connectors-webui` for Git-tagged release
|
||||||
|
dependencies. It mirrors the owning `webui/package.json` version, public
|
||||||
|
TypeScript/CSS exports and peer requirements, with entry paths under
|
||||||
|
`webui/src`. Consumers provide the shared Core/React peers; the facade runs no
|
||||||
|
development or install scripts. The source archive contains `webui/src`, this
|
||||||
|
README and any repository license file. Run module development checks from `webui/`; Python
|
||||||
|
installation remains governed by `pyproject.toml`.
|
||||||
|
|
||||||
|
Das Repository stellt `@govoplan/connectors-webui` am Wurzelpfad für versionierte
|
||||||
|
Git-Abhängigkeiten bereit. Version, öffentliche TypeScript-/CSS-Exporte und
|
||||||
|
Peer-Anforderungen entsprechen `webui/package.json`; die Einstiegspfade liegen
|
||||||
|
unter `webui/src`. Gemeinsame Core-/React-Peers stellt die einbindende Anwendung
|
||||||
|
bereit. Die Fassade führt keine Entwicklungs- oder Installationsskripte aus.
|
||||||
|
Entwicklungsprüfungen bleiben in `webui/`, die Python-Installation weiterhin in
|
||||||
|
`pyproject.toml` definiert.
|
||||||
|
|||||||
@@ -132,6 +132,25 @@ explicit refresh. Missing Files capability, revoked file access, quarantine,
|
|||||||
oversized or malformed content, inactive/stale credentials, unreachable SQL,
|
oversized or malformed content, inactive/stale credentials, unreachable SQL,
|
||||||
and timeout failures produce sanitized unavailable/validation diagnostics.
|
and timeout failures produce sanitized unavailable/validation diagnostics.
|
||||||
|
|
||||||
|
XLSX archive checks and workbook parsing run in a fresh disposable Core worker.
|
||||||
|
Each invocation allows 15 seconds wall time, 10 CPU seconds and 512 MiB virtual
|
||||||
|
address space, with no file output. The typed transport allows 8 MiB input and
|
||||||
|
64 MiB result bytes, at most 64 nesting levels and 1,000,000 value nodes. These
|
||||||
|
transport bounds include serialization overhead. The existing workbook bounds
|
||||||
|
remain 5,000,000 raw bytes, 50,000,000 expanded bytes, 5,000 archive entries,
|
||||||
|
100:1 compression ratio, 500 columns and 10,000 row positions after the header.
|
||||||
|
Authorization and exact-version file reads happen in the parent; credentials,
|
||||||
|
SQL sessions and durable changes are never passed to the parser.
|
||||||
|
|
||||||
|
Users receive an explicit failure rather than a partial source when resource
|
||||||
|
or transport limits are exceeded; reduce workbook size or complexity before
|
||||||
|
retrying. The Core `GOVOPLAN_ISOLATED_PROCESS_CONCURRENCY` setting limits active work per
|
||||||
|
API/worker process without queuing. Busy capacity may be retried later. Missing
|
||||||
|
POSIX resource controls, cancellation and worker failure produce sanitized
|
||||||
|
unavailable diagnostics; there is no in-process fallback. Operators must keep
|
||||||
|
the Core worker API available and account for the aggregate memory of all
|
||||||
|
active worker slots across API/worker replicas.
|
||||||
|
|
||||||
All three current providers declare projection and pagination pushdown only.
|
All three current providers declare projection and pagination pushdown only.
|
||||||
Filters, aggregations, and sorting remain in Dataflow until an adapter explicitly
|
Filters, aggregations, and sorting remain in Dataflow until an adapter explicitly
|
||||||
declares and tests those operations.
|
declares and tests those operations.
|
||||||
@@ -178,9 +197,10 @@ start through the connector mutation recovery contract with a stable
|
|||||||
idempotency key, SHA-256 request digest, source revision/cursor, and dry-run
|
idempotency key, SHA-256 request digest, source revision/cursor, and dry-run
|
||||||
evidence. Definitive rejection is terminal. A timeout or lost acknowledgement
|
evidence. Definitive rejection is terminal. A timeout or lost acknowledgement
|
||||||
after dispatch is `outcome_unknown` and blocks replay until the owning connector
|
after dispatch is `outcome_unknown` and blocks replay until the owning connector
|
||||||
verifies provider state. The contract and conformance tests exist; no current
|
verifies provider state. The MediaWiki/BlueSpice page publisher implements this
|
||||||
connector advertises a production external mutation, so write/delete adoption
|
path for revision-checked edits. Other connector types do not thereby acquire a
|
||||||
remains explicitly planned rather than implied.
|
write capability; each adapter must declare and prove its own recovery and
|
||||||
|
reconciliation behavior.
|
||||||
|
|
||||||
## Provider Declaration
|
## Provider Declaration
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,134 @@
|
|||||||
|
# MediaWiki and BlueSpice Knowledge Connector
|
||||||
|
|
||||||
|
The knowledge connector integrates an external MediaWiki or BlueSpice instance
|
||||||
|
without making Connectors the owner of native knowledge semantics. Connectors
|
||||||
|
owns endpoint and credential governance, Action API transport, discovery,
|
||||||
|
synchronization, identity-stable snapshots, diagnostics, and publication
|
||||||
|
recovery. Wiki continues to own GovOPlaN-native spaces, pages, drafts,
|
||||||
|
revisions, comments, and publishing policy.
|
||||||
|
|
||||||
|
## Configure and discover
|
||||||
|
|
||||||
|
Create an active governed connector definition and configuration with:
|
||||||
|
|
||||||
|
- provider `mediawiki` or `bluespice`;
|
||||||
|
- protocol `mediawiki`, `action_api`, or `mediawiki_action_api`;
|
||||||
|
- an absolute HTTP(S) endpoint without embedded credentials; and
|
||||||
|
- an optional Core credential-envelope reference. Publication requires the
|
||||||
|
credential; public read-only sources may omit it.
|
||||||
|
|
||||||
|
Core validates the endpoint and every redirect against the deployment egress,
|
||||||
|
DNS/IP-pinning, peer, downgrade, and response-size policies. Credentials can be
|
||||||
|
bearer/access tokens or username/password values resolved only for the current
|
||||||
|
tenant, actor, module, target scope, and server. They never enter connector
|
||||||
|
profiles, snapshots, diagnostics, URLs, or API responses.
|
||||||
|
|
||||||
|
A knowledge profile selects desired maturity and authority and maps each
|
||||||
|
included source namespace to a target Wiki space reference and path prefix.
|
||||||
|
Every profile also supplies a tenant or restricted fallback visibility. A
|
||||||
|
restricted fallback needs at least one normalized Search ACL token such as
|
||||||
|
`group:<id>`, `role:<id>`, `membership:<id>`, `account:<id>`,
|
||||||
|
`identity:<id>`, `function:<id>`, or `scope:<permission>`.
|
||||||
|
|
||||||
|
Discovery queries site and current-user metadata. It records the product,
|
||||||
|
version, API/PHP version, extensions, namespaces, advertised edit right,
|
||||||
|
capabilities, a stable discovery digest, and sanitized diagnostics. BlueSpice
|
||||||
|
permission and discussion extensions are detected where advertised. Standard
|
||||||
|
MediaWiki does not expose complete page ACL semantics through the base Action
|
||||||
|
API, so discovery reports that the configured namespace fallback will be used.
|
||||||
|
|
||||||
|
## Synchronize and preserve identity
|
||||||
|
|
||||||
|
Run one bounded full backfill and then bounded recent-change deltas. Every run
|
||||||
|
requires an idempotency key; exact replay returns the original evidence without
|
||||||
|
contacting the source, while reusing the key with different inputs is rejected.
|
||||||
|
The profile stores only its latest cursor and high-watermark. Runs retain
|
||||||
|
counts, redacted effects, diagnostics, configuration/discovery provenance, and
|
||||||
|
transport evidence.
|
||||||
|
|
||||||
|
Snapshots use the MediaWiki page id as stable identity and preserve revision
|
||||||
|
id, canonical URL, content digest, namespace, title, source timestamp, and
|
||||||
|
observed timestamp. Page content mapping covers categories, links, file
|
||||||
|
references, discussion references, revision author references, redirect
|
||||||
|
targets, permission metadata, target space/path, and detected wikitext macros.
|
||||||
|
Referenced files remain external references; binary transfer and attachment
|
||||||
|
ownership remain with Files or DMS.
|
||||||
|
|
||||||
|
Moves update the existing stable page. Deletions use the stable page id when the
|
||||||
|
provider supplies it and otherwise match a previously synchronized namespace
|
||||||
|
and title. An unmatched deletion becomes a separate tombstone with an explicit
|
||||||
|
diagnostic instead of silently deleting an unrelated page. Permission changes
|
||||||
|
are part of the content digest and therefore update the current snapshot even
|
||||||
|
when page text is unchanged.
|
||||||
|
|
||||||
|
Desired maturity is an operator ceiling. A read-only profile cannot be used for
|
||||||
|
synchronization, migration preview, or publication merely because the provider
|
||||||
|
advertises those capabilities.
|
||||||
|
|
||||||
|
## Search and access safety
|
||||||
|
|
||||||
|
When Search is installed, Connectors registers `connectors.mediawiki.pages` for
|
||||||
|
the `external_knowledge_page` resource type. Backfill includes only active,
|
||||||
|
non-deleted pages from active profiles. Restricted documents carry the current
|
||||||
|
snapshot ACL; tenant documents carry no narrower ACL.
|
||||||
|
|
||||||
|
Search performs a fail-closed authorization recheck for every result. It
|
||||||
|
requires the external-knowledge read permission, an exact tenant match, an
|
||||||
|
active profile, a current non-deleted page, and—when restricted—intersection
|
||||||
|
with the current account, membership, identity, group, role, function, or scope
|
||||||
|
tokens. Consequently, an ACL change takes effect even before a deferred index
|
||||||
|
update completes. Immediate index updates remove deleted pages and refresh ACLs;
|
||||||
|
the next Search rebuild reconciles any transient writer failure. Search also
|
||||||
|
removes source projections when Connectors is disabled.
|
||||||
|
|
||||||
|
## Publish and reconcile
|
||||||
|
|
||||||
|
Publication requires the publish permission, an active profile and
|
||||||
|
configuration, discovered provider publication capability, desired maturity at
|
||||||
|
least `publish`, an idempotency key, page title/body, and an optional expected
|
||||||
|
external revision. The Action API edit carries `baserevid` when supplied and
|
||||||
|
uses a credential-provided or freshly acquired CSRF token.
|
||||||
|
|
||||||
|
Before remote I/O, Connectors prepares a durable forward-recovery operation
|
||||||
|
with request digest, expected revision, cursor, profile/configuration revision,
|
||||||
|
and resource identity. A confirmed provider response stores the stable page and
|
||||||
|
new revision and completes the recovery evidence. A definitive provider
|
||||||
|
rejection is terminal. A timeout, invalid response, or server failure after
|
||||||
|
dispatch becomes outcome-unknown: do not retry with another key until an
|
||||||
|
operator compares the provider page/revision and reconciles the recovery
|
||||||
|
operation. Local rollback cannot undo a confirmed external edit.
|
||||||
|
|
||||||
|
## Migration preview into Wiki
|
||||||
|
|
||||||
|
Migration is intentionally a dry-run in this slice. The request names a target
|
||||||
|
space, supported macro set, existing target paths/source identities, and
|
||||||
|
existing attachment names. The preview is bounded to 500 source pages and
|
||||||
|
records a source fingerprint and high-watermark. It reports create, update, or
|
||||||
|
conflict effects plus:
|
||||||
|
|
||||||
|
- target path already owned by a different external page;
|
||||||
|
- attachment filename collisions;
|
||||||
|
- unsupported macros;
|
||||||
|
- pages outside the requested target space; and
|
||||||
|
- truncation at the configured limit.
|
||||||
|
|
||||||
|
`can_apply=true` means the bounded preview contains no error or conflict and is
|
||||||
|
not truncated. It does not write Wiki pages. A future target-side migration
|
||||||
|
worker must use the optional connector capability and Wiki-owned mutation
|
||||||
|
contract, revalidate the preview fingerprint, and keep native Wiki permissions
|
||||||
|
and revision history authoritative.
|
||||||
|
|
||||||
|
## Operations and privacy
|
||||||
|
|
||||||
|
Provider state exposes profile product/maturity, active-object count, latest run
|
||||||
|
status, health, recovery attention, and last success time without endpoint,
|
||||||
|
credential, title, ACL, or content data. Existing snapshots may remain
|
||||||
|
available during an outage only to principals still authorized by current local
|
||||||
|
profile and ACL state; their health and freshness remain explicit.
|
||||||
|
|
||||||
|
Data-subject exports include only minimized operator attribution for profile
|
||||||
|
updates and synchronization/migration/publication runs. They exclude endpoints,
|
||||||
|
credentials, page content, titles, ACLs, namespace mappings, idempotency keys,
|
||||||
|
request hashes, effects, diagnostics, provenance, and transport evidence.
|
||||||
|
Attribution and external-operation evidence is retained for governance rather
|
||||||
|
than automatically erased.
|
||||||
@@ -0,0 +1,188 @@
|
|||||||
|
# Znuny and OTRS-compatible service-desk connector
|
||||||
|
|
||||||
|
## Boundary
|
||||||
|
|
||||||
|
Connectors owns the governed endpoint profile, GenericInterface REST transport,
|
||||||
|
credential hand-off, provider discovery, bounded synchronization, external
|
||||||
|
references, mapping diagnostics, health, and recovery evidence. It does not own
|
||||||
|
ticket, helpdesk, or case semantics. A synchronized provider ticket remains an
|
||||||
|
external service-desk ticket; creating or relating a GovOPlaN Ticket, Helpdesk
|
||||||
|
item, or Case is the responsibility of the corresponding optional module.
|
||||||
|
|
||||||
|
The provider interface is `connectors.external_service_desk@1.0.0`, and the
|
||||||
|
target-tested provider declaration is `connectors.znuny.tickets`.
|
||||||
|
|
||||||
|
## Governed configuration and routes
|
||||||
|
|
||||||
|
Create an active connector definition/configuration with:
|
||||||
|
|
||||||
|
- `provider`: `znuny`, `otrs`, or `znuny_otrs`;
|
||||||
|
- `protocol`: `generic_interface_rest` (the aliases `rest` and
|
||||||
|
`otrs_generic_interface_rest` are accepted);
|
||||||
|
- an HTTP(S) endpoint that passes the central outbound-request policy; and
|
||||||
|
- an optional scoped Core credential-envelope reference.
|
||||||
|
|
||||||
|
Znuny GenericInterface routes are configured by each deployment rather than
|
||||||
|
being one universal product API. The profile therefore governs relative search,
|
||||||
|
ticket-read, and optional update paths and their supported
|
||||||
|
methods. Ticket and update paths must contain `{ticket_id}`. An optional
|
||||||
|
absolute HTTP(S) browser URL template may contain `{ticket_id}` or
|
||||||
|
`{ticket_number}`. `search_filters` carries up to 100 deployment-supported,
|
||||||
|
secret-free GenericTicket search criteria such as queue identifiers. It cannot
|
||||||
|
override synchronization bounds, ordering, change cursors, or authentication.
|
||||||
|
Routes cannot change authority or contain credentials.
|
||||||
|
|
||||||
|
Header authentication is the recommended default. The adapter supports the
|
||||||
|
documented `X-OTRS-Header-UserLogin`, `X-OTRS-Header-Password`,
|
||||||
|
`X-OTRS-Header-SessionID`, and customer-login headers. A legacy credential may
|
||||||
|
declare `auth_mode: body`, but then every affected route must use POST. Secret
|
||||||
|
fields are never placed in a GET URL, persisted projection, diagnostic, or API
|
||||||
|
response. Core also treats these provider headers as redirect-sensitive and
|
||||||
|
removes them before following any cross-origin redirect.
|
||||||
|
|
||||||
|
Reference configuration examples:
|
||||||
|
|
||||||
|
- [Znuny GenericTicketConnectorREST example](https://doc.znuny.org/znuny/admin/webservices/examples/GenericTicketConnectorREST/index.html)
|
||||||
|
- [Znuny provider and header authentication](https://doc.znuny.org/znuny-7_1/admin/webservices/provider/)
|
||||||
|
- [Znuny web-service configuration](https://doc.znuny.org/znuny-7_3/admin/webservices/config/index.html)
|
||||||
|
|
||||||
|
## Profile policy
|
||||||
|
|
||||||
|
The integration and authority choices are intentionally separate but bounded:
|
||||||
|
|
||||||
|
| Integration mode | Allowed authority | Maturity |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| `link` | `linked_reference` | `link` through `search` |
|
||||||
|
| `import` | `external_authoritative`, `external_mirror` | exactly `read` |
|
||||||
|
| `synchronize` | `external_authoritative`, `governed_sync` | exactly `synchronize` |
|
||||||
|
|
||||||
|
`link` retains stable identity and routing facts and asks the provider not to
|
||||||
|
return articles, attachments, or dynamic fields; any unexpectedly returned
|
||||||
|
content is still discarded. `import` is a deliberate bounded full snapshot and
|
||||||
|
never silently changes to delta synchronization.
|
||||||
|
`synchronize` completes a full reconciliation and then advances to overlap-safe
|
||||||
|
change-time deltas.
|
||||||
|
|
||||||
|
Queue mappings govern inclusion, an optional opaque target queue reference, and
|
||||||
|
tenant or restricted visibility with ACL tokens. A restricted default requires
|
||||||
|
at least one ACL token. Provider-supplied `GovOPlaNVisibility` and
|
||||||
|
`GovOPlaNACL` fields win when valid. Otherwise a reviewed queue mapping wins,
|
||||||
|
then the profile default. Standard GenericInterface installations do not expose
|
||||||
|
a portable ticket-ACL contract, so every fallback is visible as a diagnostic.
|
||||||
|
|
||||||
|
Dynamic-field mappings govern source name, optional target name, inclusion, and
|
||||||
|
`string`, `number`, `boolean`, `date`, or `json` conversion. Conversion loss,
|
||||||
|
unreturned configured fields, provider-specific ticket fields, synthesized
|
||||||
|
identities, and truncation are structured diagnostics rather than silent loss.
|
||||||
|
|
||||||
|
## Discovery and synchronization
|
||||||
|
|
||||||
|
Discovery performs a bounded ticket search and records health, API family,
|
||||||
|
route hash, the exact governed configuration revision/hash, product/version
|
||||||
|
evidence, capabilities, maturity, and diagnostics. A changed endpoint,
|
||||||
|
configuration revision, or route map invalidates that evidence: synchronization,
|
||||||
|
and updates fail closed until discovery is repeated, while prior projections are
|
||||||
|
invalidated and Search stays closed until a new full reconciliation verifies
|
||||||
|
them. Integration, route, queue, or dynamic-field mapping changes also reset the
|
||||||
|
cursor and require a full reconciliation. Recognized Znuny or OTRS major
|
||||||
|
versions 6 and later can reach synchronization maturity. An unverified
|
||||||
|
product/version stays at read maturity. An update route adds the technical
|
||||||
|
`publish` capability, but does not override profile authority.
|
||||||
|
|
||||||
|
Full synchronization first obtains a stable ordered identity set, then reads at
|
||||||
|
most 500 tickets per call. A continued full run stores its offset, identity-set
|
||||||
|
fingerprint, and cumulative high-watermark. If the provider identity set changes
|
||||||
|
mid-run, the cursor is rejected and the operator must restart the full run. A
|
||||||
|
completed full run reconciles local removals and, for `synchronize` mode,
|
||||||
|
transitions to an overlap-safe delta cursor.
|
||||||
|
|
||||||
|
An explicit `full` request always restarts at the beginning; `auto` continues a
|
||||||
|
committed full cursor or advances a completed delta cursor. A caller-supplied
|
||||||
|
cursor must exactly match the profile's committed cursor, and delta mode cannot
|
||||||
|
bootstrap a profile that has not completed its full synchronization.
|
||||||
|
|
||||||
|
Delta synchronization reads the bounded candidate set, orders changes by
|
||||||
|
provider change time and ticket id, and suppresses only the exact ticket
|
||||||
|
revisions already observed at the current timestamp boundary. A previously seen
|
||||||
|
ticket that has changed again is therefore not lost. Every run requires a
|
||||||
|
profile-wide idempotency key. An exact replay returns the committed run without
|
||||||
|
contacting the provider; reuse for a different request or changed profile policy
|
||||||
|
is rejected.
|
||||||
|
|
||||||
|
Operational bounds:
|
||||||
|
|
||||||
|
- at most 10,000 ticket identities per profile search;
|
||||||
|
- at most 500 ticket reads per API call;
|
||||||
|
- at most 10 MB per provider response;
|
||||||
|
- a 20-second outbound timeout; and
|
||||||
|
- a 4,000-character cursor, including timestamp-boundary identities.
|
||||||
|
|
||||||
|
Partition larger or unusually bursty providers into queue-scoped profiles by
|
||||||
|
combining provider-side queue `search_filters` with matching queue mappings. A
|
||||||
|
full-run fingerprint conflict requires a restart. A timestamp-boundary overflow
|
||||||
|
requires a narrower partition. These are explicit safety stops, not partial
|
||||||
|
success claims.
|
||||||
|
|
||||||
|
## Mapping and attachment policy
|
||||||
|
|
||||||
|
Each ticket projection retains stable ticket id/number, title, type, queue,
|
||||||
|
target queue reference, state, priority, owner, responsible user, customer user,
|
||||||
|
organization, service, SLA, creation/change times, mapped dynamic fields,
|
||||||
|
articles, attachment metadata, permission source, ACLs, canonical URL, content
|
||||||
|
hash, provider revision, cursor, observation time, and source provenance.
|
||||||
|
|
||||||
|
Article and attachment references use Core `ExternalObjectReference` values.
|
||||||
|
Article bodies are capped at 200,000 characters. Attachment content is never
|
||||||
|
retained; only stable identity, filename, media type, size, disposition, content
|
||||||
|
id, article/ticket relationship, version, and provenance are mapped. Returned
|
||||||
|
bytes produce an `attachment_content_omitted` diagnostic.
|
||||||
|
|
||||||
|
## Search authorization
|
||||||
|
|
||||||
|
When Search is installed and both desired and discovered maturity permit it,
|
||||||
|
active non-deleted ticket projections are indexed. Search documents carry the
|
||||||
|
current visibility, ACL tokens, external reference, source revision, routing
|
||||||
|
metadata, article text, and bounded dynamic-field keywords.
|
||||||
|
|
||||||
|
Authorization always fails closed unless all of these remain true:
|
||||||
|
|
||||||
|
- the requesting principal belongs to the exact tenant;
|
||||||
|
- the principal has `connectors:service_desk:read`;
|
||||||
|
- the profile remains active and search-capable;
|
||||||
|
- the ticket remains active; and
|
||||||
|
- tenant visibility applies or a current account, membership, identity, group,
|
||||||
|
role, function, or scope ACL token intersects.
|
||||||
|
|
||||||
|
Pausing a profile or changing fallback/queue ACLs updates or removes Search
|
||||||
|
projections immediately. Search also rechecks the current database state for
|
||||||
|
every result, so a delayed index update does not grant access.
|
||||||
|
|
||||||
|
## Governed external updates and recovery
|
||||||
|
|
||||||
|
An external update is allowed only when the profile is active, authority is
|
||||||
|
`governed_sync`, discovery recorded the `publish` capability, and the caller has
|
||||||
|
`connectors:service_desk:update`. The request must include the synchronized
|
||||||
|
provider revision and a new idempotency key. Supported governed fields are
|
||||||
|
title, queue, state, priority, owner, responsible user, and explicitly mapped
|
||||||
|
dynamic fields.
|
||||||
|
|
||||||
|
Before dispatch, the adapter refetches the ticket and rejects a stale revision.
|
||||||
|
After dispatch, it refetches again and verifies both the changed revision and
|
||||||
|
every requested field value. A transport failure before a conclusive provider
|
||||||
|
response, an unchanged revision, or a requested value that cannot be confirmed
|
||||||
|
becomes `outcome_unknown`. Do not retry with another key. Inspect the provider
|
||||||
|
ticket and reconcile its accepted revision through the Core recovery evidence
|
||||||
|
first. Local database rollback cannot undo a remote provider mutation.
|
||||||
|
|
||||||
|
## Administrator verification
|
||||||
|
|
||||||
|
1. Create the governed definition/configuration and scoped credential envelope.
|
||||||
|
2. Create a restricted profile with reviewed routes, queue partitions, dynamic
|
||||||
|
fields, authority, and fallback ACLs.
|
||||||
|
3. Discover and confirm product/version, maturity, capabilities, and diagnostics.
|
||||||
|
4. Finish a keyed full run; continue while its cursor kind is `full`.
|
||||||
|
5. Run a new keyed automatic delta and inspect effects, losses, and health.
|
||||||
|
6. Verify one allowed and one denied Search principal against a restricted ticket.
|
||||||
|
7. If governed writes are enabled, update a non-production ticket with its
|
||||||
|
current revision, then verify provider and recovery evidence.
|
||||||
|
8. Reconcile every `outcome_unknown` run before any retry.
|
||||||
@@ -0,0 +1,31 @@
|
|||||||
|
{
|
||||||
|
"name": "@govoplan/connectors-webui",
|
||||||
|
"version": "0.1.27",
|
||||||
|
"private": true,
|
||||||
|
"type": "module",
|
||||||
|
"main": "webui/src/index.ts",
|
||||||
|
"module": "webui/src/index.ts",
|
||||||
|
"types": "webui/src/index.ts",
|
||||||
|
"exports": {
|
||||||
|
".": {
|
||||||
|
"types": "./webui/src/index.ts",
|
||||||
|
"import": "./webui/src/index.ts"
|
||||||
|
},
|
||||||
|
"./styles/connectors.css": "./webui/src/styles/connectors.css"
|
||||||
|
},
|
||||||
|
"peerDependencies": {
|
||||||
|
"@govoplan/core-webui": "^0.1.18",
|
||||||
|
"react": ">=19.2.7 <20",
|
||||||
|
"react-dom": ">=19.2.7 <20"
|
||||||
|
},
|
||||||
|
"peerDependenciesMeta": {
|
||||||
|
"@govoplan/core-webui": {
|
||||||
|
"optional": true
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"files": [
|
||||||
|
"webui/src",
|
||||||
|
"README.md",
|
||||||
|
"LICENSE"
|
||||||
|
]
|
||||||
|
}
|
||||||
+2
-2
@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
|
|||||||
|
|
||||||
[project]
|
[project]
|
||||||
name = "govoplan-connectors"
|
name = "govoplan-connectors"
|
||||||
version = "0.1.20"
|
version = "0.1.27"
|
||||||
description = "Governed connector catalogue and tabular source capabilities for GovOPlaN."
|
description = "Governed connector catalogue and tabular source capabilities for GovOPlaN."
|
||||||
readme = "README.md"
|
readme = "README.md"
|
||||||
requires-python = ">=3.12"
|
requires-python = ">=3.12"
|
||||||
@@ -12,7 +12,7 @@ license = "AGPL-3.0-or-later"
|
|||||||
authors = [{ name = "GovOPlaN" }]
|
authors = [{ name = "GovOPlaN" }]
|
||||||
dependencies = [
|
dependencies = [
|
||||||
"defusedxml>=0.7,<1",
|
"defusedxml>=0.7,<1",
|
||||||
"govoplan-core>=0.1.19",
|
"govoplan-core>=0.1.46",
|
||||||
"openpyxl>=3.1.5,<4",
|
"openpyxl>=3.1.5,<4",
|
||||||
]
|
]
|
||||||
|
|
||||||
|
|||||||
@@ -47,6 +47,8 @@ class ConnectorTabularSource(Base, TimestampMixin):
|
|||||||
row_count: Mapped[int] = mapped_column(Integer, nullable=False)
|
row_count: Mapped[int] = mapped_column(Integer, nullable=False)
|
||||||
byte_count: Mapped[int] = mapped_column(Integer, nullable=False)
|
byte_count: Mapped[int] = mapped_column(Integer, nullable=False)
|
||||||
metadata_: Mapped[dict[str, Any]] = mapped_column("metadata", JSON, default=dict, nullable=False)
|
metadata_: Mapped[dict[str, Any]] = mapped_column("metadata", JSON, default=dict, nullable=False)
|
||||||
|
# Original upload content is deliberately excluded from ordinary catalogue loads/DTOs.
|
||||||
|
csv_source_: Mapped[dict[str, Any] | None] = mapped_column("csv_source", JSON, nullable=True, deferred=True)
|
||||||
created_by: Mapped[str | None] = mapped_column(String(255), nullable=True, index=True)
|
created_by: Mapped[str | None] = mapped_column(String(255), nullable=True, index=True)
|
||||||
updated_by: Mapped[str | None] = mapped_column(String(255), nullable=True, index=True)
|
updated_by: Mapped[str | None] = mapped_column(String(255), nullable=True, index=True)
|
||||||
deleted_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True, index=True)
|
deleted_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True, index=True)
|
||||||
@@ -456,10 +458,373 @@ class ConnectorSimulationRun(Base, TimestampMixin):
|
|||||||
review_reason: Mapped[str | None] = mapped_column(Text)
|
review_reason: Mapped[str | None] = mapped_column(Text)
|
||||||
|
|
||||||
|
|
||||||
|
class ConnectorKnowledgeProfile(Base, TimestampMixin):
|
||||||
|
__tablename__ = "connector_knowledge_profiles"
|
||||||
|
__table_args__ = (
|
||||||
|
UniqueConstraint(
|
||||||
|
"tenant_id",
|
||||||
|
"configuration_id",
|
||||||
|
name="uq_connector_knowledge_profile_configuration",
|
||||||
|
),
|
||||||
|
Index(
|
||||||
|
"ix_connector_knowledge_profiles_tenant_status",
|
||||||
|
"tenant_id",
|
||||||
|
"status",
|
||||||
|
),
|
||||||
|
)
|
||||||
|
|
||||||
|
id: Mapped[str] = mapped_column(String(36), primary_key=True, default=new_uuid)
|
||||||
|
tenant_id: Mapped[str] = mapped_column(String(36), nullable=False, index=True)
|
||||||
|
configuration_id: Mapped[str] = mapped_column(
|
||||||
|
ForeignKey("connector_configurations.id", ondelete="RESTRICT"),
|
||||||
|
nullable=False,
|
||||||
|
index=True,
|
||||||
|
)
|
||||||
|
status: Mapped[str] = mapped_column(
|
||||||
|
String(30), default="active", nullable=False, index=True
|
||||||
|
)
|
||||||
|
product: Mapped[str] = mapped_column(
|
||||||
|
String(50), default="unknown", nullable=False, index=True
|
||||||
|
)
|
||||||
|
product_version: Mapped[str | None] = mapped_column(String(100))
|
||||||
|
desired_maturity: Mapped[str] = mapped_column(
|
||||||
|
String(30), default="read", nullable=False
|
||||||
|
)
|
||||||
|
discovered_maturity: Mapped[str] = mapped_column(
|
||||||
|
String(30), default="discover", nullable=False
|
||||||
|
)
|
||||||
|
source_authority_mode: Mapped[str] = mapped_column(
|
||||||
|
String(40), default="external_mirror", nullable=False
|
||||||
|
)
|
||||||
|
default_visibility: Mapped[str] = mapped_column(
|
||||||
|
String(30), default="restricted", nullable=False
|
||||||
|
)
|
||||||
|
default_acl_tokens: Mapped[list[str]] = mapped_column(
|
||||||
|
JSON, default=list, nullable=False
|
||||||
|
)
|
||||||
|
namespace_mappings: Mapped[list[dict[str, Any]]] = mapped_column(
|
||||||
|
JSON, default=list, nullable=False
|
||||||
|
)
|
||||||
|
capabilities: Mapped[list[str]] = mapped_column(JSON, default=list, nullable=False)
|
||||||
|
discovery_revision: Mapped[str | None] = mapped_column(String(255), index=True)
|
||||||
|
discovery_evidence: Mapped[dict[str, Any]] = mapped_column(
|
||||||
|
JSON, default=dict, nullable=False
|
||||||
|
)
|
||||||
|
health_status: Mapped[str] = mapped_column(
|
||||||
|
String(30), default="unknown", nullable=False, index=True
|
||||||
|
)
|
||||||
|
health_details: Mapped[dict[str, Any]] = mapped_column(
|
||||||
|
JSON, default=dict, nullable=False
|
||||||
|
)
|
||||||
|
discovered_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
|
||||||
|
last_sync_cursor: Mapped[str | None] = mapped_column(String(500))
|
||||||
|
last_high_watermark: Mapped[str | None] = mapped_column(String(500))
|
||||||
|
resource_revision: Mapped[int] = mapped_column(Integer, default=1, nullable=False)
|
||||||
|
updated_by: Mapped[str | None] = mapped_column(String(255), index=True)
|
||||||
|
|
||||||
|
|
||||||
|
class ConnectorKnowledgeObject(Base, TimestampMixin):
|
||||||
|
__tablename__ = "connector_knowledge_objects"
|
||||||
|
__table_args__ = (
|
||||||
|
UniqueConstraint(
|
||||||
|
"profile_id",
|
||||||
|
"object_type",
|
||||||
|
"external_id",
|
||||||
|
name="uq_connector_knowledge_object_identity",
|
||||||
|
),
|
||||||
|
Index(
|
||||||
|
"ix_connector_knowledge_objects_tenant_profile_status",
|
||||||
|
"tenant_id",
|
||||||
|
"profile_id",
|
||||||
|
"status",
|
||||||
|
),
|
||||||
|
Index(
|
||||||
|
"ix_connector_knowledge_objects_tenant_updated",
|
||||||
|
"tenant_id",
|
||||||
|
"source_updated_at",
|
||||||
|
),
|
||||||
|
)
|
||||||
|
|
||||||
|
id: Mapped[str] = mapped_column(String(36), primary_key=True, default=new_uuid)
|
||||||
|
tenant_id: Mapped[str] = mapped_column(String(36), nullable=False, index=True)
|
||||||
|
profile_id: Mapped[str] = mapped_column(
|
||||||
|
ForeignKey("connector_knowledge_profiles.id", ondelete="CASCADE"),
|
||||||
|
nullable=False,
|
||||||
|
index=True,
|
||||||
|
)
|
||||||
|
object_type: Mapped[str] = mapped_column(String(40), nullable=False, index=True)
|
||||||
|
external_id: Mapped[str] = mapped_column(String(255), nullable=False)
|
||||||
|
external_page_id: Mapped[str | None] = mapped_column(String(255), index=True)
|
||||||
|
external_revision_id: Mapped[str | None] = mapped_column(String(255), index=True)
|
||||||
|
namespace_id: Mapped[int | None] = mapped_column(Integer, index=True)
|
||||||
|
title: Mapped[str] = mapped_column(String(500), nullable=False)
|
||||||
|
canonical_url: Mapped[str | None] = mapped_column(String(1500))
|
||||||
|
status: Mapped[str] = mapped_column(
|
||||||
|
String(30), default="active", nullable=False, index=True
|
||||||
|
)
|
||||||
|
redirect_target_external_id: Mapped[str | None] = mapped_column(String(255))
|
||||||
|
source_revision: Mapped[str] = mapped_column(String(255), nullable=False)
|
||||||
|
content_hash: Mapped[str] = mapped_column(String(64), nullable=False, index=True)
|
||||||
|
visibility: Mapped[str] = mapped_column(
|
||||||
|
String(30), default="restricted", nullable=False
|
||||||
|
)
|
||||||
|
acl_tokens: Mapped[list[str]] = mapped_column(JSON, default=list, nullable=False)
|
||||||
|
mapped_data: Mapped[dict[str, Any]] = mapped_column(
|
||||||
|
JSON, default=dict, nullable=False
|
||||||
|
)
|
||||||
|
provenance: Mapped[dict[str, Any]] = mapped_column(
|
||||||
|
JSON, default=dict, nullable=False
|
||||||
|
)
|
||||||
|
change_cursor: Mapped[str | None] = mapped_column(String(500), index=True)
|
||||||
|
source_updated_at: Mapped[datetime | None] = mapped_column(
|
||||||
|
DateTime(timezone=True), index=True
|
||||||
|
)
|
||||||
|
observed_at: Mapped[datetime] = mapped_column(
|
||||||
|
DateTime(timezone=True), nullable=False, index=True
|
||||||
|
)
|
||||||
|
resource_revision: Mapped[int] = mapped_column(Integer, default=1, nullable=False)
|
||||||
|
|
||||||
|
|
||||||
|
class ConnectorKnowledgeSyncRun(Base, TimestampMixin):
|
||||||
|
__tablename__ = "connector_knowledge_sync_runs"
|
||||||
|
__table_args__ = (
|
||||||
|
UniqueConstraint(
|
||||||
|
"tenant_id",
|
||||||
|
"profile_id",
|
||||||
|
"mode",
|
||||||
|
"idempotency_key",
|
||||||
|
name="uq_connector_knowledge_sync_run_idempotency",
|
||||||
|
),
|
||||||
|
Index(
|
||||||
|
"ix_connector_knowledge_sync_runs_profile_started",
|
||||||
|
"tenant_id",
|
||||||
|
"profile_id",
|
||||||
|
"started_at",
|
||||||
|
),
|
||||||
|
)
|
||||||
|
|
||||||
|
id: Mapped[str] = mapped_column(String(36), primary_key=True, default=new_uuid)
|
||||||
|
tenant_id: Mapped[str] = mapped_column(String(36), nullable=False, index=True)
|
||||||
|
profile_id: Mapped[str] = mapped_column(
|
||||||
|
ForeignKey("connector_knowledge_profiles.id", ondelete="CASCADE"),
|
||||||
|
nullable=False,
|
||||||
|
index=True,
|
||||||
|
)
|
||||||
|
mode: Mapped[str] = mapped_column(String(40), nullable=False, index=True)
|
||||||
|
idempotency_key: Mapped[str] = mapped_column(String(255), nullable=False)
|
||||||
|
request_hash: Mapped[str] = mapped_column(String(64), nullable=False)
|
||||||
|
status: Mapped[str] = mapped_column(String(30), nullable=False, index=True)
|
||||||
|
cursor_before: Mapped[str | None] = mapped_column(String(500))
|
||||||
|
cursor_after: Mapped[str | None] = mapped_column(String(500))
|
||||||
|
high_watermark: Mapped[str | None] = mapped_column(String(500))
|
||||||
|
counts: Mapped[dict[str, Any]] = mapped_column(JSON, default=dict, nullable=False)
|
||||||
|
effects: Mapped[list[dict[str, Any]]] = mapped_column(
|
||||||
|
JSON, default=list, nullable=False
|
||||||
|
)
|
||||||
|
diagnostics: Mapped[list[dict[str, Any]]] = mapped_column(
|
||||||
|
JSON, default=list, nullable=False
|
||||||
|
)
|
||||||
|
provenance: Mapped[dict[str, Any]] = mapped_column(
|
||||||
|
JSON, default=dict, nullable=False
|
||||||
|
)
|
||||||
|
created_by: Mapped[str | None] = mapped_column(String(255), index=True)
|
||||||
|
started_at: Mapped[datetime] = mapped_column(
|
||||||
|
DateTime(timezone=True), nullable=False, index=True
|
||||||
|
)
|
||||||
|
finished_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
|
||||||
|
|
||||||
|
|
||||||
|
class ConnectorServiceDeskProfile(Base, TimestampMixin):
|
||||||
|
__tablename__ = "connector_service_desk_profiles"
|
||||||
|
__table_args__ = (
|
||||||
|
UniqueConstraint(
|
||||||
|
"tenant_id",
|
||||||
|
"configuration_id",
|
||||||
|
name="uq_connector_service_desk_profile_configuration",
|
||||||
|
),
|
||||||
|
Index(
|
||||||
|
"ix_connector_service_desk_profiles_tenant_status",
|
||||||
|
"tenant_id",
|
||||||
|
"status",
|
||||||
|
),
|
||||||
|
)
|
||||||
|
|
||||||
|
id: Mapped[str] = mapped_column(String(36), primary_key=True, default=new_uuid)
|
||||||
|
tenant_id: Mapped[str] = mapped_column(String(36), nullable=False, index=True)
|
||||||
|
configuration_id: Mapped[str] = mapped_column(
|
||||||
|
ForeignKey("connector_configurations.id", ondelete="RESTRICT"),
|
||||||
|
nullable=False,
|
||||||
|
index=True,
|
||||||
|
)
|
||||||
|
status: Mapped[str] = mapped_column(
|
||||||
|
String(30), default="active", nullable=False, index=True
|
||||||
|
)
|
||||||
|
integration_mode: Mapped[str] = mapped_column(
|
||||||
|
String(30), default="synchronize", nullable=False, index=True
|
||||||
|
)
|
||||||
|
product: Mapped[str] = mapped_column(
|
||||||
|
String(50), default="unknown", nullable=False, index=True
|
||||||
|
)
|
||||||
|
product_version: Mapped[str | None] = mapped_column(String(100))
|
||||||
|
desired_maturity: Mapped[str] = mapped_column(
|
||||||
|
String(30), default="synchronize", nullable=False
|
||||||
|
)
|
||||||
|
discovered_maturity: Mapped[str] = mapped_column(
|
||||||
|
String(30), default="discover", nullable=False
|
||||||
|
)
|
||||||
|
source_authority_mode: Mapped[str] = mapped_column(
|
||||||
|
String(40), default="external_authoritative", nullable=False
|
||||||
|
)
|
||||||
|
default_visibility: Mapped[str] = mapped_column(
|
||||||
|
String(30), default="restricted", nullable=False
|
||||||
|
)
|
||||||
|
default_acl_tokens: Mapped[list[str]] = mapped_column(
|
||||||
|
JSON, default=list, nullable=False
|
||||||
|
)
|
||||||
|
routes: Mapped[dict[str, Any]] = mapped_column(JSON, default=dict, nullable=False)
|
||||||
|
queue_mappings: Mapped[list[dict[str, Any]]] = mapped_column(
|
||||||
|
JSON, default=list, nullable=False
|
||||||
|
)
|
||||||
|
dynamic_field_mappings: Mapped[list[dict[str, Any]]] = mapped_column(
|
||||||
|
JSON, default=list, nullable=False
|
||||||
|
)
|
||||||
|
capabilities: Mapped[list[str]] = mapped_column(JSON, default=list, nullable=False)
|
||||||
|
discovery_revision: Mapped[str | None] = mapped_column(String(255), index=True)
|
||||||
|
discovered_configuration_revision: Mapped[int | None] = mapped_column(Integer)
|
||||||
|
discovered_configuration_hash: Mapped[str | None] = mapped_column(String(64))
|
||||||
|
discovery_evidence: Mapped[dict[str, Any]] = mapped_column(
|
||||||
|
JSON, default=dict, nullable=False
|
||||||
|
)
|
||||||
|
health_status: Mapped[str] = mapped_column(
|
||||||
|
String(30), default="unknown", nullable=False, index=True
|
||||||
|
)
|
||||||
|
health_details: Mapped[dict[str, Any]] = mapped_column(
|
||||||
|
JSON, default=dict, nullable=False
|
||||||
|
)
|
||||||
|
discovered_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
|
||||||
|
last_sync_cursor: Mapped[str | None] = mapped_column(String(4000))
|
||||||
|
last_high_watermark: Mapped[str | None] = mapped_column(String(500))
|
||||||
|
resource_revision: Mapped[int] = mapped_column(Integer, default=1, nullable=False)
|
||||||
|
updated_by: Mapped[str | None] = mapped_column(String(255), index=True)
|
||||||
|
|
||||||
|
|
||||||
|
class ConnectorServiceDeskObject(Base, TimestampMixin):
|
||||||
|
__tablename__ = "connector_service_desk_objects"
|
||||||
|
__table_args__ = (
|
||||||
|
UniqueConstraint(
|
||||||
|
"profile_id",
|
||||||
|
"object_type",
|
||||||
|
"external_id",
|
||||||
|
name="uq_connector_service_desk_object_identity",
|
||||||
|
),
|
||||||
|
Index(
|
||||||
|
"ix_connector_service_desk_objects_tenant_profile_status",
|
||||||
|
"tenant_id",
|
||||||
|
"profile_id",
|
||||||
|
"status",
|
||||||
|
),
|
||||||
|
Index(
|
||||||
|
"ix_connector_service_desk_objects_tenant_updated",
|
||||||
|
"tenant_id",
|
||||||
|
"source_updated_at",
|
||||||
|
),
|
||||||
|
)
|
||||||
|
|
||||||
|
id: Mapped[str] = mapped_column(String(36), primary_key=True, default=new_uuid)
|
||||||
|
tenant_id: Mapped[str] = mapped_column(String(36), nullable=False, index=True)
|
||||||
|
profile_id: Mapped[str] = mapped_column(
|
||||||
|
ForeignKey("connector_service_desk_profiles.id", ondelete="CASCADE"),
|
||||||
|
nullable=False,
|
||||||
|
index=True,
|
||||||
|
)
|
||||||
|
object_type: Mapped[str] = mapped_column(String(40), nullable=False, index=True)
|
||||||
|
external_id: Mapped[str] = mapped_column(String(255), nullable=False)
|
||||||
|
external_ticket_number: Mapped[str | None] = mapped_column(String(255), index=True)
|
||||||
|
title: Mapped[str] = mapped_column(String(500), nullable=False)
|
||||||
|
canonical_url: Mapped[str | None] = mapped_column(String(1500))
|
||||||
|
status: Mapped[str] = mapped_column(
|
||||||
|
String(30), default="active", nullable=False, index=True
|
||||||
|
)
|
||||||
|
source_revision: Mapped[str] = mapped_column(String(255), nullable=False)
|
||||||
|
content_hash: Mapped[str] = mapped_column(String(64), nullable=False, index=True)
|
||||||
|
visibility: Mapped[str] = mapped_column(
|
||||||
|
String(30), default="restricted", nullable=False
|
||||||
|
)
|
||||||
|
acl_tokens: Mapped[list[str]] = mapped_column(JSON, default=list, nullable=False)
|
||||||
|
mapped_data: Mapped[dict[str, Any]] = mapped_column(
|
||||||
|
JSON, default=dict, nullable=False
|
||||||
|
)
|
||||||
|
provenance: Mapped[dict[str, Any]] = mapped_column(
|
||||||
|
JSON, default=dict, nullable=False
|
||||||
|
)
|
||||||
|
change_cursor: Mapped[str | None] = mapped_column(String(4000), index=True)
|
||||||
|
source_updated_at: Mapped[datetime | None] = mapped_column(
|
||||||
|
DateTime(timezone=True), index=True
|
||||||
|
)
|
||||||
|
observed_at: Mapped[datetime] = mapped_column(
|
||||||
|
DateTime(timezone=True), nullable=False, index=True
|
||||||
|
)
|
||||||
|
resource_revision: Mapped[int] = mapped_column(Integer, default=1, nullable=False)
|
||||||
|
|
||||||
|
|
||||||
|
class ConnectorServiceDeskSyncRun(Base, TimestampMixin):
|
||||||
|
__tablename__ = "connector_service_desk_sync_runs"
|
||||||
|
__table_args__ = (
|
||||||
|
UniqueConstraint(
|
||||||
|
"tenant_id",
|
||||||
|
"profile_id",
|
||||||
|
"idempotency_key",
|
||||||
|
name="uq_connector_service_desk_sync_run_idempotency",
|
||||||
|
),
|
||||||
|
Index(
|
||||||
|
"ix_connector_service_desk_runs_profile_started",
|
||||||
|
"tenant_id",
|
||||||
|
"profile_id",
|
||||||
|
"started_at",
|
||||||
|
),
|
||||||
|
)
|
||||||
|
|
||||||
|
id: Mapped[str] = mapped_column(String(36), primary_key=True, default=new_uuid)
|
||||||
|
tenant_id: Mapped[str] = mapped_column(String(36), nullable=False, index=True)
|
||||||
|
profile_id: Mapped[str] = mapped_column(
|
||||||
|
ForeignKey("connector_service_desk_profiles.id", ondelete="CASCADE"),
|
||||||
|
nullable=False,
|
||||||
|
index=True,
|
||||||
|
)
|
||||||
|
mode: Mapped[str] = mapped_column(String(40), nullable=False, index=True)
|
||||||
|
idempotency_key: Mapped[str] = mapped_column(String(255), nullable=False)
|
||||||
|
request_hash: Mapped[str] = mapped_column(String(64), nullable=False)
|
||||||
|
status: Mapped[str] = mapped_column(String(30), nullable=False, index=True)
|
||||||
|
cursor_before: Mapped[str | None] = mapped_column(String(4000))
|
||||||
|
cursor_after: Mapped[str | None] = mapped_column(String(4000))
|
||||||
|
high_watermark: Mapped[str | None] = mapped_column(String(500))
|
||||||
|
counts: Mapped[dict[str, Any]] = mapped_column(JSON, default=dict, nullable=False)
|
||||||
|
effects: Mapped[list[dict[str, Any]]] = mapped_column(
|
||||||
|
JSON, default=list, nullable=False
|
||||||
|
)
|
||||||
|
diagnostics: Mapped[list[dict[str, Any]]] = mapped_column(
|
||||||
|
JSON, default=list, nullable=False
|
||||||
|
)
|
||||||
|
provenance: Mapped[dict[str, Any]] = mapped_column(
|
||||||
|
JSON, default=dict, nullable=False
|
||||||
|
)
|
||||||
|
created_by: Mapped[str | None] = mapped_column(String(255), index=True)
|
||||||
|
started_at: Mapped[datetime] = mapped_column(
|
||||||
|
DateTime(timezone=True), nullable=False, index=True
|
||||||
|
)
|
||||||
|
finished_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
|
||||||
|
|
||||||
|
|
||||||
__all__ = [
|
__all__ = [
|
||||||
"ConnectorConfiguration",
|
"ConnectorConfiguration",
|
||||||
"ConnectorDefinition",
|
"ConnectorDefinition",
|
||||||
"ConnectorDefinitionRevision",
|
"ConnectorDefinitionRevision",
|
||||||
|
"ConnectorKnowledgeObject",
|
||||||
|
"ConnectorKnowledgeProfile",
|
||||||
|
"ConnectorKnowledgeSyncRun",
|
||||||
|
"ConnectorServiceDeskObject",
|
||||||
|
"ConnectorServiceDeskProfile",
|
||||||
|
"ConnectorServiceDeskSyncRun",
|
||||||
"ConnectorSanctionsAcquisitionRun",
|
"ConnectorSanctionsAcquisitionRun",
|
||||||
"ConnectorSanctionsSnapshot",
|
"ConnectorSanctionsSnapshot",
|
||||||
"ConnectorSimulationRun",
|
"ConnectorSimulationRun",
|
||||||
|
|||||||
@@ -18,6 +18,10 @@ from govoplan_connectors.backend.db.models import (
|
|||||||
ConnectorConfiguration,
|
ConnectorConfiguration,
|
||||||
ConnectorDefinition,
|
ConnectorDefinition,
|
||||||
ConnectorDefinitionRevision,
|
ConnectorDefinitionRevision,
|
||||||
|
ConnectorKnowledgeProfile,
|
||||||
|
ConnectorKnowledgeSyncRun,
|
||||||
|
ConnectorServiceDeskProfile,
|
||||||
|
ConnectorServiceDeskSyncRun,
|
||||||
ConnectorSanctionsAcquisitionRun,
|
ConnectorSanctionsAcquisitionRun,
|
||||||
ConnectorSimulationRun,
|
ConnectorSimulationRun,
|
||||||
ConnectorTabularSource,
|
ConnectorTabularSource,
|
||||||
@@ -37,6 +41,10 @@ class _SubjectSelectors:
|
|||||||
definition_id: str | None
|
definition_id: str | None
|
||||||
configuration_id: str | None
|
configuration_id: str | None
|
||||||
simulation_id: str | None
|
simulation_id: str | None
|
||||||
|
knowledge_profile_id: str | None
|
||||||
|
knowledge_run_id: str | None
|
||||||
|
service_desk_profile_id: str | None
|
||||||
|
service_desk_run_id: str | None
|
||||||
|
|
||||||
@property
|
@property
|
||||||
def narrowed(self) -> bool:
|
def narrowed(self) -> bool:
|
||||||
@@ -47,6 +55,10 @@ class _SubjectSelectors:
|
|||||||
self.definition_id,
|
self.definition_id,
|
||||||
self.configuration_id,
|
self.configuration_id,
|
||||||
self.simulation_id,
|
self.simulation_id,
|
||||||
|
self.knowledge_profile_id,
|
||||||
|
self.knowledge_run_id,
|
||||||
|
self.service_desk_profile_id,
|
||||||
|
self.service_desk_run_id,
|
||||||
)
|
)
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -172,6 +184,83 @@ class ConnectorsDsarProvider:
|
|||||||
)
|
)
|
||||||
)
|
)
|
||||||
|
|
||||||
|
if not selectors.narrowed or selectors.knowledge_profile_id:
|
||||||
|
query = db.query(ConnectorKnowledgeProfile).filter(
|
||||||
|
ConnectorKnowledgeProfile.tenant_id == tenant_id,
|
||||||
|
ConnectorKnowledgeProfile.updated_by == selectors.account_id,
|
||||||
|
)
|
||||||
|
if selectors.knowledge_profile_id:
|
||||||
|
query = query.filter(
|
||||||
|
ConnectorKnowledgeProfile.id == selectors.knowledge_profile_id
|
||||||
|
)
|
||||||
|
records.extend(
|
||||||
|
_knowledge_profile_attribution(row)
|
||||||
|
for row in _limited(
|
||||||
|
query,
|
||||||
|
ConnectorKnowledgeProfile.created_at,
|
||||||
|
ConnectorKnowledgeProfile.id,
|
||||||
|
label="knowledge profile attribution",
|
||||||
|
)
|
||||||
|
)
|
||||||
|
|
||||||
|
if not selectors.narrowed or selectors.knowledge_run_id:
|
||||||
|
query = db.query(ConnectorKnowledgeSyncRun).filter(
|
||||||
|
ConnectorKnowledgeSyncRun.tenant_id == tenant_id,
|
||||||
|
ConnectorKnowledgeSyncRun.created_by == selectors.account_id,
|
||||||
|
)
|
||||||
|
if selectors.knowledge_run_id:
|
||||||
|
query = query.filter(
|
||||||
|
ConnectorKnowledgeSyncRun.id == selectors.knowledge_run_id
|
||||||
|
)
|
||||||
|
records.extend(
|
||||||
|
_knowledge_run_attribution(row)
|
||||||
|
for row in _limited(
|
||||||
|
query,
|
||||||
|
ConnectorKnowledgeSyncRun.started_at,
|
||||||
|
ConnectorKnowledgeSyncRun.id,
|
||||||
|
label="knowledge run attribution",
|
||||||
|
)
|
||||||
|
)
|
||||||
|
|
||||||
|
if not selectors.narrowed or selectors.service_desk_profile_id:
|
||||||
|
query = db.query(ConnectorServiceDeskProfile).filter(
|
||||||
|
ConnectorServiceDeskProfile.tenant_id == tenant_id,
|
||||||
|
ConnectorServiceDeskProfile.updated_by == selectors.account_id,
|
||||||
|
)
|
||||||
|
if selectors.service_desk_profile_id:
|
||||||
|
query = query.filter(
|
||||||
|
ConnectorServiceDeskProfile.id
|
||||||
|
== selectors.service_desk_profile_id
|
||||||
|
)
|
||||||
|
records.extend(
|
||||||
|
_service_desk_profile_attribution(row)
|
||||||
|
for row in _limited(
|
||||||
|
query,
|
||||||
|
ConnectorServiceDeskProfile.created_at,
|
||||||
|
ConnectorServiceDeskProfile.id,
|
||||||
|
label="service-desk profile attribution",
|
||||||
|
)
|
||||||
|
)
|
||||||
|
|
||||||
|
if not selectors.narrowed or selectors.service_desk_run_id:
|
||||||
|
query = db.query(ConnectorServiceDeskSyncRun).filter(
|
||||||
|
ConnectorServiceDeskSyncRun.tenant_id == tenant_id,
|
||||||
|
ConnectorServiceDeskSyncRun.created_by == selectors.account_id,
|
||||||
|
)
|
||||||
|
if selectors.service_desk_run_id:
|
||||||
|
query = query.filter(
|
||||||
|
ConnectorServiceDeskSyncRun.id == selectors.service_desk_run_id
|
||||||
|
)
|
||||||
|
records.extend(
|
||||||
|
_service_desk_run_attribution(row)
|
||||||
|
for row in _limited(
|
||||||
|
query,
|
||||||
|
ConnectorServiceDeskSyncRun.started_at,
|
||||||
|
ConnectorServiceDeskSyncRun.id,
|
||||||
|
label="service-desk run attribution",
|
||||||
|
)
|
||||||
|
)
|
||||||
|
|
||||||
if len(records) > _MAX_RECORDS:
|
if len(records) > _MAX_RECORDS:
|
||||||
raise ValueError("Connectors DSAR result limit exceeded; narrow selectors.")
|
raise ValueError("Connectors DSAR result limit exceeded; narrow selectors.")
|
||||||
return tuple(
|
return tuple(
|
||||||
@@ -270,6 +359,22 @@ def _subject_selectors(subject: DsarSubjectRef) -> _SubjectSelectors | None:
|
|||||||
references.get("connectors.simulation"),
|
references.get("connectors.simulation"),
|
||||||
references.get("connectors.simulation_id"),
|
references.get("connectors.simulation_id"),
|
||||||
),
|
),
|
||||||
|
"knowledge_profile_id": _coalesce(
|
||||||
|
references.get("connectors.knowledge_profile"),
|
||||||
|
references.get("connectors.knowledge_profile_id"),
|
||||||
|
),
|
||||||
|
"knowledge_run_id": _coalesce(
|
||||||
|
references.get("connectors.knowledge_run"),
|
||||||
|
references.get("connectors.knowledge_run_id"),
|
||||||
|
),
|
||||||
|
"service_desk_profile_id": _coalesce(
|
||||||
|
references.get("connectors.service_desk_profile"),
|
||||||
|
references.get("connectors.service_desk_profile_id"),
|
||||||
|
),
|
||||||
|
"service_desk_run_id": _coalesce(
|
||||||
|
references.get("connectors.service_desk_run"),
|
||||||
|
references.get("connectors.service_desk_run_id"),
|
||||||
|
),
|
||||||
}
|
}
|
||||||
if account is _CONFLICT or any(value is _CONFLICT for value in values.values()):
|
if account is _CONFLICT or any(value is _CONFLICT for value in values.values()):
|
||||||
return None
|
return None
|
||||||
@@ -283,6 +388,10 @@ def _subject_selectors(subject: DsarSubjectRef) -> _SubjectSelectors | None:
|
|||||||
definition_id=_optional_string(values["definition_id"]),
|
definition_id=_optional_string(values["definition_id"]),
|
||||||
configuration_id=_optional_string(values["configuration_id"]),
|
configuration_id=_optional_string(values["configuration_id"]),
|
||||||
simulation_id=_optional_string(values["simulation_id"]),
|
simulation_id=_optional_string(values["simulation_id"]),
|
||||||
|
knowledge_profile_id=_optional_string(values["knowledge_profile_id"]),
|
||||||
|
knowledge_run_id=_optional_string(values["knowledge_run_id"]),
|
||||||
|
service_desk_profile_id=_optional_string(values["service_desk_profile_id"]),
|
||||||
|
service_desk_run_id=_optional_string(values["service_desk_run_id"]),
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
@@ -396,6 +505,91 @@ def _simulation_attribution(
|
|||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _knowledge_profile_attribution(
|
||||||
|
row: ConnectorKnowledgeProfile,
|
||||||
|
) -> DsarRecordRef:
|
||||||
|
return _record(
|
||||||
|
resource_type="knowledge_profile_actor_attribution",
|
||||||
|
resource_id=row.id,
|
||||||
|
title="External knowledge profile actor attribution",
|
||||||
|
data={
|
||||||
|
"knowledge_profile_id": row.id,
|
||||||
|
"configuration_id": row.configuration_id,
|
||||||
|
"status": row.status,
|
||||||
|
"desired_maturity": row.desired_maturity,
|
||||||
|
"source_authority_mode": row.source_authority_mode,
|
||||||
|
"resource_revision": row.resource_revision,
|
||||||
|
"activity": "updated_external_knowledge_profile",
|
||||||
|
"created_at": _iso(row.created_at),
|
||||||
|
"updated_at": _iso(row.updated_at),
|
||||||
|
},
|
||||||
|
observed_at=row.updated_at,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _knowledge_run_attribution(
|
||||||
|
row: ConnectorKnowledgeSyncRun,
|
||||||
|
) -> DsarRecordRef:
|
||||||
|
return _record(
|
||||||
|
resource_type="knowledge_run_actor_attribution",
|
||||||
|
resource_id=row.id,
|
||||||
|
title="External knowledge operation actor attribution",
|
||||||
|
data={
|
||||||
|
"knowledge_run_id": row.id,
|
||||||
|
"knowledge_profile_id": row.profile_id,
|
||||||
|
"mode": row.mode,
|
||||||
|
"status": row.status,
|
||||||
|
"started_at": _iso(row.started_at),
|
||||||
|
"finished_at": _iso(row.finished_at),
|
||||||
|
"activity": "started_external_knowledge_operation",
|
||||||
|
},
|
||||||
|
observed_at=row.finished_at or row.started_at,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _service_desk_profile_attribution(
|
||||||
|
row: ConnectorServiceDeskProfile,
|
||||||
|
) -> DsarRecordRef:
|
||||||
|
return _record(
|
||||||
|
resource_type="service_desk_profile_actor_attribution",
|
||||||
|
resource_id=row.id,
|
||||||
|
title="External service-desk profile actor attribution",
|
||||||
|
data={
|
||||||
|
"service_desk_profile_id": row.id,
|
||||||
|
"configuration_id": row.configuration_id,
|
||||||
|
"status": row.status,
|
||||||
|
"integration_mode": row.integration_mode,
|
||||||
|
"desired_maturity": row.desired_maturity,
|
||||||
|
"source_authority_mode": row.source_authority_mode,
|
||||||
|
"resource_revision": row.resource_revision,
|
||||||
|
"activity": "updated_external_service_desk_profile",
|
||||||
|
"created_at": _iso(row.created_at),
|
||||||
|
"updated_at": _iso(row.updated_at),
|
||||||
|
},
|
||||||
|
observed_at=row.updated_at,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _service_desk_run_attribution(
|
||||||
|
row: ConnectorServiceDeskSyncRun,
|
||||||
|
) -> DsarRecordRef:
|
||||||
|
return _record(
|
||||||
|
resource_type="service_desk_run_actor_attribution",
|
||||||
|
resource_id=row.id,
|
||||||
|
title="External service-desk operation actor attribution",
|
||||||
|
data={
|
||||||
|
"service_desk_run_id": row.id,
|
||||||
|
"service_desk_profile_id": row.profile_id,
|
||||||
|
"mode": row.mode,
|
||||||
|
"status": row.status,
|
||||||
|
"started_at": _iso(row.started_at),
|
||||||
|
"finished_at": _iso(row.finished_at),
|
||||||
|
"activity": "started_external_service_desk_operation",
|
||||||
|
},
|
||||||
|
observed_at=row.finished_at or row.started_at,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
def _record(
|
def _record(
|
||||||
*,
|
*,
|
||||||
resource_type: str,
|
resource_type: str,
|
||||||
@@ -462,6 +656,10 @@ _RESOURCE_TYPES = {
|
|||||||
"definition_actor_attribution",
|
"definition_actor_attribution",
|
||||||
"configuration_actor_attribution",
|
"configuration_actor_attribution",
|
||||||
"simulation_actor_attribution",
|
"simulation_actor_attribution",
|
||||||
|
"knowledge_profile_actor_attribution",
|
||||||
|
"knowledge_run_actor_attribution",
|
||||||
|
"service_desk_profile_actor_attribution",
|
||||||
|
"service_desk_run_actor_attribution",
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,66 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from typing import Iterable
|
||||||
|
|
||||||
|
from govoplan_core.core.modules import DocumentationTopic, localize_documentation_topics as _localize_topics
|
||||||
|
|
||||||
|
|
||||||
|
_TRANSLATIONS = {
|
||||||
|
"connectors.data-subject-requests": {
|
||||||
|
"title": "Datenschutzanfragen zu Connector-Daten",
|
||||||
|
"summary": "Nachvollziehbare Connector-Aktivitäten exportieren, ohne Zugangsdaten oder externe Inhalte offenzulegen.",
|
||||||
|
"body": (
|
||||||
|
"Connectors gleicht ausschließlich eine eindeutige, mandantenbezogene Konto-ID ab und kann eine bereits verifizierte Suche auf eine Quelle, einen Abruf, eine Definition, Konfiguration, Simulation, ein externes Wissens- oder Service-Desk-Profil oder einen Connector-Vorgang begrenzen. Der Export weist Konfigurations-, Abruf-, Simulations-, externe Vorgangs- und Prüfaktivitäten anhand begrenzter Lebenszyklusmetadaten aus. Zugangsdaten, Endpunktreferenzen, Quelldatensätze, externe Antworten, Anfrageinhalte, Mapping- oder Konfigurationsdokumente, Diagnosen, Provenienz, Hashwerte und Transportnachweise werden nie ausgegeben. Die Zuordnung zu handelnden Personen bleibt unveränderlicher Governance- und Vorgangsnachweis und wird nicht automatisch gelöscht. E-Mail- oder Objektkennungen ohne verifizierte Konto-ID begründen keinen Treffer."
|
||||||
|
),
|
||||||
|
},
|
||||||
|
"connectors.governed-configuration": {
|
||||||
|
"title": "Connector-Definitionen und Simulationen steuern",
|
||||||
|
"summary": "Connector-Schemata und Mappings versionieren und dabei lokale Überschreibungen sowie Prüfnachweise erhalten.",
|
||||||
|
"body": (
|
||||||
|
"Connector-Administrationen erstellen paketverwaltete oder lokale Definitionen, die Anbieter, Protokoll, Fähigkeiten, Schemata, Mapping-Regeln, Validierung, Vorschau, Audit-Anforderungen, Datenschutz, Aufbewahrung, Grenzen und Wiederholungsverhalten ausdrücklich beschreiben. Jede Änderung erzeugt eine unveränderliche Revision. Eine Mandantenkonfiguration bindet genau eine Revision und speichert nur eine Zugangsdatenreferenz; Paketaktualisierungen ändern die wirksame Konfiguration erst nach bewusster Übernahme. Lokale Überschreibungen werden als geschützt angezeigt und bei der Übernahme erneut angewendet. Testläufe und Simulationen sind begrenzt, schwärzen konfigurierte Felder, sind über den Aufrufschlüssel idempotent und bewahren Konfigurations-, Mapping-, Eingabe- und externe Revisionsprovenienz. Mehrdeutige Ergebnisse folgen der Richtlinie für manuelle Prüfung, Quarantäne oder Ablehnung. Offene und quarantänisierte Nachweise müssen mit Begründung freigegeben oder abgelehnt werden. Eine erfolgreiche generische Simulation verspricht keinen anbieterspezifischen Live-Schreibzugriff."
|
||||||
|
),
|
||||||
|
},
|
||||||
|
"connectors.authority-and-effects": {
|
||||||
|
"title": "Autorität und Wirkungen eines Connectors verstehen",
|
||||||
|
"summary": "Richtung, technische Reife und konfigurierte Quellenautorität getrennt und sichtbar behandeln.",
|
||||||
|
"body": (
|
||||||
|
"Ein Connector kann Daten beziehen, veröffentlichen oder bidirektional arbeiten und von reiner Erkennung bis zum vollständigen Ersatz reifen. Jede Bindung legt getrennt fest, ob GovOPlaN führend ist, einer externen Autorität folgt, einen Spiegel führt, nach Konfliktregeln synchronisiert, eine Governance-Schicht ergänzt oder nur einen Verweis bewahrt. Schreibende Anbieter müssen Revisionen, Grenzen, Idempotenz, unbekannte Ergebnisse, Nachweise, Abgleich, Korrektur, Ausfallverhalten und Anforderungen an Geheimnisse erläutern."
|
||||||
|
),
|
||||||
|
},
|
||||||
|
"connectors.runtime-preview-contract": {
|
||||||
|
"title": "Connector-Vorschauen und Diagnosen",
|
||||||
|
"summary": "Für externe Transporte ein einheitliches, begrenztes und geschwärztes Testlaufformat verwenden.",
|
||||||
|
"body": (
|
||||||
|
"Connectors verantwortet Endpunkterkennung, Authentifizierungsübergabe, Transportgrenzen, Wiederholungen und Protokollzustand. Fachmodule verantworten Feldzuordnung, Validierung, Abgleich und Datensatzänderungen. Der gemeinsame Core-Laufzeitvertrag meldet geschwärzte Wirkungen und Diagnosen zusammen mit Quellrevisionen, Fingerabdrücken und unveränderlichen Eingabe-Hashes. Tabellarische Vorschauen begrenzen Zeilen, serialisierte Bytes und Laufzeit und melden Abschneidungen strukturiert. Eine Übernahme muss veraltete, abgeschnittene, widersprüchliche oder fehlerhafte Vorschauen ablehnen; Zugangsdaten erscheinen weder in URLs noch in Beispielen."
|
||||||
|
),
|
||||||
|
},
|
||||||
|
"connectors.tabular-sources": {
|
||||||
|
"title": "Gesteuerte tabellarische Quellen",
|
||||||
|
"summary": "Anbieterneutrale Quellenerkennung und begrenzte Lesezugriffe für Dataflow bereitstellen.",
|
||||||
|
"body": (
|
||||||
|
"Connectors verantwortet Quellkonfiguration, Zugriffsprüfung, Schemaerkennung, Fingerabdrücke und begrenzte Lesevorgänge. Dataflow speichert nur undurchsichtige Quellreferenzen und erwartete Fingerabdrücke. Jede Quelle weist ihren Live-, Cache-, Datei- oder statischen Modus, einen strukturierten Zustand und unterstützte Projektion, Filterung, Aggregation, Sortierung und Seitennavigation aus. Unveränderliche JSON- und CSV-Snapshots bleiben verfügbar. Verwaltete CSV- und XLSX-Quellen nutzen optional Files, binden eine exakt autorisierte Version, erzwingen Archiv- und Entpackgrenzen und übernehmen neuere Versionen erst nach ausdrücklicher Aktualisierung. "
|
||||||
|
"XLSX-Lesevorgänge prüfen die tatsächlichen Koordinaten des ausgewählten Arbeitsblatts vor dem Aufbau des Zellrasters: höchstens 500 Spalten und 10.000 Zeilenpositionen nach der Kopfzeile einschließlich leerer Zwischenräume. Unzuverlässige Dimensionsangaben vergrößern weder das Raster noch verbergen sie Zellen; übergroße oder widersprüchliche Koordinaten führen zu einem Validierungsfehler statt zu still abgeschnittenen Daten. "
|
||||||
|
"Der PostgreSQL-Adapter nutzt eine aktive gesteuerte Konfiguration und eine eingegrenzte Core-Zugangsdatenhülle, liest nur einfache Schema- und Tabellenkennungen und blockiert bei Konfigurations-, Zugangsdaten- oder Schemadrift bis zur geprüften Aktualisierung. Zugangsdaten, Endpunkte, Speicherschlüssel und interne Dateiinhalte werden nie über die Quelle offengelegt."
|
||||||
|
),
|
||||||
|
},
|
||||||
|
"connectors.sanctions-snapshots": {
|
||||||
|
"title": "Snapshots von Sanktionsquellen",
|
||||||
|
"summary": "Unveränderliche, per Prüfsumme verifizierbare Sanktionslistennachweise abrufen, ohne Prüfsachverhalte zu übertragen.",
|
||||||
|
"body": (
|
||||||
|
"Connectors stellt eine deterministische synthetische Testquelle und die offizielle konsolidierte XML-Liste des Sicherheitsrats der Vereinten Nationen bereit. Jeder Abruf zeichnet bedingte Transportnachweise, begrenzte Wiederholungen, Zustand, Quellmetadaten, Rohbeleg und SHA-256-Prüfsumme auf. Vor Anbieterzugriffen erwirbt die Aktualisierung eine verteilte Recovery-Sperre; unveränderlicher Snapshot und abschließender Recovery-Prüfpunkt werden anschließend in einer Transaktion gespeichert. Derselbe Anfrageschlüssel liefert dasselbe Ergebnis. Risk Compliance verantwortet Normalisierung, Abgleich, rechtliche Prüfung und Entscheidungen."
|
||||||
|
),
|
||||||
|
},
|
||||||
|
"connectors.rss-atom": {
|
||||||
|
"title": "RSS- und Atom-Feeds",
|
||||||
|
"summary": "Gesteuerte Feed-Snapshots importieren und nach Sichtbarkeit gefilterte Feeds ausgeben.",
|
||||||
|
"body": (
|
||||||
|
"Connectors verantwortet begrenzten, gegen SSRF geschützten RSS-/Atom-Transport und XML-Verarbeitung. Importierte Einträge werden zu unveränderlichen tabellarischen Snapshots in Datasources und enthalten Abruf, Aktualität, ETag, Inhaltsdigest und Quellprovenienz. Ausgaben akzeptieren nur provenienzbelegte Ereignis-, Veröffentlichungs-, Fall- oder Berichtsauswahlen einer verantwortlichen Oberfläche. Die Zielgruppe setzt die Sichtbarkeitsobergrenze: Öffentliche Feeds enthalten nur öffentliche Einträge; mandantenbezogene oder private Feeds benötigen eine eigene Berechtigung. Aufrufende Stellen dürfen keine eigene Sichtbarkeitsliste vorgeben. Portal oder Reporting verantwortet dauerhafte Veröffentlichungsrouten und autorisiert jeden Zugriff auf eingeschränkte Feeds neu. Ein eigenes RSS-Modul ist erst erforderlich, wenn GovOPlaN später eine eigenständige Feed-Reader-Oberfläche benötigt."
|
||||||
|
),
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def localize_documentation_topics(
|
||||||
|
topics: Iterable[DocumentationTopic],
|
||||||
|
) -> tuple[DocumentationTopic, ...]:
|
||||||
|
return _localize_topics(topics, locale="de", translations=_TRANSLATIONS)
|
||||||
@@ -0,0 +1,77 @@
|
|||||||
|
"""German translations for public structured documentation metadata."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
|
||||||
|
GERMAN_STRUCTURED_TRANSLATIONS: dict[str, dict[str, Any]] = {'connectors.data-subject-requests': {'consequence_classes': {'exclude_connector_secrets': 'Gibt '
|
||||||
|
'niemals '
|
||||||
|
'Anmeldeinformationen, '
|
||||||
|
'Endpunkte, '
|
||||||
|
'externe '
|
||||||
|
'Zeilen '
|
||||||
|
'oder '
|
||||||
|
'Beweisnutzlasten '
|
||||||
|
'zurück.',
|
||||||
|
'export_operator_attribution': 'Retouren '
|
||||||
|
'minimierte '
|
||||||
|
'Connector-Aktivität '
|
||||||
|
'für '
|
||||||
|
'das '
|
||||||
|
'genaue '
|
||||||
|
'Konto.',
|
||||||
|
'retain_connector_evidence': 'Bewahrt '
|
||||||
|
'die '
|
||||||
|
'Konfiguration '
|
||||||
|
'und '
|
||||||
|
'die '
|
||||||
|
'Rechenschaftspflicht '
|
||||||
|
'für '
|
||||||
|
'externe '
|
||||||
|
'Operationen '
|
||||||
|
'bei.'}},
|
||||||
|
'connectors.governed-configuration': {'outcome': 'Das aktive Konnektorverhalten ist inspizierbar, '
|
||||||
|
'versionengebunden, testbar und überprüfbar, '
|
||||||
|
'bevor ein anbieterspezifisches Schreiben '
|
||||||
|
'erfolgt.',
|
||||||
|
'prerequisites': ['Eine Konnektordefinition wurde '
|
||||||
|
'installiert oder erstellt.',
|
||||||
|
'Anmeldematerial wird außerhalb der '
|
||||||
|
'Connector-URL gespeichert und durch eine '
|
||||||
|
'genehmigte geheime Kennung '
|
||||||
|
'referenziert.'],
|
||||||
|
'verification': 'Laden Sie die Konfiguration neu, prüfen '
|
||||||
|
'Sie geschützte Pfade und effektiven Hash, '
|
||||||
|
'führen Sie eine Simulation mit einem neuen '
|
||||||
|
'idempotency-Schlüssel aus und lösen Sie '
|
||||||
|
'alle ausstehenden Überprüfungsergebnisse.'},
|
||||||
|
'connectors.mediawiki-bluespice': {'outcome': 'Externes Wissen bleibt identitätsstabil, '
|
||||||
|
'verlustsichtbar, ACL-sicher und migrationsfähig.',
|
||||||
|
'prerequisites': ['Eine aktive verwaltete MediaWiki Action '
|
||||||
|
'API-Konfiguration existiert.',
|
||||||
|
'Namespace-Ziele und Fallback-ACLs wurden '
|
||||||
|
'überprüft.',
|
||||||
|
'Die Such- und Wiki-Module sind optional und '
|
||||||
|
'bleiben fähigkeitsgetrennt.'],
|
||||||
|
'verification': 'Entdecken Sie das Profil neu, führen Sie ein '
|
||||||
|
'Delta mit Schlüsseln aus, inspizieren Sie '
|
||||||
|
'Gesundheit und Diagnose, überprüfen Sie einen '
|
||||||
|
'erlaubten und verweigerten Suchprinzipal und '
|
||||||
|
'führen Sie einen Migrations-Dry-Run aus, '
|
||||||
|
'bevor Sie zielseitig arbeiten.'},
|
||||||
|
'connectors.znuny-otrs': {'outcome': 'Externe Tickets bleiben identitätsstabil, verlustsichtbar, '
|
||||||
|
'ACL-sicher, wiederherstellbar und semantisch getrennt von '
|
||||||
|
'GovOPlaN-Domäneneinträgen.',
|
||||||
|
'prerequisites': ['Eine aktiv gesteuerte Znuny/OTRS GenericInterface '
|
||||||
|
'REST-Konfiguration existiert.',
|
||||||
|
'Bereitstellungsdefinierte Routen, '
|
||||||
|
'Warteschlangenpartitionen, Autorität und '
|
||||||
|
'Fallback-ACLs wurden überprüft.',
|
||||||
|
'Tickets, Helpdesk, Cases und Search bleiben '
|
||||||
|
'optionale funktionsgetrennte Verbraucher.'],
|
||||||
|
'verification': 'Entdecke das Profil neu, beende einen Keyed-Full-Run, '
|
||||||
|
'führe ein Keyed-Delta aus, inspiziere die '
|
||||||
|
'Mapping-Diagnose, überprüfe einen erlaubten und '
|
||||||
|
'verweigerten Suchprinzipal und versöhne jedes '
|
||||||
|
'ergebnisunbekannte Update vor dem erneuten Versuch.'}}
|
||||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,303 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from datetime import datetime
|
||||||
|
from typing import Any, Literal
|
||||||
|
|
||||||
|
from pydantic import BaseModel, ConfigDict, Field, field_validator, model_validator
|
||||||
|
|
||||||
|
|
||||||
|
KnowledgeMaturity = Literal[
|
||||||
|
"discover",
|
||||||
|
"link",
|
||||||
|
"search",
|
||||||
|
"read",
|
||||||
|
"publish",
|
||||||
|
"synchronize",
|
||||||
|
"migrate",
|
||||||
|
]
|
||||||
|
KnowledgeVisibility = Literal["tenant", "restricted"]
|
||||||
|
|
||||||
|
|
||||||
|
class KnowledgeNamespaceMapping(BaseModel):
|
||||||
|
model_config = ConfigDict(extra="forbid")
|
||||||
|
|
||||||
|
source_namespace_id: int
|
||||||
|
source_name: str = Field(default="", max_length=200)
|
||||||
|
target_space_ref: str = Field(min_length=1, max_length=255)
|
||||||
|
target_path_prefix: str = Field(default="", max_length=500)
|
||||||
|
include: bool = True
|
||||||
|
visibility: KnowledgeVisibility | None = None
|
||||||
|
acl_tokens: list[str] = Field(default_factory=list, max_length=500)
|
||||||
|
|
||||||
|
@field_validator("acl_tokens")
|
||||||
|
@classmethod
|
||||||
|
def normalize_acl_tokens(cls, values: list[str]) -> list[str]:
|
||||||
|
return _normalized_tokens(values)
|
||||||
|
|
||||||
|
@model_validator(mode="after")
|
||||||
|
def restricted_mapping_requires_acl(self) -> "KnowledgeNamespaceMapping":
|
||||||
|
if self.visibility == "restricted" and not self.acl_tokens:
|
||||||
|
raise ValueError("Restricted namespace mappings require ACL tokens")
|
||||||
|
return self
|
||||||
|
|
||||||
|
|
||||||
|
class KnowledgeProfileCreateRequest(BaseModel):
|
||||||
|
model_config = ConfigDict(extra="forbid")
|
||||||
|
|
||||||
|
configuration_id: str = Field(min_length=1, max_length=36)
|
||||||
|
desired_maturity: KnowledgeMaturity = "read"
|
||||||
|
source_authority_mode: Literal[
|
||||||
|
"external_authoritative",
|
||||||
|
"external_mirror",
|
||||||
|
"governed_sync",
|
||||||
|
"linked_reference",
|
||||||
|
] = "external_mirror"
|
||||||
|
default_visibility: KnowledgeVisibility = "restricted"
|
||||||
|
default_acl_tokens: list[str] = Field(default_factory=list, max_length=500)
|
||||||
|
namespace_mappings: list[KnowledgeNamespaceMapping] = Field(
|
||||||
|
min_length=1, max_length=500
|
||||||
|
)
|
||||||
|
|
||||||
|
@field_validator("default_acl_tokens")
|
||||||
|
@classmethod
|
||||||
|
def normalize_acl_tokens(cls, values: list[str]) -> list[str]:
|
||||||
|
return _normalized_tokens(values)
|
||||||
|
|
||||||
|
@model_validator(mode="after")
|
||||||
|
def restricted_profile_requires_acl(self) -> "KnowledgeProfileCreateRequest":
|
||||||
|
if self.default_visibility == "restricted" and not self.default_acl_tokens:
|
||||||
|
raise ValueError("Restricted knowledge profiles require default ACL tokens")
|
||||||
|
return self
|
||||||
|
|
||||||
|
|
||||||
|
class KnowledgeProfileUpdateRequest(BaseModel):
|
||||||
|
model_config = ConfigDict(extra="forbid")
|
||||||
|
|
||||||
|
expected_resource_revision: int = Field(ge=1)
|
||||||
|
status: Literal["active", "paused"] | None = None
|
||||||
|
desired_maturity: KnowledgeMaturity | None = None
|
||||||
|
source_authority_mode: Literal[
|
||||||
|
"external_authoritative",
|
||||||
|
"external_mirror",
|
||||||
|
"governed_sync",
|
||||||
|
"linked_reference",
|
||||||
|
] | None = None
|
||||||
|
default_visibility: KnowledgeVisibility | None = None
|
||||||
|
default_acl_tokens: list[str] | None = Field(default=None, max_length=500)
|
||||||
|
namespace_mappings: list[KnowledgeNamespaceMapping] | None = Field(
|
||||||
|
default=None, min_length=1, max_length=500
|
||||||
|
)
|
||||||
|
|
||||||
|
@field_validator("default_acl_tokens")
|
||||||
|
@classmethod
|
||||||
|
def normalize_acl_tokens(cls, values: list[str] | None) -> list[str] | None:
|
||||||
|
return _normalized_tokens(values) if values is not None else None
|
||||||
|
|
||||||
|
|
||||||
|
class KnowledgeProfileItem(BaseModel):
|
||||||
|
id: str
|
||||||
|
tenant_id: str
|
||||||
|
configuration_id: str
|
||||||
|
status: str
|
||||||
|
product: str
|
||||||
|
product_version: str | None = None
|
||||||
|
desired_maturity: str
|
||||||
|
discovered_maturity: str
|
||||||
|
source_authority_mode: str
|
||||||
|
default_visibility: str
|
||||||
|
default_acl_tokens: list[str]
|
||||||
|
namespace_mappings: list[dict[str, Any]]
|
||||||
|
capabilities: list[str]
|
||||||
|
discovery_revision: str | None = None
|
||||||
|
health_status: str
|
||||||
|
health_details: dict[str, Any]
|
||||||
|
discovered_at: datetime | None = None
|
||||||
|
last_sync_cursor: str | None = None
|
||||||
|
last_high_watermark: str | None = None
|
||||||
|
resource_revision: int
|
||||||
|
credential_reference_present: bool = False
|
||||||
|
endpoint_configured: bool = False
|
||||||
|
created_at: datetime
|
||||||
|
updated_at: datetime
|
||||||
|
|
||||||
|
|
||||||
|
class KnowledgeProfileListResponse(BaseModel):
|
||||||
|
items: list[KnowledgeProfileItem]
|
||||||
|
|
||||||
|
|
||||||
|
class KnowledgeDiagnostic(BaseModel):
|
||||||
|
severity: Literal["info", "warning", "error"]
|
||||||
|
code: str
|
||||||
|
message: str
|
||||||
|
object_ref: str | None = None
|
||||||
|
field: str | None = None
|
||||||
|
retryable: bool = False
|
||||||
|
details: dict[str, Any] = Field(default_factory=dict)
|
||||||
|
|
||||||
|
|
||||||
|
class KnowledgeDiscoveryResponse(BaseModel):
|
||||||
|
profile: KnowledgeProfileItem
|
||||||
|
product: str
|
||||||
|
product_version: str | None = None
|
||||||
|
api_version: str | None = None
|
||||||
|
capabilities: list[str]
|
||||||
|
namespaces: list[dict[str, Any]]
|
||||||
|
extensions: list[dict[str, Any]]
|
||||||
|
maturity: str
|
||||||
|
health_status: str
|
||||||
|
diagnostics: list[KnowledgeDiagnostic]
|
||||||
|
revision: str
|
||||||
|
|
||||||
|
|
||||||
|
class KnowledgeSyncRequest(BaseModel):
|
||||||
|
model_config = ConfigDict(extra="forbid")
|
||||||
|
|
||||||
|
idempotency_key: str = Field(min_length=1, max_length=255)
|
||||||
|
cursor: str | None = Field(default=None, max_length=500)
|
||||||
|
force_full: bool = False
|
||||||
|
limit: int = Field(default=100, ge=1, le=500)
|
||||||
|
|
||||||
|
|
||||||
|
class KnowledgeSyncRunItem(BaseModel):
|
||||||
|
id: str
|
||||||
|
tenant_id: str
|
||||||
|
profile_id: str
|
||||||
|
mode: str
|
||||||
|
idempotency_key: str
|
||||||
|
status: str
|
||||||
|
cursor_before: str | None = None
|
||||||
|
cursor_after: str | None = None
|
||||||
|
high_watermark: str | None = None
|
||||||
|
counts: dict[str, Any]
|
||||||
|
effects: list[dict[str, Any]]
|
||||||
|
diagnostics: list[KnowledgeDiagnostic]
|
||||||
|
provenance: dict[str, Any]
|
||||||
|
started_at: datetime
|
||||||
|
finished_at: datetime | None = None
|
||||||
|
created_at: datetime
|
||||||
|
|
||||||
|
|
||||||
|
class KnowledgeSyncRunListResponse(BaseModel):
|
||||||
|
items: list[KnowledgeSyncRunItem]
|
||||||
|
|
||||||
|
|
||||||
|
class KnowledgeExternalReferenceResponse(BaseModel):
|
||||||
|
system: str
|
||||||
|
object_type: str
|
||||||
|
object_id: str
|
||||||
|
maturity: str
|
||||||
|
authority_mode: str
|
||||||
|
connector_id: str | None = None
|
||||||
|
canonical_url: str | None = None
|
||||||
|
version: str | None = None
|
||||||
|
etag: str | None = None
|
||||||
|
observed_at: str | None = None
|
||||||
|
metadata: dict[str, Any]
|
||||||
|
|
||||||
|
|
||||||
|
class KnowledgeObjectItem(BaseModel):
|
||||||
|
id: str
|
||||||
|
profile_id: str
|
||||||
|
object_type: str
|
||||||
|
external_id: str
|
||||||
|
external_page_id: str | None = None
|
||||||
|
external_revision_id: str | None = None
|
||||||
|
namespace_id: int | None = None
|
||||||
|
title: str
|
||||||
|
canonical_url: str | None = None
|
||||||
|
status: str
|
||||||
|
redirect_target_external_id: str | None = None
|
||||||
|
source_revision: str
|
||||||
|
visibility: str
|
||||||
|
acl_tokens: list[str]
|
||||||
|
mapped_data: dict[str, Any]
|
||||||
|
external_reference: KnowledgeExternalReferenceResponse
|
||||||
|
source_updated_at: datetime | None = None
|
||||||
|
observed_at: datetime
|
||||||
|
resource_revision: int
|
||||||
|
|
||||||
|
|
||||||
|
class KnowledgeObjectListResponse(BaseModel):
|
||||||
|
items: list[KnowledgeObjectItem]
|
||||||
|
next_cursor: str | None = None
|
||||||
|
|
||||||
|
|
||||||
|
class KnowledgeMigrationTargetState(BaseModel):
|
||||||
|
model_config = ConfigDict(extra="forbid")
|
||||||
|
|
||||||
|
path: str = Field(min_length=1, max_length=500)
|
||||||
|
source_external_id: str | None = Field(default=None, max_length=255)
|
||||||
|
attachment_names: list[str] = Field(default_factory=list, max_length=500)
|
||||||
|
|
||||||
|
|
||||||
|
class KnowledgeMigrationDryRunRequest(BaseModel):
|
||||||
|
model_config = ConfigDict(extra="forbid")
|
||||||
|
|
||||||
|
idempotency_key: str = Field(min_length=1, max_length=255)
|
||||||
|
target_space_ref: str = Field(min_length=1, max_length=255)
|
||||||
|
max_items: int = Field(default=100, ge=1, le=500)
|
||||||
|
supported_macros: list[str] = Field(default_factory=list, max_length=200)
|
||||||
|
existing_targets: list[KnowledgeMigrationTargetState] = Field(
|
||||||
|
default_factory=list, max_length=5_000
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
class KnowledgeMigrationDryRunResponse(BaseModel):
|
||||||
|
run: KnowledgeSyncRunItem
|
||||||
|
target_space_ref: str
|
||||||
|
source_revision: str
|
||||||
|
source_fingerprint: str
|
||||||
|
summary: dict[str, int]
|
||||||
|
effects: list[dict[str, Any]]
|
||||||
|
diagnostics: list[KnowledgeDiagnostic]
|
||||||
|
truncated: bool
|
||||||
|
can_apply: bool
|
||||||
|
|
||||||
|
|
||||||
|
class KnowledgePublishRequest(BaseModel):
|
||||||
|
model_config = ConfigDict(extra="forbid")
|
||||||
|
|
||||||
|
idempotency_key: str = Field(min_length=1, max_length=255)
|
||||||
|
title: str = Field(min_length=1, max_length=500)
|
||||||
|
body: str = Field(max_length=200_000)
|
||||||
|
summary: str = Field(default="", max_length=500)
|
||||||
|
expected_external_revision: str | None = Field(default=None, max_length=255)
|
||||||
|
minor: bool = False
|
||||||
|
|
||||||
|
|
||||||
|
class KnowledgePublishResponse(BaseModel):
|
||||||
|
run: KnowledgeSyncRunItem
|
||||||
|
external_reference: KnowledgeExternalReferenceResponse
|
||||||
|
accepted: bool
|
||||||
|
outcome_unknown: bool = False
|
||||||
|
|
||||||
|
|
||||||
|
def _normalized_tokens(values: list[str]) -> list[str]:
|
||||||
|
normalized = [str(item).strip() for item in values]
|
||||||
|
if any(not item or len(item) > 500 for item in normalized):
|
||||||
|
raise ValueError("ACL tokens must contain 1 to 500 characters")
|
||||||
|
if len(normalized) != len(set(normalized)):
|
||||||
|
raise ValueError("ACL tokens must be unique")
|
||||||
|
return normalized
|
||||||
|
|
||||||
|
|
||||||
|
__all__ = [
|
||||||
|
"KnowledgeDiagnostic",
|
||||||
|
"KnowledgeDiscoveryResponse",
|
||||||
|
"KnowledgeExternalReferenceResponse",
|
||||||
|
"KnowledgeMigrationDryRunRequest",
|
||||||
|
"KnowledgeMigrationDryRunResponse",
|
||||||
|
"KnowledgeMigrationTargetState",
|
||||||
|
"KnowledgeNamespaceMapping",
|
||||||
|
"KnowledgeObjectItem",
|
||||||
|
"KnowledgeObjectListResponse",
|
||||||
|
"KnowledgeProfileCreateRequest",
|
||||||
|
"KnowledgeProfileItem",
|
||||||
|
"KnowledgeProfileListResponse",
|
||||||
|
"KnowledgeProfileUpdateRequest",
|
||||||
|
"KnowledgePublishRequest",
|
||||||
|
"KnowledgePublishResponse",
|
||||||
|
"KnowledgeSyncRequest",
|
||||||
|
"KnowledgeSyncRunItem",
|
||||||
|
"KnowledgeSyncRunListResponse",
|
||||||
|
]
|
||||||
@@ -0,0 +1,247 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from govoplan_connectors.backend.search_principal import principal_acl_tokens as _principal_acl_tokens
|
||||||
|
|
||||||
|
from collections.abc import Mapping, Sequence
|
||||||
|
from urllib.parse import quote
|
||||||
|
|
||||||
|
from sqlalchemy import func, select
|
||||||
|
from sqlalchemy.orm import Session
|
||||||
|
|
||||||
|
from govoplan_core.core.external_references import ExternalObjectReference
|
||||||
|
from govoplan_core.core.modules import ModuleContext
|
||||||
|
from govoplan_core.core.search import (
|
||||||
|
SearchAuthorizationRequest,
|
||||||
|
SearchBackfillPage,
|
||||||
|
SearchBackfillRequest,
|
||||||
|
SearchDocument,
|
||||||
|
SearchResourceType,
|
||||||
|
)
|
||||||
|
from govoplan_connectors.backend.db.models import (
|
||||||
|
ConnectorKnowledgeObject,
|
||||||
|
ConnectorKnowledgeProfile,
|
||||||
|
)
|
||||||
|
from govoplan_connectors.backend.knowledge_connector import (
|
||||||
|
KNOWLEDGE_PROVIDER_ID,
|
||||||
|
KNOWLEDGE_READ_SCOPE,
|
||||||
|
KNOWLEDGE_RESOURCE_TYPE,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
class ExternalKnowledgeSearchSource:
|
||||||
|
def resource_types(self) -> Sequence[SearchResourceType]:
|
||||||
|
return (
|
||||||
|
SearchResourceType(
|
||||||
|
provider_id=KNOWLEDGE_PROVIDER_ID,
|
||||||
|
module_id="connectors",
|
||||||
|
resource_type=KNOWLEDGE_RESOURCE_TYPE,
|
||||||
|
label="External knowledge pages",
|
||||||
|
requires_authorization_recheck=True,
|
||||||
|
),
|
||||||
|
)
|
||||||
|
|
||||||
|
def backfill(
|
||||||
|
self, session: object, *, request: SearchBackfillRequest
|
||||||
|
) -> SearchBackfillPage:
|
||||||
|
_assert_source(request.provider_id, request.resource_type)
|
||||||
|
db = _session(session)
|
||||||
|
query = (
|
||||||
|
select(ConnectorKnowledgeObject, ConnectorKnowledgeProfile)
|
||||||
|
.join(
|
||||||
|
ConnectorKnowledgeProfile,
|
||||||
|
ConnectorKnowledgeProfile.id == ConnectorKnowledgeObject.profile_id,
|
||||||
|
)
|
||||||
|
.where(
|
||||||
|
ConnectorKnowledgeObject.tenant_id == request.tenant_id,
|
||||||
|
ConnectorKnowledgeObject.object_type == "page",
|
||||||
|
ConnectorKnowledgeObject.status != "deleted",
|
||||||
|
ConnectorKnowledgeProfile.tenant_id == request.tenant_id,
|
||||||
|
ConnectorKnowledgeProfile.status == "active",
|
||||||
|
)
|
||||||
|
)
|
||||||
|
if request.cursor:
|
||||||
|
query = query.where(ConnectorKnowledgeObject.id > request.cursor)
|
||||||
|
rows = tuple(
|
||||||
|
db.execute(
|
||||||
|
query.order_by(ConnectorKnowledgeObject.id.asc()).limit(
|
||||||
|
request.limit + 1
|
||||||
|
)
|
||||||
|
)
|
||||||
|
)
|
||||||
|
has_more = len(rows) > request.limit
|
||||||
|
selected = rows[: request.limit]
|
||||||
|
watermark = db.scalar(
|
||||||
|
select(func.max(ConnectorKnowledgeObject.updated_at))
|
||||||
|
.join(
|
||||||
|
ConnectorKnowledgeProfile,
|
||||||
|
ConnectorKnowledgeProfile.id == ConnectorKnowledgeObject.profile_id,
|
||||||
|
)
|
||||||
|
.where(
|
||||||
|
ConnectorKnowledgeObject.tenant_id == request.tenant_id,
|
||||||
|
ConnectorKnowledgeObject.object_type == "page",
|
||||||
|
ConnectorKnowledgeObject.status != "deleted",
|
||||||
|
ConnectorKnowledgeProfile.tenant_id == request.tenant_id,
|
||||||
|
ConnectorKnowledgeProfile.status == "active",
|
||||||
|
)
|
||||||
|
)
|
||||||
|
return SearchBackfillPage(
|
||||||
|
documents=tuple(search_document(db, row, profile) for row, profile in selected),
|
||||||
|
next_cursor=selected[-1][0].id if has_more and selected else None,
|
||||||
|
complete=not has_more,
|
||||||
|
high_watermark=watermark.isoformat() if watermark else None,
|
||||||
|
)
|
||||||
|
|
||||||
|
def authorize(
|
||||||
|
self,
|
||||||
|
session: object,
|
||||||
|
principal: object,
|
||||||
|
*,
|
||||||
|
requests: Sequence[SearchAuthorizationRequest],
|
||||||
|
) -> Mapping[str, bool]:
|
||||||
|
db = _session(session)
|
||||||
|
tenant_id = str(getattr(principal, "tenant_id", "") or "")
|
||||||
|
can_read = _has_scope(principal, KNOWLEDGE_READ_SCOPE)
|
||||||
|
tokens = set(_principal_acl_tokens(principal))
|
||||||
|
decisions = {item.reference.key: False for item in requests}
|
||||||
|
if not tenant_id or not can_read:
|
||||||
|
return decisions
|
||||||
|
for request in requests:
|
||||||
|
reference = request.reference
|
||||||
|
if (
|
||||||
|
reference.tenant_id != tenant_id
|
||||||
|
or reference.module_id != "connectors"
|
||||||
|
or reference.resource_type != KNOWLEDGE_RESOURCE_TYPE
|
||||||
|
):
|
||||||
|
continue
|
||||||
|
row = db.execute(
|
||||||
|
select(ConnectorKnowledgeObject, ConnectorKnowledgeProfile)
|
||||||
|
.join(
|
||||||
|
ConnectorKnowledgeProfile,
|
||||||
|
ConnectorKnowledgeProfile.id
|
||||||
|
== ConnectorKnowledgeObject.profile_id,
|
||||||
|
)
|
||||||
|
.where(
|
||||||
|
ConnectorKnowledgeObject.tenant_id == tenant_id,
|
||||||
|
ConnectorKnowledgeObject.id == reference.resource_id,
|
||||||
|
ConnectorKnowledgeObject.object_type == "page",
|
||||||
|
ConnectorKnowledgeObject.status != "deleted",
|
||||||
|
ConnectorKnowledgeProfile.tenant_id == tenant_id,
|
||||||
|
ConnectorKnowledgeProfile.status == "active",
|
||||||
|
)
|
||||||
|
).first()
|
||||||
|
if row is None:
|
||||||
|
continue
|
||||||
|
page, _profile = row
|
||||||
|
decisions[reference.key] = page.visibility == "tenant" or bool(
|
||||||
|
tokens.intersection(str(value) for value in page.acl_tokens or ())
|
||||||
|
)
|
||||||
|
return decisions
|
||||||
|
|
||||||
|
|
||||||
|
def create_external_knowledge_search_source(
|
||||||
|
_context: ModuleContext,
|
||||||
|
) -> ExternalKnowledgeSearchSource:
|
||||||
|
return ExternalKnowledgeSearchSource()
|
||||||
|
|
||||||
|
|
||||||
|
def search_document(
|
||||||
|
session: Session,
|
||||||
|
row: ConnectorKnowledgeObject,
|
||||||
|
profile: ConnectorKnowledgeProfile | None = None,
|
||||||
|
) -> SearchDocument:
|
||||||
|
if profile is None:
|
||||||
|
profile = session.scalar(
|
||||||
|
select(ConnectorKnowledgeProfile).where(
|
||||||
|
ConnectorKnowledgeProfile.tenant_id == row.tenant_id,
|
||||||
|
ConnectorKnowledgeProfile.id == row.profile_id,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
if profile is None:
|
||||||
|
raise ValueError("External knowledge profile is unavailable.")
|
||||||
|
data = dict(row.mapped_data or {})
|
||||||
|
categories = tuple(str(value)[:200] for value in data.get("categories") or ())
|
||||||
|
links = tuple(
|
||||||
|
str(value.get("title") or value.get("external_id") or "")[:200]
|
||||||
|
for value in data.get("links") or ()
|
||||||
|
if isinstance(value, Mapping)
|
||||||
|
)
|
||||||
|
external_reference = ExternalObjectReference(
|
||||||
|
system=profile.product if profile.product != "unknown" else "mediawiki",
|
||||||
|
object_type=row.object_type,
|
||||||
|
object_id=row.external_id,
|
||||||
|
maturity=profile.discovered_maturity,
|
||||||
|
authority_mode=profile.source_authority_mode,
|
||||||
|
connector_id=profile.id,
|
||||||
|
canonical_url=row.canonical_url,
|
||||||
|
version=row.source_revision,
|
||||||
|
etag=row.content_hash,
|
||||||
|
observed_at=row.observed_at,
|
||||||
|
metadata={
|
||||||
|
"title": row.title,
|
||||||
|
"namespace_id": row.namespace_id,
|
||||||
|
"external_revision_id": row.external_revision_id,
|
||||||
|
},
|
||||||
|
)
|
||||||
|
return SearchDocument(
|
||||||
|
tenant_id=row.tenant_id,
|
||||||
|
module_id="connectors",
|
||||||
|
provider_id=KNOWLEDGE_PROVIDER_ID,
|
||||||
|
resource_type=KNOWLEDGE_RESOURCE_TYPE,
|
||||||
|
resource_id=row.id,
|
||||||
|
title=row.title,
|
||||||
|
url=(
|
||||||
|
"/connectors/knowledge?profileId="
|
||||||
|
f"{quote(row.profile_id, safe='')}&objectId={quote(row.id, safe='')}"
|
||||||
|
),
|
||||||
|
summary=str(data.get("summary") or data.get("body") or "")[:4_000] or None,
|
||||||
|
body=str(data.get("body") or "")[:200_000] or None,
|
||||||
|
keywords=tuple(dict.fromkeys((*categories, *links)))[:100],
|
||||||
|
visibility=row.visibility,
|
||||||
|
acl_tokens=(
|
||||||
|
tuple(str(value) for value in row.acl_tokens or ())
|
||||||
|
if row.visibility == "restricted"
|
||||||
|
else ()
|
||||||
|
),
|
||||||
|
external_reference=external_reference,
|
||||||
|
metadata={
|
||||||
|
"profile_id": row.profile_id,
|
||||||
|
"external_page_id": row.external_page_id,
|
||||||
|
"namespace_id": row.namespace_id,
|
||||||
|
"target_space_ref": data.get("target_space_ref"),
|
||||||
|
"target_path": data.get("target_path"),
|
||||||
|
"status": row.status,
|
||||||
|
"redirect_target_external_id": row.redirect_target_external_id,
|
||||||
|
},
|
||||||
|
source_revision=row.source_revision,
|
||||||
|
change_cursor=row.change_cursor,
|
||||||
|
source_updated_at=row.source_updated_at or row.observed_at,
|
||||||
|
requires_authorization_recheck=True,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _has_scope(principal: object, required: str) -> bool:
|
||||||
|
check = getattr(principal, "has", None)
|
||||||
|
if callable(check):
|
||||||
|
return bool(check(required))
|
||||||
|
return required in getattr(principal, "scopes", ())
|
||||||
|
|
||||||
|
|
||||||
|
def _assert_source(provider_id: str, resource_type: str) -> None:
|
||||||
|
if (
|
||||||
|
provider_id != KNOWLEDGE_PROVIDER_ID
|
||||||
|
or resource_type != KNOWLEDGE_RESOURCE_TYPE
|
||||||
|
):
|
||||||
|
raise ValueError("Unsupported external knowledge search source.")
|
||||||
|
|
||||||
|
|
||||||
|
def _session(value: object) -> Session:
|
||||||
|
if not isinstance(value, Session):
|
||||||
|
raise TypeError("External knowledge Search requires a SQLAlchemy session.")
|
||||||
|
return value
|
||||||
|
|
||||||
|
|
||||||
|
__all__ = [
|
||||||
|
"ExternalKnowledgeSearchSource",
|
||||||
|
"create_external_knowledge_search_source",
|
||||||
|
"search_document",
|
||||||
|
]
|
||||||
@@ -1,5 +1,8 @@
|
|||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from govoplan_core.core.modules import with_documentation_structured_translations
|
||||||
|
from govoplan_connectors.backend.german_structured_documentation import GERMAN_STRUCTURED_TRANSLATIONS
|
||||||
|
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
|
|
||||||
from govoplan_core.core.access import (
|
from govoplan_core.core.access import (
|
||||||
@@ -14,6 +17,7 @@ from govoplan_core.core.datasources import CAPABILITY_DATASOURCE_ORIGINS
|
|||||||
from govoplan_core.core.feeds import CAPABILITY_CONNECTORS_FEEDS
|
from govoplan_core.core.feeds import CAPABILITY_CONNECTORS_FEEDS
|
||||||
from govoplan_core.core.modules import (
|
from govoplan_core.core.modules import (
|
||||||
CapabilityDocumentation,
|
CapabilityDocumentation,
|
||||||
|
DocumentationCondition,
|
||||||
DocumentationTopic,
|
DocumentationTopic,
|
||||||
FrontendModule,
|
FrontendModule,
|
||||||
MigrationSpec,
|
MigrationSpec,
|
||||||
@@ -41,16 +45,50 @@ from govoplan_core.core.files import CAPABILITY_FILES_TABULAR_CONTENT
|
|||||||
from govoplan_core.core.sanctions import (
|
from govoplan_core.core.sanctions import (
|
||||||
CAPABILITY_CONNECTORS_SANCTIONS_SNAPSHOTS,
|
CAPABILITY_CONNECTORS_SANCTIONS_SNAPSHOTS,
|
||||||
)
|
)
|
||||||
|
from govoplan_core.core.search import SearchSourceProviderRegistration
|
||||||
from govoplan_core.db.base import Base
|
from govoplan_core.db.base import Base
|
||||||
from govoplan_connectors.backend.db.models import (
|
from govoplan_connectors.backend.db.models import (
|
||||||
ConnectorConfiguration,
|
ConnectorConfiguration,
|
||||||
ConnectorDefinition,
|
ConnectorDefinition,
|
||||||
ConnectorDefinitionRevision,
|
ConnectorDefinitionRevision,
|
||||||
|
ConnectorKnowledgeObject,
|
||||||
|
ConnectorKnowledgeProfile,
|
||||||
|
ConnectorKnowledgeSyncRun,
|
||||||
|
ConnectorServiceDeskObject,
|
||||||
|
ConnectorServiceDeskProfile,
|
||||||
|
ConnectorServiceDeskSyncRun,
|
||||||
ConnectorSanctionsAcquisitionRun,
|
ConnectorSanctionsAcquisitionRun,
|
||||||
ConnectorSanctionsSnapshot,
|
ConnectorSanctionsSnapshot,
|
||||||
ConnectorSimulationRun,
|
ConnectorSimulationRun,
|
||||||
ConnectorTabularSource,
|
ConnectorTabularSource,
|
||||||
)
|
)
|
||||||
|
from govoplan_connectors.backend.knowledge_connector import (
|
||||||
|
KNOWLEDGE_ADMIN_SCOPE,
|
||||||
|
KNOWLEDGE_CAPABILITY,
|
||||||
|
KNOWLEDGE_INTERFACE_VERSION,
|
||||||
|
KNOWLEDGE_MIGRATE_SCOPE,
|
||||||
|
KNOWLEDGE_PROVIDER_ID,
|
||||||
|
KNOWLEDGE_PUBLISH_SCOPE,
|
||||||
|
KNOWLEDGE_READ_SCOPE,
|
||||||
|
KNOWLEDGE_SYNC_SCOPE,
|
||||||
|
ExternalKnowledgeCapability,
|
||||||
|
)
|
||||||
|
from govoplan_connectors.backend.knowledge_search import (
|
||||||
|
create_external_knowledge_search_source,
|
||||||
|
)
|
||||||
|
from govoplan_connectors.backend.service_desk_connector import (
|
||||||
|
SERVICE_DESK_ADMIN_SCOPE,
|
||||||
|
SERVICE_DESK_CAPABILITY,
|
||||||
|
SERVICE_DESK_INTERFACE_VERSION,
|
||||||
|
SERVICE_DESK_PROVIDER_ID,
|
||||||
|
SERVICE_DESK_READ_SCOPE,
|
||||||
|
SERVICE_DESK_SYNC_SCOPE,
|
||||||
|
SERVICE_DESK_UPDATE_SCOPE,
|
||||||
|
ExternalServiceDeskCapability,
|
||||||
|
)
|
||||||
|
from govoplan_connectors.backend.service_desk_search import (
|
||||||
|
create_external_service_desk_search_source,
|
||||||
|
)
|
||||||
from govoplan_connectors.backend.dsar_provider import (
|
from govoplan_connectors.backend.dsar_provider import (
|
||||||
CONNECTORS_DSAR_CAPABILITY,
|
CONNECTORS_DSAR_CAPABILITY,
|
||||||
ConnectorsDsarProvider,
|
ConnectorsDsarProvider,
|
||||||
@@ -77,13 +115,18 @@ from govoplan_connectors.backend.feeds import (
|
|||||||
from govoplan_connectors.backend.provider_state import (
|
from govoplan_connectors.backend.provider_state import (
|
||||||
SANCTIONS_PROVIDER_ID,
|
SANCTIONS_PROVIDER_ID,
|
||||||
TABULAR_PROVIDER_ID,
|
TABULAR_PROVIDER_ID,
|
||||||
|
knowledge_provider_states,
|
||||||
sanctions_provider_states,
|
sanctions_provider_states,
|
||||||
|
service_desk_provider_states,
|
||||||
tabular_provider_states,
|
tabular_provider_states,
|
||||||
)
|
)
|
||||||
|
from govoplan_connectors.backend.german_documentation import (
|
||||||
|
localize_documentation_topics,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
MODULE_ID = "connectors"
|
MODULE_ID = "connectors"
|
||||||
MODULE_VERSION = "0.1.20"
|
MODULE_VERSION = "0.1.27"
|
||||||
TABULAR_SOURCE_INTERFACE_VERSION = "0.1.0"
|
TABULAR_SOURCE_INTERFACE_VERSION = "0.1.0"
|
||||||
DATASOURCE_ORIGIN_INTERFACE_VERSION = "0.1.0"
|
DATASOURCE_ORIGIN_INTERFACE_VERSION = "0.1.0"
|
||||||
SANCTIONS_SNAPSHOT_INTERFACE_VERSION = "1.0.0"
|
SANCTIONS_SNAPSHOT_INTERFACE_VERSION = "1.0.0"
|
||||||
@@ -125,6 +168,16 @@ ARCHITECTURE = ModuleArchitectureDeclaration(
|
|||||||
reference="tests/test_governed_runtime.py",
|
reference="tests/test_governed_runtime.py",
|
||||||
summary="Exercises immutable definition revisions, protected local overrides, idempotent simulations, and explicit ambiguity review.",
|
summary="Exercises immutable definition revisions, protected local overrides, idempotent simulations, and explicit ambiguity review.",
|
||||||
),
|
),
|
||||||
|
ModuleMaturityEvidence(
|
||||||
|
kind="test",
|
||||||
|
reference="tests/test_mediawiki_connector.py",
|
||||||
|
summary="Exercises deterministic MediaWiki/BlueSpice discovery, stable mapping, bounded deltas, ACL-safe Search, migration loss diagnostics, and publication recovery states.",
|
||||||
|
),
|
||||||
|
ModuleMaturityEvidence(
|
||||||
|
kind="test",
|
||||||
|
reference="tests/test_service_desk_connector.py",
|
||||||
|
summary="Exercises Znuny/OTRS profile policy, stable ticket mapping, bounded synchronization, Search authorization, governed updates, and unknown-outcome evidence.",
|
||||||
|
),
|
||||||
ModuleMaturityEvidence(
|
ModuleMaturityEvidence(
|
||||||
kind="documentation",
|
kind="documentation",
|
||||||
reference="docs/CONNECTOR_SOURCE_LIFECYCLE.md",
|
reference="docs/CONNECTOR_SOURCE_LIFECYCLE.md",
|
||||||
@@ -134,11 +187,14 @@ ARCHITECTURE = ModuleArchitectureDeclaration(
|
|||||||
known_limits=(
|
known_limits=(
|
||||||
"Tabular origins support immutable snapshots, exact managed CSV/XLSX versions, and read-only PostgreSQL tables; arbitrary REST and other database adapters remain future providers.",
|
"Tabular origins support immutable snapshots, exact managed CSV/XLSX versions, and read-only PostgreSQL tables; arbitrary REST and other database adapters remain future providers.",
|
||||||
"Feed publication renders a governed document but does not yet push it to an external publishing endpoint.",
|
"Feed publication renders a governed document but does not yet push it to an external publishing endpoint.",
|
||||||
"The generic governed runtime simulates deterministic mapping and validation; provider-specific live writes remain owned by explicit connector adapters.",
|
"MediaWiki/BlueSpice publication and governed-sync Znuny/OTRS ticket updates are explicit revision-checked write paths; generic simulations and other providers do not imply a live write capability.",
|
||||||
|
"Migration into native Wiki is preview-only; a target-side write worker and Wiki-owned mutation contract remain future work.",
|
||||||
|
"The Znuny/OTRS GenericInterface route map is deployment-defined; queues must be partitioned below the 10000-ticket identity bound and attachment bytes remain provider-side.",
|
||||||
),
|
),
|
||||||
supported_authority_modes=(
|
supported_authority_modes=(
|
||||||
"external_authoritative",
|
"external_authoritative",
|
||||||
"external_mirror",
|
"external_mirror",
|
||||||
|
"governed_sync",
|
||||||
"linked_reference",
|
"linked_reference",
|
||||||
),
|
),
|
||||||
owned_concepts=(
|
owned_concepts=(
|
||||||
@@ -146,23 +202,45 @@ ARCHITECTURE = ModuleArchitectureDeclaration(
|
|||||||
"protocol interaction",
|
"protocol interaction",
|
||||||
"immutable connector snapshots",
|
"immutable connector snapshots",
|
||||||
"connector acquisition health",
|
"connector acquisition health",
|
||||||
|
"external knowledge synchronization evidence",
|
||||||
|
"external service-desk transport and synchronization evidence",
|
||||||
),
|
),
|
||||||
non_owned_concepts=(
|
non_owned_concepts=(
|
||||||
"datasource catalogue identity and lifecycle",
|
"datasource catalogue identity and lifecycle",
|
||||||
"domain records and business semantics",
|
"domain records and business semantics",
|
||||||
"data transformations",
|
"data transformations",
|
||||||
"screening dispositions",
|
"screening dispositions",
|
||||||
|
"native Wiki spaces, pages, and revision semantics",
|
||||||
|
"ticket, article, customer, case, and helpdesk business semantics",
|
||||||
),
|
),
|
||||||
target_tested_providers=(
|
target_tested_providers=(
|
||||||
TABULAR_PROVIDER_ID,
|
TABULAR_PROVIDER_ID,
|
||||||
SANCTIONS_PROVIDER_ID,
|
SANCTIONS_PROVIDER_ID,
|
||||||
|
KNOWLEDGE_PROVIDER_ID,
|
||||||
|
SERVICE_DESK_PROVIDER_ID,
|
||||||
),
|
),
|
||||||
documentation=ModuleArchitectureDocumentation(
|
documentation=ModuleArchitectureDocumentation(
|
||||||
migration=("src/govoplan_connectors/backend/migrations/versions",),
|
migration=("src/govoplan_connectors/backend/migrations/versions",),
|
||||||
upgrade=("docs/CONNECTOR_SOURCE_LIFECYCLE.md",),
|
upgrade=(
|
||||||
recovery=("docs/CONNECTOR_SOURCE_LIFECYCLE.md",),
|
"docs/CONNECTOR_SOURCE_LIFECYCLE.md",
|
||||||
security=("docs/CONNECTOR_SOURCE_LIFECYCLE.md",),
|
"docs/MEDIAWIKI_BLUESPICE_CONNECTOR.md",
|
||||||
operations=("docs/CONNECTOR_SOURCE_LIFECYCLE.md",),
|
"docs/ZNUNY_OTRS_CONNECTOR.md",
|
||||||
|
),
|
||||||
|
recovery=(
|
||||||
|
"docs/CONNECTOR_SOURCE_LIFECYCLE.md",
|
||||||
|
"docs/MEDIAWIKI_BLUESPICE_CONNECTOR.md",
|
||||||
|
"docs/ZNUNY_OTRS_CONNECTOR.md",
|
||||||
|
),
|
||||||
|
security=(
|
||||||
|
"docs/CONNECTOR_SOURCE_LIFECYCLE.md",
|
||||||
|
"docs/MEDIAWIKI_BLUESPICE_CONNECTOR.md",
|
||||||
|
"docs/ZNUNY_OTRS_CONNECTOR.md",
|
||||||
|
),
|
||||||
|
operations=(
|
||||||
|
"docs/CONNECTOR_SOURCE_LIFECYCLE.md",
|
||||||
|
"docs/MEDIAWIKI_BLUESPICE_CONNECTOR.md",
|
||||||
|
"docs/ZNUNY_OTRS_CONNECTOR.md",
|
||||||
|
),
|
||||||
),
|
),
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -261,6 +339,159 @@ EXTERNAL_PROVIDERS = (
|
|||||||
"connectors.sanctions-snapshots",
|
"connectors.sanctions-snapshots",
|
||||||
),
|
),
|
||||||
),
|
),
|
||||||
|
ExternalProviderDeclaration(
|
||||||
|
id=KNOWLEDGE_PROVIDER_ID,
|
||||||
|
module_id=MODULE_ID,
|
||||||
|
label="MediaWiki and BlueSpice knowledge provider",
|
||||||
|
maturity="migrate",
|
||||||
|
operations=(
|
||||||
|
"discover",
|
||||||
|
"link",
|
||||||
|
"search",
|
||||||
|
"read",
|
||||||
|
"publish",
|
||||||
|
"synchronize",
|
||||||
|
"migrate",
|
||||||
|
"preview",
|
||||||
|
"dry_run",
|
||||||
|
),
|
||||||
|
objects=(
|
||||||
|
ProviderObjectDeclaration(
|
||||||
|
object_type="external_knowledge_page",
|
||||||
|
field_groups=(
|
||||||
|
"stable_identity",
|
||||||
|
"revisions",
|
||||||
|
"namespaces",
|
||||||
|
"categories",
|
||||||
|
"links",
|
||||||
|
"files",
|
||||||
|
"discussions",
|
||||||
|
"permissions",
|
||||||
|
"source_provenance",
|
||||||
|
),
|
||||||
|
authority_modes=(
|
||||||
|
"external_authoritative",
|
||||||
|
"external_mirror",
|
||||||
|
"governed_sync",
|
||||||
|
"linked_reference",
|
||||||
|
),
|
||||||
|
default_authority_mode="external_mirror",
|
||||||
|
),
|
||||||
|
),
|
||||||
|
behavior=ProviderBehaviorDeclaration(
|
||||||
|
revision_tokens="Stable MediaWiki page ids, revision ids, discovery revisions, recent-change cursors, content digests, and provider timestamps are retained.",
|
||||||
|
concurrency="Profile changes use optimistic revisions; publication supplies an expected provider revision and a durable idempotency fence.",
|
||||||
|
freshness="The latest discovery time, sync high-watermark, recent-change cursor, source update time, and health state are exposed.",
|
||||||
|
health="Product discovery, authentication, transport, mapping loss, ACL degradation, Search deferral, and outcome-unknown publication are explicit without exposing credentials.",
|
||||||
|
max_read_items=500,
|
||||||
|
idempotency="Backfill, delta synchronization, migration previews, and publication require caller keys; exact replays return the committed evidence and mismatched reuse is rejected.",
|
||||||
|
retry="Read-only discovery and synchronization may be retried with a new key; a publication with an unknown outcome must be reconciled before retry.",
|
||||||
|
timeout_seconds=30,
|
||||||
|
conflicts="Namespace and path mappings are explicit; migration previews block target-path and attachment conflicts, while publication rejects stale expected revisions.",
|
||||||
|
outcome_unknown="A timed-out publication remains outcome-unknown behind a durable recovery fence until the institutional operator verifies the provider revision.",
|
||||||
|
outcome_unknown_supported=True,
|
||||||
|
evidence="Mapped snapshots retain stable external references, revision identity, source hashes, transport provenance, ACLs, and bounded loss diagnostics.",
|
||||||
|
audit_event_types=(
|
||||||
|
"connectors.knowledge.profile.created",
|
||||||
|
"connectors.knowledge.profile.updated",
|
||||||
|
"connectors.knowledge.profile.discovered",
|
||||||
|
"connectors.knowledge.profile.synchronized",
|
||||||
|
"connectors.knowledge.migration.previewed",
|
||||||
|
"connectors.knowledge.page.published",
|
||||||
|
),
|
||||||
|
correction="A later provider revision or explicit publication creates a new connector snapshot revision while synchronization-run evidence remains retained.",
|
||||||
|
rollback="Local snapshot and terminal recovery evidence commit atomically; remote publication cannot be rolled back by a local transaction.",
|
||||||
|
reconciliation="Rediscover capabilities, compare stable page and revision ids, run a bounded delta or full backfill, and inspect unresolved publication evidence.",
|
||||||
|
outage="Existing snapshots remain visible only through current tenant and ACL authorization and are marked stale or unavailable; no provider freshness claim is made.",
|
||||||
|
classifications=("public", "internal", "confidential", "restricted"),
|
||||||
|
purposes=(
|
||||||
|
"external knowledge discovery",
|
||||||
|
"authorized federated search",
|
||||||
|
"knowledge synchronization",
|
||||||
|
"migration planning",
|
||||||
|
"governed publication",
|
||||||
|
),
|
||||||
|
retention="The tenant's connector, Records, and target Wiki policies determine snapshot and operation-evidence retention.",
|
||||||
|
secret_handling="Credentials are resolved from a scoped Core credential envelope, never placed in endpoint URLs, persisted snapshots, diagnostics, or API responses.",
|
||||||
|
),
|
||||||
|
capability_names=(KNOWLEDGE_CAPABILITY,),
|
||||||
|
interface_names=(KNOWLEDGE_CAPABILITY,),
|
||||||
|
documentation_topic_ids=(
|
||||||
|
"connectors.authority-and-effects",
|
||||||
|
"connectors.mediawiki-bluespice",
|
||||||
|
),
|
||||||
|
),
|
||||||
|
ExternalProviderDeclaration(
|
||||||
|
id=SERVICE_DESK_PROVIDER_ID,
|
||||||
|
module_id=MODULE_ID,
|
||||||
|
label="Znuny and OTRS-compatible service-desk provider",
|
||||||
|
maturity="synchronize",
|
||||||
|
operations=("discover", "link", "search", "read", "publish", "synchronize"),
|
||||||
|
objects=(
|
||||||
|
ProviderObjectDeclaration(
|
||||||
|
object_type="external_service_desk_ticket",
|
||||||
|
field_groups=(
|
||||||
|
"stable_identity",
|
||||||
|
"queue_and_routing",
|
||||||
|
"state_and_priority",
|
||||||
|
"users_and_organizations",
|
||||||
|
"articles",
|
||||||
|
"attachment_metadata",
|
||||||
|
"dynamic_fields",
|
||||||
|
"permissions",
|
||||||
|
"source_provenance",
|
||||||
|
),
|
||||||
|
authority_modes=(
|
||||||
|
"external_authoritative",
|
||||||
|
"external_mirror",
|
||||||
|
"governed_sync",
|
||||||
|
"linked_reference",
|
||||||
|
),
|
||||||
|
default_authority_mode="external_authoritative",
|
||||||
|
),
|
||||||
|
),
|
||||||
|
behavior=ProviderBehaviorDeclaration(
|
||||||
|
revision_tokens="Stable ticket, ticket-number, article, and attachment identities plus provider change timestamps, discovery revisions, cursors, and content hashes are retained.",
|
||||||
|
concurrency="Profiles use optimistic revisions; governed ticket updates require the synchronized provider revision and a durable idempotency fence.",
|
||||||
|
freshness="Discovery time, synchronization high-watermark, cursor, provider change time, observation time, and profile health remain explicit.",
|
||||||
|
health="Product/version verification, authentication, transport, route policy, mapping loss, ACL fallback, Search deferral, cursor bounds, and unknown update outcomes are explicit without exposing credentials.",
|
||||||
|
max_read_items=500,
|
||||||
|
idempotency="Every full, delta, or update operation requires a profile-wide caller key; exact replays return committed evidence and mismatched reuse is rejected.",
|
||||||
|
retry="Discovery and read-only synchronization can be retried deliberately; an update with an unknown outcome must be reconciled against the provider before retry.",
|
||||||
|
timeout_seconds=20,
|
||||||
|
conflicts="Queue inclusion, target refs, ACLs, dynamic fields, authority, and route mappings are explicit; updates reject stale provider revisions.",
|
||||||
|
outcome_unknown="A timed-out or inconclusive remote update remains outcome-unknown behind durable recovery evidence until an operator verifies the provider ticket revision.",
|
||||||
|
outcome_unknown_supported=True,
|
||||||
|
evidence="Mapped tickets retain stable external references, revisions, transport and mapping provenance, current ACLs, loss diagnostics, and operation effects; attachment bytes are never retained.",
|
||||||
|
audit_event_types=(
|
||||||
|
"connectors.service_desk.profile.created",
|
||||||
|
"connectors.service_desk.profile.updated",
|
||||||
|
"connectors.service_desk.profile.discovered",
|
||||||
|
"connectors.service_desk.profile.synchronized",
|
||||||
|
"connectors.service_desk.ticket.updated",
|
||||||
|
),
|
||||||
|
correction="A later provider revision updates or restores the connector projection while prior synchronization and mutation evidence stays retained.",
|
||||||
|
rollback="Local projection and terminal recovery evidence commit atomically; a remote provider update cannot be rolled back by a local transaction.",
|
||||||
|
reconciliation="Rediscover the deployment-defined GenericInterface routes, compare stable ticket revisions, finish or restart bounded synchronization, and inspect unresolved remote mutations.",
|
||||||
|
outage="Authorized existing projections remain visible with explicit stale health; no provider freshness or write-success claim is made during an outage.",
|
||||||
|
classifications=("internal", "confidential", "restricted"),
|
||||||
|
purposes=(
|
||||||
|
"external service-desk discovery",
|
||||||
|
"authorized federated search",
|
||||||
|
"ticket reference or import",
|
||||||
|
"bounded synchronization",
|
||||||
|
"governed ticket update",
|
||||||
|
),
|
||||||
|
retention="The tenant's connector, Tickets, Helpdesk, Cases, and Records policies determine projection and operation-evidence retention.",
|
||||||
|
secret_handling="Credentials resolve from a scoped Core envelope and use approved headers or POST bodies; they never enter GET URLs, snapshots, diagnostics, or API responses.",
|
||||||
|
),
|
||||||
|
capability_names=(SERVICE_DESK_CAPABILITY,),
|
||||||
|
interface_names=(SERVICE_DESK_CAPABILITY,),
|
||||||
|
documentation_topic_ids=(
|
||||||
|
"connectors.authority-and-effects",
|
||||||
|
"connectors.znuny-otrs",
|
||||||
|
),
|
||||||
|
),
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
@@ -314,6 +545,51 @@ PERMISSIONS = (
|
|||||||
"Refresh sanctions sources",
|
"Refresh sanctions sources",
|
||||||
"Acquire a new immutable sanctions source snapshot.",
|
"Acquire a new immutable sanctions source snapshot.",
|
||||||
),
|
),
|
||||||
|
_permission(
|
||||||
|
KNOWLEDGE_READ_SCOPE,
|
||||||
|
"View external knowledge",
|
||||||
|
"View authorized MediaWiki and BlueSpice profiles, pages, and synchronization evidence.",
|
||||||
|
),
|
||||||
|
_permission(
|
||||||
|
KNOWLEDGE_ADMIN_SCOPE,
|
||||||
|
"Administer external knowledge",
|
||||||
|
"Configure knowledge profiles, namespace mappings, authority, visibility, and discovery.",
|
||||||
|
),
|
||||||
|
_permission(
|
||||||
|
KNOWLEDGE_SYNC_SCOPE,
|
||||||
|
"Synchronize external knowledge",
|
||||||
|
"Run bounded MediaWiki and BlueSpice backfills and change synchronization.",
|
||||||
|
),
|
||||||
|
_permission(
|
||||||
|
KNOWLEDGE_PUBLISH_SCOPE,
|
||||||
|
"Publish external knowledge",
|
||||||
|
"Publish a governed page revision with concurrency and recovery evidence.",
|
||||||
|
),
|
||||||
|
_permission(
|
||||||
|
KNOWLEDGE_MIGRATE_SCOPE,
|
||||||
|
"Preview knowledge migration",
|
||||||
|
"Dry-run a bounded migration into Wiki and inspect loss or conflict diagnostics.",
|
||||||
|
),
|
||||||
|
_permission(
|
||||||
|
SERVICE_DESK_READ_SCOPE,
|
||||||
|
"View external service-desk tickets",
|
||||||
|
"View authorized Znuny/OTRS profiles, mapped tickets, and synchronization evidence.",
|
||||||
|
),
|
||||||
|
_permission(
|
||||||
|
SERVICE_DESK_ADMIN_SCOPE,
|
||||||
|
"Administer service-desk connectors",
|
||||||
|
"Configure GenericInterface routes, queues, fields, authority, visibility, and discovery.",
|
||||||
|
),
|
||||||
|
_permission(
|
||||||
|
SERVICE_DESK_SYNC_SCOPE,
|
||||||
|
"Synchronize service-desk tickets",
|
||||||
|
"Run bounded Znuny/OTRS backfills and change synchronization.",
|
||||||
|
),
|
||||||
|
_permission(
|
||||||
|
SERVICE_DESK_UPDATE_SCOPE,
|
||||||
|
"Update external service-desk tickets",
|
||||||
|
"Apply governed revision-checked ticket updates with durable recovery evidence.",
|
||||||
|
),
|
||||||
)
|
)
|
||||||
|
|
||||||
ROLE_TEMPLATES = (
|
ROLE_TEMPLATES = (
|
||||||
@@ -329,6 +605,15 @@ ROLE_TEMPLATES = (
|
|||||||
FEED_PRIVATE_PUBLISH_SCOPE,
|
FEED_PRIVATE_PUBLISH_SCOPE,
|
||||||
SANCTIONS_READ_SCOPE,
|
SANCTIONS_READ_SCOPE,
|
||||||
SANCTIONS_REFRESH_SCOPE,
|
SANCTIONS_REFRESH_SCOPE,
|
||||||
|
KNOWLEDGE_READ_SCOPE,
|
||||||
|
KNOWLEDGE_ADMIN_SCOPE,
|
||||||
|
KNOWLEDGE_SYNC_SCOPE,
|
||||||
|
KNOWLEDGE_PUBLISH_SCOPE,
|
||||||
|
KNOWLEDGE_MIGRATE_SCOPE,
|
||||||
|
SERVICE_DESK_READ_SCOPE,
|
||||||
|
SERVICE_DESK_ADMIN_SCOPE,
|
||||||
|
SERVICE_DESK_SYNC_SCOPE,
|
||||||
|
SERVICE_DESK_UPDATE_SCOPE,
|
||||||
),
|
),
|
||||||
),
|
),
|
||||||
RoleTemplate(
|
RoleTemplate(
|
||||||
@@ -341,13 +626,23 @@ ROLE_TEMPLATES = (
|
|||||||
FEED_PUBLISH_SCOPE,
|
FEED_PUBLISH_SCOPE,
|
||||||
SANCTIONS_READ_SCOPE,
|
SANCTIONS_READ_SCOPE,
|
||||||
SANCTIONS_REFRESH_SCOPE,
|
SANCTIONS_REFRESH_SCOPE,
|
||||||
|
KNOWLEDGE_READ_SCOPE,
|
||||||
|
KNOWLEDGE_SYNC_SCOPE,
|
||||||
|
KNOWLEDGE_MIGRATE_SCOPE,
|
||||||
|
SERVICE_DESK_READ_SCOPE,
|
||||||
|
SERVICE_DESK_SYNC_SCOPE,
|
||||||
),
|
),
|
||||||
),
|
),
|
||||||
RoleTemplate(
|
RoleTemplate(
|
||||||
slug="connector_source_reader",
|
slug="connector_source_reader",
|
||||||
name="Connector source reader",
|
name="Connector source reader",
|
||||||
description="Discover and preview tabular connector sources.",
|
description="Discover and preview tabular connector sources.",
|
||||||
permissions=(READ_SCOPE, SANCTIONS_READ_SCOPE),
|
permissions=(
|
||||||
|
READ_SCOPE,
|
||||||
|
SANCTIONS_READ_SCOPE,
|
||||||
|
KNOWLEDGE_READ_SCOPE,
|
||||||
|
SERVICE_DESK_READ_SCOPE,
|
||||||
|
),
|
||||||
),
|
),
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -382,6 +677,14 @@ def _dsar_provider(_context) -> ConnectorsDsarProvider:
|
|||||||
return ConnectorsDsarProvider()
|
return ConnectorsDsarProvider()
|
||||||
|
|
||||||
|
|
||||||
|
def _knowledge_provider(_context) -> ExternalKnowledgeCapability:
|
||||||
|
return ExternalKnowledgeCapability()
|
||||||
|
|
||||||
|
|
||||||
|
def _service_desk_provider(_context) -> ExternalServiceDeskCapability:
|
||||||
|
return ExternalServiceDeskCapability()
|
||||||
|
|
||||||
|
|
||||||
def _tenant_summary(session, tenant_id: str) -> dict[str, int]:
|
def _tenant_summary(session, tenant_id: str) -> dict[str, int]:
|
||||||
return {
|
return {
|
||||||
"connector_definitions": (
|
"connector_definitions": (
|
||||||
@@ -417,6 +720,42 @@ def _tenant_summary(session, tenant_id: str) -> dict[str, int]:
|
|||||||
.filter(ConnectorSanctionsAcquisitionRun.tenant_id == tenant_id)
|
.filter(ConnectorSanctionsAcquisitionRun.tenant_id == tenant_id)
|
||||||
.count()
|
.count()
|
||||||
),
|
),
|
||||||
|
"connector_knowledge_profiles": (
|
||||||
|
session.query(ConnectorKnowledgeProfile)
|
||||||
|
.filter(ConnectorKnowledgeProfile.tenant_id == tenant_id)
|
||||||
|
.count()
|
||||||
|
),
|
||||||
|
"connector_knowledge_objects": (
|
||||||
|
session.query(ConnectorKnowledgeObject)
|
||||||
|
.filter(
|
||||||
|
ConnectorKnowledgeObject.tenant_id == tenant_id,
|
||||||
|
ConnectorKnowledgeObject.status != "deleted",
|
||||||
|
)
|
||||||
|
.count()
|
||||||
|
),
|
||||||
|
"connector_knowledge_runs": (
|
||||||
|
session.query(ConnectorKnowledgeSyncRun)
|
||||||
|
.filter(ConnectorKnowledgeSyncRun.tenant_id == tenant_id)
|
||||||
|
.count()
|
||||||
|
),
|
||||||
|
"connector_service_desk_profiles": (
|
||||||
|
session.query(ConnectorServiceDeskProfile)
|
||||||
|
.filter(ConnectorServiceDeskProfile.tenant_id == tenant_id)
|
||||||
|
.count()
|
||||||
|
),
|
||||||
|
"connector_service_desk_objects": (
|
||||||
|
session.query(ConnectorServiceDeskObject)
|
||||||
|
.filter(
|
||||||
|
ConnectorServiceDeskObject.tenant_id == tenant_id,
|
||||||
|
ConnectorServiceDeskObject.status != "deleted",
|
||||||
|
)
|
||||||
|
.count()
|
||||||
|
),
|
||||||
|
"connector_service_desk_runs": (
|
||||||
|
session.query(ConnectorServiceDeskSyncRun)
|
||||||
|
.filter(ConnectorServiceDeskSyncRun.tenant_id == tenant_id)
|
||||||
|
.count()
|
||||||
|
),
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
@@ -433,6 +772,11 @@ manifest = ModuleManifest(
|
|||||||
"portal",
|
"portal",
|
||||||
"reporting",
|
"reporting",
|
||||||
"risk_compliance",
|
"risk_compliance",
|
||||||
|
"search",
|
||||||
|
"tickets",
|
||||||
|
"helpdesk",
|
||||||
|
"cases",
|
||||||
|
"wiki",
|
||||||
),
|
),
|
||||||
required_capabilities=(
|
required_capabilities=(
|
||||||
CAPABILITY_AUTH_PRINCIPAL_RESOLVER,
|
CAPABILITY_AUTH_PRINCIPAL_RESOLVER,
|
||||||
@@ -463,6 +807,14 @@ manifest = ModuleManifest(
|
|||||||
name="connectors.runtime_contract",
|
name="connectors.runtime_contract",
|
||||||
version=CONNECTOR_RUNTIME_INTERFACE_VERSION,
|
version=CONNECTOR_RUNTIME_INTERFACE_VERSION,
|
||||||
),
|
),
|
||||||
|
ModuleInterfaceProvider(
|
||||||
|
name=KNOWLEDGE_CAPABILITY,
|
||||||
|
version=KNOWLEDGE_INTERFACE_VERSION,
|
||||||
|
),
|
||||||
|
ModuleInterfaceProvider(
|
||||||
|
name=SERVICE_DESK_CAPABILITY,
|
||||||
|
version=SERVICE_DESK_INTERFACE_VERSION,
|
||||||
|
),
|
||||||
ModuleInterfaceProvider(name=CONNECTORS_DSAR_CAPABILITY, version="0.1.0"),
|
ModuleInterfaceProvider(name=CONNECTORS_DSAR_CAPABILITY, version="0.1.0"),
|
||||||
),
|
),
|
||||||
requires_interfaces=(
|
requires_interfaces=(
|
||||||
@@ -495,6 +847,22 @@ manifest = ModuleManifest(
|
|||||||
parent_id="connectors.admin.governed-configurations",
|
parent_id="connectors.admin.governed-configurations",
|
||||||
order=20,
|
order=20,
|
||||||
),
|
),
|
||||||
|
ViewSurface(
|
||||||
|
id="connectors.admin.external-knowledge",
|
||||||
|
module_id=MODULE_ID,
|
||||||
|
kind="section",
|
||||||
|
label="External knowledge",
|
||||||
|
parent_id="connectors.admin.governed-configurations",
|
||||||
|
order=30,
|
||||||
|
),
|
||||||
|
ViewSurface(
|
||||||
|
id="connectors.admin.external-service-desk",
|
||||||
|
module_id=MODULE_ID,
|
||||||
|
kind="section",
|
||||||
|
label="External service desk",
|
||||||
|
parent_id="connectors.admin.governed-configurations",
|
||||||
|
order=40,
|
||||||
|
),
|
||||||
),
|
),
|
||||||
),
|
),
|
||||||
capability_factories={
|
capability_factories={
|
||||||
@@ -503,6 +871,8 @@ manifest = ModuleManifest(
|
|||||||
CAPABILITY_DATASOURCE_ORIGINS: _datasource_origin_provider,
|
CAPABILITY_DATASOURCE_ORIGINS: _datasource_origin_provider,
|
||||||
CAPABILITY_CONNECTORS_SANCTIONS_SNAPSHOTS: (_sanctions_snapshot_provider),
|
CAPABILITY_CONNECTORS_SANCTIONS_SNAPSHOTS: (_sanctions_snapshot_provider),
|
||||||
CAPABILITY_CONNECTORS_FEEDS: _feed_provider,
|
CAPABILITY_CONNECTORS_FEEDS: _feed_provider,
|
||||||
|
KNOWLEDGE_CAPABILITY: _knowledge_provider,
|
||||||
|
SERVICE_DESK_CAPABILITY: _service_desk_provider,
|
||||||
CONNECTORS_DSAR_CAPABILITY: _dsar_provider,
|
CONNECTORS_DSAR_CAPABILITY: _dsar_provider,
|
||||||
},
|
},
|
||||||
capability_documentation={
|
capability_documentation={
|
||||||
@@ -516,6 +886,18 @@ manifest = ModuleManifest(
|
|||||||
),
|
),
|
||||||
},
|
},
|
||||||
tenant_summary_providers=(_tenant_summary,),
|
tenant_summary_providers=(_tenant_summary,),
|
||||||
|
search_sources=(
|
||||||
|
SearchSourceProviderRegistration(
|
||||||
|
id=KNOWLEDGE_PROVIDER_ID,
|
||||||
|
factory=create_external_knowledge_search_source,
|
||||||
|
order=65,
|
||||||
|
),
|
||||||
|
SearchSourceProviderRegistration(
|
||||||
|
id=SERVICE_DESK_PROVIDER_ID,
|
||||||
|
factory=create_external_service_desk_search_source,
|
||||||
|
order=66,
|
||||||
|
),
|
||||||
|
),
|
||||||
architecture=ARCHITECTURE,
|
architecture=ARCHITECTURE,
|
||||||
external_providers=EXTERNAL_PROVIDERS,
|
external_providers=EXTERNAL_PROVIDERS,
|
||||||
external_provider_state_providers=(
|
external_provider_state_providers=(
|
||||||
@@ -529,6 +911,16 @@ manifest = ModuleManifest(
|
|||||||
provider_id=SANCTIONS_PROVIDER_ID,
|
provider_id=SANCTIONS_PROVIDER_ID,
|
||||||
provider=sanctions_provider_states,
|
provider=sanctions_provider_states,
|
||||||
),
|
),
|
||||||
|
ExternalProviderStateProviderRegistration(
|
||||||
|
module_id=MODULE_ID,
|
||||||
|
provider_id=KNOWLEDGE_PROVIDER_ID,
|
||||||
|
provider=knowledge_provider_states,
|
||||||
|
),
|
||||||
|
ExternalProviderStateProviderRegistration(
|
||||||
|
module_id=MODULE_ID,
|
||||||
|
provider_id=SERVICE_DESK_PROVIDER_ID,
|
||||||
|
provider=service_desk_provider_states,
|
||||||
|
),
|
||||||
),
|
),
|
||||||
migration_spec=MigrationSpec(
|
migration_spec=MigrationSpec(
|
||||||
module_id=MODULE_ID,
|
module_id=MODULE_ID,
|
||||||
@@ -536,6 +928,12 @@ manifest = ModuleManifest(
|
|||||||
script_location=str(Path(__file__).with_name("migrations") / "versions"),
|
script_location=str(Path(__file__).with_name("migrations") / "versions"),
|
||||||
retirement_supported=True,
|
retirement_supported=True,
|
||||||
retirement_provider=drop_table_retirement_provider(
|
retirement_provider=drop_table_retirement_provider(
|
||||||
|
ConnectorServiceDeskSyncRun,
|
||||||
|
ConnectorServiceDeskObject,
|
||||||
|
ConnectorServiceDeskProfile,
|
||||||
|
ConnectorKnowledgeSyncRun,
|
||||||
|
ConnectorKnowledgeObject,
|
||||||
|
ConnectorKnowledgeProfile,
|
||||||
ConnectorSimulationRun,
|
ConnectorSimulationRun,
|
||||||
ConnectorConfiguration,
|
ConnectorConfiguration,
|
||||||
ConnectorDefinitionRevision,
|
ConnectorDefinitionRevision,
|
||||||
@@ -552,6 +950,12 @@ manifest = ModuleManifest(
|
|||||||
),
|
),
|
||||||
uninstall_guard_providers=(
|
uninstall_guard_providers=(
|
||||||
persistent_table_uninstall_guard(
|
persistent_table_uninstall_guard(
|
||||||
|
ConnectorServiceDeskSyncRun,
|
||||||
|
ConnectorServiceDeskObject,
|
||||||
|
ConnectorServiceDeskProfile,
|
||||||
|
ConnectorKnowledgeSyncRun,
|
||||||
|
ConnectorKnowledgeObject,
|
||||||
|
ConnectorKnowledgeProfile,
|
||||||
ConnectorSimulationRun,
|
ConnectorSimulationRun,
|
||||||
ConnectorConfiguration,
|
ConnectorConfiguration,
|
||||||
ConnectorDefinitionRevision,
|
ConnectorDefinitionRevision,
|
||||||
@@ -562,7 +966,39 @@ manifest = ModuleManifest(
|
|||||||
label="Connectors",
|
label="Connectors",
|
||||||
),
|
),
|
||||||
),
|
),
|
||||||
documentation=(
|
documentation=localize_documentation_topics((
|
||||||
|
DocumentationTopic(
|
||||||
|
id="connectors.csv-source-fidelity",
|
||||||
|
title="Preserving original CSV input",
|
||||||
|
summary="Choose explicit text or legacy type inference, and retrieve the verified original of new CSV snapshots.",
|
||||||
|
body=(
|
||||||
|
"CSV snapshot and managed-file creation accept csv_value_mode=text to preserve field whitespace, decimal digits, large identifiers, boolean-looking text and explicit empty records as strings. "
|
||||||
|
"Text mode rejects missing/extra fields and malformed quoting rather than dropping values. Headers remain normalized and blank physical lines are not table rows; the original upload preserves these lexical details. "
|
||||||
|
"Omitting csv_value_mode keeps legacy_typed API behavior for existing clients. Legacy inference can trim values, infer numbers/booleans and omit empty rows; choose text when exact values matter. "
|
||||||
|
"New CSV snapshots retain the original supplied text, delimiter, parser profile and UTF-8 SHA-256 separately from catalogue responses; csv_source metadata is reserved verified evidence, not a place to supply content. "
|
||||||
|
"GET /api/v1/connectors/tabular-sources/{source_ref}/original-csv requires current connectors:source:read or connectors:source:admin authority in the same tenant, verifies the checksum, audits the export without source content, and returns an uncached attachment. "
|
||||||
|
"The export preserves the text submitted to the API as UTF-8, not an earlier file encoding. It does not sanitize spreadsheet formulas: treat imported originals as untrusted data. "
|
||||||
|
"Original text and parsed rows each remain limited to 5 MB, with at most 10,000 parsed rows. Each snapshot can retain up to 5 MB of original text, plus serialization and metadata storage overhead. Source text follows the snapshot lifecycle; retired/deleted sources are not downloadable. "
|
||||||
|
"Managed files retain their original through the exact Files version and its access/lifecycle policy; text mode is pinned in source metadata and preserved on refresh. Existing snapshot fingerprints and historical rows are never rewritten. "
|
||||||
|
"Original text cannot be reconstructed for older snapshots: the endpoint reports it unavailable. Schema inference now shares one ordered, bounded-state mechanism with Datasources while preserving legacy provider type classifications."
|
||||||
|
),
|
||||||
|
layer="always", documentation_types=("user", "admin"), audience=("user", "module_admin", "operator"), order=8,
|
||||||
|
translations={"de": {
|
||||||
|
"title": "Ursprüngliche CSV-Eingaben erhalten",
|
||||||
|
"summary": "Textwerte oder bisherige Typableitung ausdrücklich wählen und das geprüfte Original neuer CSV-Snapshots abrufen.",
|
||||||
|
"body": (
|
||||||
|
"CSV-Snapshots und verwaltete Dateiquellen unterstützen csv_value_mode=text. Dieser Modus erhält Leerzeichen in Feldwerten, Dezimalstellen, große Kennungen, boolesch wirkenden Text und ausdrücklich leere Datensätze als Zeichenketten. "
|
||||||
|
"Fehlende oder zusätzliche Felder und fehlerhafte Anführungszeichen werden abgelehnt. Überschriften werden weiterhin normalisiert; vollständig leere physische Zeilen sind keine Tabellenzeilen. Das Original erhält auch diese Texteigenschaften. "
|
||||||
|
"Ohne csv_value_mode bleibt für bestehende API-Aufrufe legacy_typed aktiv. Dabei können Werte gekürzt, Zahlen/Wahrheitswerte abgeleitet und leere Zeilen ausgelassen werden. Für genaue Werte wählen Sie Text. "
|
||||||
|
"Neue CSV-Snapshots speichern den gelieferten Originaltext, Trennzeichen, Parserprofil und UTF-8-SHA-256 getrennt vom Katalog. csv_source-Metadaten sind reservierter geprüfter Nachweis. "
|
||||||
|
"GET /api/v1/connectors/tabular-sources/{source_ref}/original-csv benötigt aktuelle Rechte connectors:source:read oder connectors:source:admin im selben Mandanten, prüft die Prüfsumme, protokolliert den Export ohne Quellinhalt und liefert einen nicht zwischengespeicherten Download. "
|
||||||
|
"Exportiert wird der an die API übermittelte Text als UTF-8, nicht eine frühere Dateikodierung. Tabellenformeln werden nicht verändert; behandeln Sie Originaldateien als nicht vertrauenswürdige Daten. "
|
||||||
|
"Originaltext und verarbeitete Zeilen sind jeweils auf 5 MB begrenzt; höchstens 10.000 Zeilen werden angenommen. Pro Snapshot werden bis zu 5 MB Originaltext zuzüglich Speicher für Serialisierung und Metadaten aufbewahrt. Der Originaltext folgt dem Snapshot-Lebenszyklus; stillgelegte oder gelöschte Quellen sind nicht abrufbar. "
|
||||||
|
"Verwaltete Dateien behalten ihr Original in der genauen Files-Version mit deren Zugriffs- und Lebenszyklusregeln; der Textmodus bleibt bei Aktualisierungen erhalten. Bestehende Fingerprints und historische Zeilen werden nicht umgeschrieben. "
|
||||||
|
"Für ältere Snapshots kann das Original nicht rekonstruiert werden; der Abruf meldet es als nicht verfügbar. Die gemeinsame Schemaableitung erhält die bisherigen Typregeln und Spaltenreihenfolge."
|
||||||
|
),
|
||||||
|
}},
|
||||||
|
),
|
||||||
DocumentationTopic(
|
DocumentationTopic(
|
||||||
id="connectors.data-subject-requests",
|
id="connectors.data-subject-requests",
|
||||||
title="Connector data-subject requests",
|
title="Connector data-subject requests",
|
||||||
@@ -573,8 +1009,9 @@ manifest = ModuleManifest(
|
|||||||
body=(
|
body=(
|
||||||
"Connectors correlates only an exact tenant account identifier and can "
|
"Connectors correlates only an exact tenant account identifier and can "
|
||||||
"narrow an already verified search to one source, acquisition, "
|
"narrow an already verified search to one source, acquisition, "
|
||||||
"definition, configuration, or simulation. The export identifies the "
|
"definition, configuration, simulation, external-knowledge or service-desk "
|
||||||
"subject's configuration, acquisition, simulation, and review activity "
|
"profile, or connector operation. The export identifies the subject's "
|
||||||
|
"configuration, acquisition, simulation, external-operation, and review activity "
|
||||||
"using bounded lifecycle metadata. It never includes credential or "
|
"using bounded lifecycle metadata. It never includes credential or "
|
||||||
"endpoint references, source rows, external responses, request payloads, "
|
"endpoint references, source rows, external responses, request payloads, "
|
||||||
"mapping and configuration documents, diagnostics, provenance, hashes, "
|
"mapping and configuration documents, diagnostics, provenance, hashes, "
|
||||||
@@ -616,7 +1053,7 @@ manifest = ModuleManifest(
|
|||||||
related_modules=("policy", "audit", "dataflow", "ops"),
|
related_modules=("policy", "audit", "dataflow", "ops"),
|
||||||
order=38,
|
order=38,
|
||||||
metadata={
|
metadata={
|
||||||
"kind": "guide",
|
"kind": "workflow",
|
||||||
"help_contexts": ["connectors.admin.governed-configurations"],
|
"help_contexts": ["connectors.admin.governed-configurations"],
|
||||||
"prerequisites": [
|
"prerequisites": [
|
||||||
"A connector definition has been installed or authored.",
|
"A connector definition has been installed or authored.",
|
||||||
@@ -625,6 +1062,12 @@ manifest = ModuleManifest(
|
|||||||
"outcome": "The active connector behavior is inspectable, version-pinned, testable, and reviewable before any provider-specific write.",
|
"outcome": "The active connector behavior is inspectable, version-pinned, testable, and reviewable before any provider-specific write.",
|
||||||
"verification": "Reload the configuration, inspect protected paths and effective hash, run a simulation with a new idempotency key, and resolve any pending review result.",
|
"verification": "Reload the configuration, inspect protected paths and effective hash, run a simulation with a new idempotency key, and resolve any pending review result.",
|
||||||
},
|
},
|
||||||
|
conditions=(
|
||||||
|
DocumentationCondition(
|
||||||
|
required_modules=("connectors",),
|
||||||
|
required_scopes=(ADMIN_SCOPE,),
|
||||||
|
),
|
||||||
|
),
|
||||||
),
|
),
|
||||||
DocumentationTopic(
|
DocumentationTopic(
|
||||||
id="connectors.authority-and-effects",
|
id="connectors.authority-and-effects",
|
||||||
@@ -640,6 +1083,38 @@ manifest = ModuleManifest(
|
|||||||
audience=("operator", "module_admin", "power_user", "product_owner"),
|
audience=("operator", "module_admin", "power_user", "product_owner"),
|
||||||
related_modules=("datasources", "dataflow", "ops", "policy", "audit"),
|
related_modules=("datasources", "dataflow", "ops", "policy", "audit"),
|
||||||
order=39,
|
order=39,
|
||||||
|
metadata={
|
||||||
|
"kind": "reference",
|
||||||
|
"help_contexts": [
|
||||||
|
"connectors.action.reject-ambiguous-result",
|
||||||
|
"connectors.action.approve-ambiguous-result",
|
||||||
|
],
|
||||||
|
"fields": [
|
||||||
|
"Direction and technical maturity",
|
||||||
|
"Configured source authority",
|
||||||
|
"Revision and idempotency behavior",
|
||||||
|
"Reconciliation, outage, and secret requirements",
|
||||||
|
],
|
||||||
|
"consequences": [
|
||||||
|
"The authority mode determines which side may change business state.",
|
||||||
|
"A writable provider must preserve evidence and reconcile unknown outcomes before retry.",
|
||||||
|
],
|
||||||
|
},
|
||||||
|
structured_translation_version="1",
|
||||||
|
structured_translations={
|
||||||
|
"de": {
|
||||||
|
"fields": [
|
||||||
|
"Richtung und technische Reife",
|
||||||
|
"Konfigurierte Quellenautorität",
|
||||||
|
"Revisions- und Idempotenzverhalten",
|
||||||
|
"Anforderungen an Abgleich, Ausfallverhalten und Geheimnisse",
|
||||||
|
],
|
||||||
|
"consequences": [
|
||||||
|
"Der Autoritätsmodus bestimmt, welche Seite den Fachzustand ändern darf.",
|
||||||
|
"Ein schreibender Anbieter muss Nachweise bewahren und unbekannte Ergebnisse vor einer Wiederholung abgleichen.",
|
||||||
|
],
|
||||||
|
}
|
||||||
|
},
|
||||||
),
|
),
|
||||||
DocumentationTopic(
|
DocumentationTopic(
|
||||||
id="connectors.runtime-preview-contract",
|
id="connectors.runtime-preview-contract",
|
||||||
@@ -672,7 +1147,9 @@ manifest = ModuleManifest(
|
|||||||
"Immutable JSON/CSV snapshots remain available. Managed CSV/XLSX "
|
"Immutable JSON/CSV snapshots remain available. Managed CSV/XLSX "
|
||||||
"sources use the optional Files capability, pin an exact authorized "
|
"sources use the optional Files capability, pin an exact authorized "
|
||||||
"version, apply archive and expansion limits, and require explicit "
|
"version, apply archive and expansion limits, and require explicit "
|
||||||
"refresh before adopting a newer version. The PostgreSQL adapter uses "
|
"refresh before adopting a newer version. XLSX reads validate actual selected-worksheet coordinates before grid allocation: "
|
||||||
|
"at most 500 columns and 10,000 row positions after the header, including blank gaps. Unreliable declared dimensions "
|
||||||
|
"neither expand the grid nor conceal cells; oversized or inconsistent coordinates fail validation instead of truncating data. The PostgreSQL adapter uses "
|
||||||
"an active governed configuration and scoped Core credential envelope, "
|
"an active governed configuration and scoped Core credential envelope, "
|
||||||
"reflects simple schema/table identifiers, runs read-only bounded "
|
"reflects simple schema/table identifiers, runs read-only bounded "
|
||||||
"projection and pagination, and blocks configuration, credential, or "
|
"projection and pagination, and blocks configuration, credential, or "
|
||||||
@@ -685,6 +1162,44 @@ manifest = ModuleManifest(
|
|||||||
related_modules=("dataflow", "files", "reporting", "risk_compliance"),
|
related_modules=("dataflow", "files", "reporting", "risk_compliance"),
|
||||||
order=40,
|
order=40,
|
||||||
),
|
),
|
||||||
|
DocumentationTopic(
|
||||||
|
id="connectors.xlsx-worker-limits",
|
||||||
|
title="XLSX processing resource limits",
|
||||||
|
summary="Understand isolated workbook parsing and explicit capacity failures.",
|
||||||
|
body=(
|
||||||
|
"Authorized XLSX bytes are parsed in a fresh disposable process with a 15-second wall limit, 10 CPU seconds, "
|
||||||
|
"512 MiB address-space limit, no file output, and 8 MiB input/64 MiB result transport ceilings. Existing limits remain "
|
||||||
|
"5,000,000 input bytes, 50,000,000 expanded archive bytes, 5,000 archive entries, 100:1 expansion ratio, "
|
||||||
|
"500 columns and 10,000 row positions after the header. Typed transport additionally limits nesting to 64 levels "
|
||||||
|
"and 1,000,000 value nodes. Limit failures reject the whole parse; reduce the workbook before retrying. "
|
||||||
|
"The shared Core GOVOPLAN_ISOLATED_PROCESS_CONCURRENCY capacity is per API/worker process and does not queue: busy work "
|
||||||
|
"fails explicitly and may be retried later. POSIX resource controls are required; missing controls, cancellation "
|
||||||
|
"or worker failure produce sanitized unavailable diagnostics. No in-process fallback occurs. File authorization, "
|
||||||
|
"credentials, SQL sessions and durable source changes remain in the parent. Operators must budget aggregate "
|
||||||
|
"memory across process slots and keep the required Core worker API available."
|
||||||
|
),
|
||||||
|
layer="static",
|
||||||
|
documentation_types=("user", "admin"),
|
||||||
|
audience=("user", "module_admin", "operator"),
|
||||||
|
translations={"de": {
|
||||||
|
"title": "Ressourcengrenzen der XLSX-Verarbeitung",
|
||||||
|
"summary": "Isoliertes Einlesen von Arbeitsmappen und ausdrückliche Kapazitätsfehler verstehen.",
|
||||||
|
"body": (
|
||||||
|
"Autorisierte XLSX-Bytes werden in einem neuen kurzlebigen Prozess verarbeitet: höchstens 15 Sekunden Gesamtdauer, "
|
||||||
|
"10 CPU-Sekunden, 512 MiB Adressraum, keine Dateiausgabe und 8 MiB Eingabe-/64 MiB Ergebnistransport. Weiterhin gelten "
|
||||||
|
"5.000.000 Eingabebytes, 50.000.000 entpackte Archivbytes, 5.000 Archiveinträge, ein Entpackverhältnis von 100:1, "
|
||||||
|
"500 Spalten und 10.000 Zeilenpositionen nach der Kopfzeile. Der typisierte Transport begrenzt zusätzlich die "
|
||||||
|
"Verschachtelung auf 64 Ebenen und 1.000.000 Wertknoten. Grenzverletzungen lehnen den gesamten Lesevorgang ab; "
|
||||||
|
"verkleinern Sie die Arbeitsmappe vor einem erneuten Versuch. Die gemeinsame Core-Einstellung GOVOPLAN_ISOLATED_PROCESS_CONCURRENCY "
|
||||||
|
"gilt pro API-/Worker-Prozess und bildet keine Warteschlange: Bei belegter Kapazität erfolgt ein ausdrücklicher Fehler; "
|
||||||
|
"versuchen Sie es später erneut. POSIX-Ressourcenbegrenzungen sind erforderlich. Fehlende Kontrollen, Abbruch oder "
|
||||||
|
"Worker-Fehler melden eine bereinigte Nichtverfügbarkeit. Es gibt keinen Rückfall auf Verarbeitung im Elternprozess. "
|
||||||
|
"Dateiberechtigungen, Zugangsdaten, SQL-Sitzungen und dauerhafte Quelländerungen bleiben im Elternprozess. Betreiber "
|
||||||
|
"müssen den Gesamtspeicher aller Prozessplätze berücksichtigen und die benötigte Core-Worker-API bereitstellen."
|
||||||
|
),
|
||||||
|
}},
|
||||||
|
order=41,
|
||||||
|
),
|
||||||
DocumentationTopic(
|
DocumentationTopic(
|
||||||
id="connectors.rss-atom",
|
id="connectors.rss-atom",
|
||||||
title="RSS and Atom feeds",
|
title="RSS and Atom feeds",
|
||||||
@@ -709,6 +1224,45 @@ manifest = ModuleManifest(
|
|||||||
related_modules=("datasources", "dataflow", "portal", "reporting"),
|
related_modules=("datasources", "dataflow", "portal", "reporting"),
|
||||||
order=42,
|
order=42,
|
||||||
),
|
),
|
||||||
|
DocumentationTopic(
|
||||||
|
id="connectors.mediawiki-bluespice",
|
||||||
|
title="Connect MediaWiki and BlueSpice knowledge",
|
||||||
|
summary="Discover, synchronize, search, publish, and preview migration of external knowledge without weakening source permissions.",
|
||||||
|
body=(
|
||||||
|
"A connector administrator first creates a governed MediaWiki Action API configuration whose endpoint passes Core egress and peer validation; credentials remain in a scoped credential envelope. A knowledge profile then maps each included source namespace to a target Wiki space and path prefix, selects source authority, and sets a tenant or restricted fallback ACL. Discovery records product, version, extensions, namespaces, authentication state, capabilities, and loss diagnostics. Run a bounded full backfill once, then cursor-based deltas for revisions, moves, deletions, and permission changes. Stable page and revision identifiers, canonical links, redirects, files, categories, links, discussions, provenance, and current ACLs remain attached to the connector snapshot. Search indexes only active, non-deleted pages and rechecks the current profile, tenant, permission, and ACL before returning every result; disabling Connectors removes its Search projection. Publication requires an expected external revision, an idempotency key, and durable outcome evidence. An unknown provider outcome must be reconciled before retry. Migration into native Wiki is deliberately preview-only in this slice: the dry-run reports target-path collisions, attachment-name conflicts, unsupported macros, truncation, and source fingerprints; a successful preview does not write native Wiki pages. Existing authorized snapshots can remain visible during a provider outage, but health and freshness stay explicit and no current-source claim is made."
|
||||||
|
),
|
||||||
|
layer="configured",
|
||||||
|
documentation_types=("admin", "user"),
|
||||||
|
audience=(
|
||||||
|
"operator",
|
||||||
|
"module_admin",
|
||||||
|
"integration_admin",
|
||||||
|
"knowledge_manager",
|
||||||
|
"auditor",
|
||||||
|
),
|
||||||
|
related_modules=("core", "search", "wiki", "files", "audit", "policy"),
|
||||||
|
order=43,
|
||||||
|
translations={
|
||||||
|
"de": {
|
||||||
|
"title": "Wissen aus MediaWiki und BlueSpice anbinden",
|
||||||
|
"summary": "Externes Wissen erkennen, synchronisieren, durchsuchen, veröffentlichen und eine Migration prüfen, ohne Quellberechtigungen abzuschwächen.",
|
||||||
|
"body": (
|
||||||
|
"Eine Connector-Administration erstellt zuerst eine gesteuerte Konfiguration für die MediaWiki Action API. Der Endpunkt unterliegt der zentralen Ausgangs- und Gegenstellenprüfung; Zugangsdaten bleiben in einem zweckgebundenen Zugangsdaten-Umschlag. Das Wissensprofil ordnet jeden eingeschlossenen Quellnamensraum einem Zielbereich und Pfadpräfix in Wiki zu und legt Quellhoheit sowie eine mandantenweite oder eingeschränkte Ersatz-ACL fest. Die Erkennung dokumentiert Produkt, Version, Erweiterungen, Namensräume, Authentifizierungsstatus, Fähigkeiten und Verlustdiagnosen. Nach einem begrenzten Vollabgleich folgen cursorbasierte Änderungen für Revisionen, Verschiebungen, Löschungen und Berechtigungen. Stabile Seiten- und Revisionskennungen, kanonische Links, Weiterleitungen, Dateien, Kategorien, Links, Diskussionen, Herkunft und aktuelle ACLs bleiben am Snapshot. Search indiziert nur aktive, nicht gelöschte Seiten und prüft bei jedem Treffer Profilstatus, Mandant, Berechtigung und ACL erneut; bei Deaktivierung von Connectors wird dessen Suchprojektion entfernt. Veröffentlichungen erfordern die erwartete externe Revision, einen Idempotenzschlüssel und dauerhafte Ergebnisevidenz. Ein unbekanntes Ergebnis muss vor einem erneuten Versuch abgeglichen werden. Die Migration in das native Wiki ist in diesem Ausbauschritt ausschließlich eine Vorschau: Sie meldet Pfad- und Anhangskonflikte, nicht unterstützte Makros, Begrenzungen und Quellfingerabdrücke, schreibt aber keine Wiki-Seiten. Bei einem Ausfall dürfen bestehende Snapshots nur für weiterhin Berechtigte sichtbar bleiben; Zustand und Aktualität bleiben ausdrücklich erkennbar."
|
||||||
|
),
|
||||||
|
}
|
||||||
|
},
|
||||||
|
metadata={
|
||||||
|
"kind": "guide",
|
||||||
|
"help_contexts": ["connectors.admin.external-knowledge"],
|
||||||
|
"prerequisites": [
|
||||||
|
"An active governed MediaWiki Action API configuration exists.",
|
||||||
|
"Namespace targets and fallback ACLs have been reviewed.",
|
||||||
|
"The Search and Wiki modules are optional and remain capability-separated.",
|
||||||
|
],
|
||||||
|
"outcome": "External knowledge remains identity-stable, loss-visible, ACL-safe, and migration-ready.",
|
||||||
|
"verification": "Rediscover the profile, run a keyed delta, inspect health and diagnostics, verify an allowed and denied Search principal, and run a migration dry-run before any target-side work.",
|
||||||
|
},
|
||||||
|
),
|
||||||
DocumentationTopic(
|
DocumentationTopic(
|
||||||
id="connectors.sanctions-snapshots",
|
id="connectors.sanctions-snapshots",
|
||||||
title="Sanctions source snapshots",
|
title="Sanctions source snapshots",
|
||||||
@@ -734,7 +1288,59 @@ manifest = ModuleManifest(
|
|||||||
related_modules=("risk_compliance", "dataflow"),
|
related_modules=("risk_compliance", "dataflow"),
|
||||||
order=41,
|
order=41,
|
||||||
),
|
),
|
||||||
|
DocumentationTopic(
|
||||||
|
id="connectors.znuny-otrs",
|
||||||
|
title="Connect Znuny and OTRS-compatible service desks",
|
||||||
|
summary="Link, import, synchronize, search, and govern updates to external tickets without collapsing Tickets, Helpdesk, or Cases semantics.",
|
||||||
|
body=(
|
||||||
|
"A connector administrator first creates an active governed configuration for a Znuny or OTRS-compatible GenericInterface REST endpoint and keeps credentials in a scoped Core credential envelope. Because GenericInterface route paths and methods are defined by each provider deployment, the service-desk profile explicitly maps search, ticket-read, optional update, and browser-link routes. The profile chooses link, snapshot import, or ongoing synchronization; separately it records external, mirror, linked-reference, or governed-sync authority. Queue mappings decide inclusion, optional target queue references, and current tenant or restricted ACLs. Dynamic-field mappings declare included fields, governed names, and value types. Discovery verifies endpoint health, the exact configuration revision, product/version evidence, and safe technical maturity. A changed endpoint, governed configuration, or route map invalidates discovery and prior projections: synchronization and updates require rediscovery, while Search stays closed until a new full reconciliation. Integration or mapping changes also reset the cursor for a full reconciliation. A bounded full run reconciles stable ticket identities, then synchronize mode changes to cursor-based, revision-aware deltas with overlap-safe provider timestamps; delta cannot bootstrap an unreconciled profile, and supplied cursors must match committed state. The connector maps queues, state, priority, type, owners, responsible users, customers, organizations, services, SLAs, articles, attachment metadata, dynamic fields, provenance, and structured loss diagnostics. It never retains attachment bytes. GenericInterface has no portable standard ticket ACL, so provider-supplied GovOPlaN ACL metadata wins when present; otherwise reviewed queue or restricted profile defaults apply. Search includes only active authorized projections and rechecks tenant, profile status, read scope, current configuration discovery, and current ACL for every result. A ticket remains an external ticket reference: creating or relating a GovOPlaN Ticket, Helpdesk item, or Case belongs to those modules. Remote updates are available only in governed-sync mode after discovery confirms an update route, require the synchronized external revision and a unique idempotency key, and retain durable outcome evidence. If the result is unknown, operators must inspect the provider revision before retry. Providers with more than 10000 identities must be partitioned into queue-scoped profiles; too many tickets at one timestamp also require narrower partitions. During outages, existing authorized projections remain visibly stale and never imply current provider state."
|
||||||
),
|
),
|
||||||
|
layer="configured",
|
||||||
|
documentation_types=("admin", "user"),
|
||||||
|
audience=(
|
||||||
|
"operator",
|
||||||
|
"module_admin",
|
||||||
|
"integration_admin",
|
||||||
|
"service_desk_manager",
|
||||||
|
"auditor",
|
||||||
|
),
|
||||||
|
related_modules=(
|
||||||
|
"core",
|
||||||
|
"search",
|
||||||
|
"tickets",
|
||||||
|
"helpdesk",
|
||||||
|
"cases",
|
||||||
|
"audit",
|
||||||
|
"policy",
|
||||||
|
),
|
||||||
|
order=44,
|
||||||
|
translations={
|
||||||
|
"de": {
|
||||||
|
"title": "Znuny- und OTRS-kompatible Service-Desks anbinden",
|
||||||
|
"summary": "Externe Tickets verknüpfen, importieren, synchronisieren, durchsuchen und gesteuert aktualisieren, ohne die Fachsemantik von Tickets, Helpdesk oder Cases zu vermischen.",
|
||||||
|
"body": (
|
||||||
|
"Die Connector-Administration erstellt zuerst eine aktive, gesteuerte Konfiguration für einen Znuny- oder OTRS-kompatiblen GenericInterface-REST-Endpunkt; Zugangsdaten bleiben in einem zweckgebundenen Core-Umschlag. Da Pfade und Methoden im GenericInterface je Installation festgelegt werden, ordnet das Service-Desk-Profil Suche, Ticketabruf, optionale Aktualisierung und Browserlink ausdrücklich zu. Das Profil wählt Verknüpfung, Snapshot-Import oder fortlaufende Synchronisierung und legt getrennt davon Quellhoheit, Spiegelung, Referenz oder gesteuerte beidseitige Aktualisierung fest. Warteschlangen-Zuordnungen bestimmen Einschluss, optionale Zielreferenz und aktuelle mandantenweite oder eingeschränkte ACLs. Dynamische Felder erhalten freigegebene Namen und Datentypen. Die Erkennung prüft Gesundheit, die genaue Konfigurationsrevision, Produkt-/Versionsnachweis und technische Reife. Ein geänderter Endpunkt, eine geänderte gesteuerte Konfiguration oder Routenabbildung macht Nachweis und bisherige Projektionen ungültig; Synchronisierung und Aktualisierungen erfordern erneute Erkennung, Search zusätzlich einen neuen Vollabgleich. Änderungen an Integration oder Abbildungen setzen den Cursor ebenfalls zurück. Erst nach einem begrenzten Vollabgleich verwendet der Synchronisierungsmodus revisionsbewusste, überlappungssichere Delta-Cursor; Delta kann kein unabgeglichenes Profil initialisieren und übergebene Cursor müssen dem gespeicherten Stand entsprechen. Abgebildet werden stabile Ticket-, Artikel- und Anhangskennungen, Warteschlange, Status, Priorität, Typ, Bearbeitende, Kundschaft, Organisationen, Services, SLAs, Artikel, Anhangsmetadaten, dynamische Felder, Herkunft und Verlustdiagnosen. Anhangsdaten werden nie gespeichert. Da das Standard-GenericInterface keine portable Ticket-ACL liefert, haben ausdrücklich gelieferte GovOPlaN-ACL-Metadaten Vorrang; sonst greifen geprüfte Warteschlangen- oder eingeschränkte Profilvorgaben. Search prüft bei jedem Treffer Mandant, Profilstatus, Leserecht, aktuellen Konfigurationsnachweis und aktuelle ACL neu. Ein externes Ticket bleibt eine externe Referenz; fachliche Tickets, Helpdesk-Vorgänge und Cases werden ausschließlich von den jeweiligen Modulen erzeugt oder verknüpft. Externe Änderungen sind nur im Modus der gesteuerten Synchronisierung mit erkannter Update-Route, erwarteter Quellrevision und eindeutigem Idempotenzschlüssel zulässig. Ein unbekanntes Ergebnis muss vor einem erneuten Versuch am Anbieter geprüft werden. Profile mit mehr als 10000 Ticketkennungen oder zu vielen Änderungen am selben Zeitstempel müssen nach Warteschlangen enger aufgeteilt werden. Bei einem Ausfall bleiben bestehende Projektionen nur für weiterhin Berechtigte und mit sichtbarer veralteter Gesundheit verfügbar."
|
||||||
|
),
|
||||||
|
}
|
||||||
|
},
|
||||||
|
metadata={
|
||||||
|
"kind": "guide",
|
||||||
|
"help_contexts": ["connectors.admin.external-service-desk"],
|
||||||
|
"prerequisites": [
|
||||||
|
"An active governed Znuny/OTRS GenericInterface REST configuration exists.",
|
||||||
|
"Deployment-defined routes, queue partitions, authority, and fallback ACLs have been reviewed.",
|
||||||
|
"Tickets, Helpdesk, Cases, and Search remain optional capability-separated consumers.",
|
||||||
|
],
|
||||||
|
"outcome": "External tickets remain identity-stable, loss-visible, ACL-safe, recoverable, and semantically separate from GovOPlaN domain records.",
|
||||||
|
"verification": "Rediscover the profile, finish a keyed full run, run a keyed delta, inspect mapping diagnostics, verify one allowed and denied Search principal, and reconcile every outcome-unknown update before retry.",
|
||||||
|
},
|
||||||
|
),
|
||||||
|
)),
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
manifest = with_documentation_structured_translations(
|
||||||
|
manifest, locale="de", translations=GERMAN_STRUCTURED_TRANSLATIONS
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,503 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import base64
|
||||||
|
import json
|
||||||
|
from collections.abc import Mapping, Sequence
|
||||||
|
from dataclasses import dataclass
|
||||||
|
from typing import Any, Protocol
|
||||||
|
from urllib.parse import quote, urlencode, urljoin, urlsplit, urlunsplit
|
||||||
|
|
||||||
|
from govoplan_core.security.http_fetch import fetch_http
|
||||||
|
|
||||||
|
|
||||||
|
MAX_MEDIAWIKI_RESPONSE_BYTES = 10_000_000
|
||||||
|
|
||||||
|
|
||||||
|
class MediaWikiTransportError(RuntimeError):
|
||||||
|
def __init__(
|
||||||
|
self,
|
||||||
|
code: str,
|
||||||
|
message: str,
|
||||||
|
*,
|
||||||
|
retryable: bool = False,
|
||||||
|
outcome_unknown: bool = False,
|
||||||
|
) -> None:
|
||||||
|
super().__init__(message)
|
||||||
|
self.code = code
|
||||||
|
self.retryable = retryable
|
||||||
|
self.outcome_unknown = outcome_unknown
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True, slots=True)
|
||||||
|
class MediaWikiChangeBatch:
|
||||||
|
changes: tuple[Mapping[str, Any], ...]
|
||||||
|
next_cursor: str | None
|
||||||
|
complete: bool
|
||||||
|
high_watermark: str | None
|
||||||
|
evidence: Mapping[str, Any]
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True, slots=True)
|
||||||
|
class MediaWikiPublishResult:
|
||||||
|
page_id: str
|
||||||
|
revision_id: str
|
||||||
|
title: str
|
||||||
|
canonical_url: str | None
|
||||||
|
evidence: Mapping[str, Any]
|
||||||
|
|
||||||
|
|
||||||
|
class MediaWikiTransport(Protocol):
|
||||||
|
def discover(
|
||||||
|
self,
|
||||||
|
*,
|
||||||
|
endpoint_url: str,
|
||||||
|
credential: Mapping[str, Any] | None,
|
||||||
|
) -> Mapping[str, Any]: ...
|
||||||
|
|
||||||
|
def changes(
|
||||||
|
self,
|
||||||
|
*,
|
||||||
|
endpoint_url: str,
|
||||||
|
credential: Mapping[str, Any] | None,
|
||||||
|
cursor: str | None,
|
||||||
|
limit: int,
|
||||||
|
force_full: bool,
|
||||||
|
) -> MediaWikiChangeBatch: ...
|
||||||
|
|
||||||
|
def publish(
|
||||||
|
self,
|
||||||
|
*,
|
||||||
|
endpoint_url: str,
|
||||||
|
credential: Mapping[str, Any] | None,
|
||||||
|
title: str,
|
||||||
|
body: str,
|
||||||
|
summary: str,
|
||||||
|
expected_revision: str | None,
|
||||||
|
minor: bool,
|
||||||
|
) -> MediaWikiPublishResult: ...
|
||||||
|
|
||||||
|
|
||||||
|
class HttpMediaWikiTransport:
|
||||||
|
"""Bounded MediaWiki Action API transport with Core outbound policy enforcement."""
|
||||||
|
|
||||||
|
def discover(
|
||||||
|
self,
|
||||||
|
*,
|
||||||
|
endpoint_url: str,
|
||||||
|
credential: Mapping[str, Any] | None,
|
||||||
|
) -> Mapping[str, Any]:
|
||||||
|
return self._request(
|
||||||
|
endpoint_url,
|
||||||
|
credential=credential,
|
||||||
|
params={
|
||||||
|
"action": "query",
|
||||||
|
"meta": "siteinfo|userinfo",
|
||||||
|
"siprop": "general|extensions|namespaces|namespacealiases|rightsinfo",
|
||||||
|
"uiprop": "rights|groups",
|
||||||
|
"format": "json",
|
||||||
|
"formatversion": "2",
|
||||||
|
"curtimestamp": "1",
|
||||||
|
},
|
||||||
|
)
|
||||||
|
|
||||||
|
def changes(
|
||||||
|
self,
|
||||||
|
*,
|
||||||
|
endpoint_url: str,
|
||||||
|
credential: Mapping[str, Any] | None,
|
||||||
|
cursor: str | None,
|
||||||
|
limit: int,
|
||||||
|
force_full: bool,
|
||||||
|
) -> MediaWikiChangeBatch:
|
||||||
|
if force_full:
|
||||||
|
listing = self._request(
|
||||||
|
endpoint_url,
|
||||||
|
credential=credential,
|
||||||
|
params={
|
||||||
|
"action": "query",
|
||||||
|
"list": "allpages",
|
||||||
|
"aplimit": str(limit),
|
||||||
|
"apcontinue": cursor or "",
|
||||||
|
"apfilterredir": "all",
|
||||||
|
"format": "json",
|
||||||
|
"formatversion": "2",
|
||||||
|
"curtimestamp": "1",
|
||||||
|
},
|
||||||
|
)
|
||||||
|
rows = _list(listing, "query", "allpages")
|
||||||
|
page_ids = tuple(
|
||||||
|
str(item.get("pageid"))
|
||||||
|
for item in rows
|
||||||
|
if isinstance(item, Mapping) and item.get("pageid") is not None
|
||||||
|
)
|
||||||
|
changes = self._page_details(
|
||||||
|
endpoint_url,
|
||||||
|
credential=credential,
|
||||||
|
page_ids=page_ids,
|
||||||
|
)
|
||||||
|
next_cursor = _continue_token(listing, "apcontinue")
|
||||||
|
return MediaWikiChangeBatch(
|
||||||
|
changes=changes,
|
||||||
|
next_cursor=next_cursor,
|
||||||
|
complete=next_cursor is None,
|
||||||
|
high_watermark=_optional_text(listing.get("curtimestamp")),
|
||||||
|
evidence={
|
||||||
|
"mode": "backfill",
|
||||||
|
"listed": len(rows),
|
||||||
|
"resolved": len(changes),
|
||||||
|
},
|
||||||
|
)
|
||||||
|
|
||||||
|
listing = self._request(
|
||||||
|
endpoint_url,
|
||||||
|
credential=credential,
|
||||||
|
params={
|
||||||
|
"action": "query",
|
||||||
|
"list": "recentchanges",
|
||||||
|
"rclimit": str(limit),
|
||||||
|
"rccontinue": cursor or "",
|
||||||
|
"rcdir": "newer",
|
||||||
|
"rcprop": "title|ids|sizes|flags|user|timestamp|loginfo|tags",
|
||||||
|
"rctype": "edit|new|log",
|
||||||
|
"format": "json",
|
||||||
|
"formatversion": "2",
|
||||||
|
"curtimestamp": "1",
|
||||||
|
},
|
||||||
|
)
|
||||||
|
recent = tuple(
|
||||||
|
item
|
||||||
|
for item in _list(listing, "query", "recentchanges")
|
||||||
|
if isinstance(item, Mapping)
|
||||||
|
)
|
||||||
|
page_ids = tuple(
|
||||||
|
dict.fromkeys(
|
||||||
|
str(item.get("pageid"))
|
||||||
|
for item in recent
|
||||||
|
if item.get("pageid") is not None
|
||||||
|
and str(item.get("logtype") or "") != "delete"
|
||||||
|
)
|
||||||
|
)
|
||||||
|
resolved = {
|
||||||
|
str(item.get("pageid")): item
|
||||||
|
for item in self._page_details(
|
||||||
|
endpoint_url,
|
||||||
|
credential=credential,
|
||||||
|
page_ids=page_ids,
|
||||||
|
)
|
||||||
|
}
|
||||||
|
changes: list[Mapping[str, Any]] = []
|
||||||
|
for item in recent:
|
||||||
|
page_id = _optional_text(item.get("pageid"))
|
||||||
|
if str(item.get("logtype") or "") == "delete":
|
||||||
|
changes.append(
|
||||||
|
{
|
||||||
|
"change_kind": "delete",
|
||||||
|
"pageid": page_id or f"log:{item.get('logid')}",
|
||||||
|
"title": _optional_text(item.get("title")) or "Deleted page",
|
||||||
|
"ns": item.get("ns"),
|
||||||
|
"timestamp": item.get("timestamp"),
|
||||||
|
"logid": item.get("logid"),
|
||||||
|
"change_cursor": _change_cursor(item),
|
||||||
|
}
|
||||||
|
)
|
||||||
|
continue
|
||||||
|
page = dict(resolved.get(page_id or "") or item)
|
||||||
|
page["change_kind"] = "upsert"
|
||||||
|
page["change_cursor"] = _change_cursor(item)
|
||||||
|
page["recent_change"] = _safe_recent_change(item)
|
||||||
|
changes.append(page)
|
||||||
|
next_cursor = _continue_token(listing, "rccontinue")
|
||||||
|
return MediaWikiChangeBatch(
|
||||||
|
changes=tuple(changes),
|
||||||
|
next_cursor=next_cursor,
|
||||||
|
complete=next_cursor is None,
|
||||||
|
high_watermark=_optional_text(listing.get("curtimestamp")),
|
||||||
|
evidence={
|
||||||
|
"mode": "delta",
|
||||||
|
"listed": len(recent),
|
||||||
|
"resolved": len(resolved),
|
||||||
|
},
|
||||||
|
)
|
||||||
|
|
||||||
|
def publish(
|
||||||
|
self,
|
||||||
|
*,
|
||||||
|
endpoint_url: str,
|
||||||
|
credential: Mapping[str, Any] | None,
|
||||||
|
title: str,
|
||||||
|
body: str,
|
||||||
|
summary: str,
|
||||||
|
expected_revision: str | None,
|
||||||
|
minor: bool,
|
||||||
|
) -> MediaWikiPublishResult:
|
||||||
|
if not credential:
|
||||||
|
raise MediaWikiTransportError(
|
||||||
|
"credential_required",
|
||||||
|
"Publishing requires a governed credential envelope.",
|
||||||
|
)
|
||||||
|
csrf_token = _optional_text(credential.get("csrf_token"))
|
||||||
|
if not csrf_token:
|
||||||
|
token_payload = self._request(
|
||||||
|
endpoint_url,
|
||||||
|
credential=credential,
|
||||||
|
params={
|
||||||
|
"action": "query",
|
||||||
|
"meta": "tokens",
|
||||||
|
"type": "csrf",
|
||||||
|
"format": "json",
|
||||||
|
"formatversion": "2",
|
||||||
|
},
|
||||||
|
)
|
||||||
|
query = token_payload.get("query")
|
||||||
|
tokens = query.get("tokens") if isinstance(query, Mapping) else None
|
||||||
|
csrf_token = (
|
||||||
|
_optional_text(tokens.get("csrftoken"))
|
||||||
|
if isinstance(tokens, Mapping)
|
||||||
|
else None
|
||||||
|
)
|
||||||
|
if not csrf_token:
|
||||||
|
raise MediaWikiTransportError(
|
||||||
|
"csrf_token_unavailable",
|
||||||
|
"The provider did not issue a CSRF token for the configured credential.",
|
||||||
|
)
|
||||||
|
parameters: dict[str, str] = {
|
||||||
|
"action": "edit",
|
||||||
|
"title": title,
|
||||||
|
"text": body,
|
||||||
|
"summary": summary,
|
||||||
|
"token": csrf_token,
|
||||||
|
"format": "json",
|
||||||
|
"formatversion": "2",
|
||||||
|
}
|
||||||
|
if minor:
|
||||||
|
parameters["minor"] = "1"
|
||||||
|
if expected_revision:
|
||||||
|
parameters["baserevid"] = expected_revision
|
||||||
|
payload = self._request(
|
||||||
|
endpoint_url,
|
||||||
|
credential=credential,
|
||||||
|
params=parameters,
|
||||||
|
method="POST",
|
||||||
|
)
|
||||||
|
edit = payload.get("edit")
|
||||||
|
if not isinstance(edit, Mapping) or str(edit.get("result")) != "Success":
|
||||||
|
raise MediaWikiTransportError(
|
||||||
|
"publish_rejected",
|
||||||
|
"MediaWiki rejected the page publication.",
|
||||||
|
)
|
||||||
|
page_id = _required_text(edit.get("pageid"), "MediaWiki omitted the page id")
|
||||||
|
revision_id = _required_text(
|
||||||
|
edit.get("newrevid"), "MediaWiki omitted the accepted revision id"
|
||||||
|
)
|
||||||
|
canonical_url = _canonical_page_url(endpoint_url, title)
|
||||||
|
return MediaWikiPublishResult(
|
||||||
|
page_id=page_id,
|
||||||
|
revision_id=revision_id,
|
||||||
|
title=_optional_text(edit.get("title")) or title,
|
||||||
|
canonical_url=canonical_url,
|
||||||
|
evidence={
|
||||||
|
"result": "Success",
|
||||||
|
"page_id": page_id,
|
||||||
|
"revision_id": revision_id,
|
||||||
|
"old_revision_id": _optional_text(edit.get("oldrevid")),
|
||||||
|
"new_page": bool(edit.get("new")),
|
||||||
|
},
|
||||||
|
)
|
||||||
|
|
||||||
|
def _page_details(
|
||||||
|
self,
|
||||||
|
endpoint_url: str,
|
||||||
|
*,
|
||||||
|
credential: Mapping[str, Any] | None,
|
||||||
|
page_ids: Sequence[str],
|
||||||
|
) -> tuple[Mapping[str, Any], ...]:
|
||||||
|
if not page_ids:
|
||||||
|
return ()
|
||||||
|
payload = self._request(
|
||||||
|
endpoint_url,
|
||||||
|
credential=credential,
|
||||||
|
params={
|
||||||
|
"action": "query",
|
||||||
|
"pageids": "|".join(page_ids),
|
||||||
|
"prop": "info|revisions|categories|links|images|pageprops",
|
||||||
|
"inprop": "url|displaytitle",
|
||||||
|
"rvlimit": "1",
|
||||||
|
"rvprop": "ids|timestamp|user|comment|content|contentmodel|sha1|flags",
|
||||||
|
"rvslots": "main",
|
||||||
|
"cllimit": "max",
|
||||||
|
"pllimit": "max",
|
||||||
|
"imlimit": "max",
|
||||||
|
"format": "json",
|
||||||
|
"formatversion": "2",
|
||||||
|
},
|
||||||
|
)
|
||||||
|
return tuple(
|
||||||
|
item
|
||||||
|
for item in _list(payload, "query", "pages")
|
||||||
|
if isinstance(item, Mapping)
|
||||||
|
)
|
||||||
|
|
||||||
|
def _request(
|
||||||
|
self,
|
||||||
|
endpoint_url: str,
|
||||||
|
*,
|
||||||
|
credential: Mapping[str, Any] | None,
|
||||||
|
params: Mapping[str, str],
|
||||||
|
method: str = "GET",
|
||||||
|
) -> Mapping[str, Any]:
|
||||||
|
api_url = _api_url(endpoint_url)
|
||||||
|
headers = {
|
||||||
|
"Accept": "application/json",
|
||||||
|
"User-Agent": "GovOPlaN-Connectors/MediaWiki",
|
||||||
|
**_auth_headers(credential),
|
||||||
|
}
|
||||||
|
body = None
|
||||||
|
target_url = api_url
|
||||||
|
if method == "POST":
|
||||||
|
body = urlencode(params).encode("utf-8")
|
||||||
|
headers["Content-Type"] = "application/x-www-form-urlencoded"
|
||||||
|
else:
|
||||||
|
target_url = f"{api_url}?{urlencode({key: value for key, value in params.items() if value != ''})}"
|
||||||
|
try:
|
||||||
|
response = fetch_http(
|
||||||
|
target_url,
|
||||||
|
method=method,
|
||||||
|
headers=headers,
|
||||||
|
body=body,
|
||||||
|
max_bytes=MAX_MEDIAWIKI_RESPONSE_BYTES,
|
||||||
|
timeout=30,
|
||||||
|
label="MediaWiki Action API URL",
|
||||||
|
)
|
||||||
|
except Exception as exc:
|
||||||
|
raise MediaWikiTransportError(
|
||||||
|
"transport_unavailable",
|
||||||
|
"MediaWiki transport failed before a valid response was received.",
|
||||||
|
retryable=True,
|
||||||
|
outcome_unknown=method == "POST",
|
||||||
|
) from exc
|
||||||
|
if response.status < 200 or response.status >= 300:
|
||||||
|
raise MediaWikiTransportError(
|
||||||
|
"http_error",
|
||||||
|
f"MediaWiki returned HTTP {response.status}.",
|
||||||
|
retryable=response.status >= 500,
|
||||||
|
outcome_unknown=method == "POST" and response.status >= 500,
|
||||||
|
)
|
||||||
|
try:
|
||||||
|
payload = json.loads(response.body)
|
||||||
|
except (UnicodeDecodeError, json.JSONDecodeError) as exc:
|
||||||
|
raise MediaWikiTransportError(
|
||||||
|
"invalid_json",
|
||||||
|
"MediaWiki returned an invalid JSON response.",
|
||||||
|
outcome_unknown=method == "POST",
|
||||||
|
) from exc
|
||||||
|
if not isinstance(payload, Mapping):
|
||||||
|
raise MediaWikiTransportError(
|
||||||
|
"invalid_response",
|
||||||
|
"MediaWiki returned an unsupported response shape.",
|
||||||
|
outcome_unknown=method == "POST",
|
||||||
|
)
|
||||||
|
error = payload.get("error")
|
||||||
|
if isinstance(error, Mapping):
|
||||||
|
code = _optional_text(error.get("code")) or "provider_error"
|
||||||
|
raise MediaWikiTransportError(
|
||||||
|
code,
|
||||||
|
_optional_text(error.get("info")) or "MediaWiki rejected the request.",
|
||||||
|
retryable=code in {"maxlag", "readonly", "ratelimited"},
|
||||||
|
)
|
||||||
|
return payload
|
||||||
|
|
||||||
|
|
||||||
|
def _api_url(endpoint_url: str) -> str:
|
||||||
|
normalized = endpoint_url.strip().rstrip("/")
|
||||||
|
parsed = urlsplit(normalized)
|
||||||
|
if parsed.path.endswith("/api.php"):
|
||||||
|
return normalized
|
||||||
|
path = f"{parsed.path.rstrip('/')}/api.php"
|
||||||
|
return urlunsplit((parsed.scheme, parsed.netloc, path, "", ""))
|
||||||
|
|
||||||
|
|
||||||
|
def _canonical_page_url(endpoint_url: str, title: str) -> str:
|
||||||
|
normalized = endpoint_url.strip().rstrip("/")
|
||||||
|
parsed = urlsplit(normalized)
|
||||||
|
root_path = parsed.path
|
||||||
|
if root_path.endswith("/api.php"):
|
||||||
|
root_path = root_path[: -len("/api.php")]
|
||||||
|
base = urlunsplit((parsed.scheme, parsed.netloc, f"{root_path.rstrip('/')}/", "", ""))
|
||||||
|
return urljoin(base, f"wiki/{quote(title.replace(' ', '_'), safe=':_-./~')}")
|
||||||
|
|
||||||
|
|
||||||
|
def _auth_headers(credential: Mapping[str, Any] | None) -> dict[str, str]:
|
||||||
|
if not credential:
|
||||||
|
return {}
|
||||||
|
token = _optional_text(
|
||||||
|
credential.get("bearer_token")
|
||||||
|
or credential.get("access_token")
|
||||||
|
or credential.get("token")
|
||||||
|
)
|
||||||
|
if token:
|
||||||
|
return {"Authorization": f"Bearer {token}"}
|
||||||
|
username = _optional_text(credential.get("username") or credential.get("user"))
|
||||||
|
password = _optional_text(credential.get("password"))
|
||||||
|
if username and password:
|
||||||
|
encoded = base64.b64encode(f"{username}:{password}".encode()).decode("ascii")
|
||||||
|
return {"Authorization": f"Basic {encoded}"}
|
||||||
|
return {}
|
||||||
|
|
||||||
|
|
||||||
|
def _list(payload: Mapping[str, Any], *path: str) -> list[Any]:
|
||||||
|
value: Any = payload
|
||||||
|
for part in path:
|
||||||
|
if not isinstance(value, Mapping):
|
||||||
|
return []
|
||||||
|
value = value.get(part)
|
||||||
|
return list(value) if isinstance(value, list) else []
|
||||||
|
|
||||||
|
|
||||||
|
def _continue_token(payload: Mapping[str, Any], name: str) -> str | None:
|
||||||
|
value = payload.get("continue")
|
||||||
|
return _optional_text(value.get(name)) if isinstance(value, Mapping) else None
|
||||||
|
|
||||||
|
|
||||||
|
def _change_cursor(change: Mapping[str, Any]) -> str:
|
||||||
|
for name in ("rcid", "logid", "revid", "old_revid"):
|
||||||
|
if change.get(name) is not None:
|
||||||
|
return f"{name}:{change[name]}"
|
||||||
|
return _optional_text(change.get("timestamp")) or "unknown"
|
||||||
|
|
||||||
|
|
||||||
|
def _safe_recent_change(change: Mapping[str, Any]) -> dict[str, Any]:
|
||||||
|
allowed = (
|
||||||
|
"type",
|
||||||
|
"ns",
|
||||||
|
"title",
|
||||||
|
"pageid",
|
||||||
|
"revid",
|
||||||
|
"old_revid",
|
||||||
|
"timestamp",
|
||||||
|
"logtype",
|
||||||
|
"logaction",
|
||||||
|
"tags",
|
||||||
|
)
|
||||||
|
return {key: change[key] for key in allowed if key in change}
|
||||||
|
|
||||||
|
|
||||||
|
def _required_text(value: object, message: str) -> str:
|
||||||
|
normalized = _optional_text(value)
|
||||||
|
if not normalized:
|
||||||
|
raise MediaWikiTransportError("invalid_response", message)
|
||||||
|
return normalized
|
||||||
|
|
||||||
|
|
||||||
|
def _optional_text(value: object) -> str | None:
|
||||||
|
normalized = str(value or "").strip()
|
||||||
|
return normalized or None
|
||||||
|
|
||||||
|
|
||||||
|
__all__ = [
|
||||||
|
"HttpMediaWikiTransport",
|
||||||
|
"MAX_MEDIAWIKI_RESPONSE_BYTES",
|
||||||
|
"MediaWikiChangeBatch",
|
||||||
|
"MediaWikiPublishResult",
|
||||||
|
"MediaWikiTransport",
|
||||||
|
"MediaWikiTransportError",
|
||||||
|
]
|
||||||
+207
@@ -0,0 +1,207 @@
|
|||||||
|
"""MediaWiki and BlueSpice knowledge connector state
|
||||||
|
|
||||||
|
Revision ID: b9e0f1a2c3d4
|
||||||
|
Revises: a8d9e0f1b2c3
|
||||||
|
Create Date: 2026-08-22 14:00:00.000000
|
||||||
|
"""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from alembic import op
|
||||||
|
import sqlalchemy as sa
|
||||||
|
|
||||||
|
|
||||||
|
revision = "b9e0f1a2c3d4"
|
||||||
|
down_revision = "a8d9e0f1b2c3"
|
||||||
|
branch_labels = None
|
||||||
|
depends_on = None
|
||||||
|
|
||||||
|
|
||||||
|
def upgrade() -> None:
|
||||||
|
op.create_table(
|
||||||
|
"connector_knowledge_profiles",
|
||||||
|
sa.Column("id", sa.String(length=36), nullable=False),
|
||||||
|
sa.Column("tenant_id", sa.String(length=36), nullable=False),
|
||||||
|
sa.Column("configuration_id", sa.String(length=36), nullable=False),
|
||||||
|
sa.Column("status", sa.String(length=30), nullable=False),
|
||||||
|
sa.Column("product", sa.String(length=50), nullable=False),
|
||||||
|
sa.Column("product_version", sa.String(length=100), nullable=True),
|
||||||
|
sa.Column("desired_maturity", sa.String(length=30), nullable=False),
|
||||||
|
sa.Column("discovered_maturity", sa.String(length=30), nullable=False),
|
||||||
|
sa.Column("source_authority_mode", sa.String(length=40), nullable=False),
|
||||||
|
sa.Column("default_visibility", sa.String(length=30), nullable=False),
|
||||||
|
sa.Column("default_acl_tokens", sa.JSON(), nullable=False),
|
||||||
|
sa.Column("namespace_mappings", sa.JSON(), nullable=False),
|
||||||
|
sa.Column("capabilities", sa.JSON(), nullable=False),
|
||||||
|
sa.Column("discovery_revision", sa.String(length=255), nullable=True),
|
||||||
|
sa.Column("discovery_evidence", sa.JSON(), nullable=False),
|
||||||
|
sa.Column("health_status", sa.String(length=30), nullable=False),
|
||||||
|
sa.Column("health_details", sa.JSON(), nullable=False),
|
||||||
|
sa.Column("discovered_at", sa.DateTime(timezone=True), nullable=True),
|
||||||
|
sa.Column("last_sync_cursor", sa.String(length=500), nullable=True),
|
||||||
|
sa.Column("last_high_watermark", sa.String(length=500), nullable=True),
|
||||||
|
sa.Column("resource_revision", sa.Integer(), nullable=False),
|
||||||
|
sa.Column("updated_by", sa.String(length=255), nullable=True),
|
||||||
|
sa.Column("created_at", sa.DateTime(timezone=True), nullable=False),
|
||||||
|
sa.Column("updated_at", sa.DateTime(timezone=True), nullable=False),
|
||||||
|
sa.ForeignKeyConstraint(
|
||||||
|
["configuration_id"],
|
||||||
|
["connector_configurations.id"],
|
||||||
|
name=op.f(
|
||||||
|
"fk_connector_knowledge_profiles_configuration_id_connector_configurations"
|
||||||
|
),
|
||||||
|
ondelete="RESTRICT",
|
||||||
|
),
|
||||||
|
sa.PrimaryKeyConstraint(
|
||||||
|
"id", name=op.f("pk_connector_knowledge_profiles")
|
||||||
|
),
|
||||||
|
sa.UniqueConstraint(
|
||||||
|
"tenant_id",
|
||||||
|
"configuration_id",
|
||||||
|
name="uq_connector_knowledge_profile_configuration",
|
||||||
|
),
|
||||||
|
)
|
||||||
|
for name, columns in (
|
||||||
|
("ix_connector_knowledge_profiles_tenant_id", ["tenant_id"]),
|
||||||
|
("ix_connector_knowledge_profiles_configuration_id", ["configuration_id"]),
|
||||||
|
("ix_connector_knowledge_profiles_status", ["status"]),
|
||||||
|
("ix_connector_knowledge_profiles_product", ["product"]),
|
||||||
|
("ix_connector_knowledge_profiles_discovery_revision", ["discovery_revision"]),
|
||||||
|
("ix_connector_knowledge_profiles_health_status", ["health_status"]),
|
||||||
|
("ix_connector_knowledge_profiles_updated_by", ["updated_by"]),
|
||||||
|
):
|
||||||
|
op.create_index(op.f(name), "connector_knowledge_profiles", columns)
|
||||||
|
op.create_index(
|
||||||
|
"ix_connector_knowledge_profiles_tenant_status",
|
||||||
|
"connector_knowledge_profiles",
|
||||||
|
["tenant_id", "status"],
|
||||||
|
)
|
||||||
|
|
||||||
|
op.create_table(
|
||||||
|
"connector_knowledge_objects",
|
||||||
|
sa.Column("id", sa.String(length=36), nullable=False),
|
||||||
|
sa.Column("tenant_id", sa.String(length=36), nullable=False),
|
||||||
|
sa.Column("profile_id", sa.String(length=36), nullable=False),
|
||||||
|
sa.Column("object_type", sa.String(length=40), nullable=False),
|
||||||
|
sa.Column("external_id", sa.String(length=255), nullable=False),
|
||||||
|
sa.Column("external_page_id", sa.String(length=255), nullable=True),
|
||||||
|
sa.Column("external_revision_id", sa.String(length=255), nullable=True),
|
||||||
|
sa.Column("namespace_id", sa.Integer(), nullable=True),
|
||||||
|
sa.Column("title", sa.String(length=500), nullable=False),
|
||||||
|
sa.Column("canonical_url", sa.String(length=1500), nullable=True),
|
||||||
|
sa.Column("status", sa.String(length=30), nullable=False),
|
||||||
|
sa.Column("redirect_target_external_id", sa.String(length=255), nullable=True),
|
||||||
|
sa.Column("source_revision", sa.String(length=255), nullable=False),
|
||||||
|
sa.Column("content_hash", sa.String(length=64), nullable=False),
|
||||||
|
sa.Column("visibility", sa.String(length=30), nullable=False),
|
||||||
|
sa.Column("acl_tokens", sa.JSON(), nullable=False),
|
||||||
|
sa.Column("mapped_data", sa.JSON(), nullable=False),
|
||||||
|
sa.Column("provenance", sa.JSON(), nullable=False),
|
||||||
|
sa.Column("change_cursor", sa.String(length=500), nullable=True),
|
||||||
|
sa.Column("source_updated_at", sa.DateTime(timezone=True), nullable=True),
|
||||||
|
sa.Column("observed_at", sa.DateTime(timezone=True), nullable=False),
|
||||||
|
sa.Column("resource_revision", sa.Integer(), nullable=False),
|
||||||
|
sa.Column("created_at", sa.DateTime(timezone=True), nullable=False),
|
||||||
|
sa.Column("updated_at", sa.DateTime(timezone=True), nullable=False),
|
||||||
|
sa.ForeignKeyConstraint(
|
||||||
|
["profile_id"],
|
||||||
|
["connector_knowledge_profiles.id"],
|
||||||
|
name=op.f(
|
||||||
|
"fk_connector_knowledge_objects_profile_id_connector_knowledge_profiles"
|
||||||
|
),
|
||||||
|
ondelete="CASCADE",
|
||||||
|
),
|
||||||
|
sa.PrimaryKeyConstraint(
|
||||||
|
"id", name=op.f("pk_connector_knowledge_objects")
|
||||||
|
),
|
||||||
|
sa.UniqueConstraint(
|
||||||
|
"profile_id",
|
||||||
|
"object_type",
|
||||||
|
"external_id",
|
||||||
|
name="uq_connector_knowledge_object_identity",
|
||||||
|
),
|
||||||
|
)
|
||||||
|
for name, columns in (
|
||||||
|
("ix_connector_knowledge_objects_tenant_id", ["tenant_id"]),
|
||||||
|
("ix_connector_knowledge_objects_profile_id", ["profile_id"]),
|
||||||
|
("ix_connector_knowledge_objects_object_type", ["object_type"]),
|
||||||
|
("ix_connector_knowledge_objects_external_page_id", ["external_page_id"]),
|
||||||
|
("ix_connector_knowledge_objects_external_revision_id", ["external_revision_id"]),
|
||||||
|
("ix_connector_knowledge_objects_namespace_id", ["namespace_id"]),
|
||||||
|
("ix_connector_knowledge_objects_status", ["status"]),
|
||||||
|
("ix_connector_knowledge_objects_content_hash", ["content_hash"]),
|
||||||
|
("ix_connector_knowledge_objects_change_cursor", ["change_cursor"]),
|
||||||
|
("ix_connector_knowledge_objects_source_updated_at", ["source_updated_at"]),
|
||||||
|
("ix_connector_knowledge_objects_observed_at", ["observed_at"]),
|
||||||
|
):
|
||||||
|
op.create_index(op.f(name), "connector_knowledge_objects", columns)
|
||||||
|
op.create_index(
|
||||||
|
"ix_connector_knowledge_objects_tenant_profile_status",
|
||||||
|
"connector_knowledge_objects",
|
||||||
|
["tenant_id", "profile_id", "status"],
|
||||||
|
)
|
||||||
|
op.create_index(
|
||||||
|
"ix_connector_knowledge_objects_tenant_updated",
|
||||||
|
"connector_knowledge_objects",
|
||||||
|
["tenant_id", "source_updated_at"],
|
||||||
|
)
|
||||||
|
|
||||||
|
op.create_table(
|
||||||
|
"connector_knowledge_sync_runs",
|
||||||
|
sa.Column("id", sa.String(length=36), nullable=False),
|
||||||
|
sa.Column("tenant_id", sa.String(length=36), nullable=False),
|
||||||
|
sa.Column("profile_id", sa.String(length=36), nullable=False),
|
||||||
|
sa.Column("mode", sa.String(length=40), nullable=False),
|
||||||
|
sa.Column("idempotency_key", sa.String(length=255), nullable=False),
|
||||||
|
sa.Column("request_hash", sa.String(length=64), nullable=False),
|
||||||
|
sa.Column("status", sa.String(length=30), nullable=False),
|
||||||
|
sa.Column("cursor_before", sa.String(length=500), nullable=True),
|
||||||
|
sa.Column("cursor_after", sa.String(length=500), nullable=True),
|
||||||
|
sa.Column("high_watermark", sa.String(length=500), nullable=True),
|
||||||
|
sa.Column("counts", sa.JSON(), nullable=False),
|
||||||
|
sa.Column("effects", sa.JSON(), nullable=False),
|
||||||
|
sa.Column("diagnostics", sa.JSON(), nullable=False),
|
||||||
|
sa.Column("provenance", sa.JSON(), nullable=False),
|
||||||
|
sa.Column("created_by", sa.String(length=255), nullable=True),
|
||||||
|
sa.Column("started_at", sa.DateTime(timezone=True), nullable=False),
|
||||||
|
sa.Column("finished_at", sa.DateTime(timezone=True), nullable=True),
|
||||||
|
sa.Column("created_at", sa.DateTime(timezone=True), nullable=False),
|
||||||
|
sa.Column("updated_at", sa.DateTime(timezone=True), nullable=False),
|
||||||
|
sa.ForeignKeyConstraint(
|
||||||
|
["profile_id"],
|
||||||
|
["connector_knowledge_profiles.id"],
|
||||||
|
name=op.f(
|
||||||
|
"fk_connector_knowledge_sync_runs_profile_id_connector_knowledge_profiles"
|
||||||
|
),
|
||||||
|
ondelete="CASCADE",
|
||||||
|
),
|
||||||
|
sa.PrimaryKeyConstraint(
|
||||||
|
"id", name=op.f("pk_connector_knowledge_sync_runs")
|
||||||
|
),
|
||||||
|
sa.UniqueConstraint(
|
||||||
|
"tenant_id",
|
||||||
|
"profile_id",
|
||||||
|
"mode",
|
||||||
|
"idempotency_key",
|
||||||
|
name="uq_connector_knowledge_sync_run_idempotency",
|
||||||
|
),
|
||||||
|
)
|
||||||
|
for name, columns in (
|
||||||
|
("ix_connector_knowledge_sync_runs_tenant_id", ["tenant_id"]),
|
||||||
|
("ix_connector_knowledge_sync_runs_profile_id", ["profile_id"]),
|
||||||
|
("ix_connector_knowledge_sync_runs_mode", ["mode"]),
|
||||||
|
("ix_connector_knowledge_sync_runs_status", ["status"]),
|
||||||
|
("ix_connector_knowledge_sync_runs_created_by", ["created_by"]),
|
||||||
|
("ix_connector_knowledge_sync_runs_started_at", ["started_at"]),
|
||||||
|
):
|
||||||
|
op.create_index(op.f(name), "connector_knowledge_sync_runs", columns)
|
||||||
|
op.create_index(
|
||||||
|
"ix_connector_knowledge_sync_runs_profile_started",
|
||||||
|
"connector_knowledge_sync_runs",
|
||||||
|
["tenant_id", "profile_id", "started_at"],
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def downgrade() -> None:
|
||||||
|
op.drop_table("connector_knowledge_sync_runs")
|
||||||
|
op.drop_table("connector_knowledge_objects")
|
||||||
|
op.drop_table("connector_knowledge_profiles")
|
||||||
+201
@@ -0,0 +1,201 @@
|
|||||||
|
"""Znuny and OTRS-compatible service-desk connector state
|
||||||
|
|
||||||
|
Revision ID: c0f1a2b3c4d5
|
||||||
|
Revises: b9e0f1a2c3d4
|
||||||
|
Create Date: 2026-08-22 15:15:00.000000
|
||||||
|
"""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from alembic import op
|
||||||
|
import sqlalchemy as sa
|
||||||
|
|
||||||
|
|
||||||
|
revision = "c0f1a2b3c4d5"
|
||||||
|
down_revision = "b9e0f1a2c3d4"
|
||||||
|
branch_labels = None
|
||||||
|
depends_on = None
|
||||||
|
|
||||||
|
|
||||||
|
def upgrade() -> None:
|
||||||
|
op.create_table(
|
||||||
|
"connector_service_desk_profiles",
|
||||||
|
sa.Column("id", sa.String(length=36), nullable=False),
|
||||||
|
sa.Column("tenant_id", sa.String(length=36), nullable=False),
|
||||||
|
sa.Column("configuration_id", sa.String(length=36), nullable=False),
|
||||||
|
sa.Column("status", sa.String(length=30), nullable=False),
|
||||||
|
sa.Column("integration_mode", sa.String(length=30), nullable=False),
|
||||||
|
sa.Column("product", sa.String(length=50), nullable=False),
|
||||||
|
sa.Column("product_version", sa.String(length=100), nullable=True),
|
||||||
|
sa.Column("desired_maturity", sa.String(length=30), nullable=False),
|
||||||
|
sa.Column("discovered_maturity", sa.String(length=30), nullable=False),
|
||||||
|
sa.Column("source_authority_mode", sa.String(length=40), nullable=False),
|
||||||
|
sa.Column("default_visibility", sa.String(length=30), nullable=False),
|
||||||
|
sa.Column("default_acl_tokens", sa.JSON(), nullable=False),
|
||||||
|
sa.Column("routes", sa.JSON(), nullable=False),
|
||||||
|
sa.Column("queue_mappings", sa.JSON(), nullable=False),
|
||||||
|
sa.Column("dynamic_field_mappings", sa.JSON(), nullable=False),
|
||||||
|
sa.Column("capabilities", sa.JSON(), nullable=False),
|
||||||
|
sa.Column("discovery_revision", sa.String(length=255), nullable=True),
|
||||||
|
sa.Column("discovered_configuration_revision", sa.Integer(), nullable=True),
|
||||||
|
sa.Column("discovered_configuration_hash", sa.String(length=64), nullable=True),
|
||||||
|
sa.Column("discovery_evidence", sa.JSON(), nullable=False),
|
||||||
|
sa.Column("health_status", sa.String(length=30), nullable=False),
|
||||||
|
sa.Column("health_details", sa.JSON(), nullable=False),
|
||||||
|
sa.Column("discovered_at", sa.DateTime(timezone=True), nullable=True),
|
||||||
|
sa.Column("last_sync_cursor", sa.String(length=4000), nullable=True),
|
||||||
|
sa.Column("last_high_watermark", sa.String(length=500), nullable=True),
|
||||||
|
sa.Column("resource_revision", sa.Integer(), nullable=False),
|
||||||
|
sa.Column("updated_by", sa.String(length=255), nullable=True),
|
||||||
|
sa.Column("created_at", sa.DateTime(timezone=True), nullable=False),
|
||||||
|
sa.Column("updated_at", sa.DateTime(timezone=True), nullable=False),
|
||||||
|
sa.ForeignKeyConstraint(
|
||||||
|
["configuration_id"],
|
||||||
|
["connector_configurations.id"],
|
||||||
|
name=op.f(
|
||||||
|
"fk_connector_service_desk_profiles_configuration_id_connector_configurations"
|
||||||
|
),
|
||||||
|
ondelete="RESTRICT",
|
||||||
|
),
|
||||||
|
sa.PrimaryKeyConstraint("id", name=op.f("pk_connector_service_desk_profiles")),
|
||||||
|
sa.UniqueConstraint(
|
||||||
|
"tenant_id",
|
||||||
|
"configuration_id",
|
||||||
|
name="uq_connector_service_desk_profile_configuration",
|
||||||
|
),
|
||||||
|
)
|
||||||
|
for name, columns in (
|
||||||
|
("ix_connector_service_desk_profiles_tenant_id", ["tenant_id"]),
|
||||||
|
("ix_connector_service_desk_profiles_configuration_id", ["configuration_id"]),
|
||||||
|
("ix_connector_service_desk_profiles_status", ["status"]),
|
||||||
|
("ix_connector_service_desk_profiles_integration_mode", ["integration_mode"]),
|
||||||
|
("ix_connector_service_desk_profiles_product", ["product"]),
|
||||||
|
("ix_connector_service_desk_profiles_discovery_revision", ["discovery_revision"]),
|
||||||
|
("ix_connector_service_desk_profiles_health_status", ["health_status"]),
|
||||||
|
("ix_connector_service_desk_profiles_updated_by", ["updated_by"]),
|
||||||
|
):
|
||||||
|
op.create_index(op.f(name), "connector_service_desk_profiles", columns)
|
||||||
|
op.create_index(
|
||||||
|
"ix_connector_service_desk_profiles_tenant_status",
|
||||||
|
"connector_service_desk_profiles",
|
||||||
|
["tenant_id", "status"],
|
||||||
|
)
|
||||||
|
|
||||||
|
op.create_table(
|
||||||
|
"connector_service_desk_objects",
|
||||||
|
sa.Column("id", sa.String(length=36), nullable=False),
|
||||||
|
sa.Column("tenant_id", sa.String(length=36), nullable=False),
|
||||||
|
sa.Column("profile_id", sa.String(length=36), nullable=False),
|
||||||
|
sa.Column("object_type", sa.String(length=40), nullable=False),
|
||||||
|
sa.Column("external_id", sa.String(length=255), nullable=False),
|
||||||
|
sa.Column("external_ticket_number", sa.String(length=255), nullable=True),
|
||||||
|
sa.Column("title", sa.String(length=500), nullable=False),
|
||||||
|
sa.Column("canonical_url", sa.String(length=1500), nullable=True),
|
||||||
|
sa.Column("status", sa.String(length=30), nullable=False),
|
||||||
|
sa.Column("source_revision", sa.String(length=255), nullable=False),
|
||||||
|
sa.Column("content_hash", sa.String(length=64), nullable=False),
|
||||||
|
sa.Column("visibility", sa.String(length=30), nullable=False),
|
||||||
|
sa.Column("acl_tokens", sa.JSON(), nullable=False),
|
||||||
|
sa.Column("mapped_data", sa.JSON(), nullable=False),
|
||||||
|
sa.Column("provenance", sa.JSON(), nullable=False),
|
||||||
|
sa.Column("change_cursor", sa.String(length=4000), nullable=True),
|
||||||
|
sa.Column("source_updated_at", sa.DateTime(timezone=True), nullable=True),
|
||||||
|
sa.Column("observed_at", sa.DateTime(timezone=True), nullable=False),
|
||||||
|
sa.Column("resource_revision", sa.Integer(), nullable=False),
|
||||||
|
sa.Column("created_at", sa.DateTime(timezone=True), nullable=False),
|
||||||
|
sa.Column("updated_at", sa.DateTime(timezone=True), nullable=False),
|
||||||
|
sa.ForeignKeyConstraint(
|
||||||
|
["profile_id"],
|
||||||
|
["connector_service_desk_profiles.id"],
|
||||||
|
name=op.f(
|
||||||
|
"fk_connector_service_desk_objects_profile_id_connector_service_desk_profiles"
|
||||||
|
),
|
||||||
|
ondelete="CASCADE",
|
||||||
|
),
|
||||||
|
sa.PrimaryKeyConstraint("id", name=op.f("pk_connector_service_desk_objects")),
|
||||||
|
sa.UniqueConstraint(
|
||||||
|
"profile_id",
|
||||||
|
"object_type",
|
||||||
|
"external_id",
|
||||||
|
name="uq_connector_service_desk_object_identity",
|
||||||
|
),
|
||||||
|
)
|
||||||
|
for name, columns in (
|
||||||
|
("ix_connector_service_desk_objects_tenant_id", ["tenant_id"]),
|
||||||
|
("ix_connector_service_desk_objects_profile_id", ["profile_id"]),
|
||||||
|
("ix_connector_service_desk_objects_object_type", ["object_type"]),
|
||||||
|
("ix_connector_service_desk_objects_external_ticket_number", ["external_ticket_number"]),
|
||||||
|
("ix_connector_service_desk_objects_status", ["status"]),
|
||||||
|
("ix_connector_service_desk_objects_content_hash", ["content_hash"]),
|
||||||
|
("ix_connector_service_desk_objects_change_cursor", ["change_cursor"]),
|
||||||
|
("ix_connector_service_desk_objects_source_updated_at", ["source_updated_at"]),
|
||||||
|
("ix_connector_service_desk_objects_observed_at", ["observed_at"]),
|
||||||
|
):
|
||||||
|
op.create_index(op.f(name), "connector_service_desk_objects", columns)
|
||||||
|
op.create_index(
|
||||||
|
"ix_connector_service_desk_objects_tenant_profile_status",
|
||||||
|
"connector_service_desk_objects",
|
||||||
|
["tenant_id", "profile_id", "status"],
|
||||||
|
)
|
||||||
|
op.create_index(
|
||||||
|
"ix_connector_service_desk_objects_tenant_updated",
|
||||||
|
"connector_service_desk_objects",
|
||||||
|
["tenant_id", "source_updated_at"],
|
||||||
|
)
|
||||||
|
|
||||||
|
op.create_table(
|
||||||
|
"connector_service_desk_sync_runs",
|
||||||
|
sa.Column("id", sa.String(length=36), nullable=False),
|
||||||
|
sa.Column("tenant_id", sa.String(length=36), nullable=False),
|
||||||
|
sa.Column("profile_id", sa.String(length=36), nullable=False),
|
||||||
|
sa.Column("mode", sa.String(length=40), nullable=False),
|
||||||
|
sa.Column("idempotency_key", sa.String(length=255), nullable=False),
|
||||||
|
sa.Column("request_hash", sa.String(length=64), nullable=False),
|
||||||
|
sa.Column("status", sa.String(length=30), nullable=False),
|
||||||
|
sa.Column("cursor_before", sa.String(length=4000), nullable=True),
|
||||||
|
sa.Column("cursor_after", sa.String(length=4000), nullable=True),
|
||||||
|
sa.Column("high_watermark", sa.String(length=500), nullable=True),
|
||||||
|
sa.Column("counts", sa.JSON(), nullable=False),
|
||||||
|
sa.Column("effects", sa.JSON(), nullable=False),
|
||||||
|
sa.Column("diagnostics", sa.JSON(), nullable=False),
|
||||||
|
sa.Column("provenance", sa.JSON(), nullable=False),
|
||||||
|
sa.Column("created_by", sa.String(length=255), nullable=True),
|
||||||
|
sa.Column("started_at", sa.DateTime(timezone=True), nullable=False),
|
||||||
|
sa.Column("finished_at", sa.DateTime(timezone=True), nullable=True),
|
||||||
|
sa.Column("created_at", sa.DateTime(timezone=True), nullable=False),
|
||||||
|
sa.Column("updated_at", sa.DateTime(timezone=True), nullable=False),
|
||||||
|
sa.ForeignKeyConstraint(
|
||||||
|
["profile_id"],
|
||||||
|
["connector_service_desk_profiles.id"],
|
||||||
|
name=op.f(
|
||||||
|
"fk_connector_service_desk_sync_runs_profile_id_connector_service_desk_profiles"
|
||||||
|
),
|
||||||
|
ondelete="CASCADE",
|
||||||
|
),
|
||||||
|
sa.PrimaryKeyConstraint("id", name=op.f("pk_connector_service_desk_sync_runs")),
|
||||||
|
sa.UniqueConstraint(
|
||||||
|
"tenant_id",
|
||||||
|
"profile_id",
|
||||||
|
"idempotency_key",
|
||||||
|
name="uq_connector_service_desk_sync_run_idempotency",
|
||||||
|
),
|
||||||
|
)
|
||||||
|
for name, columns in (
|
||||||
|
("ix_connector_service_desk_sync_runs_tenant_id", ["tenant_id"]),
|
||||||
|
("ix_connector_service_desk_sync_runs_profile_id", ["profile_id"]),
|
||||||
|
("ix_connector_service_desk_sync_runs_mode", ["mode"]),
|
||||||
|
("ix_connector_service_desk_sync_runs_status", ["status"]),
|
||||||
|
("ix_connector_service_desk_sync_runs_created_by", ["created_by"]),
|
||||||
|
("ix_connector_service_desk_sync_runs_started_at", ["started_at"]),
|
||||||
|
):
|
||||||
|
op.create_index(op.f(name), "connector_service_desk_sync_runs", columns)
|
||||||
|
op.create_index(
|
||||||
|
"ix_connector_service_desk_runs_profile_started",
|
||||||
|
"connector_service_desk_sync_runs",
|
||||||
|
["tenant_id", "profile_id", "started_at"],
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def downgrade() -> None:
|
||||||
|
op.drop_table("connector_service_desk_sync_runs")
|
||||||
|
op.drop_table("connector_service_desk_objects")
|
||||||
|
op.drop_table("connector_service_desk_profiles")
|
||||||
+20
@@ -0,0 +1,20 @@
|
|||||||
|
"""Retain original CSV source evidence for new durable snapshots."""
|
||||||
|
|
||||||
|
from alembic import op
|
||||||
|
import sqlalchemy as sa
|
||||||
|
|
||||||
|
revision = "d2a4c6e8f0b1"
|
||||||
|
down_revision = "c0f1a2b3c4d5"
|
||||||
|
branch_labels = None
|
||||||
|
depends_on = None
|
||||||
|
|
||||||
|
|
||||||
|
def upgrade() -> None:
|
||||||
|
op.add_column(
|
||||||
|
"connector_tabular_sources", sa.Column("csv_source", sa.JSON(), nullable=True)
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def downgrade() -> None:
|
||||||
|
with op.batch_alter_table("connector_tabular_sources") as batch:
|
||||||
|
batch.drop_column("csv_source")
|
||||||
@@ -7,6 +7,13 @@ from sqlalchemy import func, select
|
|||||||
from sqlalchemy.orm import Session
|
from sqlalchemy.orm import Session
|
||||||
|
|
||||||
from govoplan_connectors.backend.db.models import (
|
from govoplan_connectors.backend.db.models import (
|
||||||
|
ConnectorConfiguration,
|
||||||
|
ConnectorKnowledgeObject,
|
||||||
|
ConnectorKnowledgeProfile,
|
||||||
|
ConnectorKnowledgeSyncRun,
|
||||||
|
ConnectorServiceDeskObject,
|
||||||
|
ConnectorServiceDeskProfile,
|
||||||
|
ConnectorServiceDeskSyncRun,
|
||||||
ConnectorSanctionsAcquisitionRun,
|
ConnectorSanctionsAcquisitionRun,
|
||||||
ConnectorSanctionsSnapshot,
|
ConnectorSanctionsSnapshot,
|
||||||
ConnectorTabularSource,
|
ConnectorTabularSource,
|
||||||
@@ -19,6 +26,8 @@ from govoplan_core.core.provider_governance import (
|
|||||||
|
|
||||||
TABULAR_PROVIDER_ID = "connectors.tabular_snapshot"
|
TABULAR_PROVIDER_ID = "connectors.tabular_snapshot"
|
||||||
SANCTIONS_PROVIDER_ID = "connectors.sanctions_snapshot"
|
SANCTIONS_PROVIDER_ID = "connectors.sanctions_snapshot"
|
||||||
|
KNOWLEDGE_PROVIDER_ID = "connectors.mediawiki.pages"
|
||||||
|
SERVICE_DESK_PROVIDER_ID = "connectors.znuny.tickets"
|
||||||
|
|
||||||
|
|
||||||
def tabular_provider_states(
|
def tabular_provider_states(
|
||||||
@@ -85,6 +94,70 @@ def sanctions_provider_states(
|
|||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def knowledge_provider_states(
|
||||||
|
context: ExternalProviderStateContext,
|
||||||
|
) -> tuple[ExternalProviderRuntimeState, ...]:
|
||||||
|
session = _session(context)
|
||||||
|
statement = select(ConnectorKnowledgeProfile)
|
||||||
|
if context.tenant_id is not None:
|
||||||
|
statement = statement.where(
|
||||||
|
ConnectorKnowledgeProfile.tenant_id == context.tenant_id
|
||||||
|
)
|
||||||
|
profiles = tuple(
|
||||||
|
session.scalars(
|
||||||
|
statement.order_by(
|
||||||
|
ConnectorKnowledgeProfile.tenant_id,
|
||||||
|
ConnectorKnowledgeProfile.id,
|
||||||
|
).limit(context.max_items + 1)
|
||||||
|
)
|
||||||
|
)[: context.max_items]
|
||||||
|
counts = _knowledge_counts(session, profiles)
|
||||||
|
latest_runs = _latest_knowledge_runs(session, profiles)
|
||||||
|
observed_at = datetime.now(UTC)
|
||||||
|
return tuple(
|
||||||
|
_knowledge_state(
|
||||||
|
profile,
|
||||||
|
observed_at=observed_at,
|
||||||
|
object_count=counts.get(profile.id, 0),
|
||||||
|
latest_run=latest_runs.get(profile.id),
|
||||||
|
)
|
||||||
|
for profile in profiles
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def service_desk_provider_states(
|
||||||
|
context: ExternalProviderStateContext,
|
||||||
|
) -> tuple[ExternalProviderRuntimeState, ...]:
|
||||||
|
session = _session(context)
|
||||||
|
statement = select(ConnectorServiceDeskProfile)
|
||||||
|
if context.tenant_id is not None:
|
||||||
|
statement = statement.where(
|
||||||
|
ConnectorServiceDeskProfile.tenant_id == context.tenant_id
|
||||||
|
)
|
||||||
|
profiles = tuple(
|
||||||
|
session.scalars(
|
||||||
|
statement.order_by(
|
||||||
|
ConnectorServiceDeskProfile.tenant_id,
|
||||||
|
ConnectorServiceDeskProfile.id,
|
||||||
|
).limit(context.max_items + 1)
|
||||||
|
)
|
||||||
|
)[: context.max_items]
|
||||||
|
counts = _service_desk_counts(session, profiles)
|
||||||
|
latest_runs = _latest_service_desk_runs(session, profiles)
|
||||||
|
configurations = _service_desk_configurations(session, profiles)
|
||||||
|
observed_at = datetime.now(UTC)
|
||||||
|
return tuple(
|
||||||
|
_service_desk_state(
|
||||||
|
profile,
|
||||||
|
observed_at=observed_at,
|
||||||
|
object_count=counts.get(profile.id, 0),
|
||||||
|
latest_run=latest_runs.get(profile.id),
|
||||||
|
configuration=configurations.get(profile.configuration_id),
|
||||||
|
)
|
||||||
|
for profile in profiles
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
def _session(context: ExternalProviderStateContext) -> Session:
|
def _session(context: ExternalProviderStateContext) -> Session:
|
||||||
if not isinstance(context.session, Session):
|
if not isinstance(context.session, Session):
|
||||||
raise RuntimeError("Connectors provider state requires a database session.")
|
raise RuntimeError("Connectors provider state requires a database session.")
|
||||||
@@ -161,6 +234,263 @@ def _snapshot_counts(
|
|||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def _knowledge_counts(
|
||||||
|
session: Session,
|
||||||
|
profiles: tuple[ConnectorKnowledgeProfile, ...],
|
||||||
|
) -> dict[str, int]:
|
||||||
|
profile_ids = tuple(item.id for item in profiles)
|
||||||
|
if not profile_ids:
|
||||||
|
return {}
|
||||||
|
return {
|
||||||
|
str(profile_id): int(count)
|
||||||
|
for profile_id, count in session.execute(
|
||||||
|
select(
|
||||||
|
ConnectorKnowledgeObject.profile_id,
|
||||||
|
func.count(ConnectorKnowledgeObject.id),
|
||||||
|
)
|
||||||
|
.where(
|
||||||
|
ConnectorKnowledgeObject.profile_id.in_(profile_ids),
|
||||||
|
ConnectorKnowledgeObject.status != "deleted",
|
||||||
|
)
|
||||||
|
.group_by(ConnectorKnowledgeObject.profile_id)
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def _latest_knowledge_runs(
|
||||||
|
session: Session,
|
||||||
|
profiles: tuple[ConnectorKnowledgeProfile, ...],
|
||||||
|
) -> dict[str, ConnectorKnowledgeSyncRun]:
|
||||||
|
profile_ids = tuple(item.id for item in profiles)
|
||||||
|
if not profile_ids:
|
||||||
|
return {}
|
||||||
|
rows = tuple(
|
||||||
|
session.scalars(
|
||||||
|
select(ConnectorKnowledgeSyncRun)
|
||||||
|
.where(
|
||||||
|
ConnectorKnowledgeSyncRun.profile_id.in_(profile_ids),
|
||||||
|
ConnectorKnowledgeSyncRun.mode.in_(("backfill", "delta")),
|
||||||
|
)
|
||||||
|
.order_by(
|
||||||
|
ConnectorKnowledgeSyncRun.profile_id,
|
||||||
|
ConnectorKnowledgeSyncRun.started_at.desc(),
|
||||||
|
ConnectorKnowledgeSyncRun.id.desc(),
|
||||||
|
)
|
||||||
|
)
|
||||||
|
)
|
||||||
|
latest: dict[str, ConnectorKnowledgeSyncRun] = {}
|
||||||
|
for row in rows:
|
||||||
|
latest.setdefault(row.profile_id, row)
|
||||||
|
return latest
|
||||||
|
|
||||||
|
|
||||||
|
def _service_desk_counts(
|
||||||
|
session: Session,
|
||||||
|
profiles: tuple[ConnectorServiceDeskProfile, ...],
|
||||||
|
) -> dict[str, int]:
|
||||||
|
profile_ids = tuple(item.id for item in profiles)
|
||||||
|
if not profile_ids:
|
||||||
|
return {}
|
||||||
|
return {
|
||||||
|
str(profile_id): int(count)
|
||||||
|
for profile_id, count in session.execute(
|
||||||
|
select(
|
||||||
|
ConnectorServiceDeskObject.profile_id,
|
||||||
|
func.count(ConnectorServiceDeskObject.id),
|
||||||
|
)
|
||||||
|
.where(
|
||||||
|
ConnectorServiceDeskObject.profile_id.in_(profile_ids),
|
||||||
|
ConnectorServiceDeskObject.status != "deleted",
|
||||||
|
)
|
||||||
|
.group_by(ConnectorServiceDeskObject.profile_id)
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def _latest_service_desk_runs(
|
||||||
|
session: Session,
|
||||||
|
profiles: tuple[ConnectorServiceDeskProfile, ...],
|
||||||
|
) -> dict[str, ConnectorServiceDeskSyncRun]:
|
||||||
|
profile_ids = tuple(item.id for item in profiles)
|
||||||
|
if not profile_ids:
|
||||||
|
return {}
|
||||||
|
rows = tuple(
|
||||||
|
session.scalars(
|
||||||
|
select(ConnectorServiceDeskSyncRun)
|
||||||
|
.where(ConnectorServiceDeskSyncRun.profile_id.in_(profile_ids))
|
||||||
|
.order_by(
|
||||||
|
ConnectorServiceDeskSyncRun.profile_id,
|
||||||
|
ConnectorServiceDeskSyncRun.started_at.desc(),
|
||||||
|
ConnectorServiceDeskSyncRun.id.desc(),
|
||||||
|
)
|
||||||
|
)
|
||||||
|
)
|
||||||
|
latest: dict[str, ConnectorServiceDeskSyncRun] = {}
|
||||||
|
for row in rows:
|
||||||
|
latest.setdefault(row.profile_id, row)
|
||||||
|
return latest
|
||||||
|
|
||||||
|
|
||||||
|
def _service_desk_configurations(
|
||||||
|
session: Session,
|
||||||
|
profiles: tuple[ConnectorServiceDeskProfile, ...],
|
||||||
|
) -> dict[str, ConnectorConfiguration]:
|
||||||
|
configuration_ids = tuple(
|
||||||
|
dict.fromkeys(profile.configuration_id for profile in profiles)
|
||||||
|
)
|
||||||
|
if not configuration_ids:
|
||||||
|
return {}
|
||||||
|
return {
|
||||||
|
row.id: row
|
||||||
|
for row in session.scalars(
|
||||||
|
select(ConnectorConfiguration).where(
|
||||||
|
ConnectorConfiguration.id.in_(configuration_ids)
|
||||||
|
)
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def _knowledge_state(
|
||||||
|
profile: ConnectorKnowledgeProfile,
|
||||||
|
*,
|
||||||
|
observed_at: datetime,
|
||||||
|
object_count: int,
|
||||||
|
latest_run: ConnectorKnowledgeSyncRun | None,
|
||||||
|
) -> ExternalProviderRuntimeState:
|
||||||
|
active = profile.status == "active"
|
||||||
|
health = (
|
||||||
|
"inactive"
|
||||||
|
if not active
|
||||||
|
else "healthy"
|
||||||
|
if profile.health_status == "healthy"
|
||||||
|
else "warning"
|
||||||
|
if profile.health_status in {"unknown", "degraded"}
|
||||||
|
else "error"
|
||||||
|
)
|
||||||
|
last_success = (
|
||||||
|
latest_run.finished_at
|
||||||
|
if latest_run is not None and latest_run.status == "completed"
|
||||||
|
else profile.discovered_at
|
||||||
|
)
|
||||||
|
return ExternalProviderRuntimeState(
|
||||||
|
provider_id=KNOWLEDGE_PROVIDER_ID,
|
||||||
|
binding_ref=f"connectors:knowledge-profile:{profile.id}",
|
||||||
|
authority_mode=profile.source_authority_mode,
|
||||||
|
observed_at=observed_at,
|
||||||
|
configured=True,
|
||||||
|
active=active,
|
||||||
|
health=health,
|
||||||
|
freshness="unknown" if active else "not_applicable",
|
||||||
|
conflict=(
|
||||||
|
"pending"
|
||||||
|
if latest_run is not None and latest_run.status == "outcome_unknown"
|
||||||
|
else "not_applicable"
|
||||||
|
),
|
||||||
|
recovery=(
|
||||||
|
"attention"
|
||||||
|
if latest_run is not None
|
||||||
|
and latest_run.status in {"failed", "outcome_unknown"}
|
||||||
|
else "ready"
|
||||||
|
if active
|
||||||
|
else "not_applicable"
|
||||||
|
),
|
||||||
|
last_success_at=_aware(last_success),
|
||||||
|
detail=(
|
||||||
|
f"{profile.product} knowledge profile is synchronized and ACL-rechecked."
|
||||||
|
if active and profile.health_status == "healthy"
|
||||||
|
else "Knowledge profile requires discovery, synchronization, or health review."
|
||||||
|
if active
|
||||||
|
else "Knowledge profile is paused."
|
||||||
|
),
|
||||||
|
metrics={
|
||||||
|
"product": profile.product,
|
||||||
|
"product_version": profile.product_version,
|
||||||
|
"desired_maturity": profile.desired_maturity,
|
||||||
|
"discovered_maturity": profile.discovered_maturity,
|
||||||
|
"active_objects": int(object_count),
|
||||||
|
"last_run_status": latest_run.status if latest_run is not None else None,
|
||||||
|
},
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _service_desk_state(
|
||||||
|
profile: ConnectorServiceDeskProfile,
|
||||||
|
*,
|
||||||
|
observed_at: datetime,
|
||||||
|
object_count: int,
|
||||||
|
latest_run: ConnectorServiceDeskSyncRun | None,
|
||||||
|
configuration: ConnectorConfiguration | None,
|
||||||
|
) -> ExternalProviderRuntimeState:
|
||||||
|
configured = configuration is not None
|
||||||
|
active = (
|
||||||
|
profile.status == "active"
|
||||||
|
and configuration is not None
|
||||||
|
and configuration.status == "active"
|
||||||
|
)
|
||||||
|
discovery_current = bool(
|
||||||
|
configuration is not None
|
||||||
|
and profile.discovered_configuration_revision
|
||||||
|
== configuration.resource_revision
|
||||||
|
and profile.discovered_configuration_hash == configuration.effective_hash
|
||||||
|
)
|
||||||
|
health = (
|
||||||
|
"inactive"
|
||||||
|
if not active
|
||||||
|
else "warning"
|
||||||
|
if not discovery_current
|
||||||
|
else "healthy"
|
||||||
|
if profile.health_status == "healthy"
|
||||||
|
else "warning"
|
||||||
|
if profile.health_status in {"unknown", "degraded"}
|
||||||
|
else "error"
|
||||||
|
)
|
||||||
|
last_success = (
|
||||||
|
latest_run.finished_at
|
||||||
|
if latest_run is not None and latest_run.status == "completed"
|
||||||
|
else profile.discovered_at
|
||||||
|
)
|
||||||
|
unresolved = latest_run is not None and latest_run.status == "outcome_unknown"
|
||||||
|
return ExternalProviderRuntimeState(
|
||||||
|
provider_id=SERVICE_DESK_PROVIDER_ID,
|
||||||
|
binding_ref=f"connectors:service-desk-profile:{profile.id}",
|
||||||
|
authority_mode=profile.source_authority_mode,
|
||||||
|
observed_at=observed_at,
|
||||||
|
configured=configured,
|
||||||
|
active=active,
|
||||||
|
health=health,
|
||||||
|
freshness="unknown" if active else "not_applicable",
|
||||||
|
conflict="pending" if unresolved else "not_applicable",
|
||||||
|
recovery=(
|
||||||
|
"attention"
|
||||||
|
if latest_run is not None
|
||||||
|
and latest_run.status in {"failed", "outcome_unknown"}
|
||||||
|
else "ready"
|
||||||
|
if active
|
||||||
|
else "not_applicable"
|
||||||
|
),
|
||||||
|
last_success_at=_aware(last_success),
|
||||||
|
detail=(
|
||||||
|
f"{profile.product} service-desk profile is synchronized and ACL-rechecked."
|
||||||
|
if active and discovery_current and profile.health_status == "healthy"
|
||||||
|
else "Service-desk configuration changed; rediscovery is required."
|
||||||
|
if active and not discovery_current
|
||||||
|
else "Service-desk profile requires discovery, synchronization, or recovery review."
|
||||||
|
if active
|
||||||
|
else "Service-desk profile is paused."
|
||||||
|
),
|
||||||
|
metrics={
|
||||||
|
"product": profile.product,
|
||||||
|
"product_version": profile.product_version,
|
||||||
|
"integration_mode": profile.integration_mode,
|
||||||
|
"desired_maturity": profile.desired_maturity,
|
||||||
|
"discovered_maturity": profile.discovered_maturity,
|
||||||
|
"discovery_current": discovery_current,
|
||||||
|
"active_objects": int(object_count),
|
||||||
|
"last_run_status": latest_run.status if latest_run is not None else None,
|
||||||
|
},
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
def _sanctions_state(
|
def _sanctions_state(
|
||||||
run: ConnectorSanctionsAcquisitionRun,
|
run: ConnectorSanctionsAcquisitionRun,
|
||||||
*,
|
*,
|
||||||
@@ -209,8 +539,12 @@ def _aware(value: datetime | None) -> datetime | None:
|
|||||||
|
|
||||||
|
|
||||||
__all__ = [
|
__all__ = [
|
||||||
|
"KNOWLEDGE_PROVIDER_ID",
|
||||||
"SANCTIONS_PROVIDER_ID",
|
"SANCTIONS_PROVIDER_ID",
|
||||||
|
"SERVICE_DESK_PROVIDER_ID",
|
||||||
"TABULAR_PROVIDER_ID",
|
"TABULAR_PROVIDER_ID",
|
||||||
"sanctions_provider_states",
|
"sanctions_provider_states",
|
||||||
|
"knowledge_provider_states",
|
||||||
|
"service_desk_provider_states",
|
||||||
"tabular_provider_states",
|
"tabular_provider_states",
|
||||||
]
|
]
|
||||||
|
|||||||
@@ -70,7 +70,7 @@ CONNECTOR_RECOVERY_OPERATIONS = (
|
|||||||
"unknown outcomes remain unresolved until provider-backed reconciliation",
|
"unknown outcomes remain unresolved until provider-backed reconciliation",
|
||||||
"never retry the same remote effect solely to reconstruct local state",
|
"never retry the same remote effect solely to reconstruct local state",
|
||||||
),
|
),
|
||||||
implemented=False,
|
implemented=True,
|
||||||
),
|
),
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -262,6 +262,20 @@ class ConnectorExternalMutationRecovery:
|
|||||||
if self.operation is not None:
|
if self.operation is not None:
|
||||||
self.operation.succeed(evidence=provider_evidence)
|
self.operation.succeed(evidence=provider_evidence)
|
||||||
|
|
||||||
|
def commit_success(
|
||||||
|
self,
|
||||||
|
session: Session,
|
||||||
|
*,
|
||||||
|
provider_evidence: dict[str, Any],
|
||||||
|
) -> None:
|
||||||
|
if self.operation is not None:
|
||||||
|
self.operation.commit_verified_success(
|
||||||
|
session,
|
||||||
|
evidence=provider_evidence,
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
session.commit()
|
||||||
|
|
||||||
def reject(self, *, summary: str, provider_code: str) -> None:
|
def reject(self, *, summary: str, provider_code: str) -> None:
|
||||||
if self.operation is not None:
|
if self.operation is not None:
|
||||||
self.operation.reject(
|
self.operation.reject(
|
||||||
|
|||||||
@@ -10,6 +10,7 @@ from sqlalchemy.orm import Session
|
|||||||
from govoplan_core.audit.logging import audit_event
|
from govoplan_core.audit.logging import audit_event
|
||||||
from govoplan_core.auth import ApiPrincipal, get_api_principal, has_scope
|
from govoplan_core.auth import ApiPrincipal, get_api_principal, has_scope
|
||||||
from govoplan_core.core.tabular_sources import (
|
from govoplan_core.core.tabular_sources import (
|
||||||
|
TabularCsvSource,
|
||||||
TabularReadRequest,
|
TabularReadRequest,
|
||||||
TabularSnapshotInput,
|
TabularSnapshotInput,
|
||||||
TabularSource,
|
TabularSource,
|
||||||
@@ -81,10 +82,61 @@ from govoplan_connectors.backend.governed_schemas import (
|
|||||||
ConnectorRunListResponse,
|
ConnectorRunListResponse,
|
||||||
ConnectorRunRequest,
|
ConnectorRunRequest,
|
||||||
)
|
)
|
||||||
|
from govoplan_connectors.backend.knowledge_connector import (
|
||||||
|
KnowledgeConnectorError,
|
||||||
|
create_profile,
|
||||||
|
discover_profile,
|
||||||
|
list_objects as list_knowledge_objects,
|
||||||
|
list_profiles as list_knowledge_profiles,
|
||||||
|
list_runs as list_knowledge_runs,
|
||||||
|
migration_dry_run,
|
||||||
|
publish_page,
|
||||||
|
synchronize_profile,
|
||||||
|
update_profile,
|
||||||
|
)
|
||||||
|
from govoplan_connectors.backend.knowledge_schemas import (
|
||||||
|
KnowledgeDiscoveryResponse,
|
||||||
|
KnowledgeMigrationDryRunRequest,
|
||||||
|
KnowledgeMigrationDryRunResponse,
|
||||||
|
KnowledgeObjectListResponse,
|
||||||
|
KnowledgeProfileCreateRequest,
|
||||||
|
KnowledgeProfileItem,
|
||||||
|
KnowledgeProfileListResponse,
|
||||||
|
KnowledgeProfileUpdateRequest,
|
||||||
|
KnowledgePublishRequest,
|
||||||
|
KnowledgePublishResponse,
|
||||||
|
KnowledgeSyncRequest,
|
||||||
|
KnowledgeSyncRunItem,
|
||||||
|
KnowledgeSyncRunListResponse,
|
||||||
|
)
|
||||||
from govoplan_connectors.backend.recovery import (
|
from govoplan_connectors.backend.recovery import (
|
||||||
ConnectorRecoveryError,
|
ConnectorRecoveryError,
|
||||||
begin_connector_read_snapshot,
|
begin_connector_read_snapshot,
|
||||||
)
|
)
|
||||||
|
from govoplan_connectors.backend.service_desk_connector import (
|
||||||
|
ServiceDeskConnectorError,
|
||||||
|
create_profile as create_service_desk_profile,
|
||||||
|
discover_profile as discover_service_desk_profile,
|
||||||
|
list_objects as list_service_desk_objects,
|
||||||
|
list_profiles as list_service_desk_profiles,
|
||||||
|
list_runs as list_service_desk_runs,
|
||||||
|
synchronize_profile as synchronize_service_desk_profile,
|
||||||
|
update_profile as update_service_desk_profile,
|
||||||
|
update_ticket as update_service_desk_ticket,
|
||||||
|
)
|
||||||
|
from govoplan_connectors.backend.service_desk_schemas import (
|
||||||
|
ServiceDeskDiscoveryResponse,
|
||||||
|
ServiceDeskObjectListResponse,
|
||||||
|
ServiceDeskProfileCreateRequest,
|
||||||
|
ServiceDeskProfileItem,
|
||||||
|
ServiceDeskProfileListResponse,
|
||||||
|
ServiceDeskProfileUpdateRequest,
|
||||||
|
ServiceDeskSyncRequest,
|
||||||
|
ServiceDeskSyncRunItem,
|
||||||
|
ServiceDeskSyncRunListResponse,
|
||||||
|
ServiceDeskTicketUpdateRequest,
|
||||||
|
ServiceDeskTicketUpdateResponse,
|
||||||
|
)
|
||||||
from govoplan_connectors.backend.sanctions_sources import (
|
from govoplan_connectors.backend.sanctions_sources import (
|
||||||
SANCTIONS_READ_SCOPE,
|
SANCTIONS_READ_SCOPE,
|
||||||
SANCTIONS_REFRESH_SCOPE,
|
SANCTIONS_REFRESH_SCOPE,
|
||||||
@@ -188,6 +240,66 @@ def _governed_http_error(exc: GovernedConnectorError) -> HTTPException:
|
|||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _knowledge_http_error(exc: KnowledgeConnectorError) -> HTTPException:
|
||||||
|
if exc.code.endswith("_not_found"):
|
||||||
|
status_code = status.HTTP_404_NOT_FOUND
|
||||||
|
elif exc.code == "access_denied":
|
||||||
|
status_code = status.HTTP_403_FORBIDDEN
|
||||||
|
elif exc.code in {
|
||||||
|
"idempotency_conflict",
|
||||||
|
"operation_unresolved",
|
||||||
|
"profile_conflict",
|
||||||
|
"profile_paused",
|
||||||
|
"publication_conflict",
|
||||||
|
"publication_reconciliation_required",
|
||||||
|
}:
|
||||||
|
status_code = status.HTTP_409_CONFLICT
|
||||||
|
elif exc.retryable or exc.code.endswith("_unavailable"):
|
||||||
|
status_code = status.HTTP_503_SERVICE_UNAVAILABLE
|
||||||
|
else:
|
||||||
|
status_code = status.HTTP_422_UNPROCESSABLE_CONTENT
|
||||||
|
return HTTPException(
|
||||||
|
status_code=status_code,
|
||||||
|
detail={
|
||||||
|
"code": exc.code,
|
||||||
|
"message": str(exc),
|
||||||
|
"retryable": exc.retryable,
|
||||||
|
},
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _service_desk_http_error(exc: ServiceDeskConnectorError) -> HTTPException:
|
||||||
|
if exc.code.endswith("_not_found"):
|
||||||
|
status_code = status.HTTP_404_NOT_FOUND
|
||||||
|
elif exc.code == "forbidden":
|
||||||
|
status_code = status.HTTP_403_FORBIDDEN
|
||||||
|
elif exc.code in {
|
||||||
|
"authority_mode_invalid",
|
||||||
|
"cursor_conflict",
|
||||||
|
"external_authoritative",
|
||||||
|
"external_revision_conflict",
|
||||||
|
"idempotency_conflict",
|
||||||
|
"operation_unresolved",
|
||||||
|
"profile_conflict",
|
||||||
|
"profile_paused",
|
||||||
|
"rediscovery_required",
|
||||||
|
"update_outcome_unknown",
|
||||||
|
}:
|
||||||
|
status_code = status.HTTP_409_CONFLICT
|
||||||
|
elif exc.retryable or exc.code.endswith("_unavailable"):
|
||||||
|
status_code = status.HTTP_503_SERVICE_UNAVAILABLE
|
||||||
|
else:
|
||||||
|
status_code = status.HTTP_422_UNPROCESSABLE_CONTENT
|
||||||
|
return HTTPException(
|
||||||
|
status_code=status_code,
|
||||||
|
detail={
|
||||||
|
"code": exc.code,
|
||||||
|
"message": str(exc),
|
||||||
|
"retryable": exc.retryable,
|
||||||
|
},
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
@router.post("/feeds/preview", response_model=FeedDocumentResponse)
|
@router.post("/feeds/preview", response_model=FeedDocumentResponse)
|
||||||
def api_preview_feed(
|
def api_preview_feed(
|
||||||
payload: FeedAcquireRequest,
|
payload: FeedAcquireRequest,
|
||||||
@@ -428,7 +540,7 @@ def api_create_tabular_snapshot(
|
|||||||
rows = (
|
rows = (
|
||||||
tuple(payload.rows or ())
|
tuple(payload.rows or ())
|
||||||
if payload.format == "json"
|
if payload.format == "json"
|
||||||
else parse_csv_snapshot(payload.csv_text or "", delimiter=payload.delimiter)
|
else parse_csv_snapshot(payload.csv_text or "", delimiter=payload.delimiter, value_mode=payload.csv_value_mode)
|
||||||
)
|
)
|
||||||
source = provider.create_snapshot(
|
source = provider.create_snapshot(
|
||||||
session,
|
session,
|
||||||
@@ -439,6 +551,11 @@ def api_create_tabular_snapshot(
|
|||||||
description=payload.description,
|
description=payload.description,
|
||||||
rows=rows,
|
rows=rows,
|
||||||
metadata={"import_format": payload.format},
|
metadata={"import_format": payload.format},
|
||||||
|
csv_source=(TabularCsvSource(
|
||||||
|
text=payload.csv_text or "",
|
||||||
|
delimiter=payload.delimiter,
|
||||||
|
value_mode=payload.csv_value_mode,
|
||||||
|
) if payload.format == "csv" else None),
|
||||||
),
|
),
|
||||||
)
|
)
|
||||||
except TabularSourceError as exc:
|
except TabularSourceError as exc:
|
||||||
@@ -462,6 +579,35 @@ def api_create_tabular_snapshot(
|
|||||||
return _source_response(source)
|
return _source_response(source)
|
||||||
|
|
||||||
|
|
||||||
|
@router.get("/tabular-sources/{source_ref}/original-csv")
|
||||||
|
def api_original_tabular_csv(
|
||||||
|
source_ref: str,
|
||||||
|
session: Session = Depends(get_session),
|
||||||
|
principal: ApiPrincipal = Depends(get_api_principal),
|
||||||
|
) -> Response:
|
||||||
|
_require_any_scope(principal, READ_SCOPE, ADMIN_SCOPE)
|
||||||
|
try:
|
||||||
|
text = provider.original_csv(session, principal, source_ref=source_ref)
|
||||||
|
except TabularSourceError as exc:
|
||||||
|
raise _http_error(exc) from exc
|
||||||
|
audit_event(
|
||||||
|
session,
|
||||||
|
tenant_id=principal.tenant_id,
|
||||||
|
user_id=getattr(principal.user, "id", None),
|
||||||
|
api_key_id=principal.api_key_id,
|
||||||
|
action="connectors.original_csv.exported",
|
||||||
|
object_type="connector_tabular_source",
|
||||||
|
object_id=source_ref,
|
||||||
|
details={"sha256": hashlib.sha256(text.encode("utf-8")).hexdigest()},
|
||||||
|
)
|
||||||
|
session.commit()
|
||||||
|
return Response(
|
||||||
|
content=text.encode("utf-8"),
|
||||||
|
media_type="text/csv; charset=utf-8",
|
||||||
|
headers={"Content-Disposition": 'attachment; filename="original.csv"', "Cache-Control": "no-store"},
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
@router.post(
|
@router.post(
|
||||||
"/tabular-sources/files",
|
"/tabular-sources/files",
|
||||||
response_model=TabularSourceResponse,
|
response_model=TabularSourceResponse,
|
||||||
@@ -484,6 +630,7 @@ def api_create_managed_file_source(
|
|||||||
file_version_id=payload.file_version_id,
|
file_version_id=payload.file_version_id,
|
||||||
delimiter=payload.delimiter,
|
delimiter=payload.delimiter,
|
||||||
sheet_name=payload.sheet_name,
|
sheet_name=payload.sheet_name,
|
||||||
|
csv_value_mode=payload.csv_value_mode,
|
||||||
)
|
)
|
||||||
except TabularSourceError as exc:
|
except TabularSourceError as exc:
|
||||||
raise _http_error(exc) from exc
|
raise _http_error(exc) from exc
|
||||||
@@ -986,6 +1133,367 @@ def api_review_governed_run(
|
|||||||
raise _governed_http_error(exc) from exc
|
raise _governed_http_error(exc) from exc
|
||||||
|
|
||||||
|
|
||||||
|
@router.get(
|
||||||
|
"/knowledge/profiles",
|
||||||
|
response_model=KnowledgeProfileListResponse,
|
||||||
|
)
|
||||||
|
def api_list_knowledge_profiles(
|
||||||
|
session: Session = Depends(get_session),
|
||||||
|
principal: ApiPrincipal = Depends(get_api_principal),
|
||||||
|
) -> KnowledgeProfileListResponse:
|
||||||
|
try:
|
||||||
|
return KnowledgeProfileListResponse(
|
||||||
|
items=list(list_knowledge_profiles(session, principal))
|
||||||
|
)
|
||||||
|
except KnowledgeConnectorError as exc:
|
||||||
|
raise _knowledge_http_error(exc) from exc
|
||||||
|
|
||||||
|
|
||||||
|
@router.post(
|
||||||
|
"/knowledge/profiles",
|
||||||
|
response_model=KnowledgeProfileItem,
|
||||||
|
status_code=status.HTTP_201_CREATED,
|
||||||
|
)
|
||||||
|
def api_create_knowledge_profile(
|
||||||
|
payload: KnowledgeProfileCreateRequest,
|
||||||
|
session: Session = Depends(get_session),
|
||||||
|
principal: ApiPrincipal = Depends(get_api_principal),
|
||||||
|
) -> KnowledgeProfileItem:
|
||||||
|
try:
|
||||||
|
return create_profile(session, principal, payload)
|
||||||
|
except KnowledgeConnectorError as exc:
|
||||||
|
session.rollback()
|
||||||
|
raise _knowledge_http_error(exc) from exc
|
||||||
|
|
||||||
|
|
||||||
|
@router.put(
|
||||||
|
"/knowledge/profiles/{profile_id}",
|
||||||
|
response_model=KnowledgeProfileItem,
|
||||||
|
)
|
||||||
|
def api_update_knowledge_profile(
|
||||||
|
profile_id: str,
|
||||||
|
payload: KnowledgeProfileUpdateRequest,
|
||||||
|
session: Session = Depends(get_session),
|
||||||
|
principal: ApiPrincipal = Depends(get_api_principal),
|
||||||
|
) -> KnowledgeProfileItem:
|
||||||
|
try:
|
||||||
|
return update_profile(
|
||||||
|
session,
|
||||||
|
principal,
|
||||||
|
profile_id=profile_id,
|
||||||
|
payload=payload,
|
||||||
|
)
|
||||||
|
except KnowledgeConnectorError as exc:
|
||||||
|
session.rollback()
|
||||||
|
raise _knowledge_http_error(exc) from exc
|
||||||
|
|
||||||
|
|
||||||
|
@router.post(
|
||||||
|
"/knowledge/profiles/{profile_id}/discover",
|
||||||
|
response_model=KnowledgeDiscoveryResponse,
|
||||||
|
)
|
||||||
|
def api_discover_knowledge_profile(
|
||||||
|
profile_id: str,
|
||||||
|
session: Session = Depends(get_session),
|
||||||
|
principal: ApiPrincipal = Depends(get_api_principal),
|
||||||
|
) -> KnowledgeDiscoveryResponse:
|
||||||
|
try:
|
||||||
|
return discover_profile(
|
||||||
|
session,
|
||||||
|
principal,
|
||||||
|
profile_id=profile_id,
|
||||||
|
)
|
||||||
|
except KnowledgeConnectorError as exc:
|
||||||
|
raise _knowledge_http_error(exc) from exc
|
||||||
|
|
||||||
|
|
||||||
|
@router.post(
|
||||||
|
"/knowledge/profiles/{profile_id}/sync",
|
||||||
|
response_model=KnowledgeSyncRunItem,
|
||||||
|
)
|
||||||
|
def api_synchronize_knowledge_profile(
|
||||||
|
profile_id: str,
|
||||||
|
payload: KnowledgeSyncRequest,
|
||||||
|
session: Session = Depends(get_session),
|
||||||
|
principal: ApiPrincipal = Depends(get_api_principal),
|
||||||
|
) -> KnowledgeSyncRunItem:
|
||||||
|
try:
|
||||||
|
return synchronize_profile(
|
||||||
|
session,
|
||||||
|
principal,
|
||||||
|
profile_id=profile_id,
|
||||||
|
payload=payload,
|
||||||
|
)
|
||||||
|
except KnowledgeConnectorError as exc:
|
||||||
|
raise _knowledge_http_error(exc) from exc
|
||||||
|
|
||||||
|
|
||||||
|
@router.get(
|
||||||
|
"/knowledge/profiles/{profile_id}/objects",
|
||||||
|
response_model=KnowledgeObjectListResponse,
|
||||||
|
)
|
||||||
|
def api_list_knowledge_objects(
|
||||||
|
profile_id: str,
|
||||||
|
cursor: str | None = Query(default=None, max_length=500),
|
||||||
|
limit: int = Query(default=100, ge=1, le=500),
|
||||||
|
session: Session = Depends(get_session),
|
||||||
|
principal: ApiPrincipal = Depends(get_api_principal),
|
||||||
|
) -> KnowledgeObjectListResponse:
|
||||||
|
try:
|
||||||
|
items, next_cursor = list_knowledge_objects(
|
||||||
|
session,
|
||||||
|
principal,
|
||||||
|
profile_id=profile_id,
|
||||||
|
cursor=cursor,
|
||||||
|
limit=limit,
|
||||||
|
)
|
||||||
|
return KnowledgeObjectListResponse(
|
||||||
|
items=list(items),
|
||||||
|
next_cursor=next_cursor,
|
||||||
|
)
|
||||||
|
except KnowledgeConnectorError as exc:
|
||||||
|
raise _knowledge_http_error(exc) from exc
|
||||||
|
|
||||||
|
|
||||||
|
@router.get(
|
||||||
|
"/knowledge/runs",
|
||||||
|
response_model=KnowledgeSyncRunListResponse,
|
||||||
|
)
|
||||||
|
def api_list_knowledge_runs(
|
||||||
|
profile_id: str | None = Query(default=None),
|
||||||
|
limit: int = Query(default=100, ge=1, le=500),
|
||||||
|
session: Session = Depends(get_session),
|
||||||
|
principal: ApiPrincipal = Depends(get_api_principal),
|
||||||
|
) -> KnowledgeSyncRunListResponse:
|
||||||
|
try:
|
||||||
|
return KnowledgeSyncRunListResponse(
|
||||||
|
items=list(
|
||||||
|
list_knowledge_runs(
|
||||||
|
session,
|
||||||
|
principal,
|
||||||
|
profile_id=profile_id,
|
||||||
|
limit=limit,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
)
|
||||||
|
except KnowledgeConnectorError as exc:
|
||||||
|
raise _knowledge_http_error(exc) from exc
|
||||||
|
|
||||||
|
|
||||||
|
@router.post(
|
||||||
|
"/knowledge/profiles/{profile_id}/migration-dry-runs",
|
||||||
|
response_model=KnowledgeMigrationDryRunResponse,
|
||||||
|
)
|
||||||
|
def api_preview_knowledge_migration(
|
||||||
|
profile_id: str,
|
||||||
|
payload: KnowledgeMigrationDryRunRequest,
|
||||||
|
session: Session = Depends(get_session),
|
||||||
|
principal: ApiPrincipal = Depends(get_api_principal),
|
||||||
|
) -> KnowledgeMigrationDryRunResponse:
|
||||||
|
try:
|
||||||
|
return migration_dry_run(
|
||||||
|
session,
|
||||||
|
principal,
|
||||||
|
profile_id=profile_id,
|
||||||
|
payload=payload,
|
||||||
|
)
|
||||||
|
except KnowledgeConnectorError as exc:
|
||||||
|
raise _knowledge_http_error(exc) from exc
|
||||||
|
|
||||||
|
|
||||||
|
@router.post(
|
||||||
|
"/knowledge/profiles/{profile_id}/pages/{external_page_id}/publish",
|
||||||
|
response_model=KnowledgePublishResponse,
|
||||||
|
)
|
||||||
|
def api_publish_knowledge_page(
|
||||||
|
profile_id: str,
|
||||||
|
external_page_id: str,
|
||||||
|
payload: KnowledgePublishRequest,
|
||||||
|
session: Session = Depends(get_session),
|
||||||
|
principal: ApiPrincipal = Depends(get_api_principal),
|
||||||
|
) -> KnowledgePublishResponse:
|
||||||
|
try:
|
||||||
|
return publish_page(
|
||||||
|
session,
|
||||||
|
principal,
|
||||||
|
profile_id=profile_id,
|
||||||
|
external_page_id=external_page_id,
|
||||||
|
payload=payload,
|
||||||
|
)
|
||||||
|
except KnowledgeConnectorError as exc:
|
||||||
|
raise _knowledge_http_error(exc) from exc
|
||||||
|
|
||||||
|
|
||||||
|
@router.get(
|
||||||
|
"/service-desk/profiles",
|
||||||
|
response_model=ServiceDeskProfileListResponse,
|
||||||
|
)
|
||||||
|
def api_list_service_desk_profiles(
|
||||||
|
session: Session = Depends(get_session),
|
||||||
|
principal: ApiPrincipal = Depends(get_api_principal),
|
||||||
|
) -> ServiceDeskProfileListResponse:
|
||||||
|
try:
|
||||||
|
return ServiceDeskProfileListResponse(
|
||||||
|
items=list(list_service_desk_profiles(session, principal))
|
||||||
|
)
|
||||||
|
except ServiceDeskConnectorError as exc:
|
||||||
|
raise _service_desk_http_error(exc) from exc
|
||||||
|
|
||||||
|
|
||||||
|
@router.post(
|
||||||
|
"/service-desk/profiles",
|
||||||
|
response_model=ServiceDeskProfileItem,
|
||||||
|
status_code=status.HTTP_201_CREATED,
|
||||||
|
)
|
||||||
|
def api_create_service_desk_profile(
|
||||||
|
payload: ServiceDeskProfileCreateRequest,
|
||||||
|
session: Session = Depends(get_session),
|
||||||
|
principal: ApiPrincipal = Depends(get_api_principal),
|
||||||
|
) -> ServiceDeskProfileItem:
|
||||||
|
try:
|
||||||
|
return create_service_desk_profile(session, principal, payload)
|
||||||
|
except ServiceDeskConnectorError as exc:
|
||||||
|
session.rollback()
|
||||||
|
raise _service_desk_http_error(exc) from exc
|
||||||
|
|
||||||
|
|
||||||
|
@router.put(
|
||||||
|
"/service-desk/profiles/{profile_id}",
|
||||||
|
response_model=ServiceDeskProfileItem,
|
||||||
|
)
|
||||||
|
def api_update_service_desk_profile(
|
||||||
|
profile_id: str,
|
||||||
|
payload: ServiceDeskProfileUpdateRequest,
|
||||||
|
session: Session = Depends(get_session),
|
||||||
|
principal: ApiPrincipal = Depends(get_api_principal),
|
||||||
|
) -> ServiceDeskProfileItem:
|
||||||
|
try:
|
||||||
|
return update_service_desk_profile(
|
||||||
|
session,
|
||||||
|
principal,
|
||||||
|
profile_id=profile_id,
|
||||||
|
payload=payload,
|
||||||
|
)
|
||||||
|
except ServiceDeskConnectorError as exc:
|
||||||
|
session.rollback()
|
||||||
|
raise _service_desk_http_error(exc) from exc
|
||||||
|
|
||||||
|
|
||||||
|
@router.post(
|
||||||
|
"/service-desk/profiles/{profile_id}/discover",
|
||||||
|
response_model=ServiceDeskDiscoveryResponse,
|
||||||
|
)
|
||||||
|
def api_discover_service_desk_profile(
|
||||||
|
profile_id: str,
|
||||||
|
session: Session = Depends(get_session),
|
||||||
|
principal: ApiPrincipal = Depends(get_api_principal),
|
||||||
|
) -> ServiceDeskDiscoveryResponse:
|
||||||
|
try:
|
||||||
|
return discover_service_desk_profile(
|
||||||
|
session,
|
||||||
|
principal,
|
||||||
|
profile_id=profile_id,
|
||||||
|
)
|
||||||
|
except ServiceDeskConnectorError as exc:
|
||||||
|
raise _service_desk_http_error(exc) from exc
|
||||||
|
|
||||||
|
|
||||||
|
@router.post(
|
||||||
|
"/service-desk/profiles/{profile_id}/sync",
|
||||||
|
response_model=ServiceDeskSyncRunItem,
|
||||||
|
)
|
||||||
|
def api_synchronize_service_desk_profile(
|
||||||
|
profile_id: str,
|
||||||
|
payload: ServiceDeskSyncRequest,
|
||||||
|
session: Session = Depends(get_session),
|
||||||
|
principal: ApiPrincipal = Depends(get_api_principal),
|
||||||
|
) -> ServiceDeskSyncRunItem:
|
||||||
|
try:
|
||||||
|
return synchronize_service_desk_profile(
|
||||||
|
session,
|
||||||
|
principal,
|
||||||
|
profile_id=profile_id,
|
||||||
|
payload=payload,
|
||||||
|
)
|
||||||
|
except ServiceDeskConnectorError as exc:
|
||||||
|
raise _service_desk_http_error(exc) from exc
|
||||||
|
|
||||||
|
|
||||||
|
@router.get(
|
||||||
|
"/service-desk/profiles/{profile_id}/objects",
|
||||||
|
response_model=ServiceDeskObjectListResponse,
|
||||||
|
)
|
||||||
|
def api_list_service_desk_objects(
|
||||||
|
profile_id: str,
|
||||||
|
cursor: str | None = Query(default=None, max_length=36),
|
||||||
|
limit: int = Query(default=100, ge=1, le=500),
|
||||||
|
session: Session = Depends(get_session),
|
||||||
|
principal: ApiPrincipal = Depends(get_api_principal),
|
||||||
|
) -> ServiceDeskObjectListResponse:
|
||||||
|
try:
|
||||||
|
items, next_cursor = list_service_desk_objects(
|
||||||
|
session,
|
||||||
|
principal,
|
||||||
|
profile_id=profile_id,
|
||||||
|
cursor=cursor,
|
||||||
|
limit=limit,
|
||||||
|
)
|
||||||
|
return ServiceDeskObjectListResponse(
|
||||||
|
items=list(items),
|
||||||
|
next_cursor=next_cursor,
|
||||||
|
)
|
||||||
|
except ServiceDeskConnectorError as exc:
|
||||||
|
raise _service_desk_http_error(exc) from exc
|
||||||
|
|
||||||
|
|
||||||
|
@router.get(
|
||||||
|
"/service-desk/runs",
|
||||||
|
response_model=ServiceDeskSyncRunListResponse,
|
||||||
|
)
|
||||||
|
def api_list_service_desk_runs(
|
||||||
|
profile_id: str | None = Query(default=None, max_length=36),
|
||||||
|
limit: int = Query(default=100, ge=1, le=500),
|
||||||
|
session: Session = Depends(get_session),
|
||||||
|
principal: ApiPrincipal = Depends(get_api_principal),
|
||||||
|
) -> ServiceDeskSyncRunListResponse:
|
||||||
|
try:
|
||||||
|
return ServiceDeskSyncRunListResponse(
|
||||||
|
items=list(
|
||||||
|
list_service_desk_runs(
|
||||||
|
session,
|
||||||
|
principal,
|
||||||
|
profile_id=profile_id,
|
||||||
|
limit=limit,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
)
|
||||||
|
except ServiceDeskConnectorError as exc:
|
||||||
|
raise _service_desk_http_error(exc) from exc
|
||||||
|
|
||||||
|
|
||||||
|
@router.post(
|
||||||
|
"/service-desk/profiles/{profile_id}/tickets/{external_ticket_id}/update",
|
||||||
|
response_model=ServiceDeskTicketUpdateResponse,
|
||||||
|
)
|
||||||
|
def api_update_service_desk_ticket(
|
||||||
|
profile_id: str,
|
||||||
|
external_ticket_id: str,
|
||||||
|
payload: ServiceDeskTicketUpdateRequest,
|
||||||
|
session: Session = Depends(get_session),
|
||||||
|
principal: ApiPrincipal = Depends(get_api_principal),
|
||||||
|
) -> ServiceDeskTicketUpdateResponse:
|
||||||
|
try:
|
||||||
|
return update_service_desk_ticket(
|
||||||
|
session,
|
||||||
|
principal,
|
||||||
|
profile_id=profile_id,
|
||||||
|
external_ticket_id=external_ticket_id,
|
||||||
|
payload=payload,
|
||||||
|
)
|
||||||
|
except ServiceDeskConnectorError as exc:
|
||||||
|
raise _service_desk_http_error(exc) from exc
|
||||||
|
|
||||||
|
|
||||||
def _source_response(source: TabularSource) -> TabularSourceResponse:
|
def _source_response(source: TabularSource) -> TabularSourceResponse:
|
||||||
return TabularSourceResponse(
|
return TabularSourceResponse(
|
||||||
ref=source.ref,
|
ref=source.ref,
|
||||||
|
|||||||
@@ -90,6 +90,7 @@ class SnapshotCreateRequest(BaseModel):
|
|||||||
format: Literal["json", "csv"] = "json"
|
format: Literal["json", "csv"] = "json"
|
||||||
rows: list[dict[str, Any]] | None = Field(default=None, max_length=10_000)
|
rows: list[dict[str, Any]] | None = Field(default=None, max_length=10_000)
|
||||||
csv_text: str | None = Field(default=None, max_length=5_000_000)
|
csv_text: str | None = Field(default=None, max_length=5_000_000)
|
||||||
|
csv_value_mode: Literal["legacy_typed", "text"] = "legacy_typed"
|
||||||
delimiter: Literal[",", ";", "\t", "|"] = ","
|
delimiter: Literal[",", ";", "\t", "|"] = ","
|
||||||
|
|
||||||
@model_validator(mode="after")
|
@model_validator(mode="after")
|
||||||
@@ -117,6 +118,7 @@ class ManagedFileSourceCreateRequest(BaseModel):
|
|||||||
file_version_id: str | None = Field(default=None, min_length=1, max_length=36)
|
file_version_id: str | None = Field(default=None, min_length=1, max_length=36)
|
||||||
delimiter: Literal[",", ";", "\t", "|"] = ","
|
delimiter: Literal[",", ";", "\t", "|"] = ","
|
||||||
sheet_name: str | None = Field(default=None, min_length=1, max_length=255)
|
sheet_name: str | None = Field(default=None, min_length=1, max_length=255)
|
||||||
|
csv_value_mode: Literal["legacy_typed", "text"] = "legacy_typed"
|
||||||
|
|
||||||
|
|
||||||
class SqlSourceCreateRequest(BaseModel):
|
class SqlSourceCreateRequest(BaseModel):
|
||||||
|
|||||||
+26
@@ -0,0 +1,26 @@
|
|||||||
|
"""Shared connector-search ACL token projection; owner authorization stays local."""
|
||||||
|
|
||||||
|
|
||||||
|
def principal_acl_tokens(principal: object) -> tuple[str, ...]:
|
||||||
|
# Keep legacy first-seen ordering and the exact 500-token authorization cap.
|
||||||
|
values: dict[str, None] = {}
|
||||||
|
for prefix, attribute in (
|
||||||
|
("account", "account_id"),
|
||||||
|
("membership", "membership_id"),
|
||||||
|
("identity", "identity_id"),
|
||||||
|
):
|
||||||
|
value = getattr(principal, attribute, None)
|
||||||
|
if value:
|
||||||
|
values[f"{prefix}:{value}"] = None
|
||||||
|
for prefix, attribute in (
|
||||||
|
("group", "group_ids"),
|
||||||
|
("role", "role_ids"),
|
||||||
|
("function", "function_assignment_ids"),
|
||||||
|
("scope", "scopes"),
|
||||||
|
):
|
||||||
|
for value in getattr(principal, attribute, ()):
|
||||||
|
if value:
|
||||||
|
values[f"{prefix}:{value}"] = None
|
||||||
|
if len(values) >= 500:
|
||||||
|
return tuple(values)
|
||||||
|
return tuple(values)
|
||||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,371 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from datetime import datetime
|
||||||
|
import json
|
||||||
|
from typing import Any, Literal
|
||||||
|
from urllib.parse import parse_qsl, urlsplit
|
||||||
|
|
||||||
|
from pydantic import BaseModel, ConfigDict, Field, model_validator
|
||||||
|
|
||||||
|
|
||||||
|
IntegrationMode = Literal["link", "import", "synchronize"]
|
||||||
|
ServiceDeskMaturity = Literal[
|
||||||
|
"discover",
|
||||||
|
"link",
|
||||||
|
"search",
|
||||||
|
"read",
|
||||||
|
"publish",
|
||||||
|
"synchronize",
|
||||||
|
]
|
||||||
|
|
||||||
|
_CREDENTIAL_CONTROL_KEYS = {
|
||||||
|
"authorization",
|
||||||
|
"auth_mode",
|
||||||
|
"sessionid",
|
||||||
|
"userlogin",
|
||||||
|
"customeruserlogin",
|
||||||
|
"password",
|
||||||
|
"x-otrs-header-sessionid",
|
||||||
|
"x-otrs-header-userlogin",
|
||||||
|
"x-otrs-header-customeruserlogin",
|
||||||
|
"x-otrs-header-password",
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
class ServiceDeskRouteMapping(BaseModel):
|
||||||
|
model_config = ConfigDict(extra="forbid")
|
||||||
|
|
||||||
|
search_path: str = Field(default="/Ticket/Search", min_length=1, max_length=500)
|
||||||
|
ticket_path: str = Field(default="/Ticket/{ticket_id}", min_length=1, max_length=500)
|
||||||
|
update_path: str | None = Field(default=None, max_length=500)
|
||||||
|
search_method: Literal["GET", "POST"] = "POST"
|
||||||
|
ticket_method: Literal["GET", "POST"] = "GET"
|
||||||
|
update_method: Literal["PATCH", "POST", "PUT"] = "PATCH"
|
||||||
|
ticket_web_url_template: str | None = Field(default=None, max_length=1500)
|
||||||
|
search_filters: dict[str, Any] = Field(default_factory=dict)
|
||||||
|
|
||||||
|
@model_validator(mode="after")
|
||||||
|
def validate_templates(self) -> "ServiceDeskRouteMapping":
|
||||||
|
for field_name in ("search_path", "ticket_path", "update_path"):
|
||||||
|
value = getattr(self, field_name)
|
||||||
|
if value is None:
|
||||||
|
continue
|
||||||
|
parsed = urlsplit(value)
|
||||||
|
if parsed.scheme or parsed.netloc or parsed.username or parsed.password:
|
||||||
|
raise ValueError(f"{field_name} must be relative to the governed endpoint")
|
||||||
|
if _credential_query_keys(parsed.query):
|
||||||
|
raise ValueError(f"{field_name} cannot contain authentication controls")
|
||||||
|
if any(part == ".." for part in parsed.path.split("/")):
|
||||||
|
raise ValueError(f"{field_name} cannot traverse parent paths")
|
||||||
|
if "{ticket_id}" not in self.ticket_path:
|
||||||
|
raise ValueError("ticket_path must contain {ticket_id}")
|
||||||
|
if self.update_path is not None and "{ticket_id}" not in self.update_path:
|
||||||
|
raise ValueError("update_path must contain {ticket_id}")
|
||||||
|
if (
|
||||||
|
self.ticket_web_url_template is not None
|
||||||
|
and "{ticket_id}" not in self.ticket_web_url_template
|
||||||
|
and "{ticket_number}" not in self.ticket_web_url_template
|
||||||
|
):
|
||||||
|
raise ValueError(
|
||||||
|
"ticket_web_url_template must contain {ticket_id} or {ticket_number}"
|
||||||
|
)
|
||||||
|
if self.ticket_web_url_template is not None:
|
||||||
|
parsed = urlsplit(self.ticket_web_url_template)
|
||||||
|
if parsed.scheme not in {"http", "https"} or not parsed.hostname:
|
||||||
|
raise ValueError("ticket_web_url_template must be an absolute HTTP(S) URL")
|
||||||
|
if parsed.username or parsed.password:
|
||||||
|
raise ValueError("ticket_web_url_template cannot contain credentials")
|
||||||
|
if _credential_query_keys(parsed.query):
|
||||||
|
raise ValueError(
|
||||||
|
"ticket_web_url_template cannot contain authentication controls"
|
||||||
|
)
|
||||||
|
if len(self.search_filters) > 100:
|
||||||
|
raise ValueError("search_filters supports at most 100 governed criteria")
|
||||||
|
reserved = _CREDENTIAL_CONTROL_KEYS | {
|
||||||
|
"limit",
|
||||||
|
"sortby",
|
||||||
|
"orderby",
|
||||||
|
"ticketchangetimenewerdate",
|
||||||
|
}
|
||||||
|
filter_names = {str(value).strip().casefold() for value in self.search_filters}
|
||||||
|
if "" in filter_names:
|
||||||
|
raise ValueError("search_filters keys cannot be empty")
|
||||||
|
if reserved.intersection(filter_names):
|
||||||
|
raise ValueError("search_filters cannot override cursors, bounds, ordering, or authentication")
|
||||||
|
if len(json.dumps(self.search_filters, default=str)) > 20_000:
|
||||||
|
raise ValueError("search_filters exceeds the 20000-character policy limit")
|
||||||
|
return self
|
||||||
|
|
||||||
|
|
||||||
|
class ServiceDeskQueueMapping(BaseModel):
|
||||||
|
model_config = ConfigDict(extra="forbid")
|
||||||
|
|
||||||
|
source_queue: str = Field(min_length=1, max_length=300)
|
||||||
|
target_queue_ref: str | None = Field(default=None, max_length=255)
|
||||||
|
include: bool = True
|
||||||
|
visibility: Literal["tenant", "restricted"] = "restricted"
|
||||||
|
acl_tokens: list[str] = Field(default_factory=list, max_length=200)
|
||||||
|
|
||||||
|
|
||||||
|
class ServiceDeskDynamicFieldMapping(BaseModel):
|
||||||
|
model_config = ConfigDict(extra="forbid")
|
||||||
|
|
||||||
|
source_name: str = Field(min_length=1, max_length=255)
|
||||||
|
target_name: str | None = Field(default=None, max_length=255)
|
||||||
|
include: bool = True
|
||||||
|
value_type: Literal["string", "number", "boolean", "date", "json"] = "string"
|
||||||
|
|
||||||
|
|
||||||
|
class ServiceDeskProfileCreateRequest(BaseModel):
|
||||||
|
model_config = ConfigDict(extra="forbid")
|
||||||
|
|
||||||
|
configuration_id: str = Field(min_length=1, max_length=36)
|
||||||
|
integration_mode: IntegrationMode = "synchronize"
|
||||||
|
desired_maturity: ServiceDeskMaturity = "synchronize"
|
||||||
|
source_authority_mode: Literal[
|
||||||
|
"external_authoritative",
|
||||||
|
"external_mirror",
|
||||||
|
"governed_sync",
|
||||||
|
"linked_reference",
|
||||||
|
] = "external_authoritative"
|
||||||
|
default_visibility: Literal["tenant", "restricted"] = "restricted"
|
||||||
|
default_acl_tokens: list[str] = Field(default_factory=list, max_length=200)
|
||||||
|
routes: ServiceDeskRouteMapping = Field(default_factory=ServiceDeskRouteMapping)
|
||||||
|
queue_mappings: list[ServiceDeskQueueMapping] = Field(
|
||||||
|
default_factory=list, max_length=500
|
||||||
|
)
|
||||||
|
dynamic_field_mappings: list[ServiceDeskDynamicFieldMapping] = Field(
|
||||||
|
default_factory=list, max_length=500
|
||||||
|
)
|
||||||
|
|
||||||
|
@model_validator(mode="after")
|
||||||
|
def validate_policy(self) -> "ServiceDeskProfileCreateRequest":
|
||||||
|
_validate_profile_policy(
|
||||||
|
integration_mode=self.integration_mode,
|
||||||
|
authority_mode=self.source_authority_mode,
|
||||||
|
visibility=self.default_visibility,
|
||||||
|
acl_tokens=self.default_acl_tokens,
|
||||||
|
)
|
||||||
|
order = ("discover", "link", "search", "read", "publish", "synchronize")
|
||||||
|
maximum = {"link": "search", "import": "read", "synchronize": "synchronize"}[
|
||||||
|
self.integration_mode
|
||||||
|
]
|
||||||
|
minimum = {"link": "link", "import": "read", "synchronize": "synchronize"}[
|
||||||
|
self.integration_mode
|
||||||
|
]
|
||||||
|
if order.index(self.desired_maturity) < order.index(minimum):
|
||||||
|
raise ValueError(
|
||||||
|
f"{self.integration_mode} mode requires at least {minimum} maturity"
|
||||||
|
)
|
||||||
|
if order.index(self.desired_maturity) > order.index(maximum):
|
||||||
|
raise ValueError(
|
||||||
|
f"{self.integration_mode} mode cannot declare maturity above {maximum}"
|
||||||
|
)
|
||||||
|
return self
|
||||||
|
|
||||||
|
|
||||||
|
class ServiceDeskProfileUpdateRequest(BaseModel):
|
||||||
|
model_config = ConfigDict(extra="forbid")
|
||||||
|
|
||||||
|
expected_resource_revision: int = Field(ge=1)
|
||||||
|
status: Literal["active", "paused"] | None = None
|
||||||
|
integration_mode: IntegrationMode | None = None
|
||||||
|
desired_maturity: ServiceDeskMaturity | None = None
|
||||||
|
source_authority_mode: Literal[
|
||||||
|
"external_authoritative",
|
||||||
|
"external_mirror",
|
||||||
|
"governed_sync",
|
||||||
|
"linked_reference",
|
||||||
|
] | None = None
|
||||||
|
default_visibility: Literal["tenant", "restricted"] | None = None
|
||||||
|
default_acl_tokens: list[str] | None = Field(default=None, max_length=200)
|
||||||
|
routes: ServiceDeskRouteMapping | None = None
|
||||||
|
queue_mappings: list[ServiceDeskQueueMapping] | None = Field(
|
||||||
|
default=None, max_length=500
|
||||||
|
)
|
||||||
|
dynamic_field_mappings: list[ServiceDeskDynamicFieldMapping] | None = Field(
|
||||||
|
default=None, max_length=500
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
class ServiceDeskDiagnostic(BaseModel):
|
||||||
|
severity: Literal["info", "warning", "error"]
|
||||||
|
code: str
|
||||||
|
message: str
|
||||||
|
object_ref: str | None = None
|
||||||
|
field: str | None = None
|
||||||
|
retryable: bool = False
|
||||||
|
details: dict[str, Any] = Field(default_factory=dict)
|
||||||
|
|
||||||
|
|
||||||
|
class ServiceDeskProfileItem(BaseModel):
|
||||||
|
id: str
|
||||||
|
tenant_id: str
|
||||||
|
configuration_id: str
|
||||||
|
status: str
|
||||||
|
integration_mode: str
|
||||||
|
product: str
|
||||||
|
product_version: str | None = None
|
||||||
|
desired_maturity: str
|
||||||
|
discovered_maturity: str
|
||||||
|
source_authority_mode: str
|
||||||
|
default_visibility: str
|
||||||
|
default_acl_tokens: list[str]
|
||||||
|
routes: ServiceDeskRouteMapping
|
||||||
|
queue_mappings: list[ServiceDeskQueueMapping]
|
||||||
|
dynamic_field_mappings: list[ServiceDeskDynamicFieldMapping]
|
||||||
|
capabilities: list[str]
|
||||||
|
discovery_revision: str | None = None
|
||||||
|
discovered_configuration_revision: int | None = None
|
||||||
|
discovered_configuration_hash: str | None = None
|
||||||
|
health_status: str
|
||||||
|
health_details: dict[str, Any]
|
||||||
|
discovered_at: datetime | None = None
|
||||||
|
last_sync_cursor: str | None = None
|
||||||
|
last_high_watermark: str | None = None
|
||||||
|
resource_revision: int
|
||||||
|
credential_reference_present: bool
|
||||||
|
endpoint_configured: bool
|
||||||
|
created_at: datetime
|
||||||
|
updated_at: datetime
|
||||||
|
|
||||||
|
|
||||||
|
class ServiceDeskProfileListResponse(BaseModel):
|
||||||
|
items: list[ServiceDeskProfileItem]
|
||||||
|
|
||||||
|
|
||||||
|
class ServiceDeskDiscoveryResponse(BaseModel):
|
||||||
|
profile: ServiceDeskProfileItem
|
||||||
|
product: str
|
||||||
|
product_version: str | None = None
|
||||||
|
api_family: str
|
||||||
|
capabilities: list[str]
|
||||||
|
maturity: str
|
||||||
|
health_status: str
|
||||||
|
diagnostics: list[ServiceDeskDiagnostic]
|
||||||
|
revision: str
|
||||||
|
|
||||||
|
|
||||||
|
class ServiceDeskSyncRequest(BaseModel):
|
||||||
|
model_config = ConfigDict(extra="forbid")
|
||||||
|
|
||||||
|
idempotency_key: str = Field(min_length=1, max_length=255)
|
||||||
|
mode: Literal["auto", "full", "delta"] = "auto"
|
||||||
|
cursor: str | None = Field(default=None, max_length=4000)
|
||||||
|
limit: int = Field(default=100, ge=1, le=500)
|
||||||
|
|
||||||
|
|
||||||
|
class ServiceDeskObjectItem(BaseModel):
|
||||||
|
id: str
|
||||||
|
profile_id: str
|
||||||
|
object_type: str
|
||||||
|
external_id: str
|
||||||
|
external_ticket_number: str | None = None
|
||||||
|
title: str
|
||||||
|
canonical_url: str | None = None
|
||||||
|
status: str
|
||||||
|
source_revision: str
|
||||||
|
visibility: str
|
||||||
|
acl_tokens: list[str]
|
||||||
|
mapped_data: dict[str, Any]
|
||||||
|
provenance: dict[str, Any]
|
||||||
|
source_updated_at: datetime | None = None
|
||||||
|
observed_at: datetime
|
||||||
|
resource_revision: int
|
||||||
|
|
||||||
|
|
||||||
|
class ServiceDeskObjectListResponse(BaseModel):
|
||||||
|
items: list[ServiceDeskObjectItem]
|
||||||
|
next_cursor: str | None = None
|
||||||
|
|
||||||
|
|
||||||
|
class ServiceDeskSyncRunItem(BaseModel):
|
||||||
|
id: str
|
||||||
|
profile_id: str
|
||||||
|
mode: str
|
||||||
|
idempotency_key: str
|
||||||
|
status: str
|
||||||
|
cursor_before: str | None = None
|
||||||
|
cursor_after: str | None = None
|
||||||
|
high_watermark: str | None = None
|
||||||
|
counts: dict[str, int]
|
||||||
|
effects: list[dict[str, Any]]
|
||||||
|
diagnostics: list[ServiceDeskDiagnostic]
|
||||||
|
provenance: dict[str, Any]
|
||||||
|
started_at: datetime
|
||||||
|
finished_at: datetime | None = None
|
||||||
|
created_at: datetime
|
||||||
|
|
||||||
|
|
||||||
|
class ServiceDeskSyncRunListResponse(BaseModel):
|
||||||
|
items: list[ServiceDeskSyncRunItem]
|
||||||
|
|
||||||
|
|
||||||
|
class ServiceDeskTicketUpdateRequest(BaseModel):
|
||||||
|
model_config = ConfigDict(extra="forbid")
|
||||||
|
|
||||||
|
idempotency_key: str = Field(min_length=1, max_length=255)
|
||||||
|
expected_external_revision: str = Field(min_length=1, max_length=255)
|
||||||
|
title: str | None = Field(default=None, min_length=1, max_length=500)
|
||||||
|
queue: str | None = Field(default=None, min_length=1, max_length=300)
|
||||||
|
state: str | None = Field(default=None, min_length=1, max_length=200)
|
||||||
|
priority: str | None = Field(default=None, min_length=1, max_length=200)
|
||||||
|
owner: str | None = Field(default=None, min_length=1, max_length=255)
|
||||||
|
responsible: str | None = Field(default=None, min_length=1, max_length=255)
|
||||||
|
dynamic_fields: dict[str, Any] = Field(default_factory=dict)
|
||||||
|
|
||||||
|
@model_validator(mode="after")
|
||||||
|
def require_change(self) -> "ServiceDeskTicketUpdateRequest":
|
||||||
|
if not any(
|
||||||
|
(
|
||||||
|
self.title,
|
||||||
|
self.queue,
|
||||||
|
self.state,
|
||||||
|
self.priority,
|
||||||
|
self.owner,
|
||||||
|
self.responsible,
|
||||||
|
self.dynamic_fields,
|
||||||
|
)
|
||||||
|
):
|
||||||
|
raise ValueError("At least one supported ticket field must change")
|
||||||
|
if len(self.dynamic_fields) > 100:
|
||||||
|
raise ValueError("At most 100 dynamic fields may be updated")
|
||||||
|
return self
|
||||||
|
|
||||||
|
|
||||||
|
class ServiceDeskTicketUpdateResponse(BaseModel):
|
||||||
|
run: ServiceDeskSyncRunItem
|
||||||
|
object: ServiceDeskObjectItem
|
||||||
|
accepted: bool
|
||||||
|
outcome_unknown: bool
|
||||||
|
|
||||||
|
|
||||||
|
def _validate_profile_policy(
|
||||||
|
*,
|
||||||
|
integration_mode: str,
|
||||||
|
authority_mode: str,
|
||||||
|
visibility: str,
|
||||||
|
acl_tokens: list[str],
|
||||||
|
) -> None:
|
||||||
|
allowed = {
|
||||||
|
"link": {"linked_reference"},
|
||||||
|
"import": {"external_authoritative", "external_mirror"},
|
||||||
|
"synchronize": {"external_authoritative", "governed_sync"},
|
||||||
|
}
|
||||||
|
if authority_mode not in allowed[integration_mode]:
|
||||||
|
raise ValueError(
|
||||||
|
f"{integration_mode} mode does not support {authority_mode} authority"
|
||||||
|
)
|
||||||
|
if visibility == "restricted" and not acl_tokens:
|
||||||
|
raise ValueError("Restricted profiles require at least one ACL token")
|
||||||
|
|
||||||
|
|
||||||
|
def _credential_query_keys(query: str) -> set[str]:
|
||||||
|
return {
|
||||||
|
str(key).strip().casefold()
|
||||||
|
for key, _value in parse_qsl(query, keep_blank_values=True)
|
||||||
|
if str(key).strip().casefold() in _CREDENTIAL_CONTROL_KEYS
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
__all__ = [name for name in globals() if name.startswith("ServiceDesk")]
|
||||||
@@ -0,0 +1,315 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from govoplan_connectors.backend.search_principal import principal_acl_tokens as _principal_acl_tokens
|
||||||
|
|
||||||
|
from collections.abc import Mapping, Sequence
|
||||||
|
from urllib.parse import quote
|
||||||
|
|
||||||
|
from sqlalchemy import func, select
|
||||||
|
from sqlalchemy.orm import Session
|
||||||
|
|
||||||
|
from govoplan_core.core.external_references import ExternalObjectReference
|
||||||
|
from govoplan_core.core.modules import ModuleContext
|
||||||
|
from govoplan_core.core.search import (
|
||||||
|
SearchAuthorizationRequest,
|
||||||
|
SearchBackfillPage,
|
||||||
|
SearchBackfillRequest,
|
||||||
|
SearchDocument,
|
||||||
|
SearchResourceType,
|
||||||
|
)
|
||||||
|
from govoplan_connectors.backend.db.models import (
|
||||||
|
ConnectorConfiguration,
|
||||||
|
ConnectorServiceDeskObject,
|
||||||
|
ConnectorServiceDeskProfile,
|
||||||
|
)
|
||||||
|
from govoplan_connectors.backend.service_desk_connector import (
|
||||||
|
SERVICE_DESK_PROVIDER_ID,
|
||||||
|
SERVICE_DESK_READ_SCOPE,
|
||||||
|
SERVICE_DESK_RESOURCE_TYPE,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
_SEARCH_MATURITIES = ("search", "read", "publish", "synchronize", "migrate", "replace")
|
||||||
|
|
||||||
|
|
||||||
|
class ExternalServiceDeskSearchSource:
|
||||||
|
def resource_types(self) -> Sequence[SearchResourceType]:
|
||||||
|
return (
|
||||||
|
SearchResourceType(
|
||||||
|
provider_id=SERVICE_DESK_PROVIDER_ID,
|
||||||
|
module_id="connectors",
|
||||||
|
resource_type=SERVICE_DESK_RESOURCE_TYPE,
|
||||||
|
label="External service-desk tickets",
|
||||||
|
requires_authorization_recheck=True,
|
||||||
|
),
|
||||||
|
)
|
||||||
|
|
||||||
|
def backfill(
|
||||||
|
self, session: object, *, request: SearchBackfillRequest
|
||||||
|
) -> SearchBackfillPage:
|
||||||
|
_assert_source(request.provider_id, request.resource_type)
|
||||||
|
db = _session(session)
|
||||||
|
query = (
|
||||||
|
select(ConnectorServiceDeskObject, ConnectorServiceDeskProfile)
|
||||||
|
.join(
|
||||||
|
ConnectorServiceDeskProfile,
|
||||||
|
ConnectorServiceDeskProfile.id == ConnectorServiceDeskObject.profile_id,
|
||||||
|
)
|
||||||
|
.join(
|
||||||
|
ConnectorConfiguration,
|
||||||
|
ConnectorConfiguration.id
|
||||||
|
== ConnectorServiceDeskProfile.configuration_id,
|
||||||
|
)
|
||||||
|
.where(
|
||||||
|
ConnectorServiceDeskObject.tenant_id == request.tenant_id,
|
||||||
|
ConnectorServiceDeskObject.object_type == "ticket",
|
||||||
|
ConnectorServiceDeskObject.status != "deleted",
|
||||||
|
ConnectorServiceDeskProfile.tenant_id == request.tenant_id,
|
||||||
|
ConnectorServiceDeskProfile.status == "active",
|
||||||
|
ConnectorConfiguration.tenant_id == request.tenant_id,
|
||||||
|
ConnectorConfiguration.status == "active",
|
||||||
|
ConnectorServiceDeskProfile.discovered_configuration_revision
|
||||||
|
== ConnectorConfiguration.resource_revision,
|
||||||
|
ConnectorServiceDeskProfile.discovered_configuration_hash
|
||||||
|
== ConnectorConfiguration.effective_hash,
|
||||||
|
ConnectorServiceDeskProfile.desired_maturity.in_(_SEARCH_MATURITIES),
|
||||||
|
ConnectorServiceDeskProfile.discovered_maturity.in_(_SEARCH_MATURITIES),
|
||||||
|
)
|
||||||
|
)
|
||||||
|
if request.cursor:
|
||||||
|
query = query.where(ConnectorServiceDeskObject.id > request.cursor)
|
||||||
|
rows = tuple(
|
||||||
|
db.execute(
|
||||||
|
query.order_by(ConnectorServiceDeskObject.id.asc()).limit(request.limit + 1)
|
||||||
|
)
|
||||||
|
)
|
||||||
|
has_more = len(rows) > request.limit
|
||||||
|
selected = rows[: request.limit]
|
||||||
|
watermark = db.scalar(
|
||||||
|
select(func.max(ConnectorServiceDeskObject.updated_at))
|
||||||
|
.join(
|
||||||
|
ConnectorServiceDeskProfile,
|
||||||
|
ConnectorServiceDeskProfile.id == ConnectorServiceDeskObject.profile_id,
|
||||||
|
)
|
||||||
|
.join(
|
||||||
|
ConnectorConfiguration,
|
||||||
|
ConnectorConfiguration.id
|
||||||
|
== ConnectorServiceDeskProfile.configuration_id,
|
||||||
|
)
|
||||||
|
.where(
|
||||||
|
ConnectorServiceDeskObject.tenant_id == request.tenant_id,
|
||||||
|
ConnectorServiceDeskObject.object_type == "ticket",
|
||||||
|
ConnectorServiceDeskObject.status != "deleted",
|
||||||
|
ConnectorServiceDeskProfile.tenant_id == request.tenant_id,
|
||||||
|
ConnectorServiceDeskProfile.status == "active",
|
||||||
|
ConnectorConfiguration.tenant_id == request.tenant_id,
|
||||||
|
ConnectorConfiguration.status == "active",
|
||||||
|
ConnectorServiceDeskProfile.discovered_configuration_revision
|
||||||
|
== ConnectorConfiguration.resource_revision,
|
||||||
|
ConnectorServiceDeskProfile.discovered_configuration_hash
|
||||||
|
== ConnectorConfiguration.effective_hash,
|
||||||
|
ConnectorServiceDeskProfile.desired_maturity.in_(_SEARCH_MATURITIES),
|
||||||
|
ConnectorServiceDeskProfile.discovered_maturity.in_(_SEARCH_MATURITIES),
|
||||||
|
)
|
||||||
|
)
|
||||||
|
return SearchBackfillPage(
|
||||||
|
documents=tuple(search_document(db, row, profile) for row, profile in selected),
|
||||||
|
next_cursor=selected[-1][0].id if has_more and selected else None,
|
||||||
|
complete=not has_more,
|
||||||
|
high_watermark=watermark.isoformat() if watermark else None,
|
||||||
|
)
|
||||||
|
|
||||||
|
def authorize(
|
||||||
|
self,
|
||||||
|
session: object,
|
||||||
|
principal: object,
|
||||||
|
*,
|
||||||
|
requests: Sequence[SearchAuthorizationRequest],
|
||||||
|
) -> Mapping[str, bool]:
|
||||||
|
db = _session(session)
|
||||||
|
tenant_id = str(getattr(principal, "tenant_id", "") or "")
|
||||||
|
can_read = _has_scope(principal, SERVICE_DESK_READ_SCOPE)
|
||||||
|
tokens = set(_principal_acl_tokens(principal))
|
||||||
|
decisions = {item.reference.key: False for item in requests}
|
||||||
|
if not tenant_id or not can_read:
|
||||||
|
return decisions
|
||||||
|
for request in requests:
|
||||||
|
reference = request.reference
|
||||||
|
if (
|
||||||
|
reference.tenant_id != tenant_id
|
||||||
|
or reference.module_id != "connectors"
|
||||||
|
or reference.resource_type != SERVICE_DESK_RESOURCE_TYPE
|
||||||
|
):
|
||||||
|
continue
|
||||||
|
joined = db.execute(
|
||||||
|
select(ConnectorServiceDeskObject, ConnectorServiceDeskProfile)
|
||||||
|
.join(
|
||||||
|
ConnectorServiceDeskProfile,
|
||||||
|
ConnectorServiceDeskProfile.id
|
||||||
|
== ConnectorServiceDeskObject.profile_id,
|
||||||
|
)
|
||||||
|
.join(
|
||||||
|
ConnectorConfiguration,
|
||||||
|
ConnectorConfiguration.id
|
||||||
|
== ConnectorServiceDeskProfile.configuration_id,
|
||||||
|
)
|
||||||
|
.where(
|
||||||
|
ConnectorServiceDeskObject.tenant_id == tenant_id,
|
||||||
|
ConnectorServiceDeskObject.id == reference.resource_id,
|
||||||
|
ConnectorServiceDeskObject.object_type == "ticket",
|
||||||
|
ConnectorServiceDeskObject.status != "deleted",
|
||||||
|
ConnectorServiceDeskProfile.tenant_id == tenant_id,
|
||||||
|
ConnectorServiceDeskProfile.status == "active",
|
||||||
|
ConnectorConfiguration.tenant_id == tenant_id,
|
||||||
|
ConnectorConfiguration.status == "active",
|
||||||
|
ConnectorServiceDeskProfile.discovered_configuration_revision
|
||||||
|
== ConnectorConfiguration.resource_revision,
|
||||||
|
ConnectorServiceDeskProfile.discovered_configuration_hash
|
||||||
|
== ConnectorConfiguration.effective_hash,
|
||||||
|
ConnectorServiceDeskProfile.desired_maturity.in_(_SEARCH_MATURITIES),
|
||||||
|
ConnectorServiceDeskProfile.discovered_maturity.in_(_SEARCH_MATURITIES),
|
||||||
|
)
|
||||||
|
).first()
|
||||||
|
if joined is None:
|
||||||
|
continue
|
||||||
|
row, _profile = joined
|
||||||
|
decisions[reference.key] = row.visibility == "tenant" or bool(
|
||||||
|
tokens.intersection(str(value) for value in row.acl_tokens or ())
|
||||||
|
)
|
||||||
|
return decisions
|
||||||
|
|
||||||
|
|
||||||
|
def create_external_service_desk_search_source(
|
||||||
|
_context: ModuleContext,
|
||||||
|
) -> ExternalServiceDeskSearchSource:
|
||||||
|
return ExternalServiceDeskSearchSource()
|
||||||
|
|
||||||
|
|
||||||
|
def search_document(
|
||||||
|
session: Session,
|
||||||
|
row: ConnectorServiceDeskObject,
|
||||||
|
profile: ConnectorServiceDeskProfile | None = None,
|
||||||
|
) -> SearchDocument:
|
||||||
|
if profile is None:
|
||||||
|
profile = session.scalar(
|
||||||
|
select(ConnectorServiceDeskProfile).where(
|
||||||
|
ConnectorServiceDeskProfile.tenant_id == row.tenant_id,
|
||||||
|
ConnectorServiceDeskProfile.id == row.profile_id,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
if profile is None:
|
||||||
|
raise ValueError("External service-desk profile is unavailable.")
|
||||||
|
data = dict(row.mapped_data or {})
|
||||||
|
queue = data.get("queue") if isinstance(data.get("queue"), Mapping) else {}
|
||||||
|
state = data.get("state") if isinstance(data.get("state"), Mapping) else {}
|
||||||
|
priority = data.get("priority") if isinstance(data.get("priority"), Mapping) else {}
|
||||||
|
articles = [value for value in data.get("articles") or () if isinstance(value, Mapping)]
|
||||||
|
article_subjects = tuple(
|
||||||
|
str(value.get("subject") or "")[:500]
|
||||||
|
for value in articles
|
||||||
|
if value.get("subject")
|
||||||
|
)
|
||||||
|
article_body = "\n\n".join(
|
||||||
|
str(value.get("body") or "") for value in articles if value.get("body")
|
||||||
|
)[:200_000]
|
||||||
|
dynamic_fields = data.get("dynamic_fields")
|
||||||
|
dynamic_keywords = tuple(
|
||||||
|
f"{key}:{str(value)[:200]}"
|
||||||
|
for key, value in (
|
||||||
|
dynamic_fields.items() if isinstance(dynamic_fields, Mapping) else ()
|
||||||
|
)
|
||||||
|
)
|
||||||
|
external_reference = ExternalObjectReference(
|
||||||
|
system=profile.product if profile.product != "unknown" else "znuny_otrs",
|
||||||
|
object_type=row.object_type,
|
||||||
|
object_id=row.external_id,
|
||||||
|
maturity=profile.discovered_maturity,
|
||||||
|
authority_mode=profile.source_authority_mode,
|
||||||
|
connector_id=profile.id,
|
||||||
|
canonical_url=row.canonical_url,
|
||||||
|
version=row.source_revision,
|
||||||
|
etag=row.content_hash,
|
||||||
|
observed_at=row.observed_at,
|
||||||
|
metadata={
|
||||||
|
"ticket_number": row.external_ticket_number,
|
||||||
|
"title": row.title,
|
||||||
|
"queue": queue.get("name"),
|
||||||
|
"state": state.get("name"),
|
||||||
|
},
|
||||||
|
)
|
||||||
|
return SearchDocument(
|
||||||
|
tenant_id=row.tenant_id,
|
||||||
|
module_id="connectors",
|
||||||
|
provider_id=SERVICE_DESK_PROVIDER_ID,
|
||||||
|
resource_type=SERVICE_DESK_RESOURCE_TYPE,
|
||||||
|
resource_id=row.id,
|
||||||
|
title=(
|
||||||
|
f"{row.external_ticket_number}: {row.title}"
|
||||||
|
if row.external_ticket_number
|
||||||
|
else row.title
|
||||||
|
),
|
||||||
|
url=(
|
||||||
|
"/connectors/service-desk?profileId="
|
||||||
|
f"{quote(row.profile_id, safe='')}&objectId={quote(row.id, safe='')}"
|
||||||
|
),
|
||||||
|
summary=(article_subjects[0] if article_subjects else None),
|
||||||
|
body=article_body or None,
|
||||||
|
keywords=tuple(
|
||||||
|
dict.fromkeys(
|
||||||
|
value
|
||||||
|
for value in (
|
||||||
|
str(queue.get("name") or ""),
|
||||||
|
str(state.get("name") or ""),
|
||||||
|
str(priority.get("name") or ""),
|
||||||
|
*article_subjects,
|
||||||
|
*dynamic_keywords,
|
||||||
|
)
|
||||||
|
if value
|
||||||
|
)
|
||||||
|
)[:100],
|
||||||
|
visibility=row.visibility,
|
||||||
|
acl_tokens=(
|
||||||
|
tuple(str(value) for value in row.acl_tokens or ())
|
||||||
|
if row.visibility == "restricted"
|
||||||
|
else ()
|
||||||
|
),
|
||||||
|
external_reference=external_reference,
|
||||||
|
metadata={
|
||||||
|
"profile_id": row.profile_id,
|
||||||
|
"external_ticket_id": row.external_id,
|
||||||
|
"external_ticket_number": row.external_ticket_number,
|
||||||
|
"target_queue_ref": data.get("target_queue_ref"),
|
||||||
|
"integration_mode": profile.integration_mode,
|
||||||
|
"source_authority_mode": profile.source_authority_mode,
|
||||||
|
"status": row.status,
|
||||||
|
},
|
||||||
|
source_revision=row.source_revision,
|
||||||
|
change_cursor=row.change_cursor,
|
||||||
|
source_updated_at=row.source_updated_at or row.observed_at,
|
||||||
|
requires_authorization_recheck=True,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _has_scope(principal: object, required: str) -> bool:
|
||||||
|
check = getattr(principal, "has", None)
|
||||||
|
if callable(check):
|
||||||
|
return bool(check(required))
|
||||||
|
return required in getattr(principal, "scopes", ())
|
||||||
|
|
||||||
|
|
||||||
|
def _assert_source(provider_id: str, resource_type: str) -> None:
|
||||||
|
if provider_id != SERVICE_DESK_PROVIDER_ID or resource_type != SERVICE_DESK_RESOURCE_TYPE:
|
||||||
|
raise ValueError("Unsupported external service-desk Search source.")
|
||||||
|
|
||||||
|
|
||||||
|
def _session(value: object) -> Session:
|
||||||
|
if not isinstance(value, Session):
|
||||||
|
raise TypeError("External service-desk Search requires a SQLAlchemy session.")
|
||||||
|
return value
|
||||||
|
|
||||||
|
|
||||||
|
__all__ = [
|
||||||
|
"ExternalServiceDeskSearchSource",
|
||||||
|
"create_external_service_desk_search_source",
|
||||||
|
"search_document",
|
||||||
|
]
|
||||||
@@ -0,0 +1,896 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import hashlib
|
||||||
|
import json
|
||||||
|
import socket
|
||||||
|
import urllib.error
|
||||||
|
from collections.abc import Mapping, Sequence
|
||||||
|
from dataclasses import dataclass
|
||||||
|
from datetime import datetime, timedelta, timezone
|
||||||
|
from typing import Any, Protocol
|
||||||
|
from urllib.parse import quote, urlencode, urljoin, urlsplit
|
||||||
|
|
||||||
|
from govoplan_core.security.http_fetch import HttpFetchResponse, fetch_http
|
||||||
|
from govoplan_core.security.outbound_http import OutboundHttpError
|
||||||
|
|
||||||
|
|
||||||
|
MAX_SERVICE_DESK_RESPONSE_BYTES = 10_000_000
|
||||||
|
MAX_SERVICE_DESK_SEARCH_IDS = 10_000
|
||||||
|
MAX_SERVICE_DESK_TICKET_READS = 500
|
||||||
|
SERVICE_DESK_SENSITIVE_HEADERS = (
|
||||||
|
"X-OTRS-Header-UserLogin",
|
||||||
|
"X-OTRS-Header-CustomerUserLogin",
|
||||||
|
"X-OTRS-Header-Password",
|
||||||
|
"X-OTRS-Header-SessionID",
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
class ServiceDeskTransportError(RuntimeError):
|
||||||
|
def __init__(
|
||||||
|
self,
|
||||||
|
code: str,
|
||||||
|
message: str,
|
||||||
|
*,
|
||||||
|
retryable: bool = False,
|
||||||
|
outcome_unknown: bool = False,
|
||||||
|
) -> None:
|
||||||
|
super().__init__(message)
|
||||||
|
self.code = code
|
||||||
|
self.retryable = retryable
|
||||||
|
self.outcome_unknown = outcome_unknown
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True, slots=True)
|
||||||
|
class ServiceDeskChangeBatch:
|
||||||
|
changes: tuple[Mapping[str, Any], ...]
|
||||||
|
next_cursor: str | None
|
||||||
|
complete: bool
|
||||||
|
high_watermark: str | None
|
||||||
|
live_ids: tuple[str, ...] | None
|
||||||
|
evidence: Mapping[str, Any]
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True, slots=True)
|
||||||
|
class ServiceDeskUpdateResult:
|
||||||
|
ticket: Mapping[str, Any]
|
||||||
|
revision: str
|
||||||
|
evidence: Mapping[str, Any]
|
||||||
|
|
||||||
|
|
||||||
|
class ServiceDeskTransport(Protocol):
|
||||||
|
def discover(
|
||||||
|
self,
|
||||||
|
*,
|
||||||
|
endpoint_url: str,
|
||||||
|
credential: Mapping[str, Any] | None,
|
||||||
|
routes: Mapping[str, Any],
|
||||||
|
) -> Mapping[str, Any]: ...
|
||||||
|
|
||||||
|
def changes(
|
||||||
|
self,
|
||||||
|
*,
|
||||||
|
endpoint_url: str,
|
||||||
|
credential: Mapping[str, Any] | None,
|
||||||
|
routes: Mapping[str, Any],
|
||||||
|
cursor: str | None,
|
||||||
|
limit: int,
|
||||||
|
force_full: bool,
|
||||||
|
) -> ServiceDeskChangeBatch: ...
|
||||||
|
|
||||||
|
def update_ticket(
|
||||||
|
self,
|
||||||
|
*,
|
||||||
|
endpoint_url: str,
|
||||||
|
credential: Mapping[str, Any] | None,
|
||||||
|
routes: Mapping[str, Any],
|
||||||
|
ticket_id: str,
|
||||||
|
expected_revision: str,
|
||||||
|
changes: Mapping[str, Any],
|
||||||
|
) -> ServiceDeskUpdateResult: ...
|
||||||
|
|
||||||
|
|
||||||
|
class HttpServiceDeskTransport:
|
||||||
|
"""Bounded Znuny/OTRS GenericInterface REST transport.
|
||||||
|
|
||||||
|
GenericInterface route names are administrator-defined. The governed profile
|
||||||
|
supplies the paths and methods while this adapter enforces outbound policy,
|
||||||
|
response bounds, credential placement, cursor stability, and write recovery
|
||||||
|
semantics.
|
||||||
|
"""
|
||||||
|
|
||||||
|
def discover(
|
||||||
|
self,
|
||||||
|
*,
|
||||||
|
endpoint_url: str,
|
||||||
|
credential: Mapping[str, Any] | None,
|
||||||
|
routes: Mapping[str, Any],
|
||||||
|
) -> Mapping[str, Any]:
|
||||||
|
payload, response = self._search(
|
||||||
|
endpoint_url,
|
||||||
|
credential=credential,
|
||||||
|
routes=routes,
|
||||||
|
criteria={"Limit": 1, "SortBy": ["Changed"], "OrderBy": ["Up"]},
|
||||||
|
)
|
||||||
|
product, version = _product_version(payload, response.headers, endpoint_url)
|
||||||
|
recognized = product in {"znuny", "otrs"} and _major(version) >= 6
|
||||||
|
capabilities = ["discover", "link", "search", "read"]
|
||||||
|
if recognized:
|
||||||
|
capabilities.append("synchronize")
|
||||||
|
if recognized and routes.get("update_path"):
|
||||||
|
capabilities.append("publish")
|
||||||
|
maturity = "synchronize" if "synchronize" in capabilities else "read"
|
||||||
|
revision = _hash(
|
||||||
|
{
|
||||||
|
"product": product,
|
||||||
|
"version": version,
|
||||||
|
"routes": dict(routes),
|
||||||
|
"capabilities": capabilities,
|
||||||
|
"status": response.status,
|
||||||
|
}
|
||||||
|
)
|
||||||
|
return {
|
||||||
|
"product": product,
|
||||||
|
"product_version": version,
|
||||||
|
"api_family": "generic_interface_rest",
|
||||||
|
"capabilities": capabilities,
|
||||||
|
"maturity": maturity,
|
||||||
|
"health_status": "healthy",
|
||||||
|
"revision": revision,
|
||||||
|
"diagnostics": (
|
||||||
|
[]
|
||||||
|
if recognized
|
||||||
|
else [
|
||||||
|
{
|
||||||
|
"severity": "warning",
|
||||||
|
"code": "product_version_unverified",
|
||||||
|
"message": (
|
||||||
|
"The GenericInterface endpoint is healthy, but its product "
|
||||||
|
"and major version could not be verified; maturity is limited to read."
|
||||||
|
),
|
||||||
|
"retryable": False,
|
||||||
|
"details": {},
|
||||||
|
}
|
||||||
|
]
|
||||||
|
),
|
||||||
|
"evidence": {
|
||||||
|
"http_status": response.status,
|
||||||
|
"response_content_type": response.headers.get("Content-Type"),
|
||||||
|
"ticket_search_shape": _search_shape(payload),
|
||||||
|
"credential_present": bool(credential),
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
def changes(
|
||||||
|
self,
|
||||||
|
*,
|
||||||
|
endpoint_url: str,
|
||||||
|
credential: Mapping[str, Any] | None,
|
||||||
|
routes: Mapping[str, Any],
|
||||||
|
cursor: str | None,
|
||||||
|
limit: int,
|
||||||
|
force_full: bool,
|
||||||
|
) -> ServiceDeskChangeBatch:
|
||||||
|
if limit < 1 or limit > MAX_SERVICE_DESK_TICKET_READS:
|
||||||
|
raise ServiceDeskTransportError(
|
||||||
|
"read_limit_invalid",
|
||||||
|
"A service-desk synchronization call must request between 1 and 500 tickets.",
|
||||||
|
)
|
||||||
|
if force_full:
|
||||||
|
return self._full_changes(
|
||||||
|
endpoint_url,
|
||||||
|
credential=credential,
|
||||||
|
routes=routes,
|
||||||
|
cursor=cursor,
|
||||||
|
limit=limit,
|
||||||
|
)
|
||||||
|
return self._delta_changes(
|
||||||
|
endpoint_url,
|
||||||
|
credential=credential,
|
||||||
|
routes=routes,
|
||||||
|
cursor=cursor,
|
||||||
|
limit=limit,
|
||||||
|
)
|
||||||
|
|
||||||
|
def update_ticket(
|
||||||
|
self,
|
||||||
|
*,
|
||||||
|
endpoint_url: str,
|
||||||
|
credential: Mapping[str, Any] | None,
|
||||||
|
routes: Mapping[str, Any],
|
||||||
|
ticket_id: str,
|
||||||
|
expected_revision: str,
|
||||||
|
changes: Mapping[str, Any],
|
||||||
|
) -> ServiceDeskUpdateResult:
|
||||||
|
update_path = _optional_text(routes.get("update_path"))
|
||||||
|
if not update_path:
|
||||||
|
raise ServiceDeskTransportError(
|
||||||
|
"update_unsupported",
|
||||||
|
"The configured GenericInterface profile has no ticket update route.",
|
||||||
|
)
|
||||||
|
current, _response = self._ticket(
|
||||||
|
endpoint_url,
|
||||||
|
credential=credential,
|
||||||
|
routes=routes,
|
||||||
|
ticket_id=ticket_id,
|
||||||
|
)
|
||||||
|
actual_revision = _ticket_revision(current)
|
||||||
|
if actual_revision != expected_revision:
|
||||||
|
raise ServiceDeskTransportError(
|
||||||
|
"external_revision_conflict",
|
||||||
|
"The external ticket changed; synchronize it before updating.",
|
||||||
|
)
|
||||||
|
url = _route_url(
|
||||||
|
endpoint_url,
|
||||||
|
update_path.replace("{ticket_id}", quote(ticket_id, safe="")),
|
||||||
|
)
|
||||||
|
method = str(routes.get("update_method") or "PATCH").upper()
|
||||||
|
request_payload = _authenticated_payload(
|
||||||
|
{"Ticket": dict(changes)}, credential
|
||||||
|
)
|
||||||
|
try:
|
||||||
|
_payload, response = self._request(
|
||||||
|
url,
|
||||||
|
method=method,
|
||||||
|
credential=credential,
|
||||||
|
payload=request_payload,
|
||||||
|
mutation=True,
|
||||||
|
)
|
||||||
|
except ServiceDeskTransportError as exc:
|
||||||
|
if exc.outcome_unknown:
|
||||||
|
raise
|
||||||
|
raise ServiceDeskTransportError(
|
||||||
|
exc.code,
|
||||||
|
str(exc),
|
||||||
|
retryable=exc.retryable,
|
||||||
|
outcome_unknown=False,
|
||||||
|
) from exc
|
||||||
|
refreshed, read_response = self._ticket(
|
||||||
|
endpoint_url,
|
||||||
|
credential=credential,
|
||||||
|
routes=routes,
|
||||||
|
ticket_id=ticket_id,
|
||||||
|
)
|
||||||
|
if _ticket_id(refreshed) != ticket_id:
|
||||||
|
raise ServiceDeskTransportError(
|
||||||
|
"update_verification_identity_mismatch",
|
||||||
|
"The provider verification returned another ticket identity.",
|
||||||
|
outcome_unknown=True,
|
||||||
|
)
|
||||||
|
revision = _ticket_revision(refreshed)
|
||||||
|
if revision == expected_revision:
|
||||||
|
raise ServiceDeskTransportError(
|
||||||
|
"update_verification_failed",
|
||||||
|
"The provider accepted the request but the ticket revision did not change.",
|
||||||
|
outcome_unknown=True,
|
||||||
|
)
|
||||||
|
mismatches = _update_mismatches(refreshed, changes)
|
||||||
|
if mismatches:
|
||||||
|
raise ServiceDeskTransportError(
|
||||||
|
"update_verification_failed",
|
||||||
|
"The provider revision changed, but the requested ticket fields could not be verified.",
|
||||||
|
outcome_unknown=True,
|
||||||
|
)
|
||||||
|
return ServiceDeskUpdateResult(
|
||||||
|
ticket=refreshed,
|
||||||
|
revision=revision,
|
||||||
|
evidence={
|
||||||
|
"update_http_status": response.status,
|
||||||
|
"verification_http_status": read_response.status,
|
||||||
|
"previous_revision": expected_revision,
|
||||||
|
"accepted_revision": revision,
|
||||||
|
"verified_fields": sorted(changes),
|
||||||
|
},
|
||||||
|
)
|
||||||
|
|
||||||
|
def _full_changes(
|
||||||
|
self,
|
||||||
|
endpoint_url: str,
|
||||||
|
*,
|
||||||
|
credential: Mapping[str, Any] | None,
|
||||||
|
routes: Mapping[str, Any],
|
||||||
|
cursor: str | None,
|
||||||
|
limit: int,
|
||||||
|
) -> ServiceDeskChangeBatch:
|
||||||
|
payload, _response = self._search(
|
||||||
|
endpoint_url,
|
||||||
|
credential=credential,
|
||||||
|
routes=routes,
|
||||||
|
criteria={
|
||||||
|
"Limit": MAX_SERVICE_DESK_SEARCH_IDS + 1,
|
||||||
|
"SortBy": ["TicketID"],
|
||||||
|
"OrderBy": ["Up"],
|
||||||
|
},
|
||||||
|
)
|
||||||
|
ids = _ticket_ids(payload)
|
||||||
|
_assert_search_bound(ids)
|
||||||
|
fingerprint = _hash(ids)
|
||||||
|
state = _decode_cursor(cursor, expected_kind="full")
|
||||||
|
offset = int(state.get("offset") or 0)
|
||||||
|
if offset > len(ids):
|
||||||
|
raise ServiceDeskTransportError(
|
||||||
|
"full_cursor_stale",
|
||||||
|
"The full synchronization cursor is beyond the current ticket set; restart the backfill.",
|
||||||
|
)
|
||||||
|
if state.get("fingerprint") is not None and state.get("fingerprint") != fingerprint:
|
||||||
|
raise ServiceDeskTransportError(
|
||||||
|
"full_cursor_stale",
|
||||||
|
"The external ticket set changed during backfill; restart the full synchronization.",
|
||||||
|
)
|
||||||
|
selected = ids[offset : offset + limit]
|
||||||
|
changes = tuple(
|
||||||
|
self._ticket(
|
||||||
|
endpoint_url,
|
||||||
|
credential=credential,
|
||||||
|
routes=routes,
|
||||||
|
ticket_id=ticket_id,
|
||||||
|
)[0]
|
||||||
|
for ticket_id in selected
|
||||||
|
)
|
||||||
|
new_offset = offset + len(selected)
|
||||||
|
complete = new_offset >= len(ids)
|
||||||
|
page_high_watermark = _latest_revision(changes)
|
||||||
|
high_watermark = max(
|
||||||
|
value
|
||||||
|
for value in (
|
||||||
|
_optional_text(state.get("high_watermark")),
|
||||||
|
page_high_watermark,
|
||||||
|
)
|
||||||
|
if value is not None
|
||||||
|
) if (_optional_text(state.get("high_watermark")) or page_high_watermark) else None
|
||||||
|
next_cursor = (
|
||||||
|
None
|
||||||
|
if complete
|
||||||
|
else _encode_cursor(
|
||||||
|
{
|
||||||
|
"kind": "full",
|
||||||
|
"offset": new_offset,
|
||||||
|
"fingerprint": fingerprint,
|
||||||
|
"high_watermark": high_watermark,
|
||||||
|
}
|
||||||
|
)
|
||||||
|
)
|
||||||
|
return ServiceDeskChangeBatch(
|
||||||
|
changes=changes,
|
||||||
|
next_cursor=next_cursor,
|
||||||
|
complete=complete,
|
||||||
|
high_watermark=high_watermark,
|
||||||
|
live_ids=tuple(ids) if complete else None,
|
||||||
|
evidence={
|
||||||
|
"mode": "backfill",
|
||||||
|
"available": len(ids),
|
||||||
|
"offset": offset,
|
||||||
|
"returned": len(changes),
|
||||||
|
"ticket_set_fingerprint": fingerprint,
|
||||||
|
},
|
||||||
|
)
|
||||||
|
|
||||||
|
def _delta_changes(
|
||||||
|
self,
|
||||||
|
endpoint_url: str,
|
||||||
|
*,
|
||||||
|
credential: Mapping[str, Any] | None,
|
||||||
|
routes: Mapping[str, Any],
|
||||||
|
cursor: str | None,
|
||||||
|
limit: int,
|
||||||
|
) -> ServiceDeskChangeBatch:
|
||||||
|
state = _decode_cursor(cursor, expected_kind="delta")
|
||||||
|
changed = _optional_text(state.get("changed"))
|
||||||
|
seen = {str(value) for value in state.get("seen") or ()}
|
||||||
|
search_limit = min(
|
||||||
|
MAX_SERVICE_DESK_TICKET_READS,
|
||||||
|
limit + len(seen) + 1,
|
||||||
|
)
|
||||||
|
criteria: dict[str, Any] = {
|
||||||
|
"Limit": search_limit,
|
||||||
|
"SortBy": ["Changed"],
|
||||||
|
"OrderBy": ["Up"],
|
||||||
|
}
|
||||||
|
if changed:
|
||||||
|
criteria["TicketChangeTimeNewerDate"] = _overlap_boundary(changed)
|
||||||
|
payload, _response = self._search(
|
||||||
|
endpoint_url,
|
||||||
|
credential=credential,
|
||||||
|
routes=routes,
|
||||||
|
criteria=criteria,
|
||||||
|
)
|
||||||
|
ids = _ticket_ids(payload)
|
||||||
|
_assert_search_bound(ids)
|
||||||
|
fetched = tuple(
|
||||||
|
self._ticket(
|
||||||
|
endpoint_url,
|
||||||
|
credential=credential,
|
||||||
|
routes=routes,
|
||||||
|
ticket_id=ticket_id,
|
||||||
|
)[0]
|
||||||
|
for ticket_id in ids
|
||||||
|
)
|
||||||
|
eligible: list[Mapping[str, Any]] = []
|
||||||
|
suppressed = 0
|
||||||
|
for ticket in fetched:
|
||||||
|
revision = _ticket_revision(ticket)
|
||||||
|
ticket_id = _ticket_id(ticket)
|
||||||
|
if changed and (
|
||||||
|
revision < changed or (revision == changed and ticket_id in seen)
|
||||||
|
):
|
||||||
|
suppressed += 1
|
||||||
|
continue
|
||||||
|
eligible.append(ticket)
|
||||||
|
ordered_eligible = sorted(
|
||||||
|
eligible,
|
||||||
|
key=lambda item: (_ticket_revision(item), _ticket_id(item)),
|
||||||
|
)
|
||||||
|
ordered = ordered_eligible[:limit]
|
||||||
|
if not ordered and len(ids) >= search_limit:
|
||||||
|
raise ServiceDeskTransportError(
|
||||||
|
"delta_boundary_overflow",
|
||||||
|
"The overlap window contains too many tickets to advance safely; narrow the profile or run a full synchronization.",
|
||||||
|
)
|
||||||
|
latest = _latest_revision(ordered) or changed
|
||||||
|
latest_seen = set()
|
||||||
|
if latest:
|
||||||
|
if latest == changed:
|
||||||
|
latest_seen.update(seen)
|
||||||
|
latest_seen.update(
|
||||||
|
_ticket_id(item)
|
||||||
|
for item in ordered
|
||||||
|
if _ticket_revision(item) == latest
|
||||||
|
)
|
||||||
|
complete = len(ordered_eligible) <= limit and len(ids) < search_limit
|
||||||
|
next_cursor = _encode_cursor(
|
||||||
|
{
|
||||||
|
"kind": "delta",
|
||||||
|
"changed": latest,
|
||||||
|
"seen": sorted(latest_seen),
|
||||||
|
}
|
||||||
|
)
|
||||||
|
return ServiceDeskChangeBatch(
|
||||||
|
changes=tuple(ordered),
|
||||||
|
next_cursor=next_cursor,
|
||||||
|
complete=complete,
|
||||||
|
high_watermark=latest,
|
||||||
|
live_ids=None,
|
||||||
|
evidence={
|
||||||
|
"mode": "delta",
|
||||||
|
"searched": len(ids),
|
||||||
|
"returned": len(ordered),
|
||||||
|
"overlap_suppressed": suppressed,
|
||||||
|
"search_limit": search_limit,
|
||||||
|
},
|
||||||
|
)
|
||||||
|
|
||||||
|
def _search(
|
||||||
|
self,
|
||||||
|
endpoint_url: str,
|
||||||
|
*,
|
||||||
|
credential: Mapping[str, Any] | None,
|
||||||
|
routes: Mapping[str, Any],
|
||||||
|
criteria: Mapping[str, Any],
|
||||||
|
) -> tuple[Mapping[str, Any], HttpFetchResponse]:
|
||||||
|
url = _route_url(endpoint_url, str(routes.get("search_path") or "/Ticket/Search"))
|
||||||
|
method = str(routes.get("search_method") or "POST").upper()
|
||||||
|
configured_filters = routes.get("search_filters")
|
||||||
|
if configured_filters is None:
|
||||||
|
configured_filters = {}
|
||||||
|
if not isinstance(configured_filters, Mapping):
|
||||||
|
raise ServiceDeskTransportError(
|
||||||
|
"search_filters_invalid",
|
||||||
|
"Governed GenericInterface search filters must be a JSON object.",
|
||||||
|
)
|
||||||
|
return self._request(
|
||||||
|
url,
|
||||||
|
method=method,
|
||||||
|
credential=credential,
|
||||||
|
payload=_authenticated_payload(
|
||||||
|
{**dict(configured_filters), **dict(criteria)}, credential
|
||||||
|
),
|
||||||
|
)
|
||||||
|
|
||||||
|
def _ticket(
|
||||||
|
self,
|
||||||
|
endpoint_url: str,
|
||||||
|
*,
|
||||||
|
credential: Mapping[str, Any] | None,
|
||||||
|
routes: Mapping[str, Any],
|
||||||
|
ticket_id: str,
|
||||||
|
) -> tuple[Mapping[str, Any], HttpFetchResponse]:
|
||||||
|
path = str(routes.get("ticket_path") or "/Ticket/{ticket_id}").replace(
|
||||||
|
"{ticket_id}", quote(ticket_id, safe="")
|
||||||
|
)
|
||||||
|
method = str(routes.get("ticket_method") or "GET").upper()
|
||||||
|
identity_only = bool(routes.get("_identity_only"))
|
||||||
|
payload: dict[str, Any] = {
|
||||||
|
"TicketID": ticket_id,
|
||||||
|
"DynamicFields": 0 if identity_only else 1,
|
||||||
|
"Extended": 1,
|
||||||
|
"AllArticles": 0 if identity_only else 1,
|
||||||
|
"Attachments": 0 if identity_only else 1,
|
||||||
|
"GetAttachmentContents": 0,
|
||||||
|
}
|
||||||
|
raw, response = self._request(
|
||||||
|
_route_url(endpoint_url, path),
|
||||||
|
method=method,
|
||||||
|
credential=credential,
|
||||||
|
payload=_authenticated_payload(payload, credential),
|
||||||
|
)
|
||||||
|
return _ticket_payload(raw), response
|
||||||
|
|
||||||
|
def _request(
|
||||||
|
self,
|
||||||
|
url: str,
|
||||||
|
*,
|
||||||
|
method: str,
|
||||||
|
credential: Mapping[str, Any] | None,
|
||||||
|
payload: Mapping[str, Any] | None,
|
||||||
|
mutation: bool = False,
|
||||||
|
) -> tuple[Mapping[str, Any], HttpFetchResponse]:
|
||||||
|
headers = {"Accept": "application/json", **_auth_headers(credential)}
|
||||||
|
body: bytes | None = None
|
||||||
|
request_url = url
|
||||||
|
if payload:
|
||||||
|
if method == "GET":
|
||||||
|
secret_keys = {
|
||||||
|
"SessionID",
|
||||||
|
"UserLogin",
|
||||||
|
"CustomerUserLogin",
|
||||||
|
"Password",
|
||||||
|
}
|
||||||
|
if secret_keys.intersection(payload):
|
||||||
|
raise ServiceDeskTransportError(
|
||||||
|
"credential_transport_unsafe",
|
||||||
|
"Body authentication cannot be used with a GET route; use header authentication or configure a POST route.",
|
||||||
|
)
|
||||||
|
separator = "&" if urlsplit(request_url).query else "?"
|
||||||
|
request_url = f"{request_url}{separator}{urlencode(payload, doseq=True)}"
|
||||||
|
else:
|
||||||
|
headers["Content-Type"] = "application/json"
|
||||||
|
body = json.dumps(payload, separators=(",", ":")).encode("utf-8")
|
||||||
|
try:
|
||||||
|
response = fetch_http(
|
||||||
|
request_url,
|
||||||
|
timeout=20,
|
||||||
|
label="Service-desk connector URL",
|
||||||
|
method=method,
|
||||||
|
headers=headers,
|
||||||
|
body=body,
|
||||||
|
max_bytes=MAX_SERVICE_DESK_RESPONSE_BYTES,
|
||||||
|
redirect_sensitive_headers=SERVICE_DESK_SENSITIVE_HEADERS,
|
||||||
|
)
|
||||||
|
except urllib.error.HTTPError as exc:
|
||||||
|
retryable = exc.code == 429 or exc.code >= 500
|
||||||
|
raise ServiceDeskTransportError(
|
||||||
|
"provider_http_error",
|
||||||
|
f"The service-desk provider returned HTTP {exc.code}.",
|
||||||
|
retryable=retryable,
|
||||||
|
outcome_unknown=mutation and retryable,
|
||||||
|
) from exc
|
||||||
|
except (urllib.error.URLError, socket.timeout, TimeoutError) as exc:
|
||||||
|
raise ServiceDeskTransportError(
|
||||||
|
"provider_unavailable",
|
||||||
|
"The service-desk provider did not return a conclusive response.",
|
||||||
|
retryable=True,
|
||||||
|
outcome_unknown=mutation,
|
||||||
|
) from exc
|
||||||
|
except (ValueError, OutboundHttpError) as exc:
|
||||||
|
raise ServiceDeskTransportError(
|
||||||
|
"transport_policy_rejected", str(exc), retryable=False
|
||||||
|
) from exc
|
||||||
|
try:
|
||||||
|
decoded = json.loads(response.body.decode("utf-8")) if response.body else {}
|
||||||
|
except (UnicodeDecodeError, json.JSONDecodeError) as exc:
|
||||||
|
raise ServiceDeskTransportError(
|
||||||
|
"invalid_provider_response",
|
||||||
|
"The service-desk provider did not return valid JSON.",
|
||||||
|
outcome_unknown=mutation,
|
||||||
|
) from exc
|
||||||
|
if not isinstance(decoded, Mapping):
|
||||||
|
raise ServiceDeskTransportError(
|
||||||
|
"invalid_provider_response",
|
||||||
|
"The service-desk provider response must be a JSON object.",
|
||||||
|
outcome_unknown=mutation,
|
||||||
|
)
|
||||||
|
error = decoded.get("Error")
|
||||||
|
if isinstance(error, Mapping):
|
||||||
|
code = _optional_text(error.get("ErrorCode")) or "provider_rejected"
|
||||||
|
message = _optional_text(error.get("ErrorMessage")) or "Provider rejected the request."
|
||||||
|
raise ServiceDeskTransportError(code, message, retryable=False)
|
||||||
|
return decoded, response
|
||||||
|
|
||||||
|
|
||||||
|
def _route_url(endpoint_url: str, route: str) -> str:
|
||||||
|
parsed = urlsplit(route)
|
||||||
|
if parsed.scheme or parsed.netloc or parsed.username or parsed.password:
|
||||||
|
raise ServiceDeskTransportError(
|
||||||
|
"route_invalid", "GenericInterface routes must be relative to the governed endpoint."
|
||||||
|
)
|
||||||
|
if any(part == ".." for part in parsed.path.split("/")):
|
||||||
|
raise ServiceDeskTransportError(
|
||||||
|
"route_invalid", "GenericInterface routes cannot traverse parent paths."
|
||||||
|
)
|
||||||
|
return urljoin(endpoint_url.rstrip("/") + "/", route.lstrip("/"))
|
||||||
|
|
||||||
|
|
||||||
|
def _auth_headers(credential: Mapping[str, Any] | None) -> dict[str, str]:
|
||||||
|
if not credential:
|
||||||
|
return {}
|
||||||
|
if str(credential.get("auth_mode") or "header").casefold() == "body":
|
||||||
|
return {}
|
||||||
|
headers: dict[str, str] = {}
|
||||||
|
session_id = _credential_value(credential, "session_id", "SessionID")
|
||||||
|
user_login = _credential_value(credential, "user_login", "UserLogin", "username")
|
||||||
|
password = _credential_value(credential, "password", "Password")
|
||||||
|
customer_login = _credential_value(
|
||||||
|
credential, "customer_user_login", "CustomerUserLogin"
|
||||||
|
)
|
||||||
|
if session_id:
|
||||||
|
headers["X-OTRS-Header-SessionID"] = session_id
|
||||||
|
if user_login:
|
||||||
|
headers["X-OTRS-Header-UserLogin"] = user_login
|
||||||
|
if customer_login:
|
||||||
|
headers["X-OTRS-Header-CustomerUserLogin"] = customer_login
|
||||||
|
if password:
|
||||||
|
headers["X-OTRS-Header-Password"] = password
|
||||||
|
return headers
|
||||||
|
|
||||||
|
|
||||||
|
def _authenticated_payload(
|
||||||
|
payload: Mapping[str, Any], credential: Mapping[str, Any] | None
|
||||||
|
) -> dict[str, Any]:
|
||||||
|
result = dict(payload)
|
||||||
|
if (
|
||||||
|
not credential
|
||||||
|
or str(credential.get("auth_mode") or "header").casefold() != "body"
|
||||||
|
):
|
||||||
|
return result
|
||||||
|
for target, aliases in (
|
||||||
|
("SessionID", ("session_id", "SessionID")),
|
||||||
|
("UserLogin", ("user_login", "UserLogin", "username")),
|
||||||
|
("CustomerUserLogin", ("customer_user_login", "CustomerUserLogin")),
|
||||||
|
("Password", ("password", "Password")),
|
||||||
|
):
|
||||||
|
value = _credential_value(credential, *aliases)
|
||||||
|
if value:
|
||||||
|
result[target] = value
|
||||||
|
return result
|
||||||
|
|
||||||
|
|
||||||
|
def _credential_value(credential: Mapping[str, Any], *keys: str) -> str | None:
|
||||||
|
for key in keys:
|
||||||
|
value = _optional_text(credential.get(key))
|
||||||
|
if value:
|
||||||
|
return value
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def _ticket_ids(payload: Mapping[str, Any]) -> list[str]:
|
||||||
|
raw = payload.get("TicketID")
|
||||||
|
if raw is None:
|
||||||
|
raw = payload.get("TicketIDs")
|
||||||
|
if raw is None:
|
||||||
|
raw = payload.get("TicketId")
|
||||||
|
if raw is None:
|
||||||
|
return []
|
||||||
|
values: Sequence[Any] = raw if isinstance(raw, Sequence) and not isinstance(raw, str) else [raw]
|
||||||
|
return list(dict.fromkeys(str(value).strip() for value in values if str(value).strip()))
|
||||||
|
|
||||||
|
|
||||||
|
def _ticket_payload(payload: Mapping[str, Any]) -> Mapping[str, Any]:
|
||||||
|
raw = payload.get("Ticket")
|
||||||
|
if isinstance(raw, Mapping):
|
||||||
|
return raw
|
||||||
|
if isinstance(raw, Sequence) and not isinstance(raw, (str, bytes)):
|
||||||
|
for value in raw:
|
||||||
|
if isinstance(value, Mapping):
|
||||||
|
return value
|
||||||
|
if any(key in payload for key in ("TicketID", "TicketNumber", "Title", "Changed")):
|
||||||
|
return payload
|
||||||
|
raise ServiceDeskTransportError(
|
||||||
|
"ticket_response_invalid", "The provider response did not contain a ticket object."
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _product_version(
|
||||||
|
payload: Mapping[str, Any], headers: Mapping[str, str], endpoint_url: str
|
||||||
|
) -> tuple[str, str | None]:
|
||||||
|
normalized_headers = {key.casefold(): value for key, value in headers.items()}
|
||||||
|
znuny_version = _optional_text(normalized_headers.get("x-znuny-version"))
|
||||||
|
otrs_version = _optional_text(normalized_headers.get("x-otrs-version"))
|
||||||
|
product = _optional_text(payload.get("Product"))
|
||||||
|
version = _optional_text(payload.get("Version"))
|
||||||
|
system_data = payload.get("SystemData")
|
||||||
|
if isinstance(system_data, Mapping):
|
||||||
|
product = product or _optional_text(system_data.get("Product"))
|
||||||
|
version = version or _optional_text(system_data.get("Version"))
|
||||||
|
if znuny_version:
|
||||||
|
return "znuny", znuny_version
|
||||||
|
if otrs_version:
|
||||||
|
return "otrs", otrs_version
|
||||||
|
folded = str(product or "").casefold()
|
||||||
|
if "znuny" in folded:
|
||||||
|
return "znuny", version
|
||||||
|
if "otrs" in folded:
|
||||||
|
return "otrs", version
|
||||||
|
path = urlsplit(endpoint_url).path.casefold()
|
||||||
|
if "/znuny/" in path:
|
||||||
|
return "znuny", version
|
||||||
|
if "/otrs/" in path:
|
||||||
|
return "otrs", version
|
||||||
|
return "znuny_otrs", version
|
||||||
|
|
||||||
|
|
||||||
|
def _major(version: str | None) -> int:
|
||||||
|
if not version:
|
||||||
|
return 0
|
||||||
|
head = version.strip().split(".", 1)[0]
|
||||||
|
return int(head) if head.isdigit() else 0
|
||||||
|
|
||||||
|
|
||||||
|
def _ticket_id(ticket: Mapping[str, Any]) -> str:
|
||||||
|
value = _optional_text(ticket.get("TicketID")) or _optional_text(ticket.get("ID"))
|
||||||
|
if not value:
|
||||||
|
raise ServiceDeskTransportError(
|
||||||
|
"ticket_identity_missing", "The provider ticket has no stable TicketID."
|
||||||
|
)
|
||||||
|
return value
|
||||||
|
|
||||||
|
|
||||||
|
def _ticket_revision(ticket: Mapping[str, Any]) -> str:
|
||||||
|
value = (
|
||||||
|
_optional_text(ticket.get("Changed"))
|
||||||
|
or _optional_text(ticket.get("ChangeTime"))
|
||||||
|
or _optional_text(ticket.get("Updated"))
|
||||||
|
)
|
||||||
|
if value:
|
||||||
|
return value
|
||||||
|
return _hash(ticket)
|
||||||
|
|
||||||
|
|
||||||
|
def _update_mismatches(
|
||||||
|
ticket: Mapping[str, Any],
|
||||||
|
changes: Mapping[str, Any],
|
||||||
|
) -> list[str]:
|
||||||
|
mismatches: list[str] = []
|
||||||
|
for field in ("Title", "Queue", "State", "Priority", "Owner", "Responsible"):
|
||||||
|
if field in changes and ticket.get(field) != changes[field]:
|
||||||
|
mismatches.append(field)
|
||||||
|
expected_dynamic = changes.get("DynamicField")
|
||||||
|
if isinstance(expected_dynamic, Sequence) and not isinstance(
|
||||||
|
expected_dynamic, (str, bytes)
|
||||||
|
):
|
||||||
|
actual_dynamic = _dynamic_field_values(ticket)
|
||||||
|
for item in expected_dynamic:
|
||||||
|
if not isinstance(item, Mapping):
|
||||||
|
mismatches.append("DynamicField")
|
||||||
|
continue
|
||||||
|
name = _optional_text(item.get("Name"))
|
||||||
|
if not name or name not in actual_dynamic or actual_dynamic[name] != item.get("Value"):
|
||||||
|
mismatches.append(f"DynamicField.{name or 'unknown'}")
|
||||||
|
return mismatches
|
||||||
|
|
||||||
|
|
||||||
|
def _dynamic_field_values(ticket: Mapping[str, Any]) -> dict[str, Any]:
|
||||||
|
raw = (
|
||||||
|
ticket.get("DynamicField")
|
||||||
|
if ticket.get("DynamicField") is not None
|
||||||
|
else ticket.get("DynamicFields")
|
||||||
|
)
|
||||||
|
if isinstance(raw, Mapping):
|
||||||
|
return {str(key): value for key, value in raw.items()}
|
||||||
|
result: dict[str, Any] = {}
|
||||||
|
if isinstance(raw, Sequence) and not isinstance(raw, (str, bytes)):
|
||||||
|
for item in raw:
|
||||||
|
if not isinstance(item, Mapping):
|
||||||
|
continue
|
||||||
|
name = _optional_text(item.get("Name"))
|
||||||
|
if name:
|
||||||
|
result[name] = item.get("Value")
|
||||||
|
return result
|
||||||
|
|
||||||
|
|
||||||
|
def _latest_revision(changes: Sequence[Mapping[str, Any]]) -> str | None:
|
||||||
|
values = [_ticket_revision(item) for item in changes]
|
||||||
|
return max(values) if values else None
|
||||||
|
|
||||||
|
|
||||||
|
def _overlap_boundary(value: str) -> str:
|
||||||
|
normalized = value.strip().replace("Z", "+00:00")
|
||||||
|
try:
|
||||||
|
parsed = datetime.fromisoformat(normalized)
|
||||||
|
except ValueError:
|
||||||
|
return value
|
||||||
|
if parsed.tzinfo is None:
|
||||||
|
parsed = parsed.replace(tzinfo=timezone.utc)
|
||||||
|
return (parsed.astimezone(timezone.utc) - timedelta(seconds=1)).isoformat().replace(
|
||||||
|
"+00:00", "Z"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _assert_search_bound(ids: Sequence[str]) -> None:
|
||||||
|
if len(ids) > MAX_SERVICE_DESK_SEARCH_IDS:
|
||||||
|
raise ServiceDeskTransportError(
|
||||||
|
"search_result_unbounded",
|
||||||
|
"The provider returned more than 10000 ticket identities; narrow the profile by queue.",
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _decode_cursor(cursor: str | None, *, expected_kind: str) -> dict[str, Any]:
|
||||||
|
if not cursor:
|
||||||
|
return {"kind": expected_kind}
|
||||||
|
try:
|
||||||
|
value = json.loads(cursor)
|
||||||
|
except json.JSONDecodeError as exc:
|
||||||
|
raise ServiceDeskTransportError("cursor_invalid", "The synchronization cursor is invalid.") from exc
|
||||||
|
if not isinstance(value, Mapping) or value.get("kind") != expected_kind:
|
||||||
|
raise ServiceDeskTransportError(
|
||||||
|
"cursor_mode_mismatch", "The synchronization cursor belongs to another mode."
|
||||||
|
)
|
||||||
|
decoded = dict(value)
|
||||||
|
if expected_kind == "full":
|
||||||
|
offset = decoded.get("offset", 0)
|
||||||
|
fingerprint = decoded.get("fingerprint")
|
||||||
|
if isinstance(offset, bool) or not isinstance(offset, int) or offset < 0:
|
||||||
|
raise ServiceDeskTransportError(
|
||||||
|
"cursor_invalid", "The full synchronization cursor offset is invalid."
|
||||||
|
)
|
||||||
|
if offset and (
|
||||||
|
not isinstance(fingerprint, str)
|
||||||
|
or len(fingerprint) != 64
|
||||||
|
or any(
|
||||||
|
character not in "0123456789abcdef"
|
||||||
|
for character in fingerprint.casefold()
|
||||||
|
)
|
||||||
|
):
|
||||||
|
raise ServiceDeskTransportError(
|
||||||
|
"cursor_invalid", "The full synchronization cursor fingerprint is invalid."
|
||||||
|
)
|
||||||
|
elif expected_kind == "delta":
|
||||||
|
changed = decoded.get("changed")
|
||||||
|
seen = decoded.get("seen", [])
|
||||||
|
if changed is not None and not isinstance(changed, str):
|
||||||
|
raise ServiceDeskTransportError(
|
||||||
|
"cursor_invalid", "The delta synchronization boundary is invalid."
|
||||||
|
)
|
||||||
|
if not isinstance(seen, list) or any(
|
||||||
|
not isinstance(item, str) or not item for item in seen
|
||||||
|
):
|
||||||
|
raise ServiceDeskTransportError(
|
||||||
|
"cursor_invalid", "The delta synchronization identity set is invalid."
|
||||||
|
)
|
||||||
|
return decoded
|
||||||
|
|
||||||
|
|
||||||
|
def _encode_cursor(value: Mapping[str, Any]) -> str:
|
||||||
|
encoded = json.dumps(value, sort_keys=True, separators=(",", ":"))
|
||||||
|
if len(encoded) > 4000:
|
||||||
|
raise ServiceDeskTransportError(
|
||||||
|
"cursor_boundary_overflow",
|
||||||
|
"Too many tickets share the same change boundary; narrow the profile by queue.",
|
||||||
|
)
|
||||||
|
return encoded
|
||||||
|
|
||||||
|
|
||||||
|
def _search_shape(payload: Mapping[str, Any]) -> str:
|
||||||
|
if "TicketID" in payload:
|
||||||
|
return "TicketID"
|
||||||
|
if "TicketIDs" in payload:
|
||||||
|
return "TicketIDs"
|
||||||
|
return "empty"
|
||||||
|
|
||||||
|
|
||||||
|
def _hash(value: Any) -> str:
|
||||||
|
return hashlib.sha256(
|
||||||
|
json.dumps(value, sort_keys=True, separators=(",", ":"), default=str).encode("utf-8")
|
||||||
|
).hexdigest()
|
||||||
|
|
||||||
|
|
||||||
|
def _optional_text(value: object) -> str | None:
|
||||||
|
normalized = str(value).strip() if value is not None else ""
|
||||||
|
return normalized or None
|
||||||
|
|
||||||
|
|
||||||
|
__all__ = [
|
||||||
|
"HttpServiceDeskTransport",
|
||||||
|
"ServiceDeskChangeBatch",
|
||||||
|
"ServiceDeskTransport",
|
||||||
|
"ServiceDeskTransportError",
|
||||||
|
"ServiceDeskUpdateResult",
|
||||||
|
]
|
||||||
@@ -12,7 +12,10 @@ from decimal import Decimal
|
|||||||
from io import BytesIO
|
from io import BytesIO
|
||||||
from typing import Any
|
from typing import Any
|
||||||
|
|
||||||
|
from defusedxml import ElementTree as SafeET
|
||||||
|
from defusedxml.common import DefusedXmlException
|
||||||
from openpyxl import load_workbook
|
from openpyxl import load_workbook
|
||||||
|
from openpyxl.utils.cell import column_index_from_string
|
||||||
from sqlalchemy import (
|
from sqlalchemy import (
|
||||||
JSON,
|
JSON,
|
||||||
BigInteger,
|
BigInteger,
|
||||||
@@ -50,13 +53,26 @@ from govoplan_core.core.tabular_sources import (
|
|||||||
TabularSourceUnavailableError,
|
TabularSourceUnavailableError,
|
||||||
TabularSourceValidationError,
|
TabularSourceValidationError,
|
||||||
parse_tabular_csv,
|
parse_tabular_csv,
|
||||||
|
CsvValueMode,
|
||||||
|
infer_tabular_schema as _infer_schema,
|
||||||
|
tabular_type_name,
|
||||||
)
|
)
|
||||||
from govoplan_core.security.credential_envelopes import (
|
from govoplan_core.security.credential_envelopes import (
|
||||||
CredentialAccessContext,
|
CredentialAccessContext,
|
||||||
CredentialEnvelopeError,
|
CredentialEnvelopeError,
|
||||||
resolve_credential_envelope,
|
resolve_credential_envelope,
|
||||||
)
|
)
|
||||||
|
from govoplan_core.security.bounded_process import (
|
||||||
|
ProcessBudgetError,
|
||||||
|
ProcessLimits,
|
||||||
|
run_bounded_operation,
|
||||||
|
)
|
||||||
from govoplan_core.security.redaction import is_sensitive_key
|
from govoplan_core.security.redaction import is_sensitive_key
|
||||||
|
from govoplan_core.security.worker_payload import (
|
||||||
|
WorkerPayloadError,
|
||||||
|
decode_worker_payload,
|
||||||
|
encode_worker_payload,
|
||||||
|
)
|
||||||
from govoplan_connectors.backend.db.models import ConnectorConfiguration
|
from govoplan_connectors.backend.db.models import ConnectorConfiguration
|
||||||
|
|
||||||
|
|
||||||
@@ -66,6 +82,13 @@ MAX_FILE_COLUMNS = 500
|
|||||||
MAX_XLSX_ENTRIES = 5_000
|
MAX_XLSX_ENTRIES = 5_000
|
||||||
MAX_XLSX_EXPANDED_BYTES = 50_000_000
|
MAX_XLSX_EXPANDED_BYTES = 50_000_000
|
||||||
MAX_XLSX_COMPRESSION_RATIO = 100
|
MAX_XLSX_COMPRESSION_RATIO = 100
|
||||||
|
XLSX_PROCESS_LIMITS = ProcessLimits(
|
||||||
|
wall_seconds=15,
|
||||||
|
cpu_seconds=10,
|
||||||
|
memory_bytes=512 * 1024 * 1024,
|
||||||
|
input_bytes=8 * 1024 * 1024,
|
||||||
|
output_bytes=64 * 1024 * 1024,
|
||||||
|
)
|
||||||
POSTGRESQL_SCHEMES = frozenset({"postgresql", "postgresql+psycopg"})
|
POSTGRESQL_SCHEMES = frozenset({"postgresql", "postgresql+psycopg"})
|
||||||
_IDENTIFIER = re.compile(r"^[A-Za-z_][A-Za-z0-9_$-]{0,127}$")
|
_IDENTIFIER = re.compile(r"^[A-Za-z_][A-Za-z0-9_$-]{0,127}$")
|
||||||
|
|
||||||
@@ -105,6 +128,7 @@ class ManagedFileTabularAdapter:
|
|||||||
file_version_id: str | None,
|
file_version_id: str | None,
|
||||||
delimiter: str = ",",
|
delimiter: str = ",",
|
||||||
sheet_name: str | None = None,
|
sheet_name: str | None = None,
|
||||||
|
csv_value_mode: CsvValueMode = "legacy_typed",
|
||||||
) -> TabularOriginInspection:
|
) -> TabularOriginInspection:
|
||||||
provider = managed_tabular_file_provider(self._registry)
|
provider = managed_tabular_file_provider(self._registry)
|
||||||
if provider is None:
|
if provider is None:
|
||||||
@@ -143,6 +167,7 @@ class ManagedFileTabularAdapter:
|
|||||||
content_type=content.file.content_type,
|
content_type=content.file.content_type,
|
||||||
delimiter=delimiter,
|
delimiter=delimiter,
|
||||||
sheet_name=sheet_name,
|
sheet_name=sheet_name,
|
||||||
|
csv_value_mode=csv_value_mode,
|
||||||
)
|
)
|
||||||
schema = infer_tabular_schema(rows)
|
schema = infer_tabular_schema(rows)
|
||||||
fingerprint = origin_fingerprint(
|
fingerprint = origin_fingerprint(
|
||||||
@@ -154,6 +179,7 @@ class ManagedFileTabularAdapter:
|
|||||||
content.file.sha256,
|
content.file.sha256,
|
||||||
resolved_sheet or "",
|
resolved_sheet or "",
|
||||||
delimiter,
|
delimiter,
|
||||||
|
*(("csv_text_values",) if csv_value_mode == "text" else ()),
|
||||||
),
|
),
|
||||||
)
|
)
|
||||||
try:
|
try:
|
||||||
@@ -207,6 +233,7 @@ class ManagedFileTabularAdapter:
|
|||||||
"content_type": content.file.content_type,
|
"content_type": content.file.content_type,
|
||||||
"format": "xlsx" if _is_xlsx(content.file.filename, content.file.content_type) else "csv",
|
"format": "xlsx" if _is_xlsx(content.file.filename, content.file.content_type) else "csv",
|
||||||
"delimiter": delimiter,
|
"delimiter": delimiter,
|
||||||
|
"csv_value_mode": csv_value_mode,
|
||||||
"sheet_name": resolved_sheet,
|
"sheet_name": resolved_sheet,
|
||||||
},
|
},
|
||||||
health=health,
|
health=health,
|
||||||
@@ -243,6 +270,7 @@ class ManagedFileTabularAdapter:
|
|||||||
file_version_id=_required_metadata(metadata, "file_version_id"),
|
file_version_id=_required_metadata(metadata, "file_version_id"),
|
||||||
delimiter=str(metadata.get("delimiter") or ","),
|
delimiter=str(metadata.get("delimiter") or ","),
|
||||||
sheet_name=_optional_text(metadata.get("sheet_name")),
|
sheet_name=_optional_text(metadata.get("sheet_name")),
|
||||||
|
csv_value_mode=str(metadata.get("csv_value_mode") or "legacy_typed"),
|
||||||
)
|
)
|
||||||
expected_sha256 = _required_metadata(metadata, "file_sha256")
|
expected_sha256 = _required_metadata(metadata, "file_sha256")
|
||||||
if inspection.metadata.get("file_sha256") != expected_sha256:
|
if inspection.metadata.get("file_sha256") != expected_sha256:
|
||||||
@@ -547,6 +575,7 @@ def parse_managed_tabular_content(
|
|||||||
content_type: str | None,
|
content_type: str | None,
|
||||||
delimiter: str,
|
delimiter: str,
|
||||||
sheet_name: str | None,
|
sheet_name: str | None,
|
||||||
|
csv_value_mode: CsvValueMode = "legacy_typed",
|
||||||
) -> tuple[tuple[Mapping[str, object], ...], str | None]:
|
) -> tuple[tuple[Mapping[str, object], ...], str | None]:
|
||||||
if len(payload) > MAX_FILE_BYTES:
|
if len(payload) > MAX_FILE_BYTES:
|
||||||
raise TabularSourceValidationError(
|
raise TabularSourceValidationError(
|
||||||
@@ -561,7 +590,7 @@ def parse_managed_tabular_content(
|
|||||||
"Managed CSV files must use UTF-8 encoding."
|
"Managed CSV files must use UTF-8 encoding."
|
||||||
) from exc
|
) from exc
|
||||||
return (
|
return (
|
||||||
tuple(parse_tabular_csv(text, delimiter=delimiter, max_rows=MAX_FILE_ROWS)),
|
tuple(parse_tabular_csv(text, delimiter=delimiter, max_rows=MAX_FILE_ROWS, max_bytes=MAX_FILE_BYTES, value_mode=csv_value_mode)),
|
||||||
None,
|
None,
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -570,6 +599,55 @@ def _parse_xlsx(
|
|||||||
payload: bytes,
|
payload: bytes,
|
||||||
*,
|
*,
|
||||||
sheet_name: str | None,
|
sheet_name: str | None,
|
||||||
|
) -> tuple[tuple[Mapping[str, object], ...], str]:
|
||||||
|
# Only already-authorized workbook bytes and a worksheet selector leave
|
||||||
|
# the parent; Files access, credentials and SQL sessions never cross.
|
||||||
|
try:
|
||||||
|
encoded = encode_worker_payload(
|
||||||
|
{"payload": payload, "sheet_name": sheet_name},
|
||||||
|
max_bytes=XLSX_PROCESS_LIMITS.input_bytes,
|
||||||
|
)
|
||||||
|
result = decode_worker_payload(
|
||||||
|
run_bounded_operation(_parse_xlsx_worker, encoded, limits=XLSX_PROCESS_LIMITS),
|
||||||
|
max_bytes=XLSX_PROCESS_LIMITS.output_bytes,
|
||||||
|
)
|
||||||
|
except ProcessBudgetError as exc:
|
||||||
|
error_type = (
|
||||||
|
TabularSourceUnavailableError
|
||||||
|
if exc.code in {"busy", "cancelled", "unavailable", "worker_failed"}
|
||||||
|
else TabularSourceValidationError
|
||||||
|
)
|
||||||
|
raise error_type(f"Managed XLSX processing failed ({exc.code}): {exc}") from exc
|
||||||
|
except (TypeError, ValueError, RecursionError) as exc:
|
||||||
|
raise TabularSourceValidationError("Managed XLSX data could not be safely transferred.") from exc
|
||||||
|
if not isinstance(result, dict):
|
||||||
|
raise TabularSourceUnavailableError("Managed XLSX worker returned an invalid result.")
|
||||||
|
if "validation_error" in result:
|
||||||
|
raise TabularSourceValidationError(str(result["validation_error"]))
|
||||||
|
rows, selected_name = result.get("rows"), result.get("sheet_name")
|
||||||
|
if not isinstance(rows, tuple) or not isinstance(selected_name, str):
|
||||||
|
raise TabularSourceUnavailableError("Managed XLSX worker returned an invalid result.")
|
||||||
|
return rows, selected_name
|
||||||
|
|
||||||
|
|
||||||
|
def _parse_xlsx_worker(payload: bytes) -> bytes:
|
||||||
|
data = decode_worker_payload(payload, max_bytes=XLSX_PROCESS_LIMITS.input_bytes)
|
||||||
|
try:
|
||||||
|
rows, sheet_name = _parse_xlsx_content(data["payload"], sheet_name=data["sheet_name"])
|
||||||
|
return encode_worker_payload(
|
||||||
|
{"rows": rows, "sheet_name": sheet_name},
|
||||||
|
max_bytes=XLSX_PROCESS_LIMITS.output_bytes,
|
||||||
|
)
|
||||||
|
except TabularSourceValidationError as exc:
|
||||||
|
return encode_worker_payload({"validation_error": str(exc)})
|
||||||
|
except WorkerPayloadError:
|
||||||
|
return encode_worker_payload({"validation_error": "Managed XLSX parsed data exceeds the worker transport limit."})
|
||||||
|
|
||||||
|
|
||||||
|
def _parse_xlsx_content(
|
||||||
|
payload: bytes,
|
||||||
|
*,
|
||||||
|
sheet_name: str | None,
|
||||||
) -> tuple[tuple[Mapping[str, object], ...], str]:
|
) -> tuple[tuple[Mapping[str, object], ...], str]:
|
||||||
_validate_xlsx_archive(payload)
|
_validate_xlsx_archive(payload)
|
||||||
try:
|
try:
|
||||||
@@ -579,6 +657,8 @@ def _parse_xlsx(
|
|||||||
data_only=True,
|
data_only=True,
|
||||||
keep_links=False,
|
keep_links=False,
|
||||||
)
|
)
|
||||||
|
except MemoryError:
|
||||||
|
raise
|
||||||
except Exception as exc:
|
except Exception as exc:
|
||||||
raise TabularSourceValidationError(
|
raise TabularSourceValidationError(
|
||||||
"Managed XLSX content could not be parsed."
|
"Managed XLSX content could not be parsed."
|
||||||
@@ -595,7 +675,12 @@ def _parse_xlsx(
|
|||||||
f"Managed XLSX worksheet {selected_name!r} was not found."
|
f"Managed XLSX worksheet {selected_name!r} was not found."
|
||||||
)
|
)
|
||||||
worksheet = workbook[selected_name]
|
worksheet = workbook[selected_name]
|
||||||
iterator = worksheet.iter_rows(values_only=True)
|
maximum_column = _validate_xlsx_worksheet(worksheet)
|
||||||
|
# Declared dimensions are not authoritative: they can inflate sparse
|
||||||
|
# rows/columns or conceal actual cells. Validate the XML coordinates
|
||||||
|
# before openpyxl synthesizes any missing cells, then ignore dimensions.
|
||||||
|
worksheet.reset_dimensions()
|
||||||
|
iterator = worksheet.iter_rows(max_col=maximum_column, values_only=True)
|
||||||
try:
|
try:
|
||||||
raw_headers = next(iterator)
|
raw_headers = next(iterator)
|
||||||
except StopIteration as exc:
|
except StopIteration as exc:
|
||||||
@@ -604,10 +689,10 @@ def _parse_xlsx(
|
|||||||
) from exc
|
) from exc
|
||||||
headers = _xlsx_headers(raw_headers)
|
headers = _xlsx_headers(raw_headers)
|
||||||
rows: list[Mapping[str, object]] = []
|
rows: list[Mapping[str, object]] = []
|
||||||
for values in iterator:
|
for row_number, values in enumerate(iterator, start=1):
|
||||||
if len(rows) >= MAX_FILE_ROWS:
|
if row_number > MAX_FILE_ROWS:
|
||||||
raise TabularSourceValidationError(
|
raise TabularSourceValidationError(
|
||||||
f"Managed XLSX worksheets are limited to {MAX_FILE_ROWS:,} data rows."
|
f"Managed XLSX worksheets are limited to {MAX_FILE_ROWS:,} row positions after the header, including blank gaps."
|
||||||
)
|
)
|
||||||
normalized = tuple(values[: len(headers)])
|
normalized = tuple(values[: len(headers)])
|
||||||
if all(value in (None, "") for value in normalized):
|
if all(value in (None, "") for value in normalized):
|
||||||
@@ -625,6 +710,60 @@ def _parse_xlsx(
|
|||||||
workbook.close()
|
workbook.close()
|
||||||
|
|
||||||
|
|
||||||
|
def _validate_xlsx_worksheet(worksheet: Any) -> int:
|
||||||
|
"""Bound actual coordinates before read-only openpyxl allocates row tuples.
|
||||||
|
|
||||||
|
The pinned openpyxl 3.x read-only source handle is streamed and closed here;
|
||||||
|
no untrusted dimensions or filesystem paths are used to allocate a grid.
|
||||||
|
"""
|
||||||
|
maximum_column = 1
|
||||||
|
row_position = 0
|
||||||
|
rows_seen = 0
|
||||||
|
column_position = 0
|
||||||
|
cells_seen = 0
|
||||||
|
try:
|
||||||
|
with worksheet._get_source() as source:
|
||||||
|
for event, element in SafeET.iterparse(source, events=("start", "end"), forbid_dtd=True):
|
||||||
|
tag = element.tag.rsplit("}", 1)[-1]
|
||||||
|
if event == "end":
|
||||||
|
element.clear()
|
||||||
|
continue
|
||||||
|
if tag == "row":
|
||||||
|
rows_seen += 1
|
||||||
|
raw_row = element.get("r", str(row_position + 1))
|
||||||
|
if len(raw_row) > 7 or not raw_row.isascii() or not raw_row.isdigit():
|
||||||
|
raise TabularSourceValidationError("Managed XLSX content has an invalid row coordinate.")
|
||||||
|
row_position = int(raw_row)
|
||||||
|
if not 1 <= row_position <= MAX_FILE_ROWS + 1 or rows_seen > MAX_FILE_ROWS + 1:
|
||||||
|
raise TabularSourceValidationError(
|
||||||
|
f"Managed XLSX worksheets are limited to {MAX_FILE_ROWS:,} row positions after the header, including blank gaps."
|
||||||
|
)
|
||||||
|
column_position = 0
|
||||||
|
cells_seen = 0
|
||||||
|
elif tag == "c":
|
||||||
|
cells_seen += 1
|
||||||
|
reference = element.get("r")
|
||||||
|
if reference is not None:
|
||||||
|
match = re.fullmatch(r"([A-Za-z]{1,3})([1-9][0-9]{0,6})", reference)
|
||||||
|
if match is None:
|
||||||
|
raise TabularSourceValidationError("Managed XLSX content has an invalid cell coordinate.")
|
||||||
|
column_position = column_index_from_string(match.group(1))
|
||||||
|
if int(match.group(2)) != row_position:
|
||||||
|
raise TabularSourceValidationError("Managed XLSX cell and row coordinates do not agree.")
|
||||||
|
else:
|
||||||
|
column_position += 1
|
||||||
|
if column_position > MAX_FILE_COLUMNS or cells_seen > MAX_FILE_COLUMNS:
|
||||||
|
raise TabularSourceValidationError(
|
||||||
|
f"Managed XLSX worksheets are limited to {MAX_FILE_COLUMNS:,} columns."
|
||||||
|
)
|
||||||
|
maximum_column = max(maximum_column, column_position)
|
||||||
|
return maximum_column
|
||||||
|
except (DefusedXmlException, SafeET.ParseError, ValueError) as exc:
|
||||||
|
if isinstance(exc, TabularSourceValidationError):
|
||||||
|
raise
|
||||||
|
raise TabularSourceValidationError("Managed XLSX worksheet XML could not be safely parsed.") from exc
|
||||||
|
|
||||||
|
|
||||||
def _validate_xlsx_archive(payload: bytes) -> None:
|
def _validate_xlsx_archive(payload: bytes) -> None:
|
||||||
try:
|
try:
|
||||||
with zipfile.ZipFile(BytesIO(payload)) as archive:
|
with zipfile.ZipFile(BytesIO(payload)) as archive:
|
||||||
@@ -679,26 +818,7 @@ def _xlsx_headers(values: Sequence[object]) -> tuple[str, ...]:
|
|||||||
def infer_tabular_schema(
|
def infer_tabular_schema(
|
||||||
rows: Sequence[Mapping[str, object]],
|
rows: Sequence[Mapping[str, object]],
|
||||||
) -> tuple[TabularColumn, ...]:
|
) -> tuple[TabularColumn, ...]:
|
||||||
names: list[str] = []
|
return _infer_schema(rows, type_name=_type_name)
|
||||||
for row in rows:
|
|
||||||
for name in row:
|
|
||||||
if name not in names:
|
|
||||||
names.append(name)
|
|
||||||
result: list[TabularColumn] = []
|
|
||||||
for name in names:
|
|
||||||
values = [row.get(name) for row in rows]
|
|
||||||
concrete = [value for value in values if value is not None]
|
|
||||||
data_type = _type_name(concrete[0]) if concrete else "unknown"
|
|
||||||
if any(_type_name(value) != data_type for value in concrete[1:]):
|
|
||||||
data_type = "mixed"
|
|
||||||
result.append(
|
|
||||||
TabularColumn(
|
|
||||||
name=name,
|
|
||||||
data_type=data_type,
|
|
||||||
nullable=len(concrete) != len(values),
|
|
||||||
)
|
|
||||||
)
|
|
||||||
return tuple(result)
|
|
||||||
|
|
||||||
|
|
||||||
def origin_fingerprint(
|
def origin_fingerprint(
|
||||||
@@ -821,19 +941,7 @@ def _json_value(value: object) -> object:
|
|||||||
|
|
||||||
|
|
||||||
def _type_name(value: object) -> str:
|
def _type_name(value: object) -> str:
|
||||||
if isinstance(value, bool):
|
return tabular_type_name(value, casefold_unknown=True)
|
||||||
return "boolean"
|
|
||||||
if isinstance(value, int):
|
|
||||||
return "integer"
|
|
||||||
if isinstance(value, (float, Decimal)):
|
|
||||||
return "number"
|
|
||||||
if isinstance(value, str):
|
|
||||||
return "string"
|
|
||||||
if isinstance(value, list):
|
|
||||||
return "array"
|
|
||||||
if isinstance(value, dict):
|
|
||||||
return "object"
|
|
||||||
return type(value).__name__.casefold()
|
|
||||||
|
|
||||||
|
|
||||||
__all__ = [
|
__all__ = [
|
||||||
|
|||||||
@@ -13,6 +13,7 @@ from sqlalchemy.orm import Session
|
|||||||
|
|
||||||
from govoplan_core.auth import ApiPrincipal, has_scope
|
from govoplan_core.auth import ApiPrincipal, has_scope
|
||||||
from govoplan_core.core.tabular_sources import (
|
from govoplan_core.core.tabular_sources import (
|
||||||
|
CsvValueMode,
|
||||||
TabularColumn,
|
TabularColumn,
|
||||||
TabularPreviewDiagnostic,
|
TabularPreviewDiagnostic,
|
||||||
TabularPushdown,
|
TabularPushdown,
|
||||||
@@ -26,6 +27,12 @@ from govoplan_core.core.tabular_sources import (
|
|||||||
TabularSourceUnavailableError,
|
TabularSourceUnavailableError,
|
||||||
TabularSourceValidationError,
|
TabularSourceValidationError,
|
||||||
parse_tabular_csv,
|
parse_tabular_csv,
|
||||||
|
csv_source_payload,
|
||||||
|
csv_source_summary,
|
||||||
|
csv_projection_matches,
|
||||||
|
verified_csv_source_text,
|
||||||
|
infer_tabular_schema,
|
||||||
|
tabular_type_name as _type_name,
|
||||||
)
|
)
|
||||||
from govoplan_core.core.runtime import get_registry
|
from govoplan_core.core.runtime import get_registry
|
||||||
from govoplan_core.db.base import utcnow
|
from govoplan_core.db.base import utcnow
|
||||||
@@ -281,6 +288,7 @@ class SqlTabularSourceProvider:
|
|||||||
description: str | None = None,
|
description: str | None = None,
|
||||||
delimiter: str = ",",
|
delimiter: str = ",",
|
||||||
sheet_name: str | None = None,
|
sheet_name: str | None = None,
|
||||||
|
csv_value_mode: CsvValueMode = "legacy_typed",
|
||||||
) -> TabularSource:
|
) -> TabularSource:
|
||||||
db, api_principal = _context(session, principal, WRITE_SCOPE)
|
db, api_principal = _context(session, principal, WRITE_SCOPE)
|
||||||
inspection = self._file_adapter().inspect(
|
inspection = self._file_adapter().inspect(
|
||||||
@@ -290,6 +298,7 @@ class SqlTabularSourceProvider:
|
|||||||
file_version_id=file_version_id,
|
file_version_id=file_version_id,
|
||||||
delimiter=delimiter,
|
delimiter=delimiter,
|
||||||
sheet_name=sheet_name,
|
sheet_name=sheet_name,
|
||||||
|
csv_value_mode=csv_value_mode,
|
||||||
)
|
)
|
||||||
return self._create_origin(
|
return self._create_origin(
|
||||||
db,
|
db,
|
||||||
@@ -353,6 +362,7 @@ class SqlTabularSourceProvider:
|
|||||||
file_version_id=None,
|
file_version_id=None,
|
||||||
delimiter=str(item.metadata_.get("delimiter") or ","),
|
delimiter=str(item.metadata_.get("delimiter") or ","),
|
||||||
sheet_name=_clean_optional(item.metadata_.get("sheet_name")),
|
sheet_name=_clean_optional(item.metadata_.get("sheet_name")),
|
||||||
|
csv_value_mode=str(item.metadata_.get("csv_value_mode") or "legacy_typed"),
|
||||||
)
|
)
|
||||||
elif item.provider == "postgresql":
|
elif item.provider == "postgresql":
|
||||||
inspection = self._sql_adapter.inspect(
|
inspection = self._sql_adapter.inspect(
|
||||||
@@ -456,6 +466,20 @@ class SqlTabularSourceProvider:
|
|||||||
f"Snapshots are limited to {MAX_SNAPSHOT_ROWS:,} rows."
|
f"Snapshots are limited to {MAX_SNAPSHOT_ROWS:,} rows."
|
||||||
)
|
)
|
||||||
rows = [_json_row(row) for row in snapshot.rows]
|
rows = [_json_row(row) for row in snapshot.rows]
|
||||||
|
csv_payload = None
|
||||||
|
if snapshot.csv_source is not None:
|
||||||
|
if "csv_source" in snapshot.metadata:
|
||||||
|
raise TabularSourceValidationError("csv_source metadata is reserved for verified CSV import evidence.")
|
||||||
|
if snapshot.csv_source.parser_profile != "core.csv.v1":
|
||||||
|
raise TabularSourceValidationError("Unsupported original CSV parser profile.")
|
||||||
|
csv_payload = csv_source_payload(snapshot.csv_source, max_bytes=MAX_SNAPSHOT_BYTES)
|
||||||
|
expected = parse_csv_snapshot(
|
||||||
|
snapshot.csv_source.text,
|
||||||
|
delimiter=snapshot.csv_source.delimiter,
|
||||||
|
value_mode=snapshot.csv_source.value_mode,
|
||||||
|
)
|
||||||
|
if not csv_projection_matches(expected, rows):
|
||||||
|
raise TabularSourceValidationError("Snapshot rows do not match their original CSV source and parsing mode.")
|
||||||
encoded = json.dumps(rows, sort_keys=True, separators=(",", ":"), default=str).encode("utf-8")
|
encoded = json.dumps(rows, sort_keys=True, separators=(",", ":"), default=str).encode("utf-8")
|
||||||
if len(encoded) > MAX_SNAPSHOT_BYTES:
|
if len(encoded) > MAX_SNAPSHOT_BYTES:
|
||||||
raise TabularSourceValidationError(
|
raise TabularSourceValidationError(
|
||||||
@@ -487,7 +511,11 @@ class SqlTabularSourceProvider:
|
|||||||
fingerprint=fingerprint,
|
fingerprint=fingerprint,
|
||||||
row_count=len(rows),
|
row_count=len(rows),
|
||||||
byte_count=len(encoded),
|
byte_count=len(encoded),
|
||||||
metadata_=dict(snapshot.metadata),
|
metadata_={
|
||||||
|
**dict(snapshot.metadata),
|
||||||
|
**({"csv_source": csv_source_summary(csv_payload)} if csv_payload is not None else {}),
|
||||||
|
},
|
||||||
|
csv_source_=csv_payload,
|
||||||
created_by=actor_id,
|
created_by=actor_id,
|
||||||
updated_by=actor_id,
|
updated_by=actor_id,
|
||||||
)
|
)
|
||||||
@@ -497,6 +525,15 @@ class SqlTabularSourceProvider:
|
|||||||
db.flush()
|
db.flush()
|
||||||
return _source_dto(item)
|
return _source_dto(item)
|
||||||
|
|
||||||
|
def original_csv(self, session: object, principal: object, *, source_ref: str) -> str:
|
||||||
|
db, api_principal = _context(session, principal, READ_SCOPE)
|
||||||
|
item = _source_record(db, tenant_id=api_principal.tenant_id, source_ref=source_ref)
|
||||||
|
if item is None or item.status != "active":
|
||||||
|
raise TabularSourceNotFoundError("Tabular source not found.")
|
||||||
|
if not item.csv_source_:
|
||||||
|
raise TabularSourceNotFoundError("Original CSV was not retained for this source; historical typed snapshots cannot reconstruct it.")
|
||||||
|
return verified_csv_source_text(item.csv_source_, expected_summary=item.metadata_.get("csv_source") or {})
|
||||||
|
|
||||||
def delete_snapshot(
|
def delete_snapshot(
|
||||||
self,
|
self,
|
||||||
session: object,
|
session: object,
|
||||||
@@ -519,35 +556,18 @@ class SqlTabularSourceProvider:
|
|||||||
return _source_dto(item)
|
return _source_dto(item)
|
||||||
|
|
||||||
|
|
||||||
def parse_csv_snapshot(csv_text: str, *, delimiter: str) -> tuple[Mapping[str, object], ...]:
|
def parse_csv_snapshot(csv_text: str, *, delimiter: str, value_mode: CsvValueMode = "legacy_typed") -> tuple[Mapping[str, object], ...]:
|
||||||
return parse_tabular_csv(
|
return parse_tabular_csv(
|
||||||
csv_text,
|
csv_text,
|
||||||
delimiter=delimiter,
|
delimiter=delimiter,
|
||||||
max_rows=MAX_SNAPSHOT_ROWS,
|
max_rows=MAX_SNAPSHOT_ROWS,
|
||||||
|
max_bytes=MAX_SNAPSHOT_BYTES,
|
||||||
|
value_mode=value_mode,
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
def infer_schema(rows: Sequence[Mapping[str, object]]) -> tuple[TabularColumn, ...]:
|
def infer_schema(rows: Sequence[Mapping[str, object]]) -> tuple[TabularColumn, ...]:
|
||||||
names: list[str] = []
|
return infer_tabular_schema(rows, type_name=_type_name)
|
||||||
for row in rows:
|
|
||||||
for name in row:
|
|
||||||
if name not in names:
|
|
||||||
names.append(name)
|
|
||||||
result: list[TabularColumn] = []
|
|
||||||
for name in names:
|
|
||||||
values = [row.get(name) for row in rows]
|
|
||||||
concrete = [value for value in values if value is not None]
|
|
||||||
data_type = _type_name(concrete[0]) if concrete else "unknown"
|
|
||||||
if any(_type_name(value) != data_type for value in concrete[1:]):
|
|
||||||
data_type = "mixed"
|
|
||||||
result.append(
|
|
||||||
TabularColumn(
|
|
||||||
name=name,
|
|
||||||
data_type=data_type,
|
|
||||||
nullable=len(concrete) != len(values),
|
|
||||||
)
|
|
||||||
)
|
|
||||||
return tuple(result)
|
|
||||||
|
|
||||||
|
|
||||||
def snapshot_fingerprint(
|
def snapshot_fingerprint(
|
||||||
@@ -717,22 +737,6 @@ def _unsupported_json(value: object) -> object:
|
|||||||
raise TypeError(f"{type(value).__name__} is not JSON serializable")
|
raise TypeError(f"{type(value).__name__} is not JSON serializable")
|
||||||
|
|
||||||
|
|
||||||
def _type_name(value: object) -> str:
|
|
||||||
if isinstance(value, bool):
|
|
||||||
return "boolean"
|
|
||||||
if isinstance(value, int):
|
|
||||||
return "integer"
|
|
||||||
if isinstance(value, (float, Decimal)):
|
|
||||||
return "number"
|
|
||||||
if isinstance(value, str):
|
|
||||||
return "string"
|
|
||||||
if isinstance(value, list):
|
|
||||||
return "array"
|
|
||||||
if isinstance(value, dict):
|
|
||||||
return "object"
|
|
||||||
return type(value).__name__.lower()
|
|
||||||
|
|
||||||
|
|
||||||
def _column_payload(column: TabularColumn) -> dict[str, object]:
|
def _column_payload(column: TabularColumn) -> dict[str, object]:
|
||||||
return {
|
return {
|
||||||
"name": column.name,
|
"name": column.name,
|
||||||
|
|||||||
@@ -0,0 +1,20 @@
|
|||||||
|
from govoplan_connectors.backend.manifest import get_manifest
|
||||||
|
|
||||||
|
|
||||||
|
def test_static_documentation_has_complete_german_reference_copy() -> None:
|
||||||
|
for topic in get_manifest().documentation:
|
||||||
|
german = topic.translations.get("de", {})
|
||||||
|
assert all(german.get(field, "").strip() for field in ("title", "summary", "body")), topic.id
|
||||||
|
|
||||||
|
|
||||||
|
def test_documentation_exposes_conditioned_workflow_and_reference() -> None:
|
||||||
|
topics = {topic.id: topic for topic in get_manifest().documentation}
|
||||||
|
workflow = topics["connectors.governed-configuration"]
|
||||||
|
assert workflow.metadata.get("kind") == "workflow"
|
||||||
|
assert any(condition.required_scopes for condition in workflow.conditions)
|
||||||
|
|
||||||
|
reference = topics["connectors.authority-and-effects"]
|
||||||
|
assert reference.metadata.get("kind") == "reference"
|
||||||
|
assert reference.metadata.get("fields")
|
||||||
|
assert reference.metadata.get("consequences")
|
||||||
|
assert reference.structured_translations.get("de")
|
||||||
@@ -17,6 +17,8 @@ from govoplan_connectors.backend.db.models import (
|
|||||||
ConnectorConfiguration,
|
ConnectorConfiguration,
|
||||||
ConnectorDefinition,
|
ConnectorDefinition,
|
||||||
ConnectorDefinitionRevision,
|
ConnectorDefinitionRevision,
|
||||||
|
ConnectorKnowledgeProfile,
|
||||||
|
ConnectorKnowledgeSyncRun,
|
||||||
ConnectorSanctionsAcquisitionRun,
|
ConnectorSanctionsAcquisitionRun,
|
||||||
ConnectorSimulationRun,
|
ConnectorSimulationRun,
|
||||||
ConnectorTabularSource,
|
ConnectorTabularSource,
|
||||||
@@ -201,6 +203,43 @@ class ConnectorsDsarProviderTests(unittest.TestCase):
|
|||||||
created_at=NOW,
|
created_at=NOW,
|
||||||
updated_at=NOW,
|
updated_at=NOW,
|
||||||
),
|
),
|
||||||
|
ConnectorKnowledgeProfile(
|
||||||
|
id="knowledge-profile-1",
|
||||||
|
tenant_id="tenant-1",
|
||||||
|
configuration_id="configuration-1",
|
||||||
|
status="active",
|
||||||
|
product="bluespice",
|
||||||
|
desired_maturity="migrate",
|
||||||
|
discovered_maturity="migrate",
|
||||||
|
source_authority_mode="external_mirror",
|
||||||
|
default_visibility="restricted",
|
||||||
|
default_acl_tokens=["group:secret-do-not-export"],
|
||||||
|
namespace_mappings=[{"secret": "mapping-do-not-export"}],
|
||||||
|
capabilities=["read", "synchronize", "migrate"],
|
||||||
|
discovery_evidence={"secret": "discovery-do-not-export"},
|
||||||
|
health_details={"secret": "health-do-not-export"},
|
||||||
|
updated_by="account-1",
|
||||||
|
created_at=NOW,
|
||||||
|
updated_at=NOW,
|
||||||
|
),
|
||||||
|
ConnectorKnowledgeSyncRun(
|
||||||
|
id="knowledge-run-1",
|
||||||
|
tenant_id="tenant-1",
|
||||||
|
profile_id="knowledge-profile-1",
|
||||||
|
mode="delta",
|
||||||
|
idempotency_key="knowledge-idempotency-do-not-export",
|
||||||
|
request_hash="knowledge-request-hash-do-not-export",
|
||||||
|
status="completed",
|
||||||
|
counts={"update": 1},
|
||||||
|
effects=[{"secret": "knowledge-effect-do-not-export"}],
|
||||||
|
diagnostics=[{"secret": "knowledge-diagnostic-do-not-export"}],
|
||||||
|
provenance={"secret": "knowledge-provenance-do-not-export"},
|
||||||
|
created_by="account-1",
|
||||||
|
started_at=NOW,
|
||||||
|
finished_at=NOW,
|
||||||
|
created_at=NOW,
|
||||||
|
updated_at=NOW,
|
||||||
|
),
|
||||||
)
|
)
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -219,6 +258,8 @@ class ConnectorsDsarProviderTests(unittest.TestCase):
|
|||||||
"definition_actor_attribution",
|
"definition_actor_attribution",
|
||||||
"configuration_actor_attribution",
|
"configuration_actor_attribution",
|
||||||
"simulation_actor_attribution",
|
"simulation_actor_attribution",
|
||||||
|
"knowledge_profile_actor_attribution",
|
||||||
|
"knowledge_run_actor_attribution",
|
||||||
},
|
},
|
||||||
{record.resource_type for record in records},
|
{record.resource_type for record in records},
|
||||||
)
|
)
|
||||||
@@ -247,6 +288,15 @@ class ConnectorsDsarProviderTests(unittest.TestCase):
|
|||||||
"diagnostic-do-not-export",
|
"diagnostic-do-not-export",
|
||||||
"provenance-do-not-export",
|
"provenance-do-not-export",
|
||||||
"review-reason-do-not-export",
|
"review-reason-do-not-export",
|
||||||
|
"secret-do-not-export",
|
||||||
|
"mapping-do-not-export",
|
||||||
|
"discovery-do-not-export",
|
||||||
|
"health-do-not-export",
|
||||||
|
"knowledge-idempotency-do-not-export",
|
||||||
|
"knowledge-request-hash-do-not-export",
|
||||||
|
"knowledge-effect-do-not-export",
|
||||||
|
"knowledge-diagnostic-do-not-export",
|
||||||
|
"knowledge-provenance-do-not-export",
|
||||||
):
|
):
|
||||||
self.assertNotIn(excluded, exported)
|
self.assertNotIn(excluded, exported)
|
||||||
|
|
||||||
@@ -332,7 +382,7 @@ class ConnectorsDsarProviderTests(unittest.TestCase):
|
|||||||
expected_revision=row.resource_revision,
|
expected_revision=row.resource_revision,
|
||||||
)
|
)
|
||||||
self.assertEqual("searched", row.status)
|
self.assertEqual("searched", row.status)
|
||||||
self.assertEqual(5, row.search_result["record_count"])
|
self.assertEqual(7, row.search_result["record_count"])
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
if __name__ == "__main__":
|
||||||
|
|||||||
@@ -10,6 +10,10 @@ from govoplan_core.core.sanctions import (
|
|||||||
CAPABILITY_CONNECTORS_SANCTIONS_SNAPSHOTS,
|
CAPABILITY_CONNECTORS_SANCTIONS_SNAPSHOTS,
|
||||||
)
|
)
|
||||||
from govoplan_connectors.backend.manifest import manifest
|
from govoplan_connectors.backend.manifest import manifest
|
||||||
|
from govoplan_connectors.backend.knowledge_connector import (
|
||||||
|
KNOWLEDGE_CAPABILITY,
|
||||||
|
KNOWLEDGE_PROVIDER_ID,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
class ConnectorsManifestTests(unittest.TestCase):
|
class ConnectorsManifestTests(unittest.TestCase):
|
||||||
@@ -41,6 +45,27 @@ class ConnectorsManifestTests(unittest.TestCase):
|
|||||||
"connectors.governed-configuration",
|
"connectors.governed-configuration",
|
||||||
{topic.id for topic in manifest.documentation},
|
{topic.id for topic in manifest.documentation},
|
||||||
)
|
)
|
||||||
|
self.assertIn(KNOWLEDGE_CAPABILITY, manifest.capability_factories)
|
||||||
|
self.assertIn(
|
||||||
|
KNOWLEDGE_CAPABILITY,
|
||||||
|
{interface.name for interface in manifest.provides_interfaces},
|
||||||
|
)
|
||||||
|
self.assertIn("search", manifest.optional_dependencies)
|
||||||
|
self.assertIn("wiki", manifest.optional_dependencies)
|
||||||
|
self.assertIn(
|
||||||
|
KNOWLEDGE_PROVIDER_ID,
|
||||||
|
{provider.id for provider in manifest.external_providers},
|
||||||
|
)
|
||||||
|
self.assertIn(
|
||||||
|
KNOWLEDGE_PROVIDER_ID,
|
||||||
|
{registration.id for registration in manifest.search_sources},
|
||||||
|
)
|
||||||
|
topic = next(
|
||||||
|
item
|
||||||
|
for item in manifest.documentation
|
||||||
|
if item.id == "connectors.mediawiki-bluespice"
|
||||||
|
)
|
||||||
|
self.assertIn("de", topic.translations)
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
if __name__ == "__main__":
|
||||||
|
|||||||
@@ -0,0 +1,659 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from types import SimpleNamespace
|
||||||
|
import unittest
|
||||||
|
from unittest.mock import patch
|
||||||
|
|
||||||
|
from sqlalchemy import create_engine, select
|
||||||
|
from sqlalchemy.orm import Session
|
||||||
|
|
||||||
|
from govoplan_core.auth import ApiPrincipal
|
||||||
|
from govoplan_core.core.access import PrincipalRef
|
||||||
|
from govoplan_core.core.search import (
|
||||||
|
SearchAuthorizationRequest,
|
||||||
|
SearchBackfillRequest,
|
||||||
|
)
|
||||||
|
from govoplan_core.core.recovery import RecoveryOperation, RecoveryStatus
|
||||||
|
from govoplan_core.core.runtime_coordination import (
|
||||||
|
RuntimeIdentity,
|
||||||
|
bind_process_runtime_identity,
|
||||||
|
)
|
||||||
|
from govoplan_core.db.base import Base
|
||||||
|
from govoplan_connectors.backend.db.models import (
|
||||||
|
ConnectorConfiguration,
|
||||||
|
ConnectorDefinition,
|
||||||
|
ConnectorDefinitionRevision,
|
||||||
|
ConnectorKnowledgeObject,
|
||||||
|
ConnectorKnowledgeProfile,
|
||||||
|
ConnectorKnowledgeSyncRun,
|
||||||
|
)
|
||||||
|
from govoplan_connectors.backend.knowledge_connector import (
|
||||||
|
KNOWLEDGE_PROVIDER_ID,
|
||||||
|
KNOWLEDGE_RESOURCE_TYPE,
|
||||||
|
KnowledgeConnectorError,
|
||||||
|
create_profile,
|
||||||
|
discover_profile,
|
||||||
|
list_objects,
|
||||||
|
migration_dry_run,
|
||||||
|
publish_page,
|
||||||
|
synchronize_profile,
|
||||||
|
update_profile,
|
||||||
|
)
|
||||||
|
from govoplan_connectors.backend.knowledge_schemas import (
|
||||||
|
KnowledgeMigrationDryRunRequest,
|
||||||
|
KnowledgeMigrationTargetState,
|
||||||
|
KnowledgeNamespaceMapping,
|
||||||
|
KnowledgeProfileCreateRequest,
|
||||||
|
KnowledgeProfileUpdateRequest,
|
||||||
|
KnowledgePublishRequest,
|
||||||
|
KnowledgeSyncRequest,
|
||||||
|
)
|
||||||
|
from govoplan_connectors.backend.knowledge_search import (
|
||||||
|
ExternalKnowledgeSearchSource,
|
||||||
|
)
|
||||||
|
from govoplan_connectors.backend.mediawiki_transport import (
|
||||||
|
MediaWikiChangeBatch,
|
||||||
|
MediaWikiPublishResult,
|
||||||
|
MediaWikiTransportError,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
ALL_SCOPES = frozenset(
|
||||||
|
{
|
||||||
|
"connectors:knowledge:read",
|
||||||
|
"connectors:knowledge:admin",
|
||||||
|
"connectors:knowledge:sync",
|
||||||
|
"connectors:knowledge:publish",
|
||||||
|
"connectors:knowledge:migrate",
|
||||||
|
}
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def principal(
|
||||||
|
tenant_id: str = "tenant-1",
|
||||||
|
*,
|
||||||
|
scopes: frozenset[str] = ALL_SCOPES,
|
||||||
|
groups: frozenset[str] = frozenset({"editors"}),
|
||||||
|
) -> ApiPrincipal:
|
||||||
|
return ApiPrincipal(
|
||||||
|
principal=PrincipalRef(
|
||||||
|
account_id="account-1",
|
||||||
|
membership_id="membership-1",
|
||||||
|
tenant_id=tenant_id,
|
||||||
|
scopes=scopes,
|
||||||
|
group_ids=groups,
|
||||||
|
),
|
||||||
|
account=SimpleNamespace(id="account-1"),
|
||||||
|
user=SimpleNamespace(id="account-1"),
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
class StaticTransport:
|
||||||
|
def __init__(self) -> None:
|
||||||
|
self.batches: list[MediaWikiChangeBatch] = []
|
||||||
|
self.change_calls = 0
|
||||||
|
self.publish_calls = 0
|
||||||
|
self.publish_error: MediaWikiTransportError | None = None
|
||||||
|
|
||||||
|
def discover(self, *, endpoint_url, credential):
|
||||||
|
del endpoint_url, credential
|
||||||
|
return {
|
||||||
|
"curtimestamp": "2026-08-22T10:00:00Z",
|
||||||
|
"query": {
|
||||||
|
"general": {
|
||||||
|
"generator": "MediaWiki 1.43.1",
|
||||||
|
"phpversion": "8.3.8",
|
||||||
|
},
|
||||||
|
"extensions": [
|
||||||
|
{"name": "BlueSpiceFoundation", "version": "4.5.2"},
|
||||||
|
{"name": "BlueSpicePermissionManager", "version": "4.5.2"},
|
||||||
|
],
|
||||||
|
"namespaces": {
|
||||||
|
"0": {"id": 0, "name": "", "content": True},
|
||||||
|
"4": {"id": 4, "name": "GovWiki", "content": True},
|
||||||
|
},
|
||||||
|
"userinfo": {
|
||||||
|
"id": 17,
|
||||||
|
"name": "govoplan",
|
||||||
|
"rights": ["read", "edit"],
|
||||||
|
"groups": ["bot"],
|
||||||
|
},
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
def changes(self, **kwargs):
|
||||||
|
del kwargs
|
||||||
|
self.change_calls += 1
|
||||||
|
if not self.batches:
|
||||||
|
raise AssertionError("No deterministic change batch remains")
|
||||||
|
return self.batches.pop(0)
|
||||||
|
|
||||||
|
def publish(self, **kwargs):
|
||||||
|
self.publish_calls += 1
|
||||||
|
if self.publish_error is not None:
|
||||||
|
raise self.publish_error
|
||||||
|
return MediaWikiPublishResult(
|
||||||
|
page_id=str(kwargs.get("expected_page_id") or "99"),
|
||||||
|
revision_id="901",
|
||||||
|
title=str(kwargs["title"]),
|
||||||
|
canonical_url="https://wiki.example.invalid/wiki/Published_Guide",
|
||||||
|
evidence={"result": "Success", "fixture": True},
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def page(
|
||||||
|
*,
|
||||||
|
page_id: str = "42",
|
||||||
|
revision_id: str = "501",
|
||||||
|
title: str = "Citizen Guide",
|
||||||
|
acl_tokens: list[str] | None = None,
|
||||||
|
) -> dict[str, object]:
|
||||||
|
return {
|
||||||
|
"change_kind": "upsert",
|
||||||
|
"change_cursor": f"rcid:{revision_id}",
|
||||||
|
"pageid": page_id,
|
||||||
|
"ns": 0,
|
||||||
|
"title": title,
|
||||||
|
"fullurl": f"https://wiki.example.invalid/wiki/{title.replace(' ', '_')}",
|
||||||
|
"lastrevid": revision_id,
|
||||||
|
"revisions": [
|
||||||
|
{
|
||||||
|
"revid": revision_id,
|
||||||
|
"timestamp": "2026-08-22T10:00:00Z",
|
||||||
|
"user": "Ada Admin",
|
||||||
|
"comment": "Reviewed guidance",
|
||||||
|
"contentmodel": "wikitext",
|
||||||
|
"sha1": f"sha-{revision_id}",
|
||||||
|
"slots": {
|
||||||
|
"main": {
|
||||||
|
"content": "Welcome {{UnsupportedBox|important}} [[Services]]"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"categories": [{"title": "Category:Citizen service"}],
|
||||||
|
"links": [{"ns": 0, "title": "Services"}],
|
||||||
|
"images": [{"ns": 6, "title": "File:guide.pdf", "pageid": 71}],
|
||||||
|
"discussions": [
|
||||||
|
{
|
||||||
|
"id": "discussion-1",
|
||||||
|
"author": "Ada Admin",
|
||||||
|
"body": "Please verify this section.",
|
||||||
|
"created_at": "2026-08-20T09:00:00Z",
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"permissions": {
|
||||||
|
"visibility": "restricted",
|
||||||
|
"acl_tokens": acl_tokens or ["group:editors"],
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
class MediaWikiConnectorTests(unittest.TestCase):
|
||||||
|
def setUp(self) -> None:
|
||||||
|
bind_process_runtime_identity(
|
||||||
|
RuntimeIdentity(
|
||||||
|
installation_id="mediawiki-connector-tests",
|
||||||
|
node_id="node-1",
|
||||||
|
incarnation="incarnation-1",
|
||||||
|
role="worker",
|
||||||
|
software_version="test",
|
||||||
|
composition_hash="a" * 64,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
self.engine = create_engine("sqlite+pysqlite:///:memory:")
|
||||||
|
Base.metadata.create_all(self.engine)
|
||||||
|
self.session = Session(self.engine)
|
||||||
|
self.audit = patch(
|
||||||
|
"govoplan_connectors.backend.knowledge_connector.audit_event"
|
||||||
|
)
|
||||||
|
self.audit.start()
|
||||||
|
self.credential = patch(
|
||||||
|
"govoplan_connectors.backend.knowledge_connector._credential",
|
||||||
|
return_value={"access_token": "fixture-token"},
|
||||||
|
)
|
||||||
|
self.credential.start()
|
||||||
|
self.transport = StaticTransport()
|
||||||
|
self.configuration_id = self._seed_configuration()
|
||||||
|
self.profile_id = self._profile()
|
||||||
|
|
||||||
|
def tearDown(self) -> None:
|
||||||
|
bind_process_runtime_identity(None)
|
||||||
|
self.credential.stop()
|
||||||
|
self.audit.stop()
|
||||||
|
self.session.close()
|
||||||
|
self.engine.dispose()
|
||||||
|
|
||||||
|
def _seed_configuration(self) -> str:
|
||||||
|
definition = ConnectorDefinition(
|
||||||
|
id="definition-1",
|
||||||
|
tenant_id="tenant-1",
|
||||||
|
definition_key="knowledge.mediawiki",
|
||||||
|
name="MediaWiki",
|
||||||
|
description="Knowledge transport",
|
||||||
|
status="active",
|
||||||
|
current_revision=1,
|
||||||
|
local_definition=True,
|
||||||
|
)
|
||||||
|
self.session.add(definition)
|
||||||
|
self.session.add(
|
||||||
|
ConnectorDefinitionRevision(
|
||||||
|
id="definition-revision-1",
|
||||||
|
definition_id=definition.id,
|
||||||
|
revision=1,
|
||||||
|
specification={
|
||||||
|
"provider": "mediawiki",
|
||||||
|
"protocol": "mediawiki_action_api",
|
||||||
|
},
|
||||||
|
definition_hash="definition-hash",
|
||||||
|
origin="local",
|
||||||
|
created_by="account-1",
|
||||||
|
)
|
||||||
|
)
|
||||||
|
configuration = ConnectorConfiguration(
|
||||||
|
id="configuration-1",
|
||||||
|
tenant_id="tenant-1",
|
||||||
|
definition_id=definition.id,
|
||||||
|
name="Institutional knowledge",
|
||||||
|
status="active",
|
||||||
|
endpoint_url="https://wiki.example.invalid",
|
||||||
|
credential_ref="credential-envelope-1",
|
||||||
|
base_definition_revision=1,
|
||||||
|
local_overrides={},
|
||||||
|
protected_paths=[],
|
||||||
|
effective_configuration={
|
||||||
|
"provider": "mediawiki",
|
||||||
|
"protocol": "mediawiki_action_api",
|
||||||
|
},
|
||||||
|
effective_hash="configuration-hash",
|
||||||
|
resource_revision=1,
|
||||||
|
ambiguity_policy="manual_review",
|
||||||
|
updated_by="account-1",
|
||||||
|
)
|
||||||
|
self.session.add(configuration)
|
||||||
|
self.session.flush()
|
||||||
|
return configuration.id
|
||||||
|
|
||||||
|
def _profile(self) -> str:
|
||||||
|
created = create_profile(
|
||||||
|
self.session,
|
||||||
|
principal(),
|
||||||
|
KnowledgeProfileCreateRequest(
|
||||||
|
configuration_id=self.configuration_id,
|
||||||
|
desired_maturity="migrate",
|
||||||
|
source_authority_mode="external_mirror",
|
||||||
|
default_visibility="restricted",
|
||||||
|
default_acl_tokens=["group:knowledge-managers"],
|
||||||
|
namespace_mappings=[
|
||||||
|
KnowledgeNamespaceMapping(
|
||||||
|
source_namespace_id=0,
|
||||||
|
source_name="",
|
||||||
|
target_space_ref="service-guidance",
|
||||||
|
target_path_prefix="imported",
|
||||||
|
)
|
||||||
|
],
|
||||||
|
),
|
||||||
|
)
|
||||||
|
return created.id
|
||||||
|
|
||||||
|
def _discover(self):
|
||||||
|
return discover_profile(
|
||||||
|
self.session,
|
||||||
|
principal(),
|
||||||
|
profile_id=self.profile_id,
|
||||||
|
transport=self.transport,
|
||||||
|
)
|
||||||
|
|
||||||
|
def _sync(self, raw_pages, *, key="sync-1", force_full=True):
|
||||||
|
self.transport.batches.append(
|
||||||
|
MediaWikiChangeBatch(
|
||||||
|
changes=tuple(raw_pages),
|
||||||
|
next_cursor=None,
|
||||||
|
complete=True,
|
||||||
|
high_watermark="2026-08-22T10:05:00Z",
|
||||||
|
evidence={"fixture": True},
|
||||||
|
)
|
||||||
|
)
|
||||||
|
return synchronize_profile(
|
||||||
|
self.session,
|
||||||
|
principal(),
|
||||||
|
profile_id=self.profile_id,
|
||||||
|
payload=KnowledgeSyncRequest(
|
||||||
|
idempotency_key=key,
|
||||||
|
force_full=force_full,
|
||||||
|
),
|
||||||
|
transport=self.transport,
|
||||||
|
registry=None,
|
||||||
|
)
|
||||||
|
|
||||||
|
def test_discovery_mapping_idempotency_and_migration_loss_diagnostics(self) -> None:
|
||||||
|
discovery = self._discover()
|
||||||
|
self.assertEqual("bluespice", discovery.product)
|
||||||
|
self.assertEqual("4.5.2", discovery.product_version)
|
||||||
|
self.assertEqual("migrate", discovery.maturity)
|
||||||
|
self.assertIn("publish", discovery.capabilities)
|
||||||
|
self.assertIn("permission_metadata", discovery.capabilities)
|
||||||
|
|
||||||
|
run = self._sync([page()])
|
||||||
|
self.assertEqual({"create": 1}, run.counts)
|
||||||
|
self.assertIn(
|
||||||
|
"attachments_reference_only", {item.code for item in run.diagnostics}
|
||||||
|
)
|
||||||
|
objects, _cursor = list_objects(
|
||||||
|
self.session, principal(), profile_id=self.profile_id
|
||||||
|
)
|
||||||
|
self.assertEqual(1, len(objects))
|
||||||
|
item = objects[0]
|
||||||
|
self.assertEqual("42", item.external_reference.object_id)
|
||||||
|
self.assertEqual("501", item.external_reference.version)
|
||||||
|
self.assertEqual("imported/Citizen-Guide", item.mapped_data["target_path"])
|
||||||
|
self.assertEqual("Ada Admin", item.mapped_data["revision_author"])
|
||||||
|
self.assertEqual(
|
||||||
|
"user", item.mapped_data["revision_author_reference"]["object_type"]
|
||||||
|
)
|
||||||
|
self.assertEqual("guide.pdf", item.mapped_data["files"][0]["name"])
|
||||||
|
self.assertEqual("discussion-1", item.mapped_data["discussions"][0]["external_id"])
|
||||||
|
|
||||||
|
replay = synchronize_profile(
|
||||||
|
self.session,
|
||||||
|
principal(),
|
||||||
|
profile_id=self.profile_id,
|
||||||
|
payload=KnowledgeSyncRequest(
|
||||||
|
idempotency_key="sync-1",
|
||||||
|
force_full=True,
|
||||||
|
),
|
||||||
|
transport=self.transport,
|
||||||
|
registry=None,
|
||||||
|
)
|
||||||
|
self.assertEqual(run.id, replay.id)
|
||||||
|
self.assertEqual(1, self.transport.change_calls)
|
||||||
|
with self.assertRaisesRegex(KnowledgeConnectorError, "different request"):
|
||||||
|
synchronize_profile(
|
||||||
|
self.session,
|
||||||
|
principal(),
|
||||||
|
profile_id=self.profile_id,
|
||||||
|
payload=KnowledgeSyncRequest(
|
||||||
|
idempotency_key="sync-1",
|
||||||
|
force_full=True,
|
||||||
|
limit=25,
|
||||||
|
),
|
||||||
|
transport=self.transport,
|
||||||
|
registry=None,
|
||||||
|
)
|
||||||
|
|
||||||
|
preview = migration_dry_run(
|
||||||
|
self.session,
|
||||||
|
principal(),
|
||||||
|
profile_id=self.profile_id,
|
||||||
|
payload=KnowledgeMigrationDryRunRequest(
|
||||||
|
idempotency_key="migration-1",
|
||||||
|
target_space_ref="service-guidance",
|
||||||
|
supported_macros=["SupportedBox"],
|
||||||
|
existing_targets=[
|
||||||
|
KnowledgeMigrationTargetState(
|
||||||
|
path="imported/Citizen-Guide",
|
||||||
|
source_external_id="different-page",
|
||||||
|
attachment_names=["guide.pdf"],
|
||||||
|
)
|
||||||
|
],
|
||||||
|
),
|
||||||
|
)
|
||||||
|
self.assertFalse(preview.can_apply)
|
||||||
|
self.assertEqual("2026-08-22T10:05:00Z", preview.source_revision)
|
||||||
|
self.assertEqual(
|
||||||
|
{"attachment_name_conflict", "target_path_conflict", "unsupported_macro"},
|
||||||
|
{item.code for item in preview.diagnostics},
|
||||||
|
)
|
||||||
|
|
||||||
|
def test_acl_changes_moves_deletes_and_search_authorization_are_current(self) -> None:
|
||||||
|
self._discover()
|
||||||
|
self._sync([page()])
|
||||||
|
source = ExternalKnowledgeSearchSource()
|
||||||
|
backfill = source.backfill(
|
||||||
|
self.session,
|
||||||
|
request=SearchBackfillRequest(
|
||||||
|
tenant_id="tenant-1",
|
||||||
|
provider_id=KNOWLEDGE_PROVIDER_ID,
|
||||||
|
resource_type=KNOWLEDGE_RESOURCE_TYPE,
|
||||||
|
rebuild_id="rebuild-1",
|
||||||
|
limit=100,
|
||||||
|
),
|
||||||
|
)
|
||||||
|
self.assertEqual(1, len(backfill.documents))
|
||||||
|
document = backfill.documents[0]
|
||||||
|
request = SearchAuthorizationRequest(
|
||||||
|
reference=document.reference,
|
||||||
|
source_revision=document.source_revision,
|
||||||
|
)
|
||||||
|
self.assertTrue(
|
||||||
|
source.authorize(self.session, principal(), requests=[request])[
|
||||||
|
document.reference.key
|
||||||
|
]
|
||||||
|
)
|
||||||
|
denied = principal(groups=frozenset({"other"}))
|
||||||
|
self.assertFalse(
|
||||||
|
source.authorize(self.session, denied, requests=[request])[
|
||||||
|
document.reference.key
|
||||||
|
]
|
||||||
|
)
|
||||||
|
|
||||||
|
self._sync(
|
||||||
|
[
|
||||||
|
page(
|
||||||
|
revision_id="502",
|
||||||
|
title="Resident Guide",
|
||||||
|
acl_tokens=["group:reviewers"],
|
||||||
|
)
|
||||||
|
],
|
||||||
|
key="sync-2",
|
||||||
|
force_full=False,
|
||||||
|
)
|
||||||
|
self.assertFalse(
|
||||||
|
source.authorize(self.session, principal(), requests=[request])[
|
||||||
|
document.reference.key
|
||||||
|
]
|
||||||
|
)
|
||||||
|
reviewers = principal(groups=frozenset({"reviewers"}))
|
||||||
|
self.assertTrue(
|
||||||
|
source.authorize(self.session, reviewers, requests=[request])[
|
||||||
|
document.reference.key
|
||||||
|
]
|
||||||
|
)
|
||||||
|
stored = self.session.scalar(
|
||||||
|
select(ConnectorKnowledgeObject).where(
|
||||||
|
ConnectorKnowledgeObject.profile_id == self.profile_id,
|
||||||
|
ConnectorKnowledgeObject.external_id == "42",
|
||||||
|
)
|
||||||
|
)
|
||||||
|
self.assertEqual("Resident Guide", stored.title)
|
||||||
|
self.assertEqual("imported/Resident-Guide", stored.mapped_data["target_path"])
|
||||||
|
|
||||||
|
self._sync(
|
||||||
|
[
|
||||||
|
{
|
||||||
|
"change_kind": "delete",
|
||||||
|
"change_cursor": "logid:700",
|
||||||
|
"pageid": 0,
|
||||||
|
"ns": 0,
|
||||||
|
"title": "Resident Guide",
|
||||||
|
"timestamp": "2026-08-22T11:00:00Z",
|
||||||
|
"logid": 700,
|
||||||
|
}
|
||||||
|
],
|
||||||
|
key="sync-3",
|
||||||
|
force_full=False,
|
||||||
|
)
|
||||||
|
self.assertEqual("deleted", stored.status)
|
||||||
|
self.assertFalse(
|
||||||
|
source.authorize(self.session, reviewers, requests=[request])[
|
||||||
|
document.reference.key
|
||||||
|
]
|
||||||
|
)
|
||||||
|
after_delete = source.backfill(
|
||||||
|
self.session,
|
||||||
|
request=SearchBackfillRequest(
|
||||||
|
tenant_id="tenant-1",
|
||||||
|
provider_id=KNOWLEDGE_PROVIDER_ID,
|
||||||
|
resource_type=KNOWLEDGE_RESOURCE_TYPE,
|
||||||
|
rebuild_id="rebuild-2",
|
||||||
|
limit=100,
|
||||||
|
),
|
||||||
|
)
|
||||||
|
self.assertEqual((), after_delete.documents)
|
||||||
|
|
||||||
|
def test_publication_replay_and_outcome_unknown_are_evidenced(self) -> None:
|
||||||
|
self._discover()
|
||||||
|
payload = KnowledgePublishRequest(
|
||||||
|
idempotency_key="publish-1",
|
||||||
|
title="Published Guide",
|
||||||
|
body="Reviewed body",
|
||||||
|
summary="Publish approved guidance",
|
||||||
|
expected_external_revision="900",
|
||||||
|
)
|
||||||
|
result = publish_page(
|
||||||
|
self.session,
|
||||||
|
principal(),
|
||||||
|
profile_id=self.profile_id,
|
||||||
|
external_page_id="99",
|
||||||
|
payload=payload,
|
||||||
|
transport=self.transport,
|
||||||
|
registry=None,
|
||||||
|
durable_recovery=True,
|
||||||
|
)
|
||||||
|
replay = publish_page(
|
||||||
|
self.session,
|
||||||
|
principal(),
|
||||||
|
profile_id=self.profile_id,
|
||||||
|
external_page_id="99",
|
||||||
|
payload=payload,
|
||||||
|
transport=self.transport,
|
||||||
|
registry=None,
|
||||||
|
durable_recovery=False,
|
||||||
|
)
|
||||||
|
self.assertTrue(result.accepted)
|
||||||
|
self.assertEqual(result.run.id, replay.run.id)
|
||||||
|
self.assertEqual("901", result.external_reference.version)
|
||||||
|
self.assertEqual(1, self.transport.publish_calls)
|
||||||
|
recovery = self.session.scalar(
|
||||||
|
select(RecoveryOperation).where(
|
||||||
|
RecoveryOperation.resource_type == "external_knowledge_page",
|
||||||
|
RecoveryOperation.resource_id == "99",
|
||||||
|
)
|
||||||
|
)
|
||||||
|
self.assertEqual(RecoveryStatus.SUCCEEDED.value, recovery.status)
|
||||||
|
|
||||||
|
self.transport.publish_error = MediaWikiTransportError(
|
||||||
|
"transport_timeout",
|
||||||
|
"Provider response timed out.",
|
||||||
|
retryable=True,
|
||||||
|
outcome_unknown=True,
|
||||||
|
)
|
||||||
|
with self.assertRaisesRegex(KnowledgeConnectorError, "outcome is unknown"):
|
||||||
|
publish_page(
|
||||||
|
self.session,
|
||||||
|
principal(),
|
||||||
|
profile_id=self.profile_id,
|
||||||
|
external_page_id="100",
|
||||||
|
payload=KnowledgePublishRequest(
|
||||||
|
idempotency_key="publish-unknown",
|
||||||
|
title="Uncertain Guide",
|
||||||
|
body="Body",
|
||||||
|
),
|
||||||
|
transport=self.transport,
|
||||||
|
registry=None,
|
||||||
|
durable_recovery=False,
|
||||||
|
)
|
||||||
|
unresolved = self.session.scalar(
|
||||||
|
select(ConnectorKnowledgeSyncRun).where(
|
||||||
|
ConnectorKnowledgeSyncRun.profile_id == self.profile_id,
|
||||||
|
ConnectorKnowledgeSyncRun.idempotency_key == "publish-unknown",
|
||||||
|
)
|
||||||
|
)
|
||||||
|
self.assertEqual("outcome_unknown", unresolved.status)
|
||||||
|
|
||||||
|
def test_fallback_acl_changes_and_profile_pause_fail_closed_immediately(self) -> None:
|
||||||
|
self._discover()
|
||||||
|
fallback_page = page(page_id="43", title="Fallback Guide")
|
||||||
|
fallback_page.pop("permissions")
|
||||||
|
self._sync([fallback_page])
|
||||||
|
source = ExternalKnowledgeSearchSource()
|
||||||
|
document = source.backfill(
|
||||||
|
self.session,
|
||||||
|
request=SearchBackfillRequest(
|
||||||
|
tenant_id="tenant-1",
|
||||||
|
provider_id=KNOWLEDGE_PROVIDER_ID,
|
||||||
|
resource_type=KNOWLEDGE_RESOURCE_TYPE,
|
||||||
|
rebuild_id="fallback-rebuild",
|
||||||
|
),
|
||||||
|
).documents[0]
|
||||||
|
request = SearchAuthorizationRequest(
|
||||||
|
reference=document.reference,
|
||||||
|
source_revision=document.source_revision,
|
||||||
|
)
|
||||||
|
managers = principal(groups=frozenset({"knowledge-managers"}))
|
||||||
|
self.assertTrue(
|
||||||
|
source.authorize(self.session, managers, requests=[request])[
|
||||||
|
document.reference.key
|
||||||
|
]
|
||||||
|
)
|
||||||
|
profile = self.session.get(ConnectorKnowledgeProfile, self.profile_id)
|
||||||
|
before_hash = self.session.get(
|
||||||
|
ConnectorKnowledgeObject, document.resource_id
|
||||||
|
).content_hash
|
||||||
|
updated = update_profile(
|
||||||
|
self.session,
|
||||||
|
principal(),
|
||||||
|
profile_id=self.profile_id,
|
||||||
|
payload=KnowledgeProfileUpdateRequest(
|
||||||
|
expected_resource_revision=profile.resource_revision,
|
||||||
|
default_acl_tokens=["group:reviewers"],
|
||||||
|
),
|
||||||
|
registry=None,
|
||||||
|
)
|
||||||
|
self.assertFalse(
|
||||||
|
source.authorize(self.session, managers, requests=[request])[
|
||||||
|
document.reference.key
|
||||||
|
]
|
||||||
|
)
|
||||||
|
reviewers = principal(groups=frozenset({"reviewers"}))
|
||||||
|
self.assertTrue(
|
||||||
|
source.authorize(self.session, reviewers, requests=[request])[
|
||||||
|
document.reference.key
|
||||||
|
]
|
||||||
|
)
|
||||||
|
self.assertNotEqual(
|
||||||
|
before_hash,
|
||||||
|
self.session.get(ConnectorKnowledgeObject, document.resource_id).content_hash,
|
||||||
|
)
|
||||||
|
|
||||||
|
update_profile(
|
||||||
|
self.session,
|
||||||
|
principal(),
|
||||||
|
profile_id=self.profile_id,
|
||||||
|
payload=KnowledgeProfileUpdateRequest(
|
||||||
|
expected_resource_revision=updated.resource_revision,
|
||||||
|
status="paused",
|
||||||
|
),
|
||||||
|
registry=None,
|
||||||
|
)
|
||||||
|
self.assertFalse(
|
||||||
|
source.authorize(self.session, reviewers, requests=[request])[
|
||||||
|
document.reference.key
|
||||||
|
]
|
||||||
|
)
|
||||||
|
|
||||||
|
def test_profiles_and_objects_are_tenant_isolated(self) -> None:
|
||||||
|
self._discover()
|
||||||
|
self._sync([page()])
|
||||||
|
with self.assertRaisesRegex(KnowledgeConnectorError, "not found"):
|
||||||
|
list_objects(
|
||||||
|
self.session,
|
||||||
|
principal("tenant-2"),
|
||||||
|
profile_id=self.profile_id,
|
||||||
|
)
|
||||||
|
profile = self.session.get(ConnectorKnowledgeProfile, self.profile_id)
|
||||||
|
self.assertEqual("tenant-1", profile.tenant_id)
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
unittest.main()
|
||||||
@@ -0,0 +1,158 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import json
|
||||||
|
from urllib.parse import parse_qs, urlsplit
|
||||||
|
import unittest
|
||||||
|
from unittest.mock import patch
|
||||||
|
|
||||||
|
from govoplan_core.security.http_fetch import HttpFetchResponse
|
||||||
|
from govoplan_connectors.backend.mediawiki_transport import HttpMediaWikiTransport
|
||||||
|
|
||||||
|
|
||||||
|
def response(payload: dict[str, object]) -> HttpFetchResponse:
|
||||||
|
return HttpFetchResponse(
|
||||||
|
status=200,
|
||||||
|
headers={"Content-Type": "application/json"},
|
||||||
|
body=json.dumps(payload).encode(),
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
class MediaWikiHttpTransportTests(unittest.TestCase):
|
||||||
|
def setUp(self) -> None:
|
||||||
|
self.transport = HttpMediaWikiTransport()
|
||||||
|
|
||||||
|
@patch("govoplan_connectors.backend.mediawiki_transport.fetch_http")
|
||||||
|
def test_discovery_uses_the_action_api_and_sanitized_auth_header(self, fetch) -> None:
|
||||||
|
fetch.return_value = response(
|
||||||
|
{"query": {"general": {"generator": "MediaWiki 1.43"}}}
|
||||||
|
)
|
||||||
|
|
||||||
|
self.transport.discover(
|
||||||
|
endpoint_url="https://wiki.example.test",
|
||||||
|
credential={"access_token": "secret-token"},
|
||||||
|
)
|
||||||
|
|
||||||
|
call = fetch.call_args
|
||||||
|
self.assertEqual("https://wiki.example.test/api.php", urlsplit(call.args[0])._replace(query="").geturl())
|
||||||
|
query = parse_qs(urlsplit(call.args[0]).query)
|
||||||
|
self.assertEqual(["query"], query["action"])
|
||||||
|
self.assertEqual(["siteinfo|userinfo"], query["meta"])
|
||||||
|
self.assertEqual("Bearer secret-token", call.kwargs["headers"]["Authorization"])
|
||||||
|
self.assertNotIn("secret-token", call.args[0])
|
||||||
|
self.assertEqual(10_000_000, call.kwargs["max_bytes"])
|
||||||
|
|
||||||
|
@patch("govoplan_connectors.backend.mediawiki_transport.fetch_http")
|
||||||
|
def test_full_backfill_resolves_page_details_and_cursor(self, fetch) -> None:
|
||||||
|
fetch.side_effect = (
|
||||||
|
response(
|
||||||
|
{
|
||||||
|
"curtimestamp": "2026-08-22T10:00:00Z",
|
||||||
|
"continue": {"apcontinue": "Next_Page"},
|
||||||
|
"query": {
|
||||||
|
"allpages": [
|
||||||
|
{"pageid": 41, "ns": 0, "title": "First"},
|
||||||
|
{"pageid": 42, "ns": 0, "title": "Second"},
|
||||||
|
]
|
||||||
|
},
|
||||||
|
}
|
||||||
|
),
|
||||||
|
response(
|
||||||
|
{
|
||||||
|
"query": {
|
||||||
|
"pages": [
|
||||||
|
{"pageid": 41, "ns": 0, "title": "First"},
|
||||||
|
{"pageid": 42, "ns": 0, "title": "Second"},
|
||||||
|
]
|
||||||
|
}
|
||||||
|
}
|
||||||
|
),
|
||||||
|
)
|
||||||
|
|
||||||
|
batch = self.transport.changes(
|
||||||
|
endpoint_url="https://wiki.example.test/api.php",
|
||||||
|
credential=None,
|
||||||
|
cursor=None,
|
||||||
|
limit=2,
|
||||||
|
force_full=True,
|
||||||
|
)
|
||||||
|
|
||||||
|
self.assertEqual(2, len(batch.changes))
|
||||||
|
self.assertEqual("Next_Page", batch.next_cursor)
|
||||||
|
self.assertFalse(batch.complete)
|
||||||
|
details = parse_qs(urlsplit(fetch.call_args_list[1].args[0]).query)
|
||||||
|
self.assertEqual(["41|42"], details["pageids"])
|
||||||
|
self.assertIn("revisions", details["prop"][0])
|
||||||
|
|
||||||
|
@patch("govoplan_connectors.backend.mediawiki_transport.fetch_http")
|
||||||
|
def test_delta_preserves_deletion_log_tombstone_and_cursor(self, fetch) -> None:
|
||||||
|
fetch.return_value = response(
|
||||||
|
{
|
||||||
|
"curtimestamp": "2026-08-22T11:00:00Z",
|
||||||
|
"query": {
|
||||||
|
"recentchanges": [
|
||||||
|
{
|
||||||
|
"type": "log",
|
||||||
|
"logtype": "delete",
|
||||||
|
"logid": 700,
|
||||||
|
"pageid": 0,
|
||||||
|
"ns": 0,
|
||||||
|
"title": "Deleted page",
|
||||||
|
"timestamp": "2026-08-22T10:59:00Z",
|
||||||
|
}
|
||||||
|
]
|
||||||
|
},
|
||||||
|
}
|
||||||
|
)
|
||||||
|
|
||||||
|
batch = self.transport.changes(
|
||||||
|
endpoint_url="https://wiki.example.test",
|
||||||
|
credential=None,
|
||||||
|
cursor="rccontinue-token",
|
||||||
|
limit=50,
|
||||||
|
force_full=False,
|
||||||
|
)
|
||||||
|
|
||||||
|
self.assertEqual("delete", batch.changes[0]["change_kind"])
|
||||||
|
self.assertEqual("logid:700", batch.changes[0]["change_cursor"])
|
||||||
|
self.assertTrue(batch.complete)
|
||||||
|
|
||||||
|
@patch("govoplan_connectors.backend.mediawiki_transport.fetch_http")
|
||||||
|
def test_publish_uses_csrf_body_expected_revision_and_canonical_url(self, fetch) -> None:
|
||||||
|
fetch.return_value = response(
|
||||||
|
{
|
||||||
|
"edit": {
|
||||||
|
"result": "Success",
|
||||||
|
"pageid": 99,
|
||||||
|
"oldrevid": 900,
|
||||||
|
"newrevid": 901,
|
||||||
|
"title": "Published Guide",
|
||||||
|
}
|
||||||
|
}
|
||||||
|
)
|
||||||
|
|
||||||
|
result = self.transport.publish(
|
||||||
|
endpoint_url="https://wiki.example.test",
|
||||||
|
credential={"csrf_token": "csrf-secret", "access_token": "token"},
|
||||||
|
title="Published Guide",
|
||||||
|
body="Reviewed body",
|
||||||
|
summary="Approved",
|
||||||
|
expected_revision="900",
|
||||||
|
minor=True,
|
||||||
|
)
|
||||||
|
|
||||||
|
call = fetch.call_args
|
||||||
|
self.assertEqual("POST", call.kwargs["method"])
|
||||||
|
body = parse_qs(call.kwargs["body"].decode())
|
||||||
|
self.assertEqual(["edit"], body["action"])
|
||||||
|
self.assertEqual(["900"], body["baserevid"])
|
||||||
|
self.assertEqual(["1"], body["minor"])
|
||||||
|
self.assertEqual(["csrf-secret"], body["token"])
|
||||||
|
self.assertEqual("99", result.page_id)
|
||||||
|
self.assertEqual(
|
||||||
|
"https://wiki.example.test/wiki/Published_Guide",
|
||||||
|
result.canonical_url,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
unittest.main()
|
||||||
@@ -2,16 +2,45 @@ from __future__ import annotations
|
|||||||
|
|
||||||
import tempfile
|
import tempfile
|
||||||
import unittest
|
import unittest
|
||||||
|
from datetime import UTC, datetime
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
|
|
||||||
|
from alembic import command
|
||||||
from alembic.runtime.migration import MigrationContext
|
from alembic.runtime.migration import MigrationContext
|
||||||
from sqlalchemy import create_engine, inspect
|
from sqlalchemy import MetaData, Table, create_engine, inspect, select
|
||||||
|
|
||||||
from govoplan_connectors.backend.manifest import get_manifest
|
from govoplan_connectors.backend.manifest import get_manifest
|
||||||
from govoplan_core.db.migrations import migrate_database
|
from govoplan_core.db.migrations import alembic_config, migrate_database
|
||||||
|
|
||||||
|
|
||||||
class ConnectorsMigrationTests(unittest.TestCase):
|
class ConnectorsMigrationTests(unittest.TestCase):
|
||||||
|
def test_csv_evidence_upgrade_preserves_legacy_snapshot_without_inventing_source(self) -> None:
|
||||||
|
with tempfile.TemporaryDirectory(prefix="govoplan-connectors-csv-migration-") as directory:
|
||||||
|
url = f"sqlite:///{Path(directory) / 'connectors.db'}"
|
||||||
|
config = alembic_config(database_url=url, enabled_modules=("connectors",), manifest_factories=(get_manifest,))
|
||||||
|
command.upgrade(config, "c0f1a2b3c4d5")
|
||||||
|
engine = create_engine(url)
|
||||||
|
try:
|
||||||
|
table = Table("connector_tabular_sources", MetaData(), autoload_with=engine)
|
||||||
|
now = datetime.now(UTC)
|
||||||
|
with engine.begin() as connection:
|
||||||
|
connection.execute(table.insert().values(
|
||||||
|
id="legacy-csv", tenant_id="tenant-1", provider="snapshot",
|
||||||
|
source_name="legacy", name="Legacy", status="active", schema_version=1,
|
||||||
|
schema=[{"name": "id", "data_type": "integer", "nullable": False}],
|
||||||
|
rows=[{"id": 1}], fingerprint="a" * 64, row_count=1, byte_count=10,
|
||||||
|
metadata={"original_label": "CSV"}, created_at=now, updated_at=now,
|
||||||
|
))
|
||||||
|
before = dict(connection.execute(select(table)).mappings().one())
|
||||||
|
command.upgrade(config, "d2a4c6e8f0b1")
|
||||||
|
upgraded = Table("connector_tabular_sources", MetaData(), autoload_with=engine)
|
||||||
|
with engine.connect() as connection:
|
||||||
|
after = dict(connection.execute(select(upgraded)).mappings().one())
|
||||||
|
self.assertIsNone(after.pop("csv_source"))
|
||||||
|
self.assertEqual(before, after)
|
||||||
|
finally:
|
||||||
|
engine.dispose()
|
||||||
|
|
||||||
def test_baseline_creates_connector_tables_and_head(self) -> None:
|
def test_baseline_creates_connector_tables_and_head(self) -> None:
|
||||||
with tempfile.TemporaryDirectory(prefix="govoplan-connectors-migration-") as directory:
|
with tempfile.TemporaryDirectory(prefix="govoplan-connectors-migration-") as directory:
|
||||||
url = f"sqlite:///{Path(directory) / 'connectors.db'}"
|
url = f"sqlite:///{Path(directory) / 'connectors.db'}"
|
||||||
@@ -24,7 +53,7 @@ class ConnectorsMigrationTests(unittest.TestCase):
|
|||||||
try:
|
try:
|
||||||
with engine.connect() as connection:
|
with engine.connect() as connection:
|
||||||
self.assertIn(
|
self.assertIn(
|
||||||
"a8d9e0f1b2c3",
|
"d2a4c6e8f0b1",
|
||||||
set(MigrationContext.configure(connection).get_current_heads()),
|
set(MigrationContext.configure(connection).get_current_heads()),
|
||||||
)
|
)
|
||||||
self.assertTrue(
|
self.assertTrue(
|
||||||
@@ -36,6 +65,12 @@ class ConnectorsMigrationTests(unittest.TestCase):
|
|||||||
"connector_definition_revisions",
|
"connector_definition_revisions",
|
||||||
"connector_configurations",
|
"connector_configurations",
|
||||||
"connector_simulation_runs",
|
"connector_simulation_runs",
|
||||||
|
"connector_knowledge_profiles",
|
||||||
|
"connector_knowledge_objects",
|
||||||
|
"connector_knowledge_sync_runs",
|
||||||
|
"connector_service_desk_profiles",
|
||||||
|
"connector_service_desk_objects",
|
||||||
|
"connector_service_desk_sync_runs",
|
||||||
}.issubset(inspect(connection).get_table_names())
|
}.issubset(inspect(connection).get_table_names())
|
||||||
)
|
)
|
||||||
finally:
|
finally:
|
||||||
|
|||||||
@@ -7,15 +7,26 @@ from sqlalchemy import create_engine
|
|||||||
from sqlalchemy.orm import sessionmaker
|
from sqlalchemy.orm import sessionmaker
|
||||||
|
|
||||||
from govoplan_connectors.backend.db.models import (
|
from govoplan_connectors.backend.db.models import (
|
||||||
|
ConnectorConfiguration,
|
||||||
|
ConnectorKnowledgeObject,
|
||||||
|
ConnectorKnowledgeProfile,
|
||||||
|
ConnectorKnowledgeSyncRun,
|
||||||
|
ConnectorServiceDeskObject,
|
||||||
|
ConnectorServiceDeskProfile,
|
||||||
|
ConnectorServiceDeskSyncRun,
|
||||||
ConnectorSanctionsAcquisitionRun,
|
ConnectorSanctionsAcquisitionRun,
|
||||||
ConnectorSanctionsSnapshot,
|
ConnectorSanctionsSnapshot,
|
||||||
ConnectorTabularSource,
|
ConnectorTabularSource,
|
||||||
)
|
)
|
||||||
from govoplan_connectors.backend.manifest import manifest
|
from govoplan_connectors.backend.manifest import manifest
|
||||||
from govoplan_connectors.backend.provider_state import (
|
from govoplan_connectors.backend.provider_state import (
|
||||||
|
KNOWLEDGE_PROVIDER_ID,
|
||||||
SANCTIONS_PROVIDER_ID,
|
SANCTIONS_PROVIDER_ID,
|
||||||
|
SERVICE_DESK_PROVIDER_ID,
|
||||||
TABULAR_PROVIDER_ID,
|
TABULAR_PROVIDER_ID,
|
||||||
|
knowledge_provider_states,
|
||||||
sanctions_provider_states,
|
sanctions_provider_states,
|
||||||
|
service_desk_provider_states,
|
||||||
tabular_provider_states,
|
tabular_provider_states,
|
||||||
)
|
)
|
||||||
from govoplan_core.core.provider_governance import ExternalProviderStateContext
|
from govoplan_core.core.provider_governance import ExternalProviderStateContext
|
||||||
@@ -27,11 +38,6 @@ class ConnectorsProviderStateTests(unittest.TestCase):
|
|||||||
self.engine = create_engine("sqlite+pysqlite:///:memory:", future=True)
|
self.engine = create_engine("sqlite+pysqlite:///:memory:", future=True)
|
||||||
Base.metadata.create_all(
|
Base.metadata.create_all(
|
||||||
self.engine,
|
self.engine,
|
||||||
tables=(
|
|
||||||
ConnectorTabularSource.__table__,
|
|
||||||
ConnectorSanctionsAcquisitionRun.__table__,
|
|
||||||
ConnectorSanctionsSnapshot.__table__,
|
|
||||||
),
|
|
||||||
)
|
)
|
||||||
self.session = sessionmaker(bind=self.engine, expire_on_commit=False)()
|
self.session = sessionmaker(bind=self.engine, expire_on_commit=False)()
|
||||||
|
|
||||||
@@ -109,10 +115,163 @@ class ConnectorsProviderStateTests(unittest.TestCase):
|
|||||||
self.assertNotIn("secret-source-name", rendered)
|
self.assertNotIn("secret-source-name", rendered)
|
||||||
self.assertNotIn("source.example.test", rendered)
|
self.assertNotIn("source.example.test", rendered)
|
||||||
self.assertEqual(
|
self.assertEqual(
|
||||||
{TABULAR_PROVIDER_ID, SANCTIONS_PROVIDER_ID},
|
{
|
||||||
|
TABULAR_PROVIDER_ID,
|
||||||
|
SANCTIONS_PROVIDER_ID,
|
||||||
|
KNOWLEDGE_PROVIDER_ID,
|
||||||
|
SERVICE_DESK_PROVIDER_ID,
|
||||||
|
},
|
||||||
{item.provider_id for item in manifest.external_provider_state_providers},
|
{item.provider_id for item in manifest.external_provider_state_providers},
|
||||||
)
|
)
|
||||||
|
|
||||||
|
def test_external_knowledge_state_reports_health_without_acl_or_endpoint_data(self) -> None:
|
||||||
|
now = datetime.now(UTC)
|
||||||
|
profile = ConnectorKnowledgeProfile(
|
||||||
|
id="knowledge-profile-1",
|
||||||
|
tenant_id="tenant-1",
|
||||||
|
configuration_id="configuration-secret",
|
||||||
|
status="active",
|
||||||
|
product="bluespice",
|
||||||
|
product_version="4.5.2",
|
||||||
|
desired_maturity="migrate",
|
||||||
|
discovered_maturity="migrate",
|
||||||
|
source_authority_mode="external_mirror",
|
||||||
|
default_visibility="restricted",
|
||||||
|
default_acl_tokens=["group:secret-acl"],
|
||||||
|
namespace_mappings=[{"secret": "mapping"}],
|
||||||
|
capabilities=["read", "synchronize", "migrate"],
|
||||||
|
health_status="healthy",
|
||||||
|
discovered_at=now,
|
||||||
|
)
|
||||||
|
page = ConnectorKnowledgeObject(
|
||||||
|
id="knowledge-object-1",
|
||||||
|
tenant_id="tenant-1",
|
||||||
|
profile_id=profile.id,
|
||||||
|
object_type="page",
|
||||||
|
external_id="42",
|
||||||
|
title="Secret page title",
|
||||||
|
status="active",
|
||||||
|
source_revision="501",
|
||||||
|
content_hash="c" * 64,
|
||||||
|
visibility="restricted",
|
||||||
|
acl_tokens=["group:secret-acl"],
|
||||||
|
observed_at=now,
|
||||||
|
)
|
||||||
|
run = ConnectorKnowledgeSyncRun(
|
||||||
|
id="knowledge-run-1",
|
||||||
|
tenant_id="tenant-1",
|
||||||
|
profile_id=profile.id,
|
||||||
|
mode="delta",
|
||||||
|
idempotency_key="secret-key",
|
||||||
|
request_hash="d" * 64,
|
||||||
|
status="completed",
|
||||||
|
started_at=now,
|
||||||
|
finished_at=now,
|
||||||
|
)
|
||||||
|
self.session.add_all((profile, page, run))
|
||||||
|
self.session.commit()
|
||||||
|
|
||||||
|
state = knowledge_provider_states(
|
||||||
|
ExternalProviderStateContext(session=self.session, tenant_id="tenant-1")
|
||||||
|
)[0]
|
||||||
|
|
||||||
|
self.assertEqual("healthy", state.health)
|
||||||
|
self.assertEqual("ready", state.recovery)
|
||||||
|
self.assertEqual(1, state.metrics["active_objects"])
|
||||||
|
rendered = str(state.to_dict())
|
||||||
|
self.assertNotIn("Secret page title", rendered)
|
||||||
|
self.assertNotIn("group:secret-acl", rendered)
|
||||||
|
self.assertNotIn("configuration-secret", rendered)
|
||||||
|
|
||||||
|
def test_service_desk_state_reports_recovery_without_ticket_or_acl_data(self) -> None:
|
||||||
|
now = datetime.now(UTC)
|
||||||
|
configuration = ConnectorConfiguration(
|
||||||
|
id="configuration-secret",
|
||||||
|
tenant_id="tenant-1",
|
||||||
|
definition_id="definition-1",
|
||||||
|
name="Secret service desk",
|
||||||
|
status="active",
|
||||||
|
base_definition_revision=1,
|
||||||
|
local_overrides={},
|
||||||
|
protected_paths=[],
|
||||||
|
effective_configuration={},
|
||||||
|
effective_hash="configuration-hash",
|
||||||
|
resource_revision=3,
|
||||||
|
ambiguity_policy="manual_review",
|
||||||
|
)
|
||||||
|
profile = ConnectorServiceDeskProfile(
|
||||||
|
id="service-desk-profile-1",
|
||||||
|
tenant_id="tenant-1",
|
||||||
|
configuration_id="configuration-secret",
|
||||||
|
status="active",
|
||||||
|
integration_mode="synchronize",
|
||||||
|
product="znuny",
|
||||||
|
product_version="7.1.4",
|
||||||
|
desired_maturity="synchronize",
|
||||||
|
discovered_maturity="synchronize",
|
||||||
|
source_authority_mode="governed_sync",
|
||||||
|
default_visibility="restricted",
|
||||||
|
default_acl_tokens=["group:secret-acl"],
|
||||||
|
routes={"secret": "route"},
|
||||||
|
queue_mappings=[{"secret": "queue"}],
|
||||||
|
dynamic_field_mappings=[{"secret": "field"}],
|
||||||
|
capabilities=["read", "search", "synchronize", "publish"],
|
||||||
|
discovered_configuration_revision=3,
|
||||||
|
discovered_configuration_hash="configuration-hash",
|
||||||
|
health_status="healthy",
|
||||||
|
discovered_at=now,
|
||||||
|
)
|
||||||
|
ticket = ConnectorServiceDeskObject(
|
||||||
|
id="service-desk-object-1",
|
||||||
|
tenant_id="tenant-1",
|
||||||
|
profile_id=profile.id,
|
||||||
|
object_type="ticket",
|
||||||
|
external_id="42",
|
||||||
|
external_ticket_number="secret-number",
|
||||||
|
title="Secret ticket title",
|
||||||
|
status="active",
|
||||||
|
source_revision="2026-08-22T10:00:00Z",
|
||||||
|
content_hash="e" * 64,
|
||||||
|
visibility="restricted",
|
||||||
|
acl_tokens=["group:secret-acl"],
|
||||||
|
mapped_data={"secret": "ticket content"},
|
||||||
|
provenance={"secret": "provider evidence"},
|
||||||
|
observed_at=now,
|
||||||
|
)
|
||||||
|
run = ConnectorServiceDeskSyncRun(
|
||||||
|
id="service-desk-run-1",
|
||||||
|
tenant_id="tenant-1",
|
||||||
|
profile_id=profile.id,
|
||||||
|
mode="update",
|
||||||
|
idempotency_key="secret-key",
|
||||||
|
request_hash="f" * 64,
|
||||||
|
status="outcome_unknown",
|
||||||
|
started_at=now,
|
||||||
|
finished_at=now,
|
||||||
|
)
|
||||||
|
self.session.add_all((configuration, profile, ticket, run))
|
||||||
|
self.session.commit()
|
||||||
|
|
||||||
|
state = service_desk_provider_states(
|
||||||
|
ExternalProviderStateContext(session=self.session, tenant_id="tenant-1")
|
||||||
|
)[0]
|
||||||
|
|
||||||
|
self.assertEqual(SERVICE_DESK_PROVIDER_ID, state.provider_id)
|
||||||
|
self.assertTrue(state.metrics["discovery_current"])
|
||||||
|
self.assertEqual("pending", state.conflict)
|
||||||
|
self.assertEqual("attention", state.recovery)
|
||||||
|
rendered = str(state.to_dict())
|
||||||
|
for secret in (
|
||||||
|
"configuration-secret",
|
||||||
|
"secret-number",
|
||||||
|
"Secret ticket title",
|
||||||
|
"group:secret-acl",
|
||||||
|
"ticket content",
|
||||||
|
"provider evidence",
|
||||||
|
"secret-key",
|
||||||
|
):
|
||||||
|
self.assertNotIn(secret, rendered)
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
if __name__ == "__main__":
|
||||||
unittest.main()
|
unittest.main()
|
||||||
|
|||||||
@@ -137,7 +137,7 @@ class ConnectorRecoveryTests(unittest.TestCase):
|
|||||||
self.assertFalse(declarations["read-snapshot"].provider_mutation)
|
self.assertFalse(declarations["read-snapshot"].provider_mutation)
|
||||||
self.assertTrue(declarations["read-snapshot"].implemented)
|
self.assertTrue(declarations["read-snapshot"].implemented)
|
||||||
self.assertTrue(declarations["external-mutation"].provider_mutation)
|
self.assertTrue(declarations["external-mutation"].provider_mutation)
|
||||||
self.assertFalse(declarations["external-mutation"].implemented)
|
self.assertTrue(declarations["external-mutation"].implemented)
|
||||||
|
|
||||||
def test_stale_atomic_connector_fence_fails_without_claiming_an_effect(self) -> None:
|
def test_stale_atomic_connector_fence_fails_without_claiming_an_effect(self) -> None:
|
||||||
recovery = begin_connector_read_snapshot(
|
recovery = begin_connector_read_snapshot(
|
||||||
|
|||||||
Executable
+52
@@ -0,0 +1,52 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import unittest
|
||||||
|
from types import SimpleNamespace
|
||||||
|
|
||||||
|
from govoplan_connectors.backend.search_principal import principal_acl_tokens
|
||||||
|
|
||||||
|
|
||||||
|
class SearchPrincipalTests(unittest.TestCase):
|
||||||
|
def test_legacy_first_seen_projection_and_cap_are_unchanged(self) -> None:
|
||||||
|
principal = SimpleNamespace(
|
||||||
|
account_id=" actor ",
|
||||||
|
membership_id="m",
|
||||||
|
identity_id="i",
|
||||||
|
group_ids=("g", "", "g", "2"),
|
||||||
|
role_ids=("r", "r"),
|
||||||
|
function_assignment_ids=("f",),
|
||||||
|
scopes=tuple(f"scope-{i}" for i in range(700)),
|
||||||
|
)
|
||||||
|
legacy = []
|
||||||
|
for prefix, attribute in (
|
||||||
|
("account", "account_id"),
|
||||||
|
("membership", "membership_id"),
|
||||||
|
("identity", "identity_id"),
|
||||||
|
):
|
||||||
|
value = getattr(principal, attribute, None)
|
||||||
|
if value:
|
||||||
|
legacy.append(f"{prefix}:{value}")
|
||||||
|
for prefix, attribute in (
|
||||||
|
("group", "group_ids"),
|
||||||
|
("role", "role_ids"),
|
||||||
|
("function", "function_assignment_ids"),
|
||||||
|
("scope", "scopes"),
|
||||||
|
):
|
||||||
|
legacy.extend(
|
||||||
|
f"{prefix}:{value}"
|
||||||
|
for value in getattr(principal, attribute, ())
|
||||||
|
if value
|
||||||
|
)
|
||||||
|
expected = tuple(dict.fromkeys(legacy))[:500]
|
||||||
|
self.assertEqual(expected, principal_acl_tokens(principal))
|
||||||
|
self.assertEqual(500, len(expected))
|
||||||
|
self.assertEqual((), principal_acl_tokens(object()))
|
||||||
|
|
||||||
|
def test_projection_stops_consuming_at_authorization_cap(self) -> None:
|
||||||
|
def bounded_scopes():
|
||||||
|
yield from (str(i) for i in range(500))
|
||||||
|
raise AssertionError("ACL projection scanned beyond its effective cap")
|
||||||
|
|
||||||
|
self.assertEqual(
|
||||||
|
500, len(principal_acl_tokens(SimpleNamespace(scopes=bounded_scopes())))
|
||||||
|
)
|
||||||
@@ -0,0 +1,806 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import json
|
||||||
|
from types import SimpleNamespace
|
||||||
|
import unittest
|
||||||
|
from unittest.mock import patch
|
||||||
|
|
||||||
|
from sqlalchemy import create_engine, select
|
||||||
|
from sqlalchemy.orm import Session
|
||||||
|
|
||||||
|
from govoplan_core.auth import ApiPrincipal
|
||||||
|
from govoplan_core.core.access import PrincipalRef
|
||||||
|
from govoplan_core.core.search import SearchAuthorizationRequest, SearchBackfillRequest
|
||||||
|
from govoplan_core.core.recovery import RecoveryOperation, RecoveryStatus
|
||||||
|
from govoplan_core.core.runtime_coordination import (
|
||||||
|
RuntimeIdentity,
|
||||||
|
bind_process_runtime_identity,
|
||||||
|
)
|
||||||
|
from govoplan_core.db.base import Base
|
||||||
|
from govoplan_connectors.backend.db.models import (
|
||||||
|
ConnectorConfiguration,
|
||||||
|
ConnectorDefinition,
|
||||||
|
ConnectorDefinitionRevision,
|
||||||
|
ConnectorServiceDeskObject,
|
||||||
|
ConnectorServiceDeskProfile,
|
||||||
|
ConnectorServiceDeskSyncRun,
|
||||||
|
)
|
||||||
|
from govoplan_connectors.backend.service_desk_connector import (
|
||||||
|
SERVICE_DESK_PROVIDER_ID,
|
||||||
|
SERVICE_DESK_RESOURCE_TYPE,
|
||||||
|
ServiceDeskConnectorError,
|
||||||
|
create_profile,
|
||||||
|
discover_profile,
|
||||||
|
list_objects,
|
||||||
|
synchronize_profile,
|
||||||
|
update_profile,
|
||||||
|
update_ticket,
|
||||||
|
)
|
||||||
|
from govoplan_connectors.backend.service_desk_schemas import (
|
||||||
|
ServiceDeskDynamicFieldMapping,
|
||||||
|
ServiceDeskProfileCreateRequest,
|
||||||
|
ServiceDeskProfileUpdateRequest,
|
||||||
|
ServiceDeskQueueMapping,
|
||||||
|
ServiceDeskRouteMapping,
|
||||||
|
ServiceDeskSyncRequest,
|
||||||
|
ServiceDeskTicketUpdateRequest,
|
||||||
|
)
|
||||||
|
from govoplan_connectors.backend.service_desk_search import ExternalServiceDeskSearchSource
|
||||||
|
from govoplan_connectors.backend.service_desk_transport import (
|
||||||
|
ServiceDeskChangeBatch,
|
||||||
|
ServiceDeskTransportError,
|
||||||
|
ServiceDeskUpdateResult,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
ALL_SCOPES = frozenset(
|
||||||
|
{
|
||||||
|
"connectors:service_desk:read",
|
||||||
|
"connectors:service_desk:admin",
|
||||||
|
"connectors:service_desk:sync",
|
||||||
|
"connectors:service_desk:update",
|
||||||
|
}
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def principal(
|
||||||
|
tenant_id: str = "tenant-1",
|
||||||
|
*,
|
||||||
|
groups: frozenset[str] = frozenset({"agents"}),
|
||||||
|
) -> ApiPrincipal:
|
||||||
|
return ApiPrincipal(
|
||||||
|
principal=PrincipalRef(
|
||||||
|
account_id="account-1",
|
||||||
|
membership_id="membership-1",
|
||||||
|
tenant_id=tenant_id,
|
||||||
|
scopes=ALL_SCOPES,
|
||||||
|
group_ids=groups,
|
||||||
|
),
|
||||||
|
account=SimpleNamespace(id="account-1"),
|
||||||
|
user=SimpleNamespace(id="account-1"),
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def ticket(
|
||||||
|
*,
|
||||||
|
ticket_id: str = "42",
|
||||||
|
revision: str = "2026-08-22T10:00:00Z",
|
||||||
|
queue: str = "Residents",
|
||||||
|
acl: list[str] | None = None,
|
||||||
|
) -> dict[str, object]:
|
||||||
|
return {
|
||||||
|
"TicketID": ticket_id,
|
||||||
|
"TicketNumber": f"20260822{ticket_id}",
|
||||||
|
"Title": "Resident parking inquiry",
|
||||||
|
"Queue": queue,
|
||||||
|
"QueueID": "3",
|
||||||
|
"State": "open",
|
||||||
|
"StateID": "4",
|
||||||
|
"Priority": "3 normal",
|
||||||
|
"PriorityID": "3",
|
||||||
|
"Owner": "agent.a",
|
||||||
|
"OwnerID": "7",
|
||||||
|
"CustomerUserID": "citizen-17",
|
||||||
|
"CustomerID": "organization-9",
|
||||||
|
"Changed": revision,
|
||||||
|
"GovOPlaNVisibility": "restricted",
|
||||||
|
"GovOPlaNACL": acl or ["group:agents"],
|
||||||
|
"DynamicField": [{"Name": "PermitKind", "Value": "resident"}],
|
||||||
|
"Article": [
|
||||||
|
{
|
||||||
|
"ArticleID": "71",
|
||||||
|
"Subject": "Question",
|
||||||
|
"Body": "Please verify the submitted address.",
|
||||||
|
"Created": "2026-08-22T09:58:00Z",
|
||||||
|
"Attachment": [
|
||||||
|
{
|
||||||
|
"AttachmentID": "91",
|
||||||
|
"Filename": "address.pdf",
|
||||||
|
"Filesize": 1234,
|
||||||
|
"ContentType": "application/pdf",
|
||||||
|
"ContentBase64": "not-retained",
|
||||||
|
}
|
||||||
|
],
|
||||||
|
}
|
||||||
|
],
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
class StaticTransport:
|
||||||
|
def __init__(self) -> None:
|
||||||
|
self.batches: list[ServiceDeskChangeBatch] = []
|
||||||
|
self.change_calls: list[dict[str, object]] = []
|
||||||
|
self.update_calls = 0
|
||||||
|
self.update_result = ServiceDeskUpdateResult(
|
||||||
|
ticket={
|
||||||
|
**ticket(revision="2026-08-22T11:00:00Z"),
|
||||||
|
"State": "pending reminder",
|
||||||
|
},
|
||||||
|
revision="2026-08-22T11:00:00Z",
|
||||||
|
evidence={"verified": True},
|
||||||
|
)
|
||||||
|
self.update_error: ServiceDeskTransportError | None = None
|
||||||
|
|
||||||
|
def discover(self, **kwargs):
|
||||||
|
del kwargs
|
||||||
|
return {
|
||||||
|
"product": "znuny",
|
||||||
|
"product_version": "7.1.4",
|
||||||
|
"api_family": "generic_interface_rest",
|
||||||
|
"capabilities": [
|
||||||
|
"discover",
|
||||||
|
"link",
|
||||||
|
"search",
|
||||||
|
"read",
|
||||||
|
"synchronize",
|
||||||
|
"publish",
|
||||||
|
],
|
||||||
|
"maturity": "synchronize",
|
||||||
|
"health_status": "healthy",
|
||||||
|
"revision": "discovery-1",
|
||||||
|
"diagnostics": [],
|
||||||
|
"evidence": {"fixture": True},
|
||||||
|
}
|
||||||
|
|
||||||
|
def changes(self, **kwargs):
|
||||||
|
self.change_calls.append(dict(kwargs))
|
||||||
|
if not self.batches:
|
||||||
|
raise AssertionError("No deterministic service-desk batch remains")
|
||||||
|
return self.batches.pop(0)
|
||||||
|
|
||||||
|
def update_ticket(self, **kwargs):
|
||||||
|
del kwargs
|
||||||
|
self.update_calls += 1
|
||||||
|
if self.update_error is not None:
|
||||||
|
raise self.update_error
|
||||||
|
return self.update_result
|
||||||
|
|
||||||
|
|
||||||
|
class RecordingSearchWriter:
|
||||||
|
def __init__(self) -> None:
|
||||||
|
self.upserts: list[object] = []
|
||||||
|
self.deletes: list[str] = []
|
||||||
|
|
||||||
|
def upsert_document(self, _session, _principal, *, document) -> None:
|
||||||
|
self.upserts.append(document)
|
||||||
|
|
||||||
|
def delete_document(
|
||||||
|
self,
|
||||||
|
_session,
|
||||||
|
_principal,
|
||||||
|
*,
|
||||||
|
tenant_id,
|
||||||
|
module_id,
|
||||||
|
resource_type,
|
||||||
|
resource_id,
|
||||||
|
) -> bool:
|
||||||
|
del tenant_id, module_id, resource_type
|
||||||
|
self.deletes.append(resource_id)
|
||||||
|
return True
|
||||||
|
|
||||||
|
def enqueue_change(self, _session, *, change) -> bool:
|
||||||
|
del change
|
||||||
|
return True
|
||||||
|
|
||||||
|
|
||||||
|
class SearchRegistry:
|
||||||
|
def __init__(self, writer: RecordingSearchWriter) -> None:
|
||||||
|
self.writer = writer
|
||||||
|
|
||||||
|
def has_capability(self, name: str) -> bool:
|
||||||
|
return name == "search.index_writer"
|
||||||
|
|
||||||
|
def capability(self, name: str):
|
||||||
|
if not self.has_capability(name):
|
||||||
|
raise KeyError(name)
|
||||||
|
return self.writer
|
||||||
|
|
||||||
|
|
||||||
|
class ServiceDeskConnectorTests(unittest.TestCase):
|
||||||
|
def setUp(self) -> None:
|
||||||
|
bind_process_runtime_identity(
|
||||||
|
RuntimeIdentity(
|
||||||
|
installation_id="service-desk-connector-tests",
|
||||||
|
node_id="node-1",
|
||||||
|
incarnation="incarnation-1",
|
||||||
|
role="worker",
|
||||||
|
software_version="test",
|
||||||
|
composition_hash="a" * 64,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
self.engine = create_engine("sqlite+pysqlite:///:memory:")
|
||||||
|
Base.metadata.create_all(self.engine)
|
||||||
|
self.session = Session(self.engine)
|
||||||
|
self.audit = patch("govoplan_connectors.backend.service_desk_connector.audit_event")
|
||||||
|
self.audit.start()
|
||||||
|
self.credential = patch(
|
||||||
|
"govoplan_connectors.backend.service_desk_connector._credential",
|
||||||
|
return_value={"user_login": "connector", "password": "secret"},
|
||||||
|
)
|
||||||
|
self.credential.start()
|
||||||
|
self.transport = StaticTransport()
|
||||||
|
self.configuration_id = self._seed_configuration()
|
||||||
|
self.profile_id = self._create_profile()
|
||||||
|
|
||||||
|
def tearDown(self) -> None:
|
||||||
|
bind_process_runtime_identity(None)
|
||||||
|
self.credential.stop()
|
||||||
|
self.audit.stop()
|
||||||
|
self.session.close()
|
||||||
|
self.engine.dispose()
|
||||||
|
|
||||||
|
def _seed_configuration(self) -> str:
|
||||||
|
definition = ConnectorDefinition(
|
||||||
|
id="definition-1",
|
||||||
|
tenant_id="tenant-1",
|
||||||
|
definition_key="service-desk.znuny",
|
||||||
|
name="Znuny",
|
||||||
|
status="active",
|
||||||
|
current_revision=1,
|
||||||
|
local_definition=True,
|
||||||
|
)
|
||||||
|
self.session.add(definition)
|
||||||
|
self.session.add(
|
||||||
|
ConnectorDefinitionRevision(
|
||||||
|
id="definition-revision-1",
|
||||||
|
definition_id=definition.id,
|
||||||
|
revision=1,
|
||||||
|
specification={"provider": "znuny", "protocol": "generic_interface_rest"},
|
||||||
|
definition_hash="definition-hash",
|
||||||
|
origin="local",
|
||||||
|
created_by="account-1",
|
||||||
|
)
|
||||||
|
)
|
||||||
|
configuration = ConnectorConfiguration(
|
||||||
|
id="configuration-1",
|
||||||
|
tenant_id="tenant-1",
|
||||||
|
definition_id=definition.id,
|
||||||
|
name="Institutional service desk",
|
||||||
|
status="active",
|
||||||
|
endpoint_url="https://support.example.invalid/znuny/nph-genericinterface.pl/Webservice/GovOPlaN",
|
||||||
|
credential_ref="credential-envelope-1",
|
||||||
|
base_definition_revision=1,
|
||||||
|
local_overrides={},
|
||||||
|
protected_paths=[],
|
||||||
|
effective_configuration={
|
||||||
|
"provider": "znuny",
|
||||||
|
"protocol": "generic_interface_rest",
|
||||||
|
},
|
||||||
|
effective_hash="configuration-hash",
|
||||||
|
resource_revision=1,
|
||||||
|
ambiguity_policy="manual_review",
|
||||||
|
updated_by="account-1",
|
||||||
|
)
|
||||||
|
self.session.add(configuration)
|
||||||
|
self.session.flush()
|
||||||
|
return configuration.id
|
||||||
|
|
||||||
|
def _create_profile(self) -> str:
|
||||||
|
item = create_profile(
|
||||||
|
self.session,
|
||||||
|
principal(),
|
||||||
|
ServiceDeskProfileCreateRequest(
|
||||||
|
configuration_id=self.configuration_id,
|
||||||
|
integration_mode="synchronize",
|
||||||
|
desired_maturity="synchronize",
|
||||||
|
source_authority_mode="governed_sync",
|
||||||
|
default_visibility="restricted",
|
||||||
|
default_acl_tokens=["group:service-desk-managers"],
|
||||||
|
routes=ServiceDeskRouteMapping(
|
||||||
|
update_path="/Ticket/{ticket_id}",
|
||||||
|
ticket_web_url_template="https://support.example.invalid/ticket/{ticket_id}",
|
||||||
|
),
|
||||||
|
queue_mappings=[
|
||||||
|
ServiceDeskQueueMapping(
|
||||||
|
source_queue="Residents",
|
||||||
|
target_queue_ref="helpdesk:residents",
|
||||||
|
visibility="restricted",
|
||||||
|
acl_tokens=["group:agents"],
|
||||||
|
)
|
||||||
|
],
|
||||||
|
dynamic_field_mappings=[
|
||||||
|
ServiceDeskDynamicFieldMapping(
|
||||||
|
source_name="PermitKind",
|
||||||
|
target_name="permit_kind",
|
||||||
|
)
|
||||||
|
],
|
||||||
|
),
|
||||||
|
)
|
||||||
|
return item.id
|
||||||
|
|
||||||
|
def _discover(self) -> None:
|
||||||
|
discover_profile(
|
||||||
|
self.session,
|
||||||
|
principal(),
|
||||||
|
profile_id=self.profile_id,
|
||||||
|
transport=self.transport,
|
||||||
|
)
|
||||||
|
|
||||||
|
def _sync(self, raw: list[dict[str, object]], *, key: str, complete: bool = True):
|
||||||
|
self.transport.batches.append(
|
||||||
|
ServiceDeskChangeBatch(
|
||||||
|
changes=tuple(raw),
|
||||||
|
next_cursor=(
|
||||||
|
None
|
||||||
|
if complete
|
||||||
|
else json.dumps(
|
||||||
|
{"kind": "full", "offset": len(raw), "fingerprint": "fixture"}
|
||||||
|
)
|
||||||
|
),
|
||||||
|
complete=complete,
|
||||||
|
high_watermark="2026-08-22T10:00:00Z",
|
||||||
|
live_ids=tuple(str(value["TicketID"]) for value in raw) if complete else None,
|
||||||
|
evidence={"fixture": True},
|
||||||
|
)
|
||||||
|
)
|
||||||
|
return synchronize_profile(
|
||||||
|
self.session,
|
||||||
|
principal(),
|
||||||
|
profile_id=self.profile_id,
|
||||||
|
payload=ServiceDeskSyncRequest(idempotency_key=key),
|
||||||
|
transport=self.transport,
|
||||||
|
registry=None,
|
||||||
|
)
|
||||||
|
|
||||||
|
def test_mapping_replay_cursor_transition_and_search_acl(self) -> None:
|
||||||
|
self._discover()
|
||||||
|
run = self._sync([ticket()], key="sync-1")
|
||||||
|
self.assertEqual({"created": 1}, run.counts)
|
||||||
|
self.assertEqual("delta", json.loads(run.cursor_after or "{}").get("kind"))
|
||||||
|
self.assertIn("attachment_content_omitted", {item.code for item in run.diagnostics})
|
||||||
|
|
||||||
|
objects, _cursor = list_objects(self.session, principal(), profile_id=self.profile_id)
|
||||||
|
self.assertEqual(1, len(objects))
|
||||||
|
item = objects[0]
|
||||||
|
self.assertEqual("helpdesk:residents", item.mapped_data["target_queue_ref"])
|
||||||
|
self.assertEqual("resident", item.mapped_data["dynamic_fields"]["permit_kind"])
|
||||||
|
self.assertFalse(item.mapped_data["attachments"][0]["content_retained"])
|
||||||
|
self.assertEqual("article", item.mapped_data["articles"][0]["reference"]["object_type"])
|
||||||
|
|
||||||
|
replay = synchronize_profile(
|
||||||
|
self.session,
|
||||||
|
principal(),
|
||||||
|
profile_id=self.profile_id,
|
||||||
|
payload=ServiceDeskSyncRequest(idempotency_key="sync-1"),
|
||||||
|
transport=self.transport,
|
||||||
|
registry=None,
|
||||||
|
)
|
||||||
|
self.assertEqual(run.id, replay.id)
|
||||||
|
self.assertEqual(1, len(self.transport.change_calls))
|
||||||
|
|
||||||
|
source = ExternalServiceDeskSearchSource()
|
||||||
|
document = source.backfill(
|
||||||
|
self.session,
|
||||||
|
request=SearchBackfillRequest(
|
||||||
|
tenant_id="tenant-1",
|
||||||
|
provider_id=SERVICE_DESK_PROVIDER_ID,
|
||||||
|
resource_type=SERVICE_DESK_RESOURCE_TYPE,
|
||||||
|
rebuild_id="rebuild-1",
|
||||||
|
),
|
||||||
|
).documents[0]
|
||||||
|
authorization = SearchAuthorizationRequest(
|
||||||
|
reference=document.reference,
|
||||||
|
source_revision=document.source_revision,
|
||||||
|
)
|
||||||
|
self.assertTrue(source.authorize(self.session, principal(), requests=[authorization])[document.reference.key])
|
||||||
|
self.assertFalse(
|
||||||
|
source.authorize(
|
||||||
|
self.session,
|
||||||
|
principal(groups=frozenset({"other"})),
|
||||||
|
requests=[authorization],
|
||||||
|
)[document.reference.key]
|
||||||
|
)
|
||||||
|
|
||||||
|
def test_paged_full_stays_full_then_switches_to_delta(self) -> None:
|
||||||
|
self._discover()
|
||||||
|
first = self._sync([ticket()], key="page-1", complete=False)
|
||||||
|
self.assertEqual("full", json.loads(first.cursor_after or "{}").get("kind"))
|
||||||
|
second = self._sync([ticket(ticket_id="43")], key="page-2", complete=True)
|
||||||
|
self.assertEqual("delta", json.loads(second.cursor_after or "{}").get("kind"))
|
||||||
|
self.assertTrue(self.transport.change_calls[0]["force_full"])
|
||||||
|
self.assertTrue(self.transport.change_calls[1]["force_full"])
|
||||||
|
|
||||||
|
def test_profile_updates_preserve_provider_acl_until_the_next_sync(self) -> None:
|
||||||
|
self._discover()
|
||||||
|
self._sync([ticket(acl=["group:provider-agents"])], key="provider-acl")
|
||||||
|
profile = self.session.get(ConnectorServiceDeskProfile, self.profile_id)
|
||||||
|
update_profile(
|
||||||
|
self.session,
|
||||||
|
principal(),
|
||||||
|
profile_id=self.profile_id,
|
||||||
|
payload=ServiceDeskProfileUpdateRequest(
|
||||||
|
expected_resource_revision=profile.resource_revision,
|
||||||
|
default_visibility="tenant",
|
||||||
|
default_acl_tokens=[],
|
||||||
|
),
|
||||||
|
registry=None,
|
||||||
|
)
|
||||||
|
stored = self.session.scalar(
|
||||||
|
select(ConnectorServiceDeskObject).where(
|
||||||
|
ConnectorServiceDeskObject.profile_id == self.profile_id,
|
||||||
|
ConnectorServiceDeskObject.external_id == "42",
|
||||||
|
)
|
||||||
|
)
|
||||||
|
self.assertEqual("restricted", stored.visibility)
|
||||||
|
self.assertEqual(["group:provider-agents"], stored.acl_tokens)
|
||||||
|
self.assertEqual("provider", stored.mapped_data["permission_source"])
|
||||||
|
|
||||||
|
def test_unchanged_editor_payload_does_not_reset_discovery_or_cursor(self) -> None:
|
||||||
|
self._discover()
|
||||||
|
self._sync([ticket()], key="before-noop-save")
|
||||||
|
profile = self.session.get(ConnectorServiceDeskProfile, self.profile_id)
|
||||||
|
cursor_before = profile.last_sync_cursor
|
||||||
|
discovered_at = profile.discovered_at
|
||||||
|
update_profile(
|
||||||
|
self.session,
|
||||||
|
principal(),
|
||||||
|
profile_id=self.profile_id,
|
||||||
|
payload=ServiceDeskProfileUpdateRequest(
|
||||||
|
expected_resource_revision=profile.resource_revision,
|
||||||
|
integration_mode=profile.integration_mode,
|
||||||
|
desired_maturity=profile.desired_maturity,
|
||||||
|
source_authority_mode=profile.source_authority_mode,
|
||||||
|
default_visibility=profile.default_visibility,
|
||||||
|
default_acl_tokens=list(profile.default_acl_tokens),
|
||||||
|
routes=ServiceDeskRouteMapping.model_validate(profile.routes),
|
||||||
|
queue_mappings=[
|
||||||
|
ServiceDeskQueueMapping.model_validate(value)
|
||||||
|
for value in profile.queue_mappings
|
||||||
|
],
|
||||||
|
dynamic_field_mappings=[
|
||||||
|
ServiceDeskDynamicFieldMapping.model_validate(value)
|
||||||
|
for value in profile.dynamic_field_mappings
|
||||||
|
],
|
||||||
|
),
|
||||||
|
registry=None,
|
||||||
|
)
|
||||||
|
self.assertEqual(cursor_before, profile.last_sync_cursor)
|
||||||
|
self.assertEqual(discovered_at, profile.discovered_at)
|
||||||
|
|
||||||
|
def test_route_or_configuration_changes_require_rediscovery(self) -> None:
|
||||||
|
self._discover()
|
||||||
|
self._sync([ticket()], key="before-route-change")
|
||||||
|
profile = self.session.get(ConnectorServiceDeskProfile, self.profile_id)
|
||||||
|
update_profile(
|
||||||
|
self.session,
|
||||||
|
principal(),
|
||||||
|
profile_id=self.profile_id,
|
||||||
|
payload=ServiceDeskProfileUpdateRequest(
|
||||||
|
expected_resource_revision=profile.resource_revision,
|
||||||
|
routes=ServiceDeskRouteMapping(
|
||||||
|
search_path="/GovOPlaN/Ticket/Search",
|
||||||
|
ticket_path="/Ticket/{ticket_id}",
|
||||||
|
update_path="/Ticket/{ticket_id}",
|
||||||
|
),
|
||||||
|
),
|
||||||
|
registry=None,
|
||||||
|
)
|
||||||
|
self.assertIsNone(profile.discovered_at)
|
||||||
|
self.assertEqual([], profile.capabilities)
|
||||||
|
stored = self.session.scalar(
|
||||||
|
select(ConnectorServiceDeskObject).where(
|
||||||
|
ConnectorServiceDeskObject.profile_id == self.profile_id,
|
||||||
|
ConnectorServiceDeskObject.external_id == "42",
|
||||||
|
)
|
||||||
|
)
|
||||||
|
self.assertEqual("deleted", stored.status)
|
||||||
|
with self.assertRaisesRegex(ServiceDeskConnectorError, "Discover the provider"):
|
||||||
|
synchronize_profile(
|
||||||
|
self.session,
|
||||||
|
principal(),
|
||||||
|
profile_id=self.profile_id,
|
||||||
|
payload=ServiceDeskSyncRequest(idempotency_key="route-stale"),
|
||||||
|
transport=self.transport,
|
||||||
|
registry=None,
|
||||||
|
)
|
||||||
|
|
||||||
|
self._discover()
|
||||||
|
self._sync([ticket()], key="after-route-rediscovery")
|
||||||
|
self.assertEqual("active", stored.status)
|
||||||
|
configuration = self.session.get(ConnectorConfiguration, self.configuration_id)
|
||||||
|
configuration.resource_revision += 1
|
||||||
|
configuration.effective_hash = "configuration-hash-changed"
|
||||||
|
self.session.commit()
|
||||||
|
self.assertEqual(
|
||||||
|
(),
|
||||||
|
ExternalServiceDeskSearchSource()
|
||||||
|
.backfill(
|
||||||
|
self.session,
|
||||||
|
request=SearchBackfillRequest(
|
||||||
|
tenant_id="tenant-1",
|
||||||
|
provider_id=SERVICE_DESK_PROVIDER_ID,
|
||||||
|
resource_type=SERVICE_DESK_RESOURCE_TYPE,
|
||||||
|
rebuild_id="stale-configuration",
|
||||||
|
),
|
||||||
|
)
|
||||||
|
.documents,
|
||||||
|
)
|
||||||
|
with self.assertRaisesRegex(ServiceDeskConnectorError, "configuration changed"):
|
||||||
|
synchronize_profile(
|
||||||
|
self.session,
|
||||||
|
principal(),
|
||||||
|
profile_id=self.profile_id,
|
||||||
|
payload=ServiceDeskSyncRequest(idempotency_key="configuration-stale"),
|
||||||
|
transport=self.transport,
|
||||||
|
registry=None,
|
||||||
|
)
|
||||||
|
self._discover()
|
||||||
|
self.assertEqual("deleted", stored.status)
|
||||||
|
self.assertIsNone(profile.last_sync_cursor)
|
||||||
|
|
||||||
|
def test_full_sync_reprojects_unchanged_tickets_into_search(self) -> None:
|
||||||
|
self._discover()
|
||||||
|
self._sync([ticket()], key="initial-search-projection")
|
||||||
|
writer = RecordingSearchWriter()
|
||||||
|
self.transport.batches.append(
|
||||||
|
ServiceDeskChangeBatch(
|
||||||
|
changes=(ticket(),),
|
||||||
|
next_cursor=None,
|
||||||
|
complete=True,
|
||||||
|
high_watermark="2026-08-22T10:00:00Z",
|
||||||
|
live_ids=("42",),
|
||||||
|
evidence={"fixture": True},
|
||||||
|
)
|
||||||
|
)
|
||||||
|
run = synchronize_profile(
|
||||||
|
self.session,
|
||||||
|
principal(),
|
||||||
|
profile_id=self.profile_id,
|
||||||
|
payload=ServiceDeskSyncRequest(
|
||||||
|
idempotency_key="full-search-reprojection",
|
||||||
|
mode="full",
|
||||||
|
),
|
||||||
|
transport=self.transport,
|
||||||
|
registry=SearchRegistry(writer),
|
||||||
|
)
|
||||||
|
self.assertEqual({"unchanged": 1}, run.counts)
|
||||||
|
self.assertEqual(1, len(writer.upserts))
|
||||||
|
self.assertIsNone(self.transport.change_calls[-1]["cursor"])
|
||||||
|
|
||||||
|
def test_delta_cannot_bootstrap_without_a_completed_full_sync(self) -> None:
|
||||||
|
self._discover()
|
||||||
|
with self.assertRaisesRegex(ServiceDeskConnectorError, "completed full"):
|
||||||
|
synchronize_profile(
|
||||||
|
self.session,
|
||||||
|
principal(),
|
||||||
|
profile_id=self.profile_id,
|
||||||
|
payload=ServiceDeskSyncRequest(
|
||||||
|
idempotency_key="unsafe-delta-bootstrap",
|
||||||
|
mode="delta",
|
||||||
|
),
|
||||||
|
transport=self.transport,
|
||||||
|
registry=None,
|
||||||
|
)
|
||||||
|
self.assertEqual([], self.transport.change_calls)
|
||||||
|
|
||||||
|
def test_link_mode_omits_top_level_attachment_metadata(self) -> None:
|
||||||
|
self._discover()
|
||||||
|
profile = self.session.get(ConnectorServiceDeskProfile, self.profile_id)
|
||||||
|
update_profile(
|
||||||
|
self.session,
|
||||||
|
principal(),
|
||||||
|
profile_id=self.profile_id,
|
||||||
|
payload=ServiceDeskProfileUpdateRequest(
|
||||||
|
expected_resource_revision=profile.resource_revision,
|
||||||
|
integration_mode="link",
|
||||||
|
desired_maturity="link",
|
||||||
|
source_authority_mode="linked_reference",
|
||||||
|
),
|
||||||
|
registry=None,
|
||||||
|
)
|
||||||
|
raw = ticket()
|
||||||
|
raw.pop("Article")
|
||||||
|
raw.pop("DynamicField")
|
||||||
|
raw["Attachment"] = [
|
||||||
|
{
|
||||||
|
"AttachmentID": "top-1",
|
||||||
|
"Filename": "metadata-only.pdf",
|
||||||
|
"Filesize": 42,
|
||||||
|
}
|
||||||
|
]
|
||||||
|
run = self._sync([raw], key="link-refresh")
|
||||||
|
objects, _cursor = list_objects(
|
||||||
|
self.session,
|
||||||
|
principal(),
|
||||||
|
profile_id=self.profile_id,
|
||||||
|
)
|
||||||
|
self.assertEqual([], objects[0].mapped_data["attachments"])
|
||||||
|
self.assertIn("link_mode_content_omitted", {item.code for item in run.diagnostics})
|
||||||
|
self.assertTrue(self.transport.change_calls[-1]["routes"]["_identity_only"])
|
||||||
|
|
||||||
|
def test_excluded_queue_removes_projection_and_profile_policy_fails_closed(self) -> None:
|
||||||
|
self._discover()
|
||||||
|
self._sync([ticket()], key="sync-active")
|
||||||
|
profile = self.session.get(ConnectorServiceDeskProfile, self.profile_id)
|
||||||
|
update_profile(
|
||||||
|
self.session,
|
||||||
|
principal(),
|
||||||
|
profile_id=self.profile_id,
|
||||||
|
payload=ServiceDeskProfileUpdateRequest(
|
||||||
|
expected_resource_revision=profile.resource_revision,
|
||||||
|
queue_mappings=[
|
||||||
|
ServiceDeskQueueMapping(
|
||||||
|
source_queue="Residents",
|
||||||
|
include=False,
|
||||||
|
visibility="restricted",
|
||||||
|
acl_tokens=["group:agents"],
|
||||||
|
)
|
||||||
|
],
|
||||||
|
),
|
||||||
|
registry=None,
|
||||||
|
)
|
||||||
|
stored = self.session.scalar(
|
||||||
|
select(ConnectorServiceDeskObject).where(
|
||||||
|
ConnectorServiceDeskObject.profile_id == self.profile_id,
|
||||||
|
ConnectorServiceDeskObject.external_id == "42",
|
||||||
|
)
|
||||||
|
)
|
||||||
|
self.assertEqual("deleted", stored.status)
|
||||||
|
self.assertEqual(
|
||||||
|
(),
|
||||||
|
ExternalServiceDeskSearchSource()
|
||||||
|
.backfill(
|
||||||
|
self.session,
|
||||||
|
request=SearchBackfillRequest(
|
||||||
|
tenant_id="tenant-1",
|
||||||
|
provider_id=SERVICE_DESK_PROVIDER_ID,
|
||||||
|
resource_type=SERVICE_DESK_RESOURCE_TYPE,
|
||||||
|
rebuild_id="excluded-rebuild",
|
||||||
|
),
|
||||||
|
)
|
||||||
|
.documents,
|
||||||
|
)
|
||||||
|
self.transport.batches.append(
|
||||||
|
ServiceDeskChangeBatch(
|
||||||
|
changes=(ticket(revision="2026-08-22T10:30:00Z"),),
|
||||||
|
next_cursor=json.dumps(
|
||||||
|
{"kind": "delta", "changed": "2026-08-22T10:30:00Z", "seen": ["42"]}
|
||||||
|
),
|
||||||
|
complete=True,
|
||||||
|
high_watermark="2026-08-22T10:30:00Z",
|
||||||
|
live_ids=None,
|
||||||
|
evidence={"fixture": True},
|
||||||
|
)
|
||||||
|
)
|
||||||
|
run = synchronize_profile(
|
||||||
|
self.session,
|
||||||
|
principal(),
|
||||||
|
profile_id=self.profile_id,
|
||||||
|
payload=ServiceDeskSyncRequest(idempotency_key="sync-excluded"),
|
||||||
|
transport=self.transport,
|
||||||
|
registry=None,
|
||||||
|
)
|
||||||
|
self.assertEqual({"ignored": 1}, run.counts)
|
||||||
|
self.assertEqual("deleted", stored.status)
|
||||||
|
|
||||||
|
with self.assertRaises(ServiceDeskConnectorError):
|
||||||
|
update_profile(
|
||||||
|
self.session,
|
||||||
|
principal(),
|
||||||
|
profile_id=self.profile_id,
|
||||||
|
payload=ServiceDeskProfileUpdateRequest(
|
||||||
|
expected_resource_revision=self.session.get(
|
||||||
|
ConnectorServiceDeskProfile, self.profile_id
|
||||||
|
).resource_revision,
|
||||||
|
integration_mode="link",
|
||||||
|
desired_maturity="synchronize",
|
||||||
|
source_authority_mode="linked_reference",
|
||||||
|
),
|
||||||
|
registry=None,
|
||||||
|
)
|
||||||
|
|
||||||
|
def test_governed_update_replay_and_unknown_outcome(self) -> None:
|
||||||
|
self._discover()
|
||||||
|
self._sync([ticket()], key="sync-before-update")
|
||||||
|
payload = ServiceDeskTicketUpdateRequest(
|
||||||
|
idempotency_key="update-1",
|
||||||
|
expected_external_revision="2026-08-22T10:00:00Z",
|
||||||
|
state="pending reminder",
|
||||||
|
)
|
||||||
|
result = update_ticket(
|
||||||
|
self.session,
|
||||||
|
principal(),
|
||||||
|
profile_id=self.profile_id,
|
||||||
|
external_ticket_id="42",
|
||||||
|
payload=payload,
|
||||||
|
transport=self.transport,
|
||||||
|
registry=None,
|
||||||
|
durable_recovery=True,
|
||||||
|
)
|
||||||
|
replay = update_ticket(
|
||||||
|
self.session,
|
||||||
|
principal(),
|
||||||
|
profile_id=self.profile_id,
|
||||||
|
external_ticket_id="42",
|
||||||
|
payload=payload,
|
||||||
|
transport=self.transport,
|
||||||
|
registry=None,
|
||||||
|
durable_recovery=False,
|
||||||
|
)
|
||||||
|
self.assertTrue(result.accepted)
|
||||||
|
self.assertEqual(result.run.id, replay.run.id)
|
||||||
|
self.assertEqual(1, self.transport.update_calls)
|
||||||
|
recovery = self.session.scalar(
|
||||||
|
select(RecoveryOperation).where(
|
||||||
|
RecoveryOperation.resource_type == SERVICE_DESK_RESOURCE_TYPE,
|
||||||
|
RecoveryOperation.resource_id == "42",
|
||||||
|
)
|
||||||
|
)
|
||||||
|
self.assertEqual(RecoveryStatus.SUCCEEDED.value, recovery.status)
|
||||||
|
|
||||||
|
self.transport.update_error = ServiceDeskTransportError(
|
||||||
|
"provider_unavailable",
|
||||||
|
"No conclusive provider response.",
|
||||||
|
retryable=True,
|
||||||
|
outcome_unknown=True,
|
||||||
|
)
|
||||||
|
with self.assertRaisesRegex(ServiceDeskConnectorError, "outcome is unknown"):
|
||||||
|
update_ticket(
|
||||||
|
self.session,
|
||||||
|
principal(),
|
||||||
|
profile_id=self.profile_id,
|
||||||
|
external_ticket_id="42",
|
||||||
|
payload=ServiceDeskTicketUpdateRequest(
|
||||||
|
idempotency_key="update-unknown",
|
||||||
|
expected_external_revision="2026-08-22T11:00:00Z",
|
||||||
|
priority="4 high",
|
||||||
|
),
|
||||||
|
transport=self.transport,
|
||||||
|
registry=None,
|
||||||
|
durable_recovery=False,
|
||||||
|
)
|
||||||
|
unresolved = self.session.scalar(
|
||||||
|
select(ConnectorServiceDeskSyncRun).where(
|
||||||
|
ConnectorServiceDeskSyncRun.idempotency_key == "update-unknown"
|
||||||
|
)
|
||||||
|
)
|
||||||
|
self.assertEqual("outcome_unknown", unresolved.status)
|
||||||
|
|
||||||
|
def test_tenant_isolation(self) -> None:
|
||||||
|
with self.assertRaisesRegex(ServiceDeskConnectorError, "not found"):
|
||||||
|
list_objects(self.session, principal("tenant-2"), profile_id=self.profile_id)
|
||||||
|
|
||||||
|
def test_malformed_continuous_batch_is_atomic_and_evidenced(self) -> None:
|
||||||
|
self._discover()
|
||||||
|
malformed = ticket()
|
||||||
|
malformed.pop("Changed")
|
||||||
|
with self.assertRaisesRegex(ServiceDeskConnectorError, "change timestamp"):
|
||||||
|
self._sync([malformed], key="sync-malformed")
|
||||||
|
self.assertIsNone(
|
||||||
|
self.session.scalar(
|
||||||
|
select(ConnectorServiceDeskObject).where(
|
||||||
|
ConnectorServiceDeskObject.profile_id == self.profile_id
|
||||||
|
)
|
||||||
|
)
|
||||||
|
)
|
||||||
|
failed = self.session.scalar(
|
||||||
|
select(ConnectorServiceDeskSyncRun).where(
|
||||||
|
ConnectorServiceDeskSyncRun.idempotency_key == "sync-malformed"
|
||||||
|
)
|
||||||
|
)
|
||||||
|
self.assertEqual("failed", failed.status)
|
||||||
|
self.assertEqual("change_timestamp_missing", failed.diagnostics[0]["code"])
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
unittest.main()
|
||||||
@@ -0,0 +1,286 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import json
|
||||||
|
from urllib.parse import parse_qs, urlsplit
|
||||||
|
import unittest
|
||||||
|
from unittest.mock import patch
|
||||||
|
|
||||||
|
from govoplan_core.security.http_fetch import HttpFetchResponse
|
||||||
|
from govoplan_connectors.backend.service_desk_transport import (
|
||||||
|
HttpServiceDeskTransport,
|
||||||
|
ServiceDeskTransportError,
|
||||||
|
)
|
||||||
|
from govoplan_connectors.backend.service_desk_schemas import ServiceDeskRouteMapping
|
||||||
|
|
||||||
|
|
||||||
|
def response(payload: dict[str, object], *, headers: dict[str, str] | None = None):
|
||||||
|
return HttpFetchResponse(
|
||||||
|
status=200,
|
||||||
|
headers={"Content-Type": "application/json", **(headers or {})},
|
||||||
|
body=json.dumps(payload).encode(),
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
class ServiceDeskTransportTests(unittest.TestCase):
|
||||||
|
def setUp(self) -> None:
|
||||||
|
self.transport = HttpServiceDeskTransport()
|
||||||
|
self.endpoint = "https://support.example.test/znuny/nph-genericinterface.pl/Webservice/GovOPlaN"
|
||||||
|
self.routes = {
|
||||||
|
"search_path": "/Ticket/Search",
|
||||||
|
"ticket_path": "/Ticket/{ticket_id}",
|
||||||
|
"search_method": "POST",
|
||||||
|
"ticket_method": "GET",
|
||||||
|
}
|
||||||
|
|
||||||
|
def test_governed_routes_reject_embedded_authentication_controls(self) -> None:
|
||||||
|
for field, value in (
|
||||||
|
("search_path", "/Ticket/Search?Password=secret"),
|
||||||
|
(
|
||||||
|
"ticket_web_url_template",
|
||||||
|
"https://desk.example.test/ticket/{ticket_id}?SessionID=secret",
|
||||||
|
),
|
||||||
|
):
|
||||||
|
with self.subTest(field=field), self.assertRaisesRegex(
|
||||||
|
ValueError, "authentication controls"
|
||||||
|
):
|
||||||
|
ServiceDeskRouteMapping(**{field: value})
|
||||||
|
|
||||||
|
@patch("govoplan_connectors.backend.service_desk_transport.fetch_http")
|
||||||
|
def test_discovery_uses_governed_route_header_auth_and_version(self, fetch) -> None:
|
||||||
|
fetch.return_value = response(
|
||||||
|
{"TicketID": []}, headers={"X-Znuny-Version": "7.1.4"}
|
||||||
|
)
|
||||||
|
discovery = self.transport.discover(
|
||||||
|
endpoint_url=self.endpoint,
|
||||||
|
credential={"user_login": "connector", "password": "secret"},
|
||||||
|
routes={**self.routes, "search_filters": {"QueueIDs": [3, 7]}},
|
||||||
|
)
|
||||||
|
call = fetch.call_args
|
||||||
|
self.assertEqual("POST", call.kwargs["method"])
|
||||||
|
self.assertTrue(call.args[0].endswith("/Ticket/Search"))
|
||||||
|
self.assertEqual("connector", call.kwargs["headers"]["X-OTRS-Header-UserLogin"])
|
||||||
|
self.assertIn(
|
||||||
|
"X-OTRS-Header-Password",
|
||||||
|
call.kwargs["redirect_sensitive_headers"],
|
||||||
|
)
|
||||||
|
self.assertNotIn("secret", call.args[0])
|
||||||
|
self.assertEqual([3, 7], json.loads(call.kwargs["body"])["QueueIDs"])
|
||||||
|
self.assertEqual("znuny", discovery["product"])
|
||||||
|
self.assertEqual("synchronize", discovery["maturity"])
|
||||||
|
|
||||||
|
@patch("govoplan_connectors.backend.service_desk_transport.fetch_http")
|
||||||
|
def test_get_ticket_flags_are_query_parameters_without_secrets(self, fetch) -> None:
|
||||||
|
fetch.side_effect = (
|
||||||
|
response({"TicketID": ["42"]}),
|
||||||
|
response({"Ticket": [{"TicketID": "42", "Changed": "2026-08-22T10:00:00Z"}]}),
|
||||||
|
)
|
||||||
|
batch = self.transport.changes(
|
||||||
|
endpoint_url=self.endpoint,
|
||||||
|
credential={"user_login": "connector", "password": "secret"},
|
||||||
|
routes=self.routes,
|
||||||
|
cursor=None,
|
||||||
|
limit=100,
|
||||||
|
force_full=True,
|
||||||
|
)
|
||||||
|
query = parse_qs(urlsplit(fetch.call_args_list[1].args[0]).query)
|
||||||
|
self.assertEqual(["1"], query["AllArticles"])
|
||||||
|
self.assertEqual(["0"], query["GetAttachmentContents"])
|
||||||
|
self.assertNotIn("UserLogin", query)
|
||||||
|
self.assertEqual(1, len(batch.changes))
|
||||||
|
|
||||||
|
@patch("govoplan_connectors.backend.service_desk_transport.fetch_http")
|
||||||
|
def test_link_identity_reads_do_not_request_articles_attachments_or_dynamic_fields(
|
||||||
|
self, fetch
|
||||||
|
) -> None:
|
||||||
|
fetch.side_effect = (
|
||||||
|
response({"TicketID": ["42"]}),
|
||||||
|
response(
|
||||||
|
{"Ticket": [{"TicketID": "42", "Changed": "2026-08-22T10:00:00Z"}]}
|
||||||
|
),
|
||||||
|
)
|
||||||
|
self.transport.changes(
|
||||||
|
endpoint_url=self.endpoint,
|
||||||
|
credential=None,
|
||||||
|
routes={**self.routes, "_identity_only": True},
|
||||||
|
cursor=None,
|
||||||
|
limit=100,
|
||||||
|
force_full=True,
|
||||||
|
)
|
||||||
|
query = parse_qs(urlsplit(fetch.call_args_list[1].args[0]).query)
|
||||||
|
self.assertEqual(["0"], query["AllArticles"])
|
||||||
|
self.assertEqual(["0"], query["Attachments"])
|
||||||
|
self.assertEqual(["0"], query["DynamicFields"])
|
||||||
|
|
||||||
|
@patch("govoplan_connectors.backend.service_desk_transport.fetch_http")
|
||||||
|
def test_paged_full_cursor_preserves_cumulative_high_watermark(self, fetch) -> None:
|
||||||
|
fetch.side_effect = (
|
||||||
|
response({"TicketID": ["41", "42"]}),
|
||||||
|
response({"Ticket": [{"TicketID": "41", "Changed": "2026-08-22T12:00:00Z"}]}),
|
||||||
|
response({"TicketID": ["41", "42"]}),
|
||||||
|
response({"Ticket": [{"TicketID": "42", "Changed": "2026-08-22T10:00:00Z"}]}),
|
||||||
|
)
|
||||||
|
first = self.transport.changes(
|
||||||
|
endpoint_url=self.endpoint,
|
||||||
|
credential=None,
|
||||||
|
routes=self.routes,
|
||||||
|
cursor=None,
|
||||||
|
limit=1,
|
||||||
|
force_full=True,
|
||||||
|
)
|
||||||
|
second = self.transport.changes(
|
||||||
|
endpoint_url=self.endpoint,
|
||||||
|
credential=None,
|
||||||
|
routes=self.routes,
|
||||||
|
cursor=first.next_cursor,
|
||||||
|
limit=1,
|
||||||
|
force_full=True,
|
||||||
|
)
|
||||||
|
self.assertFalse(first.complete)
|
||||||
|
self.assertTrue(second.complete)
|
||||||
|
self.assertEqual("2026-08-22T12:00:00Z", second.high_watermark)
|
||||||
|
|
||||||
|
@patch("govoplan_connectors.backend.service_desk_transport.fetch_http")
|
||||||
|
def test_delta_overlaps_and_pages_all_ids_at_one_timestamp(self, fetch) -> None:
|
||||||
|
shared_revision = "2026-08-22T10:00:00Z"
|
||||||
|
fetch.side_effect = (
|
||||||
|
response({"TicketID": ["41", "42"]}),
|
||||||
|
response({"Ticket": [{"TicketID": "41", "Changed": shared_revision}]}),
|
||||||
|
response({"Ticket": [{"TicketID": "42", "Changed": shared_revision}]}),
|
||||||
|
response({"TicketID": ["41", "42"]}),
|
||||||
|
response({"Ticket": [{"TicketID": "41", "Changed": shared_revision}]}),
|
||||||
|
response({"Ticket": [{"TicketID": "42", "Changed": shared_revision}]}),
|
||||||
|
)
|
||||||
|
first = self.transport.changes(
|
||||||
|
endpoint_url=self.endpoint,
|
||||||
|
credential=None,
|
||||||
|
routes=self.routes,
|
||||||
|
cursor=json.dumps(
|
||||||
|
{"kind": "delta", "changed": "2026-08-22T09:59:59Z", "seen": []}
|
||||||
|
),
|
||||||
|
limit=1,
|
||||||
|
force_full=False,
|
||||||
|
)
|
||||||
|
second = self.transport.changes(
|
||||||
|
endpoint_url=self.endpoint,
|
||||||
|
credential=None,
|
||||||
|
routes=self.routes,
|
||||||
|
cursor=first.next_cursor,
|
||||||
|
limit=1,
|
||||||
|
force_full=False,
|
||||||
|
)
|
||||||
|
self.assertEqual(["41"], [item["TicketID"] for item in first.changes])
|
||||||
|
self.assertEqual(["42"], [item["TicketID"] for item in second.changes])
|
||||||
|
second_search = json.loads(fetch.call_args_list[3].kwargs["body"])
|
||||||
|
self.assertEqual(
|
||||||
|
"2026-08-22T09:59:59Z",
|
||||||
|
second_search["TicketChangeTimeNewerDate"],
|
||||||
|
)
|
||||||
|
|
||||||
|
@patch("govoplan_connectors.backend.service_desk_transport.fetch_http")
|
||||||
|
def test_delta_does_not_suppress_a_seen_ticket_that_changed_again(self, fetch) -> None:
|
||||||
|
fetch.side_effect = (
|
||||||
|
response({"TicketID": ["41"]}),
|
||||||
|
response(
|
||||||
|
{
|
||||||
|
"Ticket": [
|
||||||
|
{"TicketID": "41", "Changed": "2026-08-22T10:05:00Z"}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
),
|
||||||
|
)
|
||||||
|
batch = self.transport.changes(
|
||||||
|
endpoint_url=self.endpoint,
|
||||||
|
credential=None,
|
||||||
|
routes=self.routes,
|
||||||
|
cursor=json.dumps(
|
||||||
|
{
|
||||||
|
"kind": "delta",
|
||||||
|
"changed": "2026-08-22T10:00:00Z",
|
||||||
|
"seen": ["41"],
|
||||||
|
}
|
||||||
|
),
|
||||||
|
limit=10,
|
||||||
|
force_full=False,
|
||||||
|
)
|
||||||
|
self.assertEqual(["41"], [item["TicketID"] for item in batch.changes])
|
||||||
|
self.assertEqual("2026-08-22T10:05:00Z", batch.high_watermark)
|
||||||
|
|
||||||
|
@patch("govoplan_connectors.backend.service_desk_transport.fetch_http")
|
||||||
|
def test_body_authentication_is_never_put_in_get_url(self, fetch) -> None:
|
||||||
|
fetch.return_value = response({"TicketID": ["42"]})
|
||||||
|
with self.assertRaisesRegex(ServiceDeskTransportError, "cannot be used with a GET"):
|
||||||
|
self.transport.changes(
|
||||||
|
endpoint_url=self.endpoint,
|
||||||
|
credential={
|
||||||
|
"auth_mode": "body",
|
||||||
|
"user_login": "connector",
|
||||||
|
"password": "secret",
|
||||||
|
},
|
||||||
|
routes={**self.routes, "search_method": "GET"},
|
||||||
|
cursor=None,
|
||||||
|
limit=1,
|
||||||
|
force_full=True,
|
||||||
|
)
|
||||||
|
fetch.assert_not_called()
|
||||||
|
|
||||||
|
@patch("govoplan_connectors.backend.service_desk_transport.fetch_http")
|
||||||
|
def test_legacy_body_authentication_is_post_only_and_not_duplicated(self, fetch) -> None:
|
||||||
|
fetch.return_value = response({"TicketID": []})
|
||||||
|
self.transport.discover(
|
||||||
|
endpoint_url=self.endpoint,
|
||||||
|
credential={
|
||||||
|
"auth_mode": "body",
|
||||||
|
"user_login": "connector",
|
||||||
|
"password": "secret",
|
||||||
|
},
|
||||||
|
routes=self.routes,
|
||||||
|
)
|
||||||
|
call = fetch.call_args
|
||||||
|
payload = json.loads(call.kwargs["body"])
|
||||||
|
self.assertEqual("connector", payload["UserLogin"])
|
||||||
|
self.assertEqual("secret", payload["Password"])
|
||||||
|
self.assertNotIn("X-OTRS-Header-UserLogin", call.kwargs["headers"])
|
||||||
|
self.assertNotIn("secret", call.args[0])
|
||||||
|
|
||||||
|
@patch("govoplan_connectors.backend.service_desk_transport.fetch_http")
|
||||||
|
def test_update_requires_revision_and_requested_field_verification(self, fetch) -> None:
|
||||||
|
fetch.side_effect = (
|
||||||
|
response(
|
||||||
|
{
|
||||||
|
"Ticket": [
|
||||||
|
{
|
||||||
|
"TicketID": "42",
|
||||||
|
"State": "open",
|
||||||
|
"Changed": "2026-08-22T10:00:00Z",
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
),
|
||||||
|
response({"Success": 1}),
|
||||||
|
response(
|
||||||
|
{
|
||||||
|
"Ticket": [
|
||||||
|
{
|
||||||
|
"TicketID": "42",
|
||||||
|
"State": "open",
|
||||||
|
"Changed": "2026-08-22T10:05:00Z",
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
),
|
||||||
|
)
|
||||||
|
with self.assertRaises(ServiceDeskTransportError) as raised:
|
||||||
|
self.transport.update_ticket(
|
||||||
|
endpoint_url=self.endpoint,
|
||||||
|
credential=None,
|
||||||
|
routes={**self.routes, "update_path": "/Ticket/{ticket_id}"},
|
||||||
|
ticket_id="42",
|
||||||
|
expected_revision="2026-08-22T10:00:00Z",
|
||||||
|
changes={"State": "pending reminder"},
|
||||||
|
)
|
||||||
|
self.assertTrue(raised.exception.outcome_unknown)
|
||||||
|
self.assertEqual("update_verification_failed", raised.exception.code)
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
unittest.main()
|
||||||
@@ -1,23 +1,27 @@
|
|||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
import unittest
|
import unittest
|
||||||
|
from unittest.mock import patch
|
||||||
|
from dataclasses import replace
|
||||||
|
|
||||||
from fastapi import HTTPException
|
from fastapi import HTTPException
|
||||||
from sqlalchemy import create_engine
|
from sqlalchemy import create_engine, inspect
|
||||||
from sqlalchemy.orm import sessionmaker
|
from sqlalchemy.orm import sessionmaker
|
||||||
|
|
||||||
from govoplan_core.auth import ApiPrincipal
|
from govoplan_core.auth import ApiPrincipal
|
||||||
from govoplan_core.core.access import PrincipalRef
|
from govoplan_core.core.access import PrincipalRef
|
||||||
from govoplan_core.core.tabular_sources import (
|
from govoplan_core.core.tabular_sources import (
|
||||||
|
TabularCsvSource,
|
||||||
TabularReadRequest,
|
TabularReadRequest,
|
||||||
TabularSnapshotInput,
|
TabularSnapshotInput,
|
||||||
TabularSourceAccessError,
|
TabularSourceAccessError,
|
||||||
TabularSourceUnavailableError,
|
TabularSourceUnavailableError,
|
||||||
TabularSourceValidationError,
|
TabularSourceValidationError,
|
||||||
|
TabularSourceNotFoundError,
|
||||||
)
|
)
|
||||||
from govoplan_core.db.base import Base
|
from govoplan_core.db.base import Base
|
||||||
from govoplan_connectors.backend.db.models import ConnectorTabularSource
|
from govoplan_connectors.backend.db.models import ConnectorTabularSource
|
||||||
from govoplan_connectors.backend.router import api_create_tabular_snapshot
|
from govoplan_connectors.backend.router import api_create_tabular_snapshot, api_original_tabular_csv
|
||||||
from govoplan_connectors.backend.schemas import SnapshotCreateRequest
|
from govoplan_connectors.backend.schemas import SnapshotCreateRequest
|
||||||
from govoplan_connectors.backend.tabular_sources import (
|
from govoplan_connectors.backend.tabular_sources import (
|
||||||
READ_SCOPE,
|
READ_SCOPE,
|
||||||
@@ -263,6 +267,54 @@ class ConnectorsTabularSourceTests(unittest.TestCase):
|
|||||||
|
|
||||||
self.assertEqual(422, raised.exception.status_code)
|
self.assertEqual(422, raised.exception.status_code)
|
||||||
|
|
||||||
|
def test_original_csv_round_trip_is_private_bounded_and_not_catalogue_content(self) -> None:
|
||||||
|
text = '\ufeffid,value\r\n9007199254740993," keep me "\r\ntrue,0.123456789012345678901234567890\r\n" ",""\r\n'
|
||||||
|
payload = SnapshotCreateRequest(name="CSV", source_name="csv", format="csv", csv_text=text)
|
||||||
|
with patch("govoplan_connectors.backend.router.audit_event"):
|
||||||
|
created = api_create_tabular_snapshot(payload, session=self.session, principal=principal())
|
||||||
|
self.session.expunge_all()
|
||||||
|
listed = self.provider.list_sources(self.session, principal())
|
||||||
|
self.assertNotIn("text", listed[0].metadata["csv_source"])
|
||||||
|
self.assertEqual("legacy_typed", listed[0].metadata["csv_source"]["value_mode"])
|
||||||
|
record = self.session.get(ConnectorTabularSource, created.ref.split(":", 1)[1])
|
||||||
|
self.assertIn("csv_source_", inspect(record).unloaded)
|
||||||
|
with patch("govoplan_connectors.backend.router.audit_event") as audit:
|
||||||
|
response = api_original_tabular_csv(created.ref, session=self.session, principal=principal())
|
||||||
|
self.assertEqual("connectors.original_csv.exported", audit.call_args.kwargs["action"])
|
||||||
|
self.assertEqual({"sha256"}, set(audit.call_args.kwargs["details"]))
|
||||||
|
with patch("govoplan_connectors.backend.router.provider.original_csv") as read, self.assertRaises(HTTPException) as denied:
|
||||||
|
api_original_tabular_csv(created.ref, session=self.session, principal=principal(scopes=()))
|
||||||
|
self.assertEqual(403, denied.exception.status_code)
|
||||||
|
read.assert_not_called()
|
||||||
|
self.assertEqual(text.encode("utf-8"), response.body)
|
||||||
|
self.assertEqual("no-store", response.headers["cache-control"])
|
||||||
|
self.assertIn("attachment", response.headers["content-disposition"])
|
||||||
|
with self.assertRaises(TabularSourceNotFoundError):
|
||||||
|
self.provider.original_csv(self.session, principal("tenant-2"), source_ref=created.ref)
|
||||||
|
with self.assertRaises(TabularSourceAccessError):
|
||||||
|
self.provider.original_csv(self.session, principal(scopes=()), source_ref=created.ref)
|
||||||
|
record.csv_source_ = {**record.csv_source_, "text": "tampered"}
|
||||||
|
self.session.flush()
|
||||||
|
with self.assertRaises(TabularSourceUnavailableError):
|
||||||
|
self.provider.original_csv(self.session, principal(), source_ref=created.ref)
|
||||||
|
|
||||||
|
def test_text_snapshot_matches_exact_projection_and_rejects_inconsistent_evidence(self) -> None:
|
||||||
|
source = TabularCsvSource(text='value\n" "\n0.123456789012345678901234567890\n', value_mode="text")
|
||||||
|
snapshot = TabularSnapshotInput(name="Text", source_name="text", rows=parse_csv_snapshot(source.text, delimiter=",", value_mode="text"), csv_source=source)
|
||||||
|
with self.assertRaises(TabularSourceValidationError):
|
||||||
|
self.provider.create_snapshot(self.session, principal(), snapshot=replace(snapshot, rows=({"value": "changed"},)))
|
||||||
|
created = self.provider.create_snapshot(self.session, principal(), snapshot=snapshot)
|
||||||
|
self.assertEqual(2, created.row_count)
|
||||||
|
self.assertEqual(source.text, self.provider.original_csv(self.session, principal(), source_ref=created.ref))
|
||||||
|
legacy = self.provider.create_snapshot(self.session, principal(), snapshot=TabularSnapshotInput(name="Old", source_name="old", rows=({"id": 1},)))
|
||||||
|
with self.assertRaises(TabularSourceNotFoundError):
|
||||||
|
self.provider.original_csv(self.session, principal(), source_ref=legacy.ref)
|
||||||
|
|
||||||
|
def test_original_csv_projection_binding_rejects_equal_but_different_types(self) -> None:
|
||||||
|
for text, value in (("value\ntrue\n", 1), ("value\n1\n", True), ("value\n1\n", 1.0)):
|
||||||
|
with self.subTest(text=text, value=value), self.assertRaises(TabularSourceValidationError):
|
||||||
|
self.provider.create_snapshot(self.session, principal(), snapshot=TabularSnapshotInput(name="Invalid", source_name="invalid", rows=({"value": value},), csv_source=TabularCsvSource(text=text)))
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
if __name__ == "__main__":
|
||||||
unittest.main()
|
unittest.main()
|
||||||
|
|||||||
@@ -0,0 +1,90 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import re
|
||||||
|
import unittest
|
||||||
|
from dataclasses import replace
|
||||||
|
from io import BytesIO
|
||||||
|
from unittest.mock import patch
|
||||||
|
from zipfile import ZipFile
|
||||||
|
|
||||||
|
from openpyxl import Workbook
|
||||||
|
|
||||||
|
from govoplan_core.core.tabular_sources import TabularSourceValidationError
|
||||||
|
from govoplan_connectors.backend import tabular_adapters as adapters
|
||||||
|
|
||||||
|
|
||||||
|
def workbook_bytes(*, dimension: str | None = None, last_cell: str = "A2") -> bytes:
|
||||||
|
workbook = Workbook()
|
||||||
|
workbook.active.append(["name"])
|
||||||
|
workbook.active.append(["Ada"])
|
||||||
|
output = BytesIO()
|
||||||
|
workbook.save(output)
|
||||||
|
workbook.close()
|
||||||
|
result = BytesIO()
|
||||||
|
with ZipFile(BytesIO(output.getvalue())) as original, ZipFile(result, "w") as modified:
|
||||||
|
for entry in original.infolist():
|
||||||
|
content = original.read(entry)
|
||||||
|
if entry.filename == "xl/worksheets/sheet1.xml":
|
||||||
|
xml = content.decode()
|
||||||
|
if dimension is not None:
|
||||||
|
xml = re.sub(r'<dimension ref="[^"]+"\s*/>', f'<dimension ref="{dimension}"/>', xml)
|
||||||
|
xml = xml.replace('r="A2"', f'r="{last_cell}"')
|
||||||
|
xml = xml.replace('<row r="2">', f'<row r="{re.search(r"[0-9]+$", last_cell).group()}">')
|
||||||
|
content = xml.encode()
|
||||||
|
modified.writestr(entry, content)
|
||||||
|
return result.getvalue()
|
||||||
|
|
||||||
|
|
||||||
|
def parse(payload: bytes):
|
||||||
|
return adapters.parse_managed_tabular_content(payload, filename="fixture.xlsx", content_type=None, delimiter=",", sheet_name=None)
|
||||||
|
|
||||||
|
|
||||||
|
class XlsxSafetyBoundsTests(unittest.TestCase):
|
||||||
|
def test_sparse_rows_cannot_bypass_limit_by_not_counting_as_data(self):
|
||||||
|
with self.assertRaisesRegex(TabularSourceValidationError, "row"):
|
||||||
|
parse(workbook_bytes(last_cell=f"A{adapters.MAX_FILE_ROWS + 2}"))
|
||||||
|
|
||||||
|
def test_forged_small_dimensions_cannot_hide_far_away_cells(self):
|
||||||
|
with self.assertRaisesRegex(TabularSourceValidationError, "row"):
|
||||||
|
parse(workbook_bytes(dimension="A1:A2", last_cell="A1000000"))
|
||||||
|
|
||||||
|
def test_forged_small_dimensions_cannot_hide_out_of_range_columns(self):
|
||||||
|
with self.assertRaisesRegex(TabularSourceValidationError, "column"):
|
||||||
|
parse(workbook_bytes(dimension="A1:A2", last_cell="XFD2"))
|
||||||
|
|
||||||
|
def test_declared_dimensions_do_not_expand_or_truncate_real_rows(self):
|
||||||
|
for dimension in ("A1:XFD1048576", "A1:A1"):
|
||||||
|
with self.subTest(dimension=dimension):
|
||||||
|
rows, sheet = parse(workbook_bytes(dimension=dimension))
|
||||||
|
self.assertEqual(({"name": "Ada"},), rows)
|
||||||
|
self.assertEqual("Sheet", sheet)
|
||||||
|
|
||||||
|
def test_small_blank_gaps_and_exact_limit_remain_usable(self):
|
||||||
|
for cell in ("A4", f"A{adapters.MAX_FILE_ROWS + 1}"):
|
||||||
|
rows, _sheet = parse(workbook_bytes(last_cell=cell))
|
||||||
|
self.assertEqual(({"name": "Ada"},), rows)
|
||||||
|
|
||||||
|
def test_workbook_is_parsed_in_a_fresh_child_not_the_parent(self):
|
||||||
|
with patch.object(adapters, "_parse_xlsx_content", side_effect=AssertionError("parent parser ran")):
|
||||||
|
rows, sheet = parse(workbook_bytes())
|
||||||
|
self.assertEqual(({"name": "Ada"},), rows)
|
||||||
|
self.assertEqual("Sheet", sheet)
|
||||||
|
|
||||||
|
def test_real_worker_timeout_fails_without_parent_fallback(self):
|
||||||
|
limits = replace(adapters.XLSX_PROCESS_LIMITS, wall_seconds=0.001)
|
||||||
|
with patch.object(adapters, "XLSX_PROCESS_LIMITS", limits), patch.object(
|
||||||
|
adapters, "_parse_xlsx_content", side_effect=AssertionError("parent parser ran")
|
||||||
|
):
|
||||||
|
with self.assertRaisesRegex(TabularSourceValidationError, "timeout"):
|
||||||
|
parse(workbook_bytes())
|
||||||
|
|
||||||
|
def test_child_validation_keeps_missing_sheet_message(self):
|
||||||
|
with self.assertRaisesRegex(TabularSourceValidationError, "worksheet 'Missing' was not found"):
|
||||||
|
adapters.parse_managed_tabular_content(
|
||||||
|
workbook_bytes(), filename="fixture.xlsx", content_type=None,
|
||||||
|
delimiter=",", sheet_name="Missing",
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
unittest.main()
|
||||||
+1
-1
@@ -1,6 +1,6 @@
|
|||||||
{
|
{
|
||||||
"name": "@govoplan/connectors-webui",
|
"name": "@govoplan/connectors-webui",
|
||||||
"version": "0.1.20",
|
"version": "0.1.27",
|
||||||
"private": true,
|
"private": true,
|
||||||
"type": "module",
|
"type": "module",
|
||||||
"main": "src/index.ts",
|
"main": "src/index.ts",
|
||||||
|
|||||||
@@ -0,0 +1,212 @@
|
|||||||
|
import { apiFetch, apiPath, type ApiSettings } from "@govoplan/core-webui";
|
||||||
|
|
||||||
|
export type KnowledgeDiagnostic = {
|
||||||
|
severity: "info" | "warning" | "error";
|
||||||
|
code: string;
|
||||||
|
message: string;
|
||||||
|
object_ref?: string | null;
|
||||||
|
field?: string | null;
|
||||||
|
retryable: boolean;
|
||||||
|
details: Record<string, unknown>;
|
||||||
|
};
|
||||||
|
|
||||||
|
export type KnowledgeNamespaceMapping = {
|
||||||
|
source_namespace_id: number;
|
||||||
|
source_name: string;
|
||||||
|
target_space_ref: string;
|
||||||
|
target_path_prefix: string;
|
||||||
|
include: boolean;
|
||||||
|
visibility?: "tenant" | "restricted" | null;
|
||||||
|
acl_tokens: string[];
|
||||||
|
};
|
||||||
|
|
||||||
|
export type KnowledgeProfile = {
|
||||||
|
id: string;
|
||||||
|
tenant_id: string;
|
||||||
|
configuration_id: string;
|
||||||
|
status: "active" | "paused";
|
||||||
|
product: string;
|
||||||
|
product_version?: string | null;
|
||||||
|
desired_maturity: "discover" | "link" | "search" | "read" | "publish" | "synchronize" | "migrate";
|
||||||
|
discovered_maturity: string;
|
||||||
|
source_authority_mode: "external_authoritative" | "external_mirror" | "governed_sync" | "linked_reference";
|
||||||
|
default_visibility: "tenant" | "restricted";
|
||||||
|
default_acl_tokens: string[];
|
||||||
|
namespace_mappings: KnowledgeNamespaceMapping[];
|
||||||
|
capabilities: string[];
|
||||||
|
discovery_revision?: string | null;
|
||||||
|
health_status: string;
|
||||||
|
health_details: Record<string, unknown>;
|
||||||
|
discovered_at?: string | null;
|
||||||
|
last_sync_cursor?: string | null;
|
||||||
|
last_high_watermark?: string | null;
|
||||||
|
resource_revision: number;
|
||||||
|
credential_reference_present: boolean;
|
||||||
|
endpoint_configured: boolean;
|
||||||
|
created_at: string;
|
||||||
|
updated_at: string;
|
||||||
|
};
|
||||||
|
|
||||||
|
export type KnowledgeObject = {
|
||||||
|
id: string;
|
||||||
|
profile_id: string;
|
||||||
|
object_type: string;
|
||||||
|
external_id: string;
|
||||||
|
external_page_id?: string | null;
|
||||||
|
external_revision_id?: string | null;
|
||||||
|
namespace_id?: number | null;
|
||||||
|
title: string;
|
||||||
|
canonical_url?: string | null;
|
||||||
|
status: string;
|
||||||
|
source_revision: string;
|
||||||
|
visibility: string;
|
||||||
|
acl_tokens: string[];
|
||||||
|
mapped_data: Record<string, unknown>;
|
||||||
|
observed_at: string;
|
||||||
|
resource_revision: number;
|
||||||
|
};
|
||||||
|
|
||||||
|
export type KnowledgeRun = {
|
||||||
|
id: string;
|
||||||
|
profile_id: string;
|
||||||
|
mode: string;
|
||||||
|
idempotency_key: string;
|
||||||
|
status: string;
|
||||||
|
cursor_before?: string | null;
|
||||||
|
cursor_after?: string | null;
|
||||||
|
high_watermark?: string | null;
|
||||||
|
counts: Record<string, number>;
|
||||||
|
effects: Array<Record<string, unknown>>;
|
||||||
|
diagnostics: KnowledgeDiagnostic[];
|
||||||
|
provenance: Record<string, unknown>;
|
||||||
|
started_at: string;
|
||||||
|
finished_at?: string | null;
|
||||||
|
created_at: string;
|
||||||
|
};
|
||||||
|
|
||||||
|
export type KnowledgeDiscovery = {
|
||||||
|
profile: KnowledgeProfile;
|
||||||
|
product: string;
|
||||||
|
product_version?: string | null;
|
||||||
|
capabilities: string[];
|
||||||
|
namespaces: Array<Record<string, unknown>>;
|
||||||
|
extensions: Array<Record<string, unknown>>;
|
||||||
|
maturity: string;
|
||||||
|
health_status: string;
|
||||||
|
diagnostics: KnowledgeDiagnostic[];
|
||||||
|
revision: string;
|
||||||
|
};
|
||||||
|
|
||||||
|
export type KnowledgeMigrationPreview = {
|
||||||
|
run: KnowledgeRun;
|
||||||
|
target_space_ref: string;
|
||||||
|
source_revision: string;
|
||||||
|
source_fingerprint: string;
|
||||||
|
summary: Record<string, number>;
|
||||||
|
effects: Array<Record<string, unknown>>;
|
||||||
|
diagnostics: KnowledgeDiagnostic[];
|
||||||
|
truncated: boolean;
|
||||||
|
can_apply: boolean;
|
||||||
|
};
|
||||||
|
|
||||||
|
export type KnowledgePublishResult = {
|
||||||
|
run: KnowledgeRun;
|
||||||
|
external_reference: Record<string, unknown>;
|
||||||
|
accepted: boolean;
|
||||||
|
outcome_unknown: boolean;
|
||||||
|
};
|
||||||
|
|
||||||
|
const ROOT = "/api/v1/connectors/knowledge";
|
||||||
|
|
||||||
|
export async function listKnowledgeProfiles(settings: ApiSettings): Promise<KnowledgeProfile[]> {
|
||||||
|
const response = await apiFetch<{ items: KnowledgeProfile[] }>(settings, `${ROOT}/profiles`);
|
||||||
|
return response.items;
|
||||||
|
}
|
||||||
|
|
||||||
|
export function createKnowledgeProfile(
|
||||||
|
settings: ApiSettings,
|
||||||
|
payload: Record<string, unknown>
|
||||||
|
): Promise<KnowledgeProfile> {
|
||||||
|
return apiFetch(settings, `${ROOT}/profiles`, {
|
||||||
|
method: "POST",
|
||||||
|
body: JSON.stringify(payload)
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
export function updateKnowledgeProfile(
|
||||||
|
settings: ApiSettings,
|
||||||
|
profileId: string,
|
||||||
|
payload: Record<string, unknown>
|
||||||
|
): Promise<KnowledgeProfile> {
|
||||||
|
return apiFetch(settings, `${ROOT}/profiles/${encodeURIComponent(profileId)}`, {
|
||||||
|
method: "PUT",
|
||||||
|
body: JSON.stringify(payload)
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
export function discoverKnowledgeProfile(
|
||||||
|
settings: ApiSettings,
|
||||||
|
profileId: string
|
||||||
|
): Promise<KnowledgeDiscovery> {
|
||||||
|
return apiFetch(settings, `${ROOT}/profiles/${encodeURIComponent(profileId)}/discover`, {
|
||||||
|
method: "POST"
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
export function synchronizeKnowledgeProfile(
|
||||||
|
settings: ApiSettings,
|
||||||
|
profileId: string,
|
||||||
|
payload: Record<string, unknown>
|
||||||
|
): Promise<KnowledgeRun> {
|
||||||
|
return apiFetch(settings, `${ROOT}/profiles/${encodeURIComponent(profileId)}/sync`, {
|
||||||
|
method: "POST",
|
||||||
|
body: JSON.stringify(payload)
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
export async function listKnowledgeObjects(
|
||||||
|
settings: ApiSettings,
|
||||||
|
profileId: string
|
||||||
|
): Promise<KnowledgeObject[]> {
|
||||||
|
const response = await apiFetch<{ items: KnowledgeObject[] }>(
|
||||||
|
settings,
|
||||||
|
apiPath(`${ROOT}/profiles/${encodeURIComponent(profileId)}/objects`, { limit: 100 })
|
||||||
|
);
|
||||||
|
return response.items;
|
||||||
|
}
|
||||||
|
|
||||||
|
export async function listKnowledgeRuns(
|
||||||
|
settings: ApiSettings,
|
||||||
|
profileId?: string
|
||||||
|
): Promise<KnowledgeRun[]> {
|
||||||
|
const response = await apiFetch<{ items: KnowledgeRun[] }>(
|
||||||
|
settings,
|
||||||
|
apiPath(`${ROOT}/runs`, { profile_id: profileId || undefined, limit: 100 })
|
||||||
|
);
|
||||||
|
return response.items;
|
||||||
|
}
|
||||||
|
|
||||||
|
export function previewKnowledgeMigration(
|
||||||
|
settings: ApiSettings,
|
||||||
|
profileId: string,
|
||||||
|
payload: Record<string, unknown>
|
||||||
|
): Promise<KnowledgeMigrationPreview> {
|
||||||
|
return apiFetch(
|
||||||
|
settings,
|
||||||
|
`${ROOT}/profiles/${encodeURIComponent(profileId)}/migration-dry-runs`,
|
||||||
|
{ method: "POST", body: JSON.stringify(payload) }
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
export function publishKnowledgePage(
|
||||||
|
settings: ApiSettings,
|
||||||
|
profileId: string,
|
||||||
|
externalPageId: string,
|
||||||
|
payload: Record<string, unknown>
|
||||||
|
): Promise<KnowledgePublishResult> {
|
||||||
|
return apiFetch(
|
||||||
|
settings,
|
||||||
|
`${ROOT}/profiles/${encodeURIComponent(profileId)}/pages/${encodeURIComponent(externalPageId)}/publish`,
|
||||||
|
{ method: "POST", body: JSON.stringify(payload) }
|
||||||
|
);
|
||||||
|
}
|
||||||
@@ -0,0 +1,208 @@
|
|||||||
|
import { apiFetch, apiPath, type ApiSettings } from "@govoplan/core-webui";
|
||||||
|
|
||||||
|
export type ServiceDeskDiagnostic = {
|
||||||
|
severity: "info" | "warning" | "error";
|
||||||
|
code: string;
|
||||||
|
message: string;
|
||||||
|
object_ref?: string | null;
|
||||||
|
field?: string | null;
|
||||||
|
retryable: boolean;
|
||||||
|
details: Record<string, unknown>;
|
||||||
|
};
|
||||||
|
|
||||||
|
export type ServiceDeskRouteMapping = {
|
||||||
|
search_path: string;
|
||||||
|
ticket_path: string;
|
||||||
|
update_path?: string | null;
|
||||||
|
search_method: "GET" | "POST";
|
||||||
|
ticket_method: "GET" | "POST";
|
||||||
|
update_method: "PATCH" | "POST" | "PUT";
|
||||||
|
ticket_web_url_template?: string | null;
|
||||||
|
search_filters: Record<string, unknown>;
|
||||||
|
};
|
||||||
|
|
||||||
|
export type ServiceDeskQueueMapping = {
|
||||||
|
source_queue: string;
|
||||||
|
target_queue_ref?: string | null;
|
||||||
|
include: boolean;
|
||||||
|
visibility: "tenant" | "restricted";
|
||||||
|
acl_tokens: string[];
|
||||||
|
};
|
||||||
|
|
||||||
|
export type ServiceDeskDynamicFieldMapping = {
|
||||||
|
source_name: string;
|
||||||
|
target_name?: string | null;
|
||||||
|
include: boolean;
|
||||||
|
value_type: "string" | "number" | "boolean" | "date" | "json";
|
||||||
|
};
|
||||||
|
|
||||||
|
export type ServiceDeskProfile = {
|
||||||
|
id: string;
|
||||||
|
tenant_id: string;
|
||||||
|
configuration_id: string;
|
||||||
|
status: "active" | "paused";
|
||||||
|
integration_mode: "link" | "import" | "synchronize";
|
||||||
|
product: string;
|
||||||
|
product_version?: string | null;
|
||||||
|
desired_maturity: "discover" | "link" | "search" | "read" | "publish" | "synchronize";
|
||||||
|
discovered_maturity: string;
|
||||||
|
source_authority_mode: "external_authoritative" | "external_mirror" | "governed_sync" | "linked_reference";
|
||||||
|
default_visibility: "tenant" | "restricted";
|
||||||
|
default_acl_tokens: string[];
|
||||||
|
routes: ServiceDeskRouteMapping;
|
||||||
|
queue_mappings: ServiceDeskQueueMapping[];
|
||||||
|
dynamic_field_mappings: ServiceDeskDynamicFieldMapping[];
|
||||||
|
capabilities: string[];
|
||||||
|
discovery_revision?: string | null;
|
||||||
|
discovered_configuration_revision?: number | null;
|
||||||
|
discovered_configuration_hash?: string | null;
|
||||||
|
health_status: string;
|
||||||
|
health_details: Record<string, unknown>;
|
||||||
|
discovered_at?: string | null;
|
||||||
|
last_sync_cursor?: string | null;
|
||||||
|
last_high_watermark?: string | null;
|
||||||
|
resource_revision: number;
|
||||||
|
credential_reference_present: boolean;
|
||||||
|
endpoint_configured: boolean;
|
||||||
|
created_at: string;
|
||||||
|
updated_at: string;
|
||||||
|
};
|
||||||
|
|
||||||
|
export type ServiceDeskObject = {
|
||||||
|
id: string;
|
||||||
|
profile_id: string;
|
||||||
|
object_type: string;
|
||||||
|
external_id: string;
|
||||||
|
external_ticket_number?: string | null;
|
||||||
|
title: string;
|
||||||
|
canonical_url?: string | null;
|
||||||
|
status: string;
|
||||||
|
source_revision: string;
|
||||||
|
visibility: string;
|
||||||
|
acl_tokens: string[];
|
||||||
|
mapped_data: Record<string, unknown>;
|
||||||
|
provenance: Record<string, unknown>;
|
||||||
|
source_updated_at?: string | null;
|
||||||
|
observed_at: string;
|
||||||
|
resource_revision: number;
|
||||||
|
};
|
||||||
|
|
||||||
|
export type ServiceDeskRun = {
|
||||||
|
id: string;
|
||||||
|
profile_id: string;
|
||||||
|
mode: string;
|
||||||
|
idempotency_key: string;
|
||||||
|
status: string;
|
||||||
|
cursor_before?: string | null;
|
||||||
|
cursor_after?: string | null;
|
||||||
|
high_watermark?: string | null;
|
||||||
|
counts: Record<string, number>;
|
||||||
|
effects: Array<Record<string, unknown>>;
|
||||||
|
diagnostics: ServiceDeskDiagnostic[];
|
||||||
|
provenance: Record<string, unknown>;
|
||||||
|
started_at: string;
|
||||||
|
finished_at?: string | null;
|
||||||
|
created_at: string;
|
||||||
|
};
|
||||||
|
|
||||||
|
export type ServiceDeskDiscovery = {
|
||||||
|
profile: ServiceDeskProfile;
|
||||||
|
product: string;
|
||||||
|
product_version?: string | null;
|
||||||
|
api_family: string;
|
||||||
|
capabilities: string[];
|
||||||
|
maturity: string;
|
||||||
|
health_status: string;
|
||||||
|
diagnostics: ServiceDeskDiagnostic[];
|
||||||
|
revision: string;
|
||||||
|
};
|
||||||
|
|
||||||
|
export type ServiceDeskTicketUpdateResult = {
|
||||||
|
run: ServiceDeskRun;
|
||||||
|
object: ServiceDeskObject;
|
||||||
|
accepted: boolean;
|
||||||
|
outcome_unknown: boolean;
|
||||||
|
};
|
||||||
|
|
||||||
|
const ROOT = "/api/v1/connectors/service-desk";
|
||||||
|
|
||||||
|
export async function listServiceDeskProfiles(settings: ApiSettings): Promise<ServiceDeskProfile[]> {
|
||||||
|
const response = await apiFetch<{ items: ServiceDeskProfile[] }>(settings, `${ROOT}/profiles`);
|
||||||
|
return response.items;
|
||||||
|
}
|
||||||
|
|
||||||
|
export function createServiceDeskProfile(
|
||||||
|
settings: ApiSettings,
|
||||||
|
payload: Record<string, unknown>
|
||||||
|
): Promise<ServiceDeskProfile> {
|
||||||
|
return apiFetch(settings, `${ROOT}/profiles`, {
|
||||||
|
method: "POST",
|
||||||
|
body: JSON.stringify(payload)
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
export function updateServiceDeskProfile(
|
||||||
|
settings: ApiSettings,
|
||||||
|
profileId: string,
|
||||||
|
payload: Record<string, unknown>
|
||||||
|
): Promise<ServiceDeskProfile> {
|
||||||
|
return apiFetch(settings, `${ROOT}/profiles/${encodeURIComponent(profileId)}`, {
|
||||||
|
method: "PUT",
|
||||||
|
body: JSON.stringify(payload)
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
export function discoverServiceDeskProfile(
|
||||||
|
settings: ApiSettings,
|
||||||
|
profileId: string
|
||||||
|
): Promise<ServiceDeskDiscovery> {
|
||||||
|
return apiFetch(settings, `${ROOT}/profiles/${encodeURIComponent(profileId)}/discover`, {
|
||||||
|
method: "POST"
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
export function synchronizeServiceDeskProfile(
|
||||||
|
settings: ApiSettings,
|
||||||
|
profileId: string,
|
||||||
|
payload: Record<string, unknown>
|
||||||
|
): Promise<ServiceDeskRun> {
|
||||||
|
return apiFetch(settings, `${ROOT}/profiles/${encodeURIComponent(profileId)}/sync`, {
|
||||||
|
method: "POST",
|
||||||
|
body: JSON.stringify(payload)
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
export async function listServiceDeskObjects(
|
||||||
|
settings: ApiSettings,
|
||||||
|
profileId: string
|
||||||
|
): Promise<ServiceDeskObject[]> {
|
||||||
|
const response = await apiFetch<{ items: ServiceDeskObject[] }>(
|
||||||
|
settings,
|
||||||
|
apiPath(`${ROOT}/profiles/${encodeURIComponent(profileId)}/objects`, { limit: 100 })
|
||||||
|
);
|
||||||
|
return response.items;
|
||||||
|
}
|
||||||
|
|
||||||
|
export async function listServiceDeskRuns(
|
||||||
|
settings: ApiSettings,
|
||||||
|
profileId?: string
|
||||||
|
): Promise<ServiceDeskRun[]> {
|
||||||
|
const response = await apiFetch<{ items: ServiceDeskRun[] }>(
|
||||||
|
settings,
|
||||||
|
apiPath(`${ROOT}/runs`, { profile_id: profileId || undefined, limit: 100 })
|
||||||
|
);
|
||||||
|
return response.items;
|
||||||
|
}
|
||||||
|
|
||||||
|
export function updateServiceDeskTicket(
|
||||||
|
settings: ApiSettings,
|
||||||
|
profileId: string,
|
||||||
|
externalTicketId: string,
|
||||||
|
payload: Record<string, unknown>
|
||||||
|
): Promise<ServiceDeskTicketUpdateResult> {
|
||||||
|
return apiFetch(
|
||||||
|
settings,
|
||||||
|
`${ROOT}/profiles/${encodeURIComponent(profileId)}/tickets/${encodeURIComponent(externalTicketId)}/update`,
|
||||||
|
{ method: "POST", body: JSON.stringify(payload) }
|
||||||
|
);
|
||||||
|
}
|
||||||
@@ -483,7 +483,7 @@ export default function ConnectorGovernancePage({ settings, auth }: Props) {
|
|||||||
<FormField label="Endpoint URL" hint="Credentials are rejected in URLs.">
|
<FormField label="Endpoint URL" hint="Credentials are rejected in URLs.">
|
||||||
<input value={draft.endpoint_url} disabled={!canAdmin || busy} placeholder="https://provider.example/api" onChange={(event) => setDraft({ ...draft, endpoint_url: event.target.value })} />
|
<input value={draft.endpoint_url} disabled={!canAdmin || busy} placeholder="https://provider.example/api" onChange={(event) => setDraft({ ...draft, endpoint_url: event.target.value })} />
|
||||||
</FormField>
|
</FormField>
|
||||||
<FormField label="Credential reference" hint="Reference an approved secret; do not paste a secret.">
|
<FormField label="Credential reference" hint="Reference an approved secret; do not paste a secret." helpContextId="connectors.admin.governed-configurations" helpModuleId="connectors">
|
||||||
<input value={draft.credential_ref} disabled={!canAdmin || busy} placeholder="vault://connectors/provider" onChange={(event) => setDraft({ ...draft, credential_ref: event.target.value })} />
|
<input value={draft.credential_ref} disabled={!canAdmin || busy} placeholder="vault://connectors/provider" onChange={(event) => setDraft({ ...draft, credential_ref: event.target.value })} />
|
||||||
</FormField>
|
</FormField>
|
||||||
<FormField label="Ambiguous-result policy">
|
<FormField label="Ambiguous-result policy">
|
||||||
@@ -574,7 +574,7 @@ export default function ConnectorGovernancePage({ settings, auth }: Props) {
|
|||||||
<FormField label="Endpoint URL">
|
<FormField label="Endpoint URL">
|
||||||
<input value={newDraft.endpoint_url} disabled={busy} onChange={(event) => setNewDraft({ ...newDraft, endpoint_url: event.target.value })} />
|
<input value={newDraft.endpoint_url} disabled={busy} onChange={(event) => setNewDraft({ ...newDraft, endpoint_url: event.target.value })} />
|
||||||
</FormField>
|
</FormField>
|
||||||
<FormField label="Credential reference">
|
<FormField label="Credential reference" helpContextId="connectors.admin.governed-configurations" helpModuleId="connectors">
|
||||||
<input value={newDraft.credential_ref} disabled={busy} onChange={(event) => setNewDraft({ ...newDraft, credential_ref: event.target.value })} />
|
<input value={newDraft.credential_ref} disabled={busy} onChange={(event) => setNewDraft({ ...newDraft, credential_ref: event.target.value })} />
|
||||||
</FormField>
|
</FormField>
|
||||||
<FormField label="Ambiguous-result policy">
|
<FormField label="Ambiguous-result policy">
|
||||||
@@ -597,8 +597,8 @@ export default function ConnectorGovernancePage({ settings, auth }: Props) {
|
|||||||
closeDisabled={busy}
|
closeDisabled={busy}
|
||||||
footer={<>
|
footer={<>
|
||||||
<Button onClick={() => setReviewRun(null)} disabled={busy}>Cancel</Button>
|
<Button onClick={() => setReviewRun(null)} disabled={busy}>Cancel</Button>
|
||||||
<Button variant="danger" onClick={() => void decideReview("rejected")} disabled={busy || reviewReason.trim().length < 5}>Reject</Button>
|
<Button variant="danger" helpContextId="connectors.action.reject-ambiguous-result" helpModuleId="connectors" onClick={() => void decideReview("rejected")} disabled={busy || reviewReason.trim().length < 5}>Reject</Button>
|
||||||
<Button variant="primary" onClick={() => void decideReview("approved")} disabled={busy || reviewReason.trim().length < 5}>Approve</Button>
|
<Button variant="primary" helpContextId="connectors.action.approve-ambiguous-result" helpModuleId="connectors" onClick={() => void decideReview("approved")} disabled={busy || reviewReason.trim().length < 5}>Approve</Button>
|
||||||
</>}
|
</>}
|
||||||
>
|
>
|
||||||
<p>Review {reviewRun?.summary.ambiguous ?? 0} ambiguous effects against the retained input and configuration hashes before deciding.</p>
|
<p>Review {reviewRun?.summary.ambiguous ?? 0} ambiguous effects against the retained input and configuration hashes before deciding.</p>
|
||||||
|
|||||||
@@ -0,0 +1,561 @@
|
|||||||
|
import { useCallback, useEffect, useMemo, useState } from "react";
|
||||||
|
import {
|
||||||
|
AdminPageLayout,
|
||||||
|
Button,
|
||||||
|
Card,
|
||||||
|
Dialog,
|
||||||
|
FilterBar,
|
||||||
|
FormField,
|
||||||
|
FormGrid,
|
||||||
|
MetricCard,
|
||||||
|
MetricGrid,
|
||||||
|
PageActionBar,
|
||||||
|
SelectionList,
|
||||||
|
SelectionListItem,
|
||||||
|
SelectionListItemContent,
|
||||||
|
StatePanel,
|
||||||
|
StatusBadge,
|
||||||
|
WorkspaceLayout,
|
||||||
|
formatDateTime,
|
||||||
|
hasScope,
|
||||||
|
useUnsavedChanges,
|
||||||
|
useUnsavedDraftGuard,
|
||||||
|
type ApiSettings,
|
||||||
|
type AuthInfo
|
||||||
|
} from "@govoplan/core-webui";
|
||||||
|
import {
|
||||||
|
createKnowledgeProfile,
|
||||||
|
discoverKnowledgeProfile,
|
||||||
|
listKnowledgeObjects,
|
||||||
|
listKnowledgeProfiles,
|
||||||
|
listKnowledgeRuns,
|
||||||
|
previewKnowledgeMigration,
|
||||||
|
publishKnowledgePage,
|
||||||
|
synchronizeKnowledgeProfile,
|
||||||
|
updateKnowledgeProfile,
|
||||||
|
type KnowledgeMigrationPreview,
|
||||||
|
type KnowledgeObject,
|
||||||
|
type KnowledgeProfile,
|
||||||
|
type KnowledgeRun
|
||||||
|
} from "../api/externalKnowledge";
|
||||||
|
|
||||||
|
type Props = {
|
||||||
|
settings: ApiSettings;
|
||||||
|
auth: AuthInfo;
|
||||||
|
};
|
||||||
|
|
||||||
|
type ProfileDraft = {
|
||||||
|
status: "active" | "paused";
|
||||||
|
desired_maturity: KnowledgeProfile["desired_maturity"];
|
||||||
|
source_authority_mode: KnowledgeProfile["source_authority_mode"];
|
||||||
|
default_visibility: KnowledgeProfile["default_visibility"];
|
||||||
|
default_acl_tokens: string;
|
||||||
|
namespace_mappings: string;
|
||||||
|
};
|
||||||
|
|
||||||
|
const EMPTY_DRAFT: ProfileDraft = {
|
||||||
|
status: "active",
|
||||||
|
desired_maturity: "migrate",
|
||||||
|
source_authority_mode: "external_mirror",
|
||||||
|
default_visibility: "restricted",
|
||||||
|
default_acl_tokens: "scope:connectors:knowledge:read",
|
||||||
|
namespace_mappings: JSON.stringify([{
|
||||||
|
source_namespace_id: 0,
|
||||||
|
source_name: "",
|
||||||
|
target_space_ref: "external-knowledge",
|
||||||
|
target_path_prefix: "",
|
||||||
|
include: true,
|
||||||
|
acl_tokens: []
|
||||||
|
}], null, 2)
|
||||||
|
};
|
||||||
|
|
||||||
|
export default function ExternalKnowledgePage({ settings, auth }: Props) {
|
||||||
|
const [profiles, setProfiles] = useState<KnowledgeProfile[]>([]);
|
||||||
|
const [objects, setObjects] = useState<KnowledgeObject[]>([]);
|
||||||
|
const [runs, setRuns] = useState<KnowledgeRun[]>([]);
|
||||||
|
const [selectedId, setSelectedId] = useState("");
|
||||||
|
const [draft, setDraft] = useState<ProfileDraft>(EMPTY_DRAFT);
|
||||||
|
const [savedKey, setSavedKey] = useState("");
|
||||||
|
const [search, setSearch] = useState("");
|
||||||
|
const [loading, setLoading] = useState(true);
|
||||||
|
const [busy, setBusy] = useState(false);
|
||||||
|
const [error, setError] = useState("");
|
||||||
|
const [success, setSuccess] = useState("");
|
||||||
|
const [createOpen, setCreateOpen] = useState(false);
|
||||||
|
const [configurationId, setConfigurationId] = useState("");
|
||||||
|
const [newDraft, setNewDraft] = useState<ProfileDraft>(EMPTY_DRAFT);
|
||||||
|
const [migrationOpen, setMigrationOpen] = useState(false);
|
||||||
|
const [targetSpace, setTargetSpace] = useState("external-knowledge");
|
||||||
|
const [supportedMacros, setSupportedMacros] = useState("");
|
||||||
|
const [existingTargets, setExistingTargets] = useState("[]");
|
||||||
|
const [migration, setMigration] = useState<KnowledgeMigrationPreview | null>(null);
|
||||||
|
const [publishOpen, setPublishOpen] = useState(false);
|
||||||
|
const [externalPageId, setExternalPageId] = useState("");
|
||||||
|
const [publishTitle, setPublishTitle] = useState("");
|
||||||
|
const [publishBody, setPublishBody] = useState("");
|
||||||
|
const [publishSummary, setPublishSummary] = useState("");
|
||||||
|
const [publishRevision, setPublishRevision] = useState("");
|
||||||
|
const { requestDiscard } = useUnsavedChanges();
|
||||||
|
|
||||||
|
const selected = profiles.find((item) => item.id === selectedId) ?? null;
|
||||||
|
const canAdmin = hasScope(auth, "connectors:knowledge:admin");
|
||||||
|
const canSync = hasScope(auth, "connectors:knowledge:sync");
|
||||||
|
const canMigrate = hasScope(auth, "connectors:knowledge:migrate");
|
||||||
|
const canPublish = hasScope(auth, "connectors:knowledge:publish");
|
||||||
|
const dirty = Boolean(selected && draftKey(draft) !== savedKey);
|
||||||
|
|
||||||
|
const applyProfile = useCallback((profile: KnowledgeProfile | null) => {
|
||||||
|
const next = profile ? draftFromProfile(profile) : EMPTY_DRAFT;
|
||||||
|
setDraft(next);
|
||||||
|
setSavedKey(profile ? draftKey(next) : "");
|
||||||
|
}, []);
|
||||||
|
|
||||||
|
const reload = useCallback(async (preferredId?: string) => {
|
||||||
|
setLoading(true);
|
||||||
|
setError("");
|
||||||
|
try {
|
||||||
|
const nextProfiles = await listKnowledgeProfiles(settings);
|
||||||
|
const nextId = preferredId && nextProfiles.some((item) => item.id === preferredId)
|
||||||
|
? preferredId
|
||||||
|
: nextProfiles.some((item) => item.id === selectedId)
|
||||||
|
? selectedId
|
||||||
|
: nextProfiles[0]?.id ?? "";
|
||||||
|
const [nextObjects, nextRuns] = nextId
|
||||||
|
? await Promise.all([
|
||||||
|
listKnowledgeObjects(settings, nextId),
|
||||||
|
listKnowledgeRuns(settings, nextId)
|
||||||
|
])
|
||||||
|
: [[], []];
|
||||||
|
setProfiles(nextProfiles);
|
||||||
|
setSelectedId(nextId);
|
||||||
|
setObjects(nextObjects);
|
||||||
|
setRuns(nextRuns);
|
||||||
|
applyProfile(nextProfiles.find((item) => item.id === nextId) ?? null);
|
||||||
|
} catch (caught) {
|
||||||
|
setError(errorMessage(caught));
|
||||||
|
} finally {
|
||||||
|
setLoading(false);
|
||||||
|
}
|
||||||
|
}, [applyProfile, selectedId, settings]);
|
||||||
|
|
||||||
|
useEffect(() => {
|
||||||
|
void reload();
|
||||||
|
}, [settings.apiBaseUrl, settings.apiKey, settings.accessToken]);
|
||||||
|
|
||||||
|
const save = async (): Promise<boolean> => {
|
||||||
|
if (!selected || !canAdmin) return false;
|
||||||
|
setBusy(true);
|
||||||
|
setError("");
|
||||||
|
try {
|
||||||
|
const updated = await updateKnowledgeProfile(settings, selected.id, {
|
||||||
|
expected_resource_revision: selected.resource_revision,
|
||||||
|
status: draft.status,
|
||||||
|
desired_maturity: draft.desired_maturity,
|
||||||
|
source_authority_mode: draft.source_authority_mode,
|
||||||
|
default_visibility: draft.default_visibility,
|
||||||
|
default_acl_tokens: lines(draft.default_acl_tokens),
|
||||||
|
namespace_mappings: parseArray(draft.namespace_mappings, "Namespace mappings")
|
||||||
|
});
|
||||||
|
setSuccess("External knowledge profile saved; fallback ACLs and Search projections were refreshed.");
|
||||||
|
await reload(updated.id);
|
||||||
|
return true;
|
||||||
|
} catch (caught) {
|
||||||
|
setError(errorMessage(caught));
|
||||||
|
return false;
|
||||||
|
} finally {
|
||||||
|
setBusy(false);
|
||||||
|
}
|
||||||
|
};
|
||||||
|
|
||||||
|
useUnsavedDraftGuard({
|
||||||
|
dirty,
|
||||||
|
onSave: save,
|
||||||
|
onDiscard: () => applyProfile(selected),
|
||||||
|
title: "Unsaved knowledge profile changes",
|
||||||
|
message: "Save or discard the profile changes before continuing."
|
||||||
|
});
|
||||||
|
|
||||||
|
const selectProfile = (profile: KnowledgeProfile) => {
|
||||||
|
if (profile.id === selectedId) return;
|
||||||
|
requestDiscard(() => {
|
||||||
|
setSelectedId(profile.id);
|
||||||
|
applyProfile(profile);
|
||||||
|
setObjects([]);
|
||||||
|
setRuns([]);
|
||||||
|
setMigration(null);
|
||||||
|
void Promise.all([
|
||||||
|
listKnowledgeObjects(settings, profile.id),
|
||||||
|
listKnowledgeRuns(settings, profile.id)
|
||||||
|
]).then(([nextObjects, nextRuns]) => {
|
||||||
|
setObjects(nextObjects);
|
||||||
|
setRuns(nextRuns);
|
||||||
|
}).catch((caught) => setError(errorMessage(caught)));
|
||||||
|
});
|
||||||
|
};
|
||||||
|
|
||||||
|
const createProfile = async () => {
|
||||||
|
if (!configurationId.trim()) return;
|
||||||
|
setBusy(true);
|
||||||
|
setError("");
|
||||||
|
try {
|
||||||
|
const created = await createKnowledgeProfile(settings, {
|
||||||
|
configuration_id: configurationId.trim(),
|
||||||
|
desired_maturity: newDraft.desired_maturity,
|
||||||
|
source_authority_mode: newDraft.source_authority_mode,
|
||||||
|
default_visibility: newDraft.default_visibility,
|
||||||
|
default_acl_tokens: lines(newDraft.default_acl_tokens),
|
||||||
|
namespace_mappings: parseArray(newDraft.namespace_mappings, "Namespace mappings")
|
||||||
|
});
|
||||||
|
setCreateOpen(false);
|
||||||
|
setConfigurationId("");
|
||||||
|
setNewDraft(EMPTY_DRAFT);
|
||||||
|
setSuccess("External knowledge profile created. Run discovery before synchronization.");
|
||||||
|
await reload(created.id);
|
||||||
|
} catch (caught) {
|
||||||
|
setError(errorMessage(caught));
|
||||||
|
} finally {
|
||||||
|
setBusy(false);
|
||||||
|
}
|
||||||
|
};
|
||||||
|
|
||||||
|
const discover = async () => {
|
||||||
|
if (!selected || dirty) return;
|
||||||
|
setBusy(true);
|
||||||
|
setError("");
|
||||||
|
try {
|
||||||
|
const result = await discoverKnowledgeProfile(settings, selected.id);
|
||||||
|
setSuccess(`Discovered ${result.product} ${result.product_version ?? ""} at ${result.maturity} maturity with ${result.diagnostics.length} diagnostics.`);
|
||||||
|
await reload(selected.id);
|
||||||
|
} catch (caught) {
|
||||||
|
setError(errorMessage(caught));
|
||||||
|
} finally {
|
||||||
|
setBusy(false);
|
||||||
|
}
|
||||||
|
};
|
||||||
|
|
||||||
|
const sync = async (forceFull: boolean) => {
|
||||||
|
if (!selected || dirty) return;
|
||||||
|
setBusy(true);
|
||||||
|
setError("");
|
||||||
|
try {
|
||||||
|
const run = await synchronizeKnowledgeProfile(settings, selected.id, {
|
||||||
|
idempotency_key: `knowledge-${forceFull ? "backfill" : "delta"}-${crypto.randomUUID()}`,
|
||||||
|
force_full: forceFull,
|
||||||
|
limit: 100
|
||||||
|
});
|
||||||
|
setSuccess(`${forceFull ? "Backfill" : "Delta"} completed with ${effectTotal(run)} effects.`);
|
||||||
|
await reload(selected.id);
|
||||||
|
} catch (caught) {
|
||||||
|
setError(errorMessage(caught));
|
||||||
|
} finally {
|
||||||
|
setBusy(false);
|
||||||
|
}
|
||||||
|
};
|
||||||
|
|
||||||
|
const previewMigration = async () => {
|
||||||
|
if (!selected || !targetSpace.trim()) return;
|
||||||
|
setBusy(true);
|
||||||
|
setError("");
|
||||||
|
try {
|
||||||
|
const result = await previewKnowledgeMigration(settings, selected.id, {
|
||||||
|
idempotency_key: `knowledge-migration-${crypto.randomUUID()}`,
|
||||||
|
target_space_ref: targetSpace.trim(),
|
||||||
|
max_items: 100,
|
||||||
|
supported_macros: lines(supportedMacros),
|
||||||
|
existing_targets: parseArray(existingTargets, "Existing targets")
|
||||||
|
});
|
||||||
|
setMigration(result);
|
||||||
|
setMigrationOpen(false);
|
||||||
|
setSuccess(result.can_apply
|
||||||
|
? "Migration preview is complete and contains no blocking conflict. It did not write Wiki pages."
|
||||||
|
: "Migration preview found conflicts, errors, or truncation. It did not write Wiki pages.");
|
||||||
|
await reload(selected.id);
|
||||||
|
} catch (caught) {
|
||||||
|
setError(errorMessage(caught));
|
||||||
|
} finally {
|
||||||
|
setBusy(false);
|
||||||
|
}
|
||||||
|
};
|
||||||
|
|
||||||
|
const publish = async () => {
|
||||||
|
if (!selected || !externalPageId.trim() || !publishTitle.trim()) return;
|
||||||
|
setBusy(true);
|
||||||
|
setError("");
|
||||||
|
try {
|
||||||
|
const result = await publishKnowledgePage(
|
||||||
|
settings,
|
||||||
|
selected.id,
|
||||||
|
externalPageId.trim(),
|
||||||
|
{
|
||||||
|
idempotency_key: `knowledge-publish-${crypto.randomUUID()}`,
|
||||||
|
title: publishTitle.trim(),
|
||||||
|
body: publishBody,
|
||||||
|
summary: publishSummary.trim(),
|
||||||
|
expected_external_revision: publishRevision.trim() || null,
|
||||||
|
minor: false
|
||||||
|
}
|
||||||
|
);
|
||||||
|
setPublishOpen(false);
|
||||||
|
setSuccess(result.outcome_unknown
|
||||||
|
? "Publication outcome is unknown. Reconcile the provider revision before retrying."
|
||||||
|
: "Provider accepted the page revision and durable recovery evidence was recorded.");
|
||||||
|
await reload(selected.id);
|
||||||
|
} catch (caught) {
|
||||||
|
setError(errorMessage(caught));
|
||||||
|
} finally {
|
||||||
|
setBusy(false);
|
||||||
|
}
|
||||||
|
};
|
||||||
|
|
||||||
|
const visibleProfiles = useMemo(() => {
|
||||||
|
const needle = search.trim().toLocaleLowerCase();
|
||||||
|
return profiles.filter((item) => !needle ||
|
||||||
|
`${item.product} ${item.product_version ?? ""} ${item.health_status} ${item.configuration_id}`
|
||||||
|
.toLocaleLowerCase()
|
||||||
|
.includes(needle));
|
||||||
|
}, [profiles, search]);
|
||||||
|
|
||||||
|
const actionBar = <PageActionBar
|
||||||
|
variant="editor"
|
||||||
|
state={busy ? "saving" : dirty ? "dirty" : "clean"}
|
||||||
|
refreshable
|
||||||
|
reloadAction={{ onReload: () => void reload(selectedId), loading }}
|
||||||
|
primaryActions={<>
|
||||||
|
<Button onClick={() => setCreateOpen(true)} disabled={!canAdmin || busy}>New profile</Button>
|
||||||
|
<Button variant="secondary" onClick={() => void discover()} disabled={!selected || !canAdmin || busy || dirty}>Discover</Button>
|
||||||
|
<Button variant="secondary" onClick={() => void sync(false)} disabled={!selected || !canSync || busy || dirty}>Run delta</Button>
|
||||||
|
<Button variant="secondary" onClick={() => void sync(true)} disabled={!selected || !canSync || busy || dirty}>Run full backfill</Button>
|
||||||
|
<Button variant="secondary" onClick={() => setMigrationOpen(true)} disabled={!selected || !canMigrate || busy || dirty}>Preview migration</Button>
|
||||||
|
<Button variant="primary" helpContextId="connectors.admin.external-knowledge" helpModuleId="connectors" onClick={() => setPublishOpen(true)} disabled={!selected || !canPublish || busy || dirty}>Publish page</Button>
|
||||||
|
</>}
|
||||||
|
discardAction={{
|
||||||
|
label: "Discard changes",
|
||||||
|
disabled: !selected,
|
||||||
|
onClick: () => applyProfile(selected)
|
||||||
|
}}
|
||||||
|
saveAction={{
|
||||||
|
label: "Save",
|
||||||
|
disabled: !selected || !canAdmin || busy,
|
||||||
|
disabledReason: !canAdmin ? "External knowledge administration permission is required." : undefined,
|
||||||
|
onClick: () => void save()
|
||||||
|
}}
|
||||||
|
/>;
|
||||||
|
|
||||||
|
return <AdminPageLayout
|
||||||
|
archetype="workspace"
|
||||||
|
title="External knowledge"
|
||||||
|
description="Discover and synchronize MediaWiki or BlueSpice, preserve stable identity and permissions, and preview migration into native Wiki."
|
||||||
|
loading={loading && !profiles.length}
|
||||||
|
error={error}
|
||||||
|
success={success}
|
||||||
|
actions={actionBar}
|
||||||
|
className="connector-knowledge-page"
|
||||||
|
helpContextId="connectors.admin.external-knowledge"
|
||||||
|
>
|
||||||
|
<MetricGrid columns={4} density="compact" minimum="compact">
|
||||||
|
<MetricCard label="Profiles" value={profiles.length} />
|
||||||
|
<MetricCard label="Active pages" value={objects.filter((item) => item.status !== "deleted").length} />
|
||||||
|
<MetricCard label="Unhealthy profiles" value={profiles.filter((item) => !["healthy", "unknown"].includes(item.health_status)).length} tone="warning" />
|
||||||
|
<MetricCard label="Unresolved runs" value={runs.filter((item) => ["failed", "outcome_unknown"].includes(item.status)).length} tone="warning" />
|
||||||
|
</MetricGrid>
|
||||||
|
|
||||||
|
<WorkspaceLayout
|
||||||
|
variant="split"
|
||||||
|
primarySize="compact"
|
||||||
|
surface="contained"
|
||||||
|
primaryScrollable={false}
|
||||||
|
contentScrollable={false}
|
||||||
|
primaryLabel="Knowledge profiles"
|
||||||
|
contentLabel="Profile details"
|
||||||
|
primary={<div className="connector-knowledge-list">
|
||||||
|
<FilterBar surface="panel">
|
||||||
|
<input type="search" value={search} onChange={(event) => setSearch(event.target.value)} placeholder="Search profiles" aria-label="Search external knowledge profiles" />
|
||||||
|
</FilterBar>
|
||||||
|
<SelectionList variant="navigation" label="External knowledge profiles">
|
||||||
|
{visibleProfiles.map((profile) => <SelectionListItem key={profile.id} selected={profile.id === selectedId} onClick={() => selectProfile(profile)}>
|
||||||
|
<SelectionListItemContent
|
||||||
|
title={`${profile.product}${profile.product_version ? ` ${profile.product_version}` : ""}`}
|
||||||
|
description={`${profile.discovered_maturity} · ${profile.configuration_id}`}
|
||||||
|
/>
|
||||||
|
<StatusBadge status={profile.status === "paused" ? "inactive" : profile.health_status} />
|
||||||
|
</SelectionListItem>)}
|
||||||
|
{!visibleProfiles.length ? <StatePanel size="compact" description="No matching knowledge profiles." /> : null}
|
||||||
|
</SelectionList>
|
||||||
|
</div>}
|
||||||
|
>
|
||||||
|
{!selected ? <StatePanel size="fill" title="External knowledge profiles" description="Create or select a profile to discover provider capabilities and inspect synchronization evidence." /> : <div className="connector-knowledge-detail">
|
||||||
|
<Card title={`${selected.product}${selected.product_version ? ` ${selected.product_version}` : ""}`}>
|
||||||
|
<div className="connector-revision-line">
|
||||||
|
<StatusBadge status={selected.status} />
|
||||||
|
<StatusBadge status={selected.health_status} />
|
||||||
|
<span>Discovered maturity: {selected.discovered_maturity}</span>
|
||||||
|
<code title={selected.discovery_revision ?? undefined}>r{selected.resource_revision}</code>
|
||||||
|
</div>
|
||||||
|
<FormGrid columns={2} collapseAt="standard" className="">
|
||||||
|
<FormField label="Status" hint="Pausing immediately makes Search authorization fail closed.">
|
||||||
|
<select value={draft.status} disabled={!canAdmin || busy} onChange={(event) => setDraft({ ...draft, status: event.target.value as ProfileDraft["status"] })}>
|
||||||
|
<option value="active">Active</option>
|
||||||
|
<option value="paused">Paused</option>
|
||||||
|
</select>
|
||||||
|
</FormField>
|
||||||
|
<FormField label="Desired maturity" hint="This is the operator ceiling even when the provider offers more.">
|
||||||
|
<select value={draft.desired_maturity} disabled={!canAdmin || busy} onChange={(event) => setDraft({ ...draft, desired_maturity: event.target.value as ProfileDraft["desired_maturity"] })}>
|
||||||
|
{['discover', 'link', 'search', 'read', 'publish', 'synchronize', 'migrate'].map((value) => <option key={value} value={value}>{value}</option>)}
|
||||||
|
</select>
|
||||||
|
</FormField>
|
||||||
|
<FormField label="Source authority">
|
||||||
|
<select value={draft.source_authority_mode} disabled={!canAdmin || busy} onChange={(event) => setDraft({ ...draft, source_authority_mode: event.target.value as ProfileDraft["source_authority_mode"] })}>
|
||||||
|
<option value="external_authoritative">External authoritative</option>
|
||||||
|
<option value="external_mirror">External mirror</option>
|
||||||
|
<option value="governed_sync">Governed sync</option>
|
||||||
|
<option value="linked_reference">Linked reference</option>
|
||||||
|
</select>
|
||||||
|
</FormField>
|
||||||
|
<FormField label="Fallback visibility">
|
||||||
|
<select value={draft.default_visibility} disabled={!canAdmin || busy} onChange={(event) => setDraft({ ...draft, default_visibility: event.target.value as ProfileDraft["default_visibility"] })}>
|
||||||
|
<option value="restricted">Restricted</option>
|
||||||
|
<option value="tenant">Tenant</option>
|
||||||
|
</select>
|
||||||
|
</FormField>
|
||||||
|
<FormField label="Fallback ACL tokens" hint="One account, membership, identity, group, role, function, or scope token per line.">
|
||||||
|
<textarea rows={6} value={draft.default_acl_tokens} disabled={!canAdmin || busy} onChange={(event) => setDraft({ ...draft, default_acl_tokens: event.target.value })} />
|
||||||
|
</FormField>
|
||||||
|
<FormField label="Namespace mappings" hint="JSON array; maps source namespace ids to target Wiki space references and path prefixes.">
|
||||||
|
<textarea rows={12} value={draft.namespace_mappings} disabled={!canAdmin || busy} onChange={(event) => setDraft({ ...draft, namespace_mappings: event.target.value })} />
|
||||||
|
</FormField>
|
||||||
|
</FormGrid>
|
||||||
|
<p className="muted">Capabilities: {selected.capabilities.length ? selected.capabilities.join(", ") : "run discovery"}</p>
|
||||||
|
<p className="muted">Last high-watermark: {selected.last_high_watermark ?? "none"} · credential reference: {selected.credential_reference_present ? "configured" : "not configured"}</p>
|
||||||
|
</Card>
|
||||||
|
|
||||||
|
<Card title="Synchronized pages">
|
||||||
|
<SelectionList variant="static" label="Synchronized external knowledge pages">
|
||||||
|
{objects.slice(0, 100).map((item) => <SelectionListItem key={item.id}>
|
||||||
|
<SelectionListItemContent title={item.title} description={`${item.status} · revision ${item.source_revision} · ${item.visibility}`} />
|
||||||
|
{item.canonical_url ? <a href={item.canonical_url} target="_blank" rel="noreferrer">Open source</a> : null}
|
||||||
|
</SelectionListItem>)}
|
||||||
|
{!objects.length ? <StatePanel size="compact" description="No synchronized pages. Run a full backfill after discovery." /> : null}
|
||||||
|
</SelectionList>
|
||||||
|
</Card>
|
||||||
|
|
||||||
|
<Card title="Synchronization and migration evidence">
|
||||||
|
<SelectionList variant="static" label="Knowledge connector runs">
|
||||||
|
{runs.map((run) => <SelectionListItem key={run.id}>
|
||||||
|
<SelectionListItemContent title={`${run.mode.replaceAll("_", " ")} · ${run.status}`} description={`${formatDateTime(run.started_at)} · ${effectTotal(run)} effects · ${run.diagnostics.length} diagnostics`} />
|
||||||
|
<StatusBadge status={run.status} />
|
||||||
|
</SelectionListItem>)}
|
||||||
|
{!runs.length ? <StatePanel size="compact" description="No knowledge connector runs have been recorded." /> : null}
|
||||||
|
</SelectionList>
|
||||||
|
</Card>
|
||||||
|
|
||||||
|
{migration ? <Card title="Latest migration preview">
|
||||||
|
<div className="connector-revision-line">
|
||||||
|
<StatusBadge status={migration.can_apply ? "success" : "warning"} label={migration.can_apply ? "No blocking conflict" : "Review required"} />
|
||||||
|
<span>{migration.effects.length} effects</span>
|
||||||
|
<span>{migration.diagnostics.length} diagnostics</span>
|
||||||
|
<code title={migration.source_fingerprint}>{migration.source_fingerprint.slice(0, 12)}</code>
|
||||||
|
</div>
|
||||||
|
<p className="muted">Preview only: no native Wiki page was written.</p>
|
||||||
|
<pre className="connector-json-preview">{JSON.stringify({ summary: migration.summary, diagnostics: migration.diagnostics, truncated: migration.truncated }, null, 2)}</pre>
|
||||||
|
</Card> : null}
|
||||||
|
</div>}
|
||||||
|
</WorkspaceLayout>
|
||||||
|
|
||||||
|
<Dialog open={createOpen} title="Create external knowledge profile" onClose={() => !busy && setCreateOpen(false)} closeDisabled={busy} footer={<>
|
||||||
|
<Button onClick={() => setCreateOpen(false)} disabled={busy}>Cancel</Button>
|
||||||
|
<Button variant="primary" onClick={() => void createProfile()} disabled={busy || !configurationId.trim()}>Create profile</Button>
|
||||||
|
</>}>
|
||||||
|
<FormGrid columns={2} collapseAt="standard" className="">
|
||||||
|
<FormField label="Governed configuration id" hint="Select an active MediaWiki Action API configuration from Connector governance.">
|
||||||
|
<input value={configurationId} disabled={busy} onChange={(event) => setConfigurationId(event.target.value)} />
|
||||||
|
</FormField>
|
||||||
|
<FormField label="Desired maturity">
|
||||||
|
<select value={newDraft.desired_maturity} disabled={busy} onChange={(event) => setNewDraft({ ...newDraft, desired_maturity: event.target.value as ProfileDraft["desired_maturity"] })}>
|
||||||
|
{['discover', 'link', 'search', 'read', 'publish', 'synchronize', 'migrate'].map((value) => <option key={value} value={value}>{value}</option>)}
|
||||||
|
</select>
|
||||||
|
</FormField>
|
||||||
|
<FormField label="Fallback visibility">
|
||||||
|
<select value={newDraft.default_visibility} disabled={busy} onChange={(event) => setNewDraft({ ...newDraft, default_visibility: event.target.value as ProfileDraft["default_visibility"] })}>
|
||||||
|
<option value="restricted">Restricted</option>
|
||||||
|
<option value="tenant">Tenant</option>
|
||||||
|
</select>
|
||||||
|
</FormField>
|
||||||
|
<FormField label="Fallback ACL tokens">
|
||||||
|
<textarea rows={5} value={newDraft.default_acl_tokens} disabled={busy} onChange={(event) => setNewDraft({ ...newDraft, default_acl_tokens: event.target.value })} />
|
||||||
|
</FormField>
|
||||||
|
<FormField label="Namespace mappings">
|
||||||
|
<textarea rows={12} value={newDraft.namespace_mappings} disabled={busy} onChange={(event) => setNewDraft({ ...newDraft, namespace_mappings: event.target.value })} />
|
||||||
|
</FormField>
|
||||||
|
</FormGrid>
|
||||||
|
</Dialog>
|
||||||
|
|
||||||
|
<Dialog open={migrationOpen} title="Preview migration into native Wiki" onClose={() => !busy && setMigrationOpen(false)} closeDisabled={busy} footer={<>
|
||||||
|
<Button onClick={() => setMigrationOpen(false)} disabled={busy}>Cancel</Button>
|
||||||
|
<Button variant="primary" onClick={() => void previewMigration()} disabled={busy || !targetSpace.trim()}>Run migration preview</Button>
|
||||||
|
</>}>
|
||||||
|
<p className="muted">This dry-run detects path, attachment, macro, and truncation problems. It never writes Wiki pages.</p>
|
||||||
|
<FormField label="Target Wiki space reference"><input value={targetSpace} disabled={busy} onChange={(event) => setTargetSpace(event.target.value)} /></FormField>
|
||||||
|
<FormField label="Supported macros" hint="One macro name per line."><textarea rows={5} value={supportedMacros} disabled={busy} onChange={(event) => setSupportedMacros(event.target.value)} /></FormField>
|
||||||
|
<FormField label="Existing targets" hint="JSON array with path, optional source_external_id, and attachment_names."><textarea rows={10} value={existingTargets} disabled={busy} onChange={(event) => setExistingTargets(event.target.value)} /></FormField>
|
||||||
|
</Dialog>
|
||||||
|
|
||||||
|
<Dialog open={publishOpen} title="Publish provider page revision" onClose={() => !busy && setPublishOpen(false)} closeDisabled={busy} footer={<>
|
||||||
|
<Button onClick={() => setPublishOpen(false)} disabled={busy}>Cancel</Button>
|
||||||
|
<Button variant="primary" helpContextId="connectors.admin.external-knowledge" helpModuleId="connectors" onClick={() => void publish()} disabled={busy || !externalPageId.trim() || !publishTitle.trim()}>Publish revision</Button>
|
||||||
|
</>}>
|
||||||
|
<p className="muted">Publication is an external effect. Supply the current provider revision where possible; an unknown outcome blocks blind retry.</p>
|
||||||
|
<FormGrid columns={2} collapseAt="standard" className="">
|
||||||
|
<FormField label="Stable external page id"><input value={externalPageId} disabled={busy} onChange={(event) => setExternalPageId(event.target.value)} /></FormField>
|
||||||
|
<FormField label="Expected provider revision"><input value={publishRevision} disabled={busy} onChange={(event) => setPublishRevision(event.target.value)} /></FormField>
|
||||||
|
<FormField label="Title"><input value={publishTitle} disabled={busy} onChange={(event) => setPublishTitle(event.target.value)} /></FormField>
|
||||||
|
<FormField label="Edit summary"><input value={publishSummary} disabled={busy} onChange={(event) => setPublishSummary(event.target.value)} /></FormField>
|
||||||
|
<FormField label="Wikitext body"><textarea rows={16} value={publishBody} disabled={busy} onChange={(event) => setPublishBody(event.target.value)} /></FormField>
|
||||||
|
</FormGrid>
|
||||||
|
</Dialog>
|
||||||
|
</AdminPageLayout>;
|
||||||
|
}
|
||||||
|
|
||||||
|
function draftFromProfile(profile: KnowledgeProfile): ProfileDraft {
|
||||||
|
return {
|
||||||
|
status: profile.status,
|
||||||
|
desired_maturity: profile.desired_maturity,
|
||||||
|
source_authority_mode: profile.source_authority_mode,
|
||||||
|
default_visibility: profile.default_visibility,
|
||||||
|
default_acl_tokens: profile.default_acl_tokens.join("\n"),
|
||||||
|
namespace_mappings: JSON.stringify(profile.namespace_mappings, null, 2)
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
function draftKey(draft: ProfileDraft): string {
|
||||||
|
return JSON.stringify({
|
||||||
|
...draft,
|
||||||
|
default_acl_tokens: lines(draft.default_acl_tokens),
|
||||||
|
namespace_mappings: normalizeJson(draft.namespace_mappings)
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
function lines(value: string): string[] {
|
||||||
|
return [...new Set(value.split(/\r?\n/).map((item) => item.trim()).filter(Boolean))];
|
||||||
|
}
|
||||||
|
|
||||||
|
function normalizeJson(value: string): unknown {
|
||||||
|
try {
|
||||||
|
return JSON.parse(value);
|
||||||
|
} catch {
|
||||||
|
return value.trim();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
function parseArray(value: string, label: string): unknown[] {
|
||||||
|
const parsed: unknown = JSON.parse(value);
|
||||||
|
if (!Array.isArray(parsed)) throw new Error(`${label} must be a JSON array.`);
|
||||||
|
return parsed;
|
||||||
|
}
|
||||||
|
|
||||||
|
function effectTotal(run: KnowledgeRun): number {
|
||||||
|
return Object.values(run.counts).reduce((total, value) => total + Number(value || 0), 0);
|
||||||
|
}
|
||||||
|
|
||||||
|
function errorMessage(error: unknown): string {
|
||||||
|
return error instanceof Error ? error.message : String(error);
|
||||||
|
}
|
||||||
@@ -0,0 +1,573 @@
|
|||||||
|
import { useCallback, useEffect, useMemo, useState } from "react";
|
||||||
|
import {
|
||||||
|
AdminPageLayout,
|
||||||
|
Button,
|
||||||
|
Card,
|
||||||
|
Dialog,
|
||||||
|
FilterBar,
|
||||||
|
FormField,
|
||||||
|
FormGrid,
|
||||||
|
MetricCard,
|
||||||
|
MetricGrid,
|
||||||
|
PageActionBar,
|
||||||
|
SelectionList,
|
||||||
|
SelectionListItem,
|
||||||
|
SelectionListItemContent,
|
||||||
|
StatePanel,
|
||||||
|
StatusBadge,
|
||||||
|
WorkspaceLayout,
|
||||||
|
formatDateTime,
|
||||||
|
hasScope,
|
||||||
|
useUnsavedChanges,
|
||||||
|
useUnsavedDraftGuard,
|
||||||
|
type ApiSettings,
|
||||||
|
type AuthInfo
|
||||||
|
} from "@govoplan/core-webui";
|
||||||
|
import {
|
||||||
|
createServiceDeskProfile,
|
||||||
|
discoverServiceDeskProfile,
|
||||||
|
listServiceDeskObjects,
|
||||||
|
listServiceDeskProfiles,
|
||||||
|
listServiceDeskRuns,
|
||||||
|
synchronizeServiceDeskProfile,
|
||||||
|
updateServiceDeskProfile,
|
||||||
|
updateServiceDeskTicket,
|
||||||
|
type ServiceDeskObject,
|
||||||
|
type ServiceDeskProfile,
|
||||||
|
type ServiceDeskRun
|
||||||
|
} from "../api/externalServiceDesk";
|
||||||
|
|
||||||
|
type Props = { settings: ApiSettings; auth: AuthInfo };
|
||||||
|
|
||||||
|
type ProfileDraft = {
|
||||||
|
status: "active" | "paused";
|
||||||
|
integration_mode: ServiceDeskProfile["integration_mode"];
|
||||||
|
desired_maturity: ServiceDeskProfile["desired_maturity"];
|
||||||
|
source_authority_mode: ServiceDeskProfile["source_authority_mode"];
|
||||||
|
default_visibility: ServiceDeskProfile["default_visibility"];
|
||||||
|
default_acl_tokens: string;
|
||||||
|
routes: string;
|
||||||
|
queue_mappings: string;
|
||||||
|
dynamic_field_mappings: string;
|
||||||
|
};
|
||||||
|
|
||||||
|
const DEFAULT_ROUTES = {
|
||||||
|
search_path: "/Ticket/Search",
|
||||||
|
ticket_path: "/Ticket/{ticket_id}",
|
||||||
|
update_path: null,
|
||||||
|
search_method: "POST",
|
||||||
|
ticket_method: "GET",
|
||||||
|
update_method: "PATCH",
|
||||||
|
ticket_web_url_template: null,
|
||||||
|
search_filters: {}
|
||||||
|
};
|
||||||
|
|
||||||
|
const EMPTY_DRAFT: ProfileDraft = {
|
||||||
|
status: "active",
|
||||||
|
integration_mode: "synchronize",
|
||||||
|
desired_maturity: "synchronize",
|
||||||
|
source_authority_mode: "external_authoritative",
|
||||||
|
default_visibility: "restricted",
|
||||||
|
default_acl_tokens: "scope:connectors:service_desk:read",
|
||||||
|
routes: JSON.stringify(DEFAULT_ROUTES, null, 2),
|
||||||
|
queue_mappings: JSON.stringify([], null, 2),
|
||||||
|
dynamic_field_mappings: JSON.stringify([], null, 2)
|
||||||
|
};
|
||||||
|
|
||||||
|
export default function ExternalServiceDeskPage({ settings, auth }: Props) {
|
||||||
|
const [profiles, setProfiles] = useState<ServiceDeskProfile[]>([]);
|
||||||
|
const [objects, setObjects] = useState<ServiceDeskObject[]>([]);
|
||||||
|
const [runs, setRuns] = useState<ServiceDeskRun[]>([]);
|
||||||
|
const [selectedId, setSelectedId] = useState("");
|
||||||
|
const [draft, setDraft] = useState<ProfileDraft>(EMPTY_DRAFT);
|
||||||
|
const [savedKey, setSavedKey] = useState("");
|
||||||
|
const [search, setSearch] = useState("");
|
||||||
|
const [loading, setLoading] = useState(true);
|
||||||
|
const [busy, setBusy] = useState(false);
|
||||||
|
const [error, setError] = useState("");
|
||||||
|
const [success, setSuccess] = useState("");
|
||||||
|
const [createOpen, setCreateOpen] = useState(false);
|
||||||
|
const [configurationId, setConfigurationId] = useState("");
|
||||||
|
const [newDraft, setNewDraft] = useState<ProfileDraft>(EMPTY_DRAFT);
|
||||||
|
const [updateOpen, setUpdateOpen] = useState(false);
|
||||||
|
const [updateObjectId, setUpdateObjectId] = useState("");
|
||||||
|
const [updateTitle, setUpdateTitle] = useState("");
|
||||||
|
const [updateQueue, setUpdateQueue] = useState("");
|
||||||
|
const [updateState, setUpdateState] = useState("");
|
||||||
|
const [updatePriority, setUpdatePriority] = useState("");
|
||||||
|
const [updateOwner, setUpdateOwner] = useState("");
|
||||||
|
const [updateResponsible, setUpdateResponsible] = useState("");
|
||||||
|
const [updateDynamicFields, setUpdateDynamicFields] = useState("{}");
|
||||||
|
const { requestDiscard } = useUnsavedChanges();
|
||||||
|
|
||||||
|
const selected = profiles.find((item) => item.id === selectedId) ?? null;
|
||||||
|
const selectedObject = objects.find((item) => item.id === updateObjectId) ?? null;
|
||||||
|
const canAdmin = hasScope(auth, "connectors:service_desk:admin");
|
||||||
|
const canSync = hasScope(auth, "connectors:service_desk:sync");
|
||||||
|
const canUpdate = hasScope(auth, "connectors:service_desk:update");
|
||||||
|
const dirty = Boolean(selected && draftKey(draft) !== savedKey);
|
||||||
|
|
||||||
|
const applyProfile = useCallback((profile: ServiceDeskProfile | null) => {
|
||||||
|
const next = profile ? draftFromProfile(profile) : EMPTY_DRAFT;
|
||||||
|
setDraft(next);
|
||||||
|
setSavedKey(profile ? draftKey(next) : "");
|
||||||
|
}, []);
|
||||||
|
|
||||||
|
const reload = useCallback(async (preferredId?: string) => {
|
||||||
|
setLoading(true);
|
||||||
|
setError("");
|
||||||
|
try {
|
||||||
|
const nextProfiles = await listServiceDeskProfiles(settings);
|
||||||
|
const nextId = preferredId && nextProfiles.some((item) => item.id === preferredId)
|
||||||
|
? preferredId
|
||||||
|
: nextProfiles.some((item) => item.id === selectedId)
|
||||||
|
? selectedId
|
||||||
|
: nextProfiles[0]?.id ?? "";
|
||||||
|
const [nextObjects, nextRuns] = nextId
|
||||||
|
? await Promise.all([
|
||||||
|
listServiceDeskObjects(settings, nextId),
|
||||||
|
listServiceDeskRuns(settings, nextId)
|
||||||
|
])
|
||||||
|
: [[], []];
|
||||||
|
setProfiles(nextProfiles);
|
||||||
|
setSelectedId(nextId);
|
||||||
|
setObjects(nextObjects);
|
||||||
|
setRuns(nextRuns);
|
||||||
|
applyProfile(nextProfiles.find((item) => item.id === nextId) ?? null);
|
||||||
|
} catch (caught) {
|
||||||
|
setError(errorMessage(caught));
|
||||||
|
} finally {
|
||||||
|
setLoading(false);
|
||||||
|
}
|
||||||
|
}, [applyProfile, selectedId, settings]);
|
||||||
|
|
||||||
|
useEffect(() => {
|
||||||
|
void reload();
|
||||||
|
}, [settings.apiBaseUrl, settings.apiKey, settings.accessToken]);
|
||||||
|
|
||||||
|
const save = async (): Promise<boolean> => {
|
||||||
|
if (!selected || !canAdmin) return false;
|
||||||
|
setBusy(true);
|
||||||
|
setError("");
|
||||||
|
try {
|
||||||
|
const updated = await updateServiceDeskProfile(settings, selected.id, {
|
||||||
|
expected_resource_revision: selected.resource_revision,
|
||||||
|
status: draft.status,
|
||||||
|
integration_mode: draft.integration_mode,
|
||||||
|
desired_maturity: draft.desired_maturity,
|
||||||
|
source_authority_mode: draft.source_authority_mode,
|
||||||
|
default_visibility: draft.default_visibility,
|
||||||
|
default_acl_tokens: lines(draft.default_acl_tokens),
|
||||||
|
routes: parseObject(draft.routes, "Routes"),
|
||||||
|
queue_mappings: parseArray(draft.queue_mappings, "Queue mappings"),
|
||||||
|
dynamic_field_mappings: parseArray(draft.dynamic_field_mappings, "Dynamic-field mappings")
|
||||||
|
});
|
||||||
|
setSuccess("Service-desk profile saved; queue ACLs and Search projections were refreshed.");
|
||||||
|
await reload(updated.id);
|
||||||
|
return true;
|
||||||
|
} catch (caught) {
|
||||||
|
setError(errorMessage(caught));
|
||||||
|
return false;
|
||||||
|
} finally {
|
||||||
|
setBusy(false);
|
||||||
|
}
|
||||||
|
};
|
||||||
|
|
||||||
|
useUnsavedDraftGuard({
|
||||||
|
dirty,
|
||||||
|
onSave: save,
|
||||||
|
onDiscard: () => applyProfile(selected),
|
||||||
|
title: "Unsaved service-desk profile changes",
|
||||||
|
message: "Save or discard the profile changes before continuing."
|
||||||
|
});
|
||||||
|
|
||||||
|
const selectProfile = (profile: ServiceDeskProfile) => {
|
||||||
|
if (profile.id === selectedId) return;
|
||||||
|
requestDiscard(() => {
|
||||||
|
setSelectedId(profile.id);
|
||||||
|
applyProfile(profile);
|
||||||
|
setObjects([]);
|
||||||
|
setRuns([]);
|
||||||
|
void Promise.all([
|
||||||
|
listServiceDeskObjects(settings, profile.id),
|
||||||
|
listServiceDeskRuns(settings, profile.id)
|
||||||
|
]).then(([nextObjects, nextRuns]) => {
|
||||||
|
setObjects(nextObjects);
|
||||||
|
setRuns(nextRuns);
|
||||||
|
}).catch((caught) => setError(errorMessage(caught)));
|
||||||
|
});
|
||||||
|
};
|
||||||
|
|
||||||
|
const createProfile = async () => {
|
||||||
|
if (!configurationId.trim()) return;
|
||||||
|
setBusy(true);
|
||||||
|
setError("");
|
||||||
|
try {
|
||||||
|
const created = await createServiceDeskProfile(settings, {
|
||||||
|
configuration_id: configurationId.trim(),
|
||||||
|
integration_mode: newDraft.integration_mode,
|
||||||
|
desired_maturity: newDraft.desired_maturity,
|
||||||
|
source_authority_mode: newDraft.source_authority_mode,
|
||||||
|
default_visibility: newDraft.default_visibility,
|
||||||
|
default_acl_tokens: lines(newDraft.default_acl_tokens),
|
||||||
|
routes: parseObject(newDraft.routes, "Routes"),
|
||||||
|
queue_mappings: parseArray(newDraft.queue_mappings, "Queue mappings"),
|
||||||
|
dynamic_field_mappings: parseArray(newDraft.dynamic_field_mappings, "Dynamic-field mappings")
|
||||||
|
});
|
||||||
|
setCreateOpen(false);
|
||||||
|
setConfigurationId("");
|
||||||
|
setNewDraft(EMPTY_DRAFT);
|
||||||
|
setSuccess("Service-desk profile created. Run discovery before synchronization.");
|
||||||
|
await reload(created.id);
|
||||||
|
} catch (caught) {
|
||||||
|
setError(errorMessage(caught));
|
||||||
|
} finally {
|
||||||
|
setBusy(false);
|
||||||
|
}
|
||||||
|
};
|
||||||
|
|
||||||
|
const discover = async () => {
|
||||||
|
if (!selected || dirty) return;
|
||||||
|
setBusy(true);
|
||||||
|
setError("");
|
||||||
|
try {
|
||||||
|
const result = await discoverServiceDeskProfile(settings, selected.id);
|
||||||
|
setSuccess(`Discovered ${result.product} ${result.product_version ?? ""} at ${result.maturity} maturity with ${result.diagnostics.length} diagnostics.`);
|
||||||
|
await reload(selected.id);
|
||||||
|
} catch (caught) {
|
||||||
|
setError(errorMessage(caught));
|
||||||
|
} finally {
|
||||||
|
setBusy(false);
|
||||||
|
}
|
||||||
|
};
|
||||||
|
|
||||||
|
const sync = async (mode: "auto" | "full") => {
|
||||||
|
if (!selected || dirty) return;
|
||||||
|
setBusy(true);
|
||||||
|
setError("");
|
||||||
|
try {
|
||||||
|
const run = await synchronizeServiceDeskProfile(settings, selected.id, {
|
||||||
|
idempotency_key: `service-desk-${mode}-${crypto.randomUUID()}`,
|
||||||
|
mode,
|
||||||
|
limit: 100
|
||||||
|
});
|
||||||
|
setSuccess(`${mode === "full" ? "Full synchronization" : "Next synchronization page"} completed with ${effectTotal(run)} effects.`);
|
||||||
|
await reload(selected.id);
|
||||||
|
} catch (caught) {
|
||||||
|
setError(errorMessage(caught));
|
||||||
|
} finally {
|
||||||
|
setBusy(false);
|
||||||
|
}
|
||||||
|
};
|
||||||
|
|
||||||
|
const openTicketUpdate = (item: ServiceDeskObject) => {
|
||||||
|
setUpdateObjectId(item.id);
|
||||||
|
setUpdateTitle("");
|
||||||
|
setUpdateQueue("");
|
||||||
|
setUpdateState("");
|
||||||
|
setUpdatePriority("");
|
||||||
|
setUpdateOwner("");
|
||||||
|
setUpdateResponsible("");
|
||||||
|
setUpdateDynamicFields("{}");
|
||||||
|
setUpdateOpen(true);
|
||||||
|
};
|
||||||
|
|
||||||
|
const submitTicketUpdate = async () => {
|
||||||
|
if (!selected || !selectedObject) return;
|
||||||
|
const dynamicFields = parseObject(updateDynamicFields, "Dynamic fields");
|
||||||
|
const changes = compact({
|
||||||
|
title: updateTitle,
|
||||||
|
queue: updateQueue,
|
||||||
|
state: updateState,
|
||||||
|
priority: updatePriority,
|
||||||
|
owner: updateOwner,
|
||||||
|
responsible: updateResponsible
|
||||||
|
});
|
||||||
|
if (!Object.keys(changes).length && !Object.keys(dynamicFields).length) {
|
||||||
|
setError("Enter at least one governed ticket change.");
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
setBusy(true);
|
||||||
|
setError("");
|
||||||
|
try {
|
||||||
|
const result = await updateServiceDeskTicket(
|
||||||
|
settings,
|
||||||
|
selected.id,
|
||||||
|
selectedObject.external_id,
|
||||||
|
{
|
||||||
|
idempotency_key: `service-desk-update-${crypto.randomUUID()}`,
|
||||||
|
expected_external_revision: selectedObject.source_revision,
|
||||||
|
...changes,
|
||||||
|
dynamic_fields: dynamicFields
|
||||||
|
}
|
||||||
|
);
|
||||||
|
setUpdateOpen(false);
|
||||||
|
setSuccess(result.outcome_unknown
|
||||||
|
? "Update outcome is unknown. Inspect the provider revision before retrying."
|
||||||
|
: "Provider accepted the revision-checked ticket update and durable evidence was recorded.");
|
||||||
|
await reload(selected.id);
|
||||||
|
} catch (caught) {
|
||||||
|
setError(errorMessage(caught));
|
||||||
|
} finally {
|
||||||
|
setBusy(false);
|
||||||
|
}
|
||||||
|
};
|
||||||
|
|
||||||
|
const visibleProfiles = useMemo(() => {
|
||||||
|
const needle = search.trim().toLocaleLowerCase();
|
||||||
|
return profiles.filter((item) => !needle ||
|
||||||
|
`${item.product} ${item.product_version ?? ""} ${item.integration_mode} ${item.health_status} ${item.configuration_id}`
|
||||||
|
.toLocaleLowerCase().includes(needle));
|
||||||
|
}, [profiles, search]);
|
||||||
|
|
||||||
|
const actionBar = <PageActionBar
|
||||||
|
variant="editor"
|
||||||
|
state={busy ? "saving" : dirty ? "dirty" : "clean"}
|
||||||
|
refreshable
|
||||||
|
reloadAction={{ onReload: () => void reload(selectedId), loading }}
|
||||||
|
primaryActions={<>
|
||||||
|
<Button onClick={() => setCreateOpen(true)} disabled={!canAdmin || busy}>New profile</Button>
|
||||||
|
<Button variant="secondary" onClick={() => void discover()} disabled={!selected || !canAdmin || busy || dirty}>Discover</Button>
|
||||||
|
<Button variant="secondary" onClick={() => void sync("auto")} disabled={!selected || !canSync || busy || dirty}>Run next page</Button>
|
||||||
|
<Button variant="secondary" onClick={() => void sync("full")} disabled={!selected || !canSync || busy || dirty}>Restart full sync</Button>
|
||||||
|
</>}
|
||||||
|
discardAction={{
|
||||||
|
label: "Discard changes",
|
||||||
|
disabled: !selected,
|
||||||
|
onClick: () => applyProfile(selected)
|
||||||
|
}}
|
||||||
|
saveAction={{
|
||||||
|
label: "Save",
|
||||||
|
disabled: !selected || !canAdmin || busy,
|
||||||
|
disabledReason: !canAdmin ? "Service-desk administration permission is required." : undefined,
|
||||||
|
onClick: () => void save()
|
||||||
|
}}
|
||||||
|
/>;
|
||||||
|
|
||||||
|
return <AdminPageLayout
|
||||||
|
archetype="workspace"
|
||||||
|
title="External service desk"
|
||||||
|
description="Connect Znuny or OTRS-compatible tickets while preserving provider identity, source authority, current ACLs, and domain-module boundaries."
|
||||||
|
loading={loading && !profiles.length}
|
||||||
|
error={error}
|
||||||
|
success={success}
|
||||||
|
actions={actionBar}
|
||||||
|
className="connector-service-desk-page"
|
||||||
|
helpContextId="connectors.admin.external-service-desk"
|
||||||
|
>
|
||||||
|
<MetricGrid columns={4} density="compact" minimum="compact">
|
||||||
|
<MetricCard label="Profiles" value={profiles.length} />
|
||||||
|
<MetricCard label="Active tickets" value={objects.filter((item) => item.status !== "deleted").length} />
|
||||||
|
<MetricCard label="Unhealthy profiles" value={profiles.filter((item) => !["healthy", "unknown"].includes(item.health_status)).length} tone="warning" />
|
||||||
|
<MetricCard label="Unresolved runs" value={runs.filter((item) => ["failed", "outcome_unknown"].includes(item.status)).length} tone="warning" />
|
||||||
|
</MetricGrid>
|
||||||
|
|
||||||
|
<WorkspaceLayout
|
||||||
|
variant="split"
|
||||||
|
primarySize="compact"
|
||||||
|
surface="contained"
|
||||||
|
primaryScrollable={false}
|
||||||
|
contentScrollable={false}
|
||||||
|
primaryLabel="Service-desk profiles"
|
||||||
|
contentLabel="Profile details"
|
||||||
|
primary={<div className="connector-knowledge-list">
|
||||||
|
<FilterBar surface="panel">
|
||||||
|
<input type="search" value={search} onChange={(event) => setSearch(event.target.value)} placeholder="Search profiles" aria-label="Search service-desk profiles" />
|
||||||
|
</FilterBar>
|
||||||
|
<SelectionList variant="navigation" label="External service-desk profiles">
|
||||||
|
{visibleProfiles.map((profile) => <SelectionListItem key={profile.id} selected={profile.id === selectedId} onClick={() => selectProfile(profile)}>
|
||||||
|
<SelectionListItemContent
|
||||||
|
title={`${profile.product}${profile.product_version ? ` ${profile.product_version}` : ""}`}
|
||||||
|
description={`${profile.integration_mode} · ${profile.discovered_maturity} · ${profile.configuration_id}`}
|
||||||
|
/>
|
||||||
|
<StatusBadge status={profile.status === "paused" ? "inactive" : profile.health_status} />
|
||||||
|
</SelectionListItem>)}
|
||||||
|
{!visibleProfiles.length ? <StatePanel size="compact" description="No matching service-desk profiles." /> : null}
|
||||||
|
</SelectionList>
|
||||||
|
</div>}
|
||||||
|
>
|
||||||
|
{!selected ? <StatePanel size="fill" title="External service-desk profiles" description="Create or select a profile to discover deployment routes and inspect synchronization evidence." /> : <div className="connector-knowledge-detail">
|
||||||
|
<Card title={`${selected.product}${selected.product_version ? ` ${selected.product_version}` : ""}`}>
|
||||||
|
<div className="connector-revision-line">
|
||||||
|
<StatusBadge status={selected.status} />
|
||||||
|
<StatusBadge status={selected.health_status} />
|
||||||
|
<span>Discovered maturity: {selected.discovered_maturity}</span>
|
||||||
|
<code title={selected.discovery_revision ?? undefined}>r{selected.resource_revision}</code>
|
||||||
|
</div>
|
||||||
|
<FormGrid columns={2} collapseAt="standard" className="">
|
||||||
|
<FormField label="Status" hint="Pausing immediately makes Search authorization fail closed.">
|
||||||
|
<select value={draft.status} disabled={!canAdmin || busy} onChange={(event) => setDraft({ ...draft, status: event.target.value as ProfileDraft["status"] })}>
|
||||||
|
<option value="active">Active</option><option value="paused">Paused</option>
|
||||||
|
</select>
|
||||||
|
</FormField>
|
||||||
|
<FormField label="Integration mode" hint="Link and import are intentionally not continuous bidirectional synchronization.">
|
||||||
|
<select value={draft.integration_mode} disabled={!canAdmin || busy} onChange={(event) => setDraft(withMode(draft, event.target.value as ProfileDraft["integration_mode"]))}>
|
||||||
|
<option value="link">Link</option><option value="import">Import snapshot</option><option value="synchronize">Synchronize</option>
|
||||||
|
</select>
|
||||||
|
</FormField>
|
||||||
|
<FormField label="Desired maturity" hint="Link permits link/search, import requires read, synchronize requires synchronize.">
|
||||||
|
<select value={draft.desired_maturity} disabled={!canAdmin || busy} onChange={(event) => setDraft({ ...draft, desired_maturity: event.target.value as ProfileDraft["desired_maturity"] })}>
|
||||||
|
{maturityOptions(draft.integration_mode).map((value) => <option key={value} value={value}>{value}</option>)}
|
||||||
|
</select>
|
||||||
|
</FormField>
|
||||||
|
<FormField label="Source authority">
|
||||||
|
<select value={draft.source_authority_mode} disabled={!canAdmin || busy} onChange={(event) => setDraft({ ...draft, source_authority_mode: event.target.value as ProfileDraft["source_authority_mode"] })}>
|
||||||
|
{authorityOptions(draft.integration_mode).map(([value, label]) => <option key={value} value={value}>{label}</option>)}
|
||||||
|
</select>
|
||||||
|
</FormField>
|
||||||
|
<FormField label="Fallback visibility">
|
||||||
|
<select value={draft.default_visibility} disabled={!canAdmin || busy} onChange={(event) => setDraft({ ...draft, default_visibility: event.target.value as ProfileDraft["default_visibility"] })}>
|
||||||
|
<option value="restricted">Restricted</option><option value="tenant">Tenant</option>
|
||||||
|
</select>
|
||||||
|
</FormField>
|
||||||
|
<FormField label="Fallback ACL tokens" hint="Used only when provider and queue mappings supply no portable ACL.">
|
||||||
|
<textarea rows={6} value={draft.default_acl_tokens} disabled={!canAdmin || busy} onChange={(event) => setDraft({ ...draft, default_acl_tokens: event.target.value })} />
|
||||||
|
</FormField>
|
||||||
|
<FormField label="GenericInterface routes" hint="JSON object; deployment-defined relative paths and methods plus optional absolute browser-link template.">
|
||||||
|
<textarea rows={14} value={draft.routes} disabled={!canAdmin || busy} onChange={(event) => setDraft({ ...draft, routes: event.target.value })} />
|
||||||
|
</FormField>
|
||||||
|
<FormField label="Queue mappings" hint="JSON array; inclusion, target queue ref, visibility, and ACL tokens.">
|
||||||
|
<textarea rows={14} value={draft.queue_mappings} disabled={!canAdmin || busy} onChange={(event) => setDraft({ ...draft, queue_mappings: event.target.value })} />
|
||||||
|
</FormField>
|
||||||
|
<FormField label="Dynamic-field mappings" hint="JSON array; source name, governed target name, inclusion, and value type.">
|
||||||
|
<textarea rows={14} value={draft.dynamic_field_mappings} disabled={!canAdmin || busy} onChange={(event) => setDraft({ ...draft, dynamic_field_mappings: event.target.value })} />
|
||||||
|
</FormField>
|
||||||
|
</FormGrid>
|
||||||
|
<p className="muted">Capabilities: {selected.capabilities.length ? selected.capabilities.join(", ") : "run discovery"}</p>
|
||||||
|
<p className="muted">Last high-watermark: {selected.last_high_watermark ?? "none"} · credential reference: {selected.credential_reference_present ? "configured" : "not configured"}</p>
|
||||||
|
</Card>
|
||||||
|
|
||||||
|
<Card title="Synchronized external tickets">
|
||||||
|
<SelectionList variant="static" label="Synchronized external service-desk tickets">
|
||||||
|
{objects.slice(0, 100).map((item) => <SelectionListItem key={item.id}>
|
||||||
|
<SelectionListItemContent title={`${item.external_ticket_number ? `${item.external_ticket_number}: ` : ""}${item.title}`} description={`${item.status} · revision ${item.source_revision} · ${item.visibility}`} />
|
||||||
|
{item.canonical_url ? <a href={item.canonical_url} target="_blank" rel="noreferrer">Open source</a> : null}
|
||||||
|
<Button variant="secondary" onClick={() => openTicketUpdate(item)} disabled={!canUpdate || busy || dirty || selected.source_authority_mode !== "governed_sync" || !selected.capabilities.includes("publish") || item.status === "deleted"}>Update</Button>
|
||||||
|
</SelectionListItem>)}
|
||||||
|
{!objects.length ? <StatePanel size="compact" description="No synchronized tickets. Run discovery and a full synchronization." /> : null}
|
||||||
|
</SelectionList>
|
||||||
|
</Card>
|
||||||
|
|
||||||
|
<Card title="Synchronization and mutation evidence">
|
||||||
|
<SelectionList variant="static" label="Service-desk connector runs">
|
||||||
|
{runs.map((run) => <SelectionListItem key={run.id}>
|
||||||
|
<SelectionListItemContent title={`${run.mode.replaceAll("_", " ")} · ${run.status}`} description={`${formatDateTime(run.started_at)} · ${effectTotal(run)} effects · ${run.diagnostics.length} diagnostics`} />
|
||||||
|
<StatusBadge status={run.status} />
|
||||||
|
</SelectionListItem>)}
|
||||||
|
{!runs.length ? <StatePanel size="compact" description="No service-desk connector runs have been recorded." /> : null}
|
||||||
|
</SelectionList>
|
||||||
|
</Card>
|
||||||
|
</div>}
|
||||||
|
</WorkspaceLayout>
|
||||||
|
|
||||||
|
<Dialog open={createOpen} title="Create external service-desk profile" onClose={() => !busy && setCreateOpen(false)} closeDisabled={busy} footer={<>
|
||||||
|
<Button onClick={() => setCreateOpen(false)} disabled={busy}>Cancel</Button>
|
||||||
|
<Button variant="primary" onClick={() => void createProfile()} disabled={busy || !configurationId.trim()}>Create profile</Button>
|
||||||
|
</>}>
|
||||||
|
<FormGrid columns={2} collapseAt="standard" className="">
|
||||||
|
<FormField label="Governed configuration id" hint="Select an active Znuny/OTRS GenericInterface REST configuration from Connector governance.">
|
||||||
|
<input value={configurationId} disabled={busy} onChange={(event) => setConfigurationId(event.target.value)} />
|
||||||
|
</FormField>
|
||||||
|
<FormField label="Integration mode"><select value={newDraft.integration_mode} disabled={busy} onChange={(event) => setNewDraft(withMode(newDraft, event.target.value as ProfileDraft["integration_mode"]))}><option value="link">Link</option><option value="import">Import snapshot</option><option value="synchronize">Synchronize</option></select></FormField>
|
||||||
|
<FormField label="Desired maturity"><select value={newDraft.desired_maturity} disabled={busy} onChange={(event) => setNewDraft({ ...newDraft, desired_maturity: event.target.value as ProfileDraft["desired_maturity"] })}>{maturityOptions(newDraft.integration_mode).map((value) => <option key={value} value={value}>{value}</option>)}</select></FormField>
|
||||||
|
<FormField label="Source authority"><select value={newDraft.source_authority_mode} disabled={busy} onChange={(event) => setNewDraft({ ...newDraft, source_authority_mode: event.target.value as ProfileDraft["source_authority_mode"] })}>{authorityOptions(newDraft.integration_mode).map(([value, label]) => <option key={value} value={value}>{label}</option>)}</select></FormField>
|
||||||
|
<FormField label="Fallback visibility"><select value={newDraft.default_visibility} disabled={busy} onChange={(event) => setNewDraft({ ...newDraft, default_visibility: event.target.value as ProfileDraft["default_visibility"] })}><option value="restricted">Restricted</option><option value="tenant">Tenant</option></select></FormField>
|
||||||
|
<FormField label="Fallback ACL tokens"><textarea rows={5} value={newDraft.default_acl_tokens} disabled={busy} onChange={(event) => setNewDraft({ ...newDraft, default_acl_tokens: event.target.value })} /></FormField>
|
||||||
|
<FormField label="GenericInterface routes"><textarea rows={12} value={newDraft.routes} disabled={busy} onChange={(event) => setNewDraft({ ...newDraft, routes: event.target.value })} /></FormField>
|
||||||
|
<FormField label="Queue mappings"><textarea rows={12} value={newDraft.queue_mappings} disabled={busy} onChange={(event) => setNewDraft({ ...newDraft, queue_mappings: event.target.value })} /></FormField>
|
||||||
|
<FormField label="Dynamic-field mappings"><textarea rows={12} value={newDraft.dynamic_field_mappings} disabled={busy} onChange={(event) => setNewDraft({ ...newDraft, dynamic_field_mappings: event.target.value })} /></FormField>
|
||||||
|
</FormGrid>
|
||||||
|
</Dialog>
|
||||||
|
|
||||||
|
<Dialog open={updateOpen} title="Update external ticket" onClose={() => !busy && setUpdateOpen(false)} closeDisabled={busy} footer={<>
|
||||||
|
<Button onClick={() => setUpdateOpen(false)} disabled={busy}>Cancel</Button>
|
||||||
|
<Button variant="primary" onClick={() => void submitTicketUpdate()} disabled={busy || !selectedObject}>Submit revision-checked update</Button>
|
||||||
|
</>}>
|
||||||
|
<p className="muted">This is an external effect. Empty fields remain unchanged; an unknown result blocks blind retry and requires provider reconciliation.</p>
|
||||||
|
<p><strong>{selectedObject?.external_ticket_number}</strong> {selectedObject?.title}<br /><span className="muted">Expected provider revision: {selectedObject?.source_revision}</span></p>
|
||||||
|
<FormGrid columns={2} collapseAt="standard" className="">
|
||||||
|
<FormField label="Title"><input value={updateTitle} disabled={busy} onChange={(event) => setUpdateTitle(event.target.value)} /></FormField>
|
||||||
|
<FormField label="Queue"><input value={updateQueue} disabled={busy} onChange={(event) => setUpdateQueue(event.target.value)} /></FormField>
|
||||||
|
<FormField label="State"><input value={updateState} disabled={busy} onChange={(event) => setUpdateState(event.target.value)} /></FormField>
|
||||||
|
<FormField label="Priority"><input value={updatePriority} disabled={busy} onChange={(event) => setUpdatePriority(event.target.value)} /></FormField>
|
||||||
|
<FormField label="Owner"><input value={updateOwner} disabled={busy} onChange={(event) => setUpdateOwner(event.target.value)} /></FormField>
|
||||||
|
<FormField label="Responsible"><input value={updateResponsible} disabled={busy} onChange={(event) => setUpdateResponsible(event.target.value)} /></FormField>
|
||||||
|
<FormField label="Governed dynamic fields" hint="JSON object keyed by configured target name."><textarea rows={8} value={updateDynamicFields} disabled={busy} onChange={(event) => setUpdateDynamicFields(event.target.value)} /></FormField>
|
||||||
|
</FormGrid>
|
||||||
|
</Dialog>
|
||||||
|
</AdminPageLayout>;
|
||||||
|
}
|
||||||
|
|
||||||
|
function draftFromProfile(profile: ServiceDeskProfile): ProfileDraft {
|
||||||
|
return {
|
||||||
|
status: profile.status,
|
||||||
|
integration_mode: profile.integration_mode,
|
||||||
|
desired_maturity: profile.desired_maturity,
|
||||||
|
source_authority_mode: profile.source_authority_mode,
|
||||||
|
default_visibility: profile.default_visibility,
|
||||||
|
default_acl_tokens: profile.default_acl_tokens.join("\n"),
|
||||||
|
routes: JSON.stringify(profile.routes, null, 2),
|
||||||
|
queue_mappings: JSON.stringify(profile.queue_mappings, null, 2),
|
||||||
|
dynamic_field_mappings: JSON.stringify(profile.dynamic_field_mappings, null, 2)
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
function withMode(draft: ProfileDraft, mode: ProfileDraft["integration_mode"]): ProfileDraft {
|
||||||
|
if (mode === "link") return { ...draft, integration_mode: mode, desired_maturity: "link", source_authority_mode: "linked_reference" };
|
||||||
|
if (mode === "import") return { ...draft, integration_mode: mode, desired_maturity: "read", source_authority_mode: "external_mirror" };
|
||||||
|
return { ...draft, integration_mode: mode, desired_maturity: "synchronize", source_authority_mode: "external_authoritative" };
|
||||||
|
}
|
||||||
|
|
||||||
|
function maturityOptions(mode: ProfileDraft["integration_mode"]): ProfileDraft["desired_maturity"][] {
|
||||||
|
if (mode === "link") return ["link", "search"];
|
||||||
|
if (mode === "import") return ["read"];
|
||||||
|
return ["synchronize"];
|
||||||
|
}
|
||||||
|
|
||||||
|
function authorityOptions(mode: ProfileDraft["integration_mode"]): Array<[ProfileDraft["source_authority_mode"], string]> {
|
||||||
|
if (mode === "link") return [["linked_reference", "Linked reference"]];
|
||||||
|
if (mode === "import") return [["external_authoritative", "External authoritative"], ["external_mirror", "External mirror"]];
|
||||||
|
return [["external_authoritative", "External authoritative"], ["governed_sync", "Governed sync"]];
|
||||||
|
}
|
||||||
|
|
||||||
|
function draftKey(draft: ProfileDraft): string {
|
||||||
|
return JSON.stringify({
|
||||||
|
...draft,
|
||||||
|
default_acl_tokens: lines(draft.default_acl_tokens),
|
||||||
|
routes: normalizeJson(draft.routes),
|
||||||
|
queue_mappings: normalizeJson(draft.queue_mappings),
|
||||||
|
dynamic_field_mappings: normalizeJson(draft.dynamic_field_mappings)
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
function lines(value: string): string[] {
|
||||||
|
return [...new Set(value.split(/\r?\n/).map((item) => item.trim()).filter(Boolean))];
|
||||||
|
}
|
||||||
|
|
||||||
|
function normalizeJson(value: string): unknown {
|
||||||
|
try { return JSON.parse(value); } catch { return value.trim(); }
|
||||||
|
}
|
||||||
|
|
||||||
|
function parseArray(value: string, label: string): unknown[] {
|
||||||
|
const parsed: unknown = JSON.parse(value);
|
||||||
|
if (!Array.isArray(parsed)) throw new Error(`${label} must be a JSON array.`);
|
||||||
|
return parsed;
|
||||||
|
}
|
||||||
|
|
||||||
|
function parseObject(value: string, label: string): Record<string, unknown> {
|
||||||
|
const parsed: unknown = JSON.parse(value);
|
||||||
|
if (!parsed || Array.isArray(parsed) || typeof parsed !== "object") throw new Error(`${label} must be a JSON object.`);
|
||||||
|
return parsed as Record<string, unknown>;
|
||||||
|
}
|
||||||
|
|
||||||
|
function compact(values: Record<string, string>): Record<string, string> {
|
||||||
|
return Object.fromEntries(Object.entries(values).map(([key, value]) => [key, value.trim()]).filter(([, value]) => value));
|
||||||
|
}
|
||||||
|
|
||||||
|
function effectTotal(run: ServiceDeskRun): number {
|
||||||
|
return Object.values(run.counts).reduce((total, value) => total + Number(value || 0), 0);
|
||||||
|
}
|
||||||
|
|
||||||
|
function errorMessage(error: unknown): string {
|
||||||
|
return error instanceof Error ? error.message : String(error);
|
||||||
|
}
|
||||||
+56
-2
@@ -8,11 +8,25 @@ import "./styles/connectors.css";
|
|||||||
const ConnectorGovernancePage = lazy(
|
const ConnectorGovernancePage = lazy(
|
||||||
() => import("./features/ConnectorGovernancePage")
|
() => import("./features/ConnectorGovernancePage")
|
||||||
);
|
);
|
||||||
|
const ExternalKnowledgePage = lazy(
|
||||||
|
() => import("./features/ExternalKnowledgePage")
|
||||||
|
);
|
||||||
|
const ExternalServiceDeskPage = lazy(
|
||||||
|
() => import("./features/ExternalServiceDeskPage")
|
||||||
|
);
|
||||||
|
|
||||||
const readScopes = [
|
const readScopes = [
|
||||||
"connectors:source:read",
|
"connectors:source:read",
|
||||||
"connectors:source:admin"
|
"connectors:source:admin"
|
||||||
];
|
];
|
||||||
|
const knowledgeReadScopes = [
|
||||||
|
"connectors:knowledge:read",
|
||||||
|
"connectors:knowledge:admin"
|
||||||
|
];
|
||||||
|
const serviceDeskReadScopes = [
|
||||||
|
"connectors:service_desk:read",
|
||||||
|
"connectors:service_desk:admin"
|
||||||
|
];
|
||||||
|
|
||||||
const adminSections: AdminSectionsUiCapability = {
|
const adminSections: AdminSectionsUiCapability = {
|
||||||
sections: [
|
sections: [
|
||||||
@@ -27,6 +41,30 @@ const adminSections: AdminSectionsUiCapability = {
|
|||||||
anyOf: readScopes,
|
anyOf: readScopes,
|
||||||
render: ({ settings, auth }) =>
|
render: ({ settings, auth }) =>
|
||||||
createElement(ConnectorGovernancePage, { settings, auth })
|
createElement(ConnectorGovernancePage, { settings, auth })
|
||||||
|
},
|
||||||
|
{
|
||||||
|
id: "connector-external-knowledge",
|
||||||
|
moduleId: "connectors",
|
||||||
|
kind: "management",
|
||||||
|
surfaceId: "connectors.admin.external-knowledge",
|
||||||
|
label: "External knowledge",
|
||||||
|
group: "SYSTEM",
|
||||||
|
order: 46,
|
||||||
|
anyOf: knowledgeReadScopes,
|
||||||
|
render: ({ settings, auth }) =>
|
||||||
|
createElement(ExternalKnowledgePage, { settings, auth })
|
||||||
|
},
|
||||||
|
{
|
||||||
|
id: "connector-external-service-desk",
|
||||||
|
moduleId: "connectors",
|
||||||
|
kind: "management",
|
||||||
|
surfaceId: "connectors.admin.external-service-desk",
|
||||||
|
label: "External service desk",
|
||||||
|
group: "SYSTEM",
|
||||||
|
order: 47,
|
||||||
|
anyOf: serviceDeskReadScopes,
|
||||||
|
render: ({ settings, auth }) =>
|
||||||
|
createElement(ExternalServiceDeskPage, { settings, auth })
|
||||||
}
|
}
|
||||||
]
|
]
|
||||||
};
|
};
|
||||||
@@ -34,9 +72,9 @@ const adminSections: AdminSectionsUiCapability = {
|
|||||||
export const connectorsModule: PlatformWebModule = {
|
export const connectorsModule: PlatformWebModule = {
|
||||||
id: "connectors",
|
id: "connectors",
|
||||||
label: "Connectors",
|
label: "Connectors",
|
||||||
version: "0.1.18",
|
version: "0.1.23",
|
||||||
dependencies: [],
|
dependencies: [],
|
||||||
optionalDependencies: ["access", "audit", "policy", "ops"],
|
optionalDependencies: ["access", "audit", "policy", "ops", "search", "wiki", "tickets", "helpdesk", "cases"],
|
||||||
viewSurfaces: [
|
viewSurfaces: [
|
||||||
{
|
{
|
||||||
id: "connectors.admin.governed-configurations",
|
id: "connectors.admin.governed-configurations",
|
||||||
@@ -52,6 +90,22 @@ export const connectorsModule: PlatformWebModule = {
|
|||||||
label: "Connector simulation review",
|
label: "Connector simulation review",
|
||||||
parentId: "connectors.admin.governed-configurations",
|
parentId: "connectors.admin.governed-configurations",
|
||||||
order: 20
|
order: 20
|
||||||
|
},
|
||||||
|
{
|
||||||
|
id: "connectors.admin.external-knowledge",
|
||||||
|
moduleId: "connectors",
|
||||||
|
kind: "section",
|
||||||
|
label: "External knowledge",
|
||||||
|
parentId: "connectors.admin.governed-configurations",
|
||||||
|
order: 30
|
||||||
|
},
|
||||||
|
{
|
||||||
|
id: "connectors.admin.external-service-desk",
|
||||||
|
moduleId: "connectors",
|
||||||
|
kind: "section",
|
||||||
|
label: "External service desk",
|
||||||
|
parentId: "connectors.admin.governed-configurations",
|
||||||
|
order: 40
|
||||||
}
|
}
|
||||||
],
|
],
|
||||||
uiCapabilities: {
|
uiCapabilities: {
|
||||||
|
|||||||
@@ -4,8 +4,15 @@
|
|||||||
gap: var(--space-4, 1rem);
|
gap: var(--space-4, 1rem);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
.connector-knowledge-page .connector-knowledge-list,
|
||||||
|
.connector-knowledge-page .connector-knowledge-detail {
|
||||||
|
display: grid;
|
||||||
|
gap: var(--space-4, 1rem);
|
||||||
|
}
|
||||||
|
|
||||||
.connector-governance-page .connector-revision-line,
|
.connector-governance-page .connector-revision-line,
|
||||||
.connector-governance-page .connector-run-actions {
|
.connector-governance-page .connector-run-actions,
|
||||||
|
.connector-knowledge-page .connector-revision-line {
|
||||||
align-items: center;
|
align-items: center;
|
||||||
display: flex;
|
display: flex;
|
||||||
flex-wrap: wrap;
|
flex-wrap: wrap;
|
||||||
@@ -13,7 +20,8 @@
|
|||||||
margin-bottom: var(--space-4, 1rem);
|
margin-bottom: var(--space-4, 1rem);
|
||||||
}
|
}
|
||||||
|
|
||||||
.connector-governance-page textarea {
|
.connector-governance-page textarea,
|
||||||
|
.connector-knowledge-page textarea {
|
||||||
font-family: var(--font-family-mono, ui-monospace, monospace);
|
font-family: var(--font-family-mono, ui-monospace, monospace);
|
||||||
resize: vertical;
|
resize: vertical;
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -4,6 +4,10 @@ import { readFileSync } from "node:fs";
|
|||||||
const moduleSource = readFileSync("src/module.ts", "utf8");
|
const moduleSource = readFileSync("src/module.ts", "utf8");
|
||||||
const page = readFileSync("src/features/ConnectorGovernancePage.tsx", "utf8");
|
const page = readFileSync("src/features/ConnectorGovernancePage.tsx", "utf8");
|
||||||
const api = readFileSync("src/api/governedConnectors.ts", "utf8");
|
const api = readFileSync("src/api/governedConnectors.ts", "utf8");
|
||||||
|
const knowledgePage = readFileSync("src/features/ExternalKnowledgePage.tsx", "utf8");
|
||||||
|
const knowledgeApi = readFileSync("src/api/externalKnowledge.ts", "utf8");
|
||||||
|
const serviceDeskPage = readFileSync("src/features/ExternalServiceDeskPage.tsx", "utf8");
|
||||||
|
const serviceDeskApi = readFileSync("src/api/externalServiceDesk.ts", "utf8");
|
||||||
|
|
||||||
assert.match(moduleSource, /"admin.sections": adminSections/);
|
assert.match(moduleSource, /"admin.sections": adminSections/);
|
||||||
assert.match(moduleSource, /connectors\.admin\.governed-configurations/);
|
assert.match(moduleSource, /connectors\.admin\.governed-configurations/);
|
||||||
@@ -21,5 +25,32 @@ assert.match(page, /Run simulation/);
|
|||||||
assert.match(page, /Review ambiguous connector result/);
|
assert.match(page, /Review ambiguous connector result/);
|
||||||
assert.match(api, /idempotency_key/);
|
assert.match(api, /idempotency_key/);
|
||||||
assert.match(api, /credential_ref/);
|
assert.match(api, /credential_ref/);
|
||||||
|
assert.match(moduleSource, /connectors\.admin\.external-knowledge/);
|
||||||
|
assert.match(knowledgePage, /<AdminPageLayout/);
|
||||||
|
assert.match(knowledgePage, /<PageActionBar/);
|
||||||
|
assert.match(knowledgePage, /refreshable/);
|
||||||
|
assert.match(knowledgePage, /saveAction=/);
|
||||||
|
assert.match(knowledgePage, /useUnsavedDraftGuard/);
|
||||||
|
assert.match(knowledgePage, /<WorkspaceLayout/);
|
||||||
|
assert.match(knowledgePage, /Run full backfill/);
|
||||||
|
assert.match(knowledgePage, /Preview migration/);
|
||||||
|
assert.match(knowledgePage, /outcome is unknown/);
|
||||||
|
assert.match(knowledgeApi, /\/knowledge/);
|
||||||
|
assert.match(knowledgeApi, /migration-dry-runs/);
|
||||||
|
assert.match(knowledgeApi, /\/publish/);
|
||||||
|
assert.match(moduleSource, /connectors\.admin\.external-service-desk/);
|
||||||
|
assert.match(serviceDeskPage, /<AdminPageLayout/);
|
||||||
|
assert.match(serviceDeskPage, /<PageActionBar/);
|
||||||
|
assert.match(serviceDeskPage, /refreshable/);
|
||||||
|
assert.match(serviceDeskPage, /saveAction=/);
|
||||||
|
assert.match(serviceDeskPage, /useUnsavedDraftGuard/);
|
||||||
|
assert.match(serviceDeskPage, /<WorkspaceLayout/);
|
||||||
|
assert.match(serviceDeskPage, /Restart full sync/);
|
||||||
|
assert.match(serviceDeskPage, /Submit revision-checked update/);
|
||||||
|
assert.match(serviceDeskPage, /outcome is unknown/);
|
||||||
|
assert.match(serviceDeskApi, /\/service-desk/);
|
||||||
|
assert.match(serviceDeskApi, /\/discover/);
|
||||||
|
assert.match(serviceDeskApi, /\/sync/);
|
||||||
|
assert.match(serviceDeskApi, /\/update/);
|
||||||
|
|
||||||
console.log("Connector governance UI structural contract passed.");
|
console.log("Connector governance UI structural contract passed.");
|
||||||
|
|||||||
Reference in New Issue
Block a user