Pin S3 and SMB connector peers

This commit is contained in:
2026-08-04 10:40:46 +02:00
parent 0c68e904cf
commit 92e649477f
15 changed files with 840 additions and 112 deletions
+30 -16
View File
@@ -405,15 +405,15 @@ in a development/test runtime.
| Seafile | Read-only native API browse/download-link import and manual sync implemented using the pinned HTTP transport; WebDAV opt-in supported |
| Nextcloud | Read-only WebDAV browse/import/manual sync implemented using the pinned HTTP transport |
| Generic WebDAV | Read-only browse/import/manual sync implemented using the pinned HTTP transport |
| SMB | Browse/import code and descriptor implemented, but every live connection fails closed until initial connections and DFS referrals can be policy-validated and pinned |
| S3 connector | Browse/import code and descriptor implemented, but every live SDK connection fails closed until botocore connections, redirects, and endpoint discovery can be validated and pinned |
| SMB | Read-only browse/import/manual sync implemented through a pinned smbprotocol transport for initial peers, reconnects, aliases, and DFS referral targets |
| S3 connector | Read-only bucket/prefix browse, import, and manual sync implemented through pinned botocore pools covering retries, redirects, endpoint discovery, and provider aliases |
| SharePoint and OneDrive | Provider keys/descriptors reserved; live Microsoft Graph browse/import is planned |
| NFS and local connector | Described as optional future providers; the local managed-storage backend is a different feature |
Provider descriptors are available from
`GET /api/v1/files/connectors/providers`. Use their `implemented`, `installed`,
and support fields for display, but expect a fail-closed transport error where
the table above says live access is disabled.
and support fields for display. An incompatible optional SDK release fails closed
before it can return a usable client or session.
## Operator runbook
@@ -478,17 +478,29 @@ The built-in HTTP transport:
- bounds structured responses to 16 MiB and file transfers to 512 MiB by
default.
The S3 SDK adapter applies the same socket rule to every botocore pool selected
for a retry, redirect, discovered endpoint, or virtual-host bucket alias. The
original authority remains in the request and TLS SNI/certificate check. S3
connector clients use no outbound proxy and never discover ambient AWS
credentials: configure both access and secret keys on the governed profile, or
use an anonymous profile for a public source.
The SMB adapter owns a separate connection cache and replaces smbprotocol's TCP
factory process-wide with the stricter pinned socket. Initial peers, reconnects,
server aliases, domain-controller connections, and DFS referral targets therefore
pass the same policy at connection time. Signing is required by default; enable
SMB encryption on the profile where the server supports it.
Override the connector response limits with
`GOVOPLAN_CONNECTOR_MAX_STRUCTURED_RESPONSE_BYTES` and
`GOVOPLAN_CONNECTOR_MAX_FILE_TRANSFER_BYTES`. The smaller applicable limit wins
when an import is also subject to `FILE_UPLOAD_MAX_BYTES`.
Never work around a connector pinning failure by adding a raw IP, disabling TLS,
or enabling private networks. SMB may redirect through DFS, and user-configured
S3 connectors may perform their own redirects or endpoint discovery; those
connector transports remain disabled until every connection peer can be
governed. The separately configured platform S3 backend is trusted only by the
deployment owner and is not selectable by a user or connector profile.
or enabling private networks. A failure means the peer policy rejected an actual
connection destination or the installed SDK no longer exposes the verified
transport seam. The separately configured platform S3 backend is trusted only by
the deployment owner and is not selectable by a user or connector profile.
### Backup and restore
@@ -620,7 +632,7 @@ Use these symptoms as routing hints:
| `Stored object does not exist` | Database/blob restore mismatch, wrong root, or missing shared storage |
| `Stored secret cannot be decrypted` | Wrong or rotated `MASTER_KEY_B64` |
| Private/non-public endpoint blocked | Deployment-wide egress policy is public-only or DNS returned a forbidden answer |
| SDK cannot pin redirects/referrals | Expected fail-closed S3/SMB boundary, not a transient connector outage |
| SDK peer-pinning seam is unavailable | Optional S3/SMB SDK is incompatible; keep access fail-closed and validate the supported dependency range before upgrade |
| Connector response exceeds limit | Remote payload exceeds connector or upload limit |
| Profile is not visible | Scope, disabled state, campaign access, or policy mismatch |
| Group removal is vetoed | The group still owns a file/folder/connector space or is a share target |
@@ -824,9 +836,11 @@ bypass the same rule.
### Pinned connector transport
Given public-only mode and DNS returns any private address, the connection must
be rejected before a socket opens. Given private mode, the HTTP connection must
still use the validated address and reject redirects. SMB and S3 must fail
before their SDK clients connect until all SDK-managed peers can be pinned.
be rejected before a socket opens. Given private mode, every HTTP, S3, and SMB
connection must still use an address from the answer validated for that exact
attempt. Botocore retries, redirects, endpoint discovery, and aliases, plus SMB
reconnects and DFS referrals, must pass through the pinned factories. A changed
or unsupported SDK seam must fail before a usable client/session is returned.
### Imported evidence and sync
@@ -867,7 +881,7 @@ returning different content or credentials.
| Deletion/retention | Soft-delete assets/folders/spaces; immediate audited connector-secret scrubbing | File restore API, hard purge, retention policy, legal hold, and blob GC ([#38](https://git.add-ideas.de/GovOPlaN/govoplan-files/issues/38)) |
| Connector governance | Scoped profiles/credentials/policies, effective source explanation, separate credentials, linked user/group spaces | Provider-owned external secret lifecycle; API `secret_ref` remains rejected |
| HTTP connectors | Pinned, bounded, no-redirect Seafile and WebDAV/Nextcloud browse/import/manual sync | Background/folder sync, remote mutation, long-running transfer workers |
| SMB and S3 connectors | Provider descriptors and browse/import logic | Live access only after SDK connections plus DFS referrals/redirects are validated and pinned ([SMB #35](https://git.add-ideas.de/GovOPlaN/govoplan-files/issues/35), [S3 #34](https://git.add-ideas.de/GovOPlaN/govoplan-files/issues/34)) |
| SMB and S3 connectors | Provider descriptors, browse/import/manual sync, pinned SDK transports, and redirect/retry/referral transport-contract tests | Live topology smoke evidence, provider-specific OAuth, remote writes, and background indexing remain separate deployment or connector-module concerns |
| Other providers | Reserved SharePoint/OneDrive keys and NFS/local descriptors | Graph/OAuth/provider paging, NFS deployment integration, DMS connectors |
| Connector spaces | User/group link, browse, manual selected-file sync, edit/disable/delete | Background sync, remote writes/deletes, full conflict-reporting jobs |
| Profile capabilities | Stored and displayed | Enforce capability flags as an independent operation gate |
@@ -890,8 +904,8 @@ Before releasing Files:
6. Exercise upload, ZIP bounds, conflict handling, download, and soft deletion.
7. Exercise connector policy explanation and one pinned HTTP provider where
configured; verify the deployment-managed or trusted external S3 backend if
selected, and verify user-configured S3 and SMB connector peers still fail
closed.
selected, and verify one configured S3 and SMB connector while recording the
actual target topology and private-network policy.
8. Verify credential deletion scrubs dependents and produces audit evidence.
9. Verify a campaign attachment snapshot still identifies its exact version and
checksum after the current file changes.