[Security] Re-enable S3 connectors only with pinned SDK peers and redirects #34

New Issue

2026-07-21T16:40:42+02:00

zemion commented

2026-07-21 16:40:42 +02:00

Outcome

S3-compatible connectors can be enabled without weakening the deployment-wide connector egress boundary. Every actual peer, redirect, endpoint-discovery result, and retry destination is authorized and DNS/IP pinned before the socket is opened.

Context

The initial S3 connector from #20 is intentionally fail-closed because boto3/botocore manages secondary connections that the current boundary cannot prove are pinned. This is a security follow-up, not a regression of the profile/catalog work already completed.

Acceptance

No boto3/botocore client is constructed while peer pinning cannot be enforced.
The implementation validates and pins the exact address used for every connection, including redirects, region/endpoint discovery, retries, and provider aliases.
TLS hostname verification and SNI retain the configured authority while the socket uses an approved address.
DNS rebinding, redirect-to-private, mixed-answer, and peer-change tests fail closed.
Deployment-wide private-network policy is applied consistently.
Operator documentation and target-provider tests describe supported topologies and residual limitations.

## Outcome S3-compatible connectors can be enabled without weakening the deployment-wide connector egress boundary. Every actual peer, redirect, endpoint-discovery result, and retry destination is authorized and DNS/IP pinned before the socket is opened. ## Context The initial S3 connector from #20 is intentionally fail-closed because boto3/botocore manages secondary connections that the current boundary cannot prove are pinned. This is a security follow-up, not a regression of the profile/catalog work already completed. ## Acceptance - No boto3/botocore client is constructed while peer pinning cannot be enforced. - The implementation validates and pins the exact address used for every connection, including redirects, region/endpoint discovery, retries, and provider aliases. - TLS hostname verification and SNI retain the configured authority while the socket uses an approved address. - DNS rebinding, redirect-to-private, mixed-answer, and peer-change tests fail closed. - Deployment-wide private-network policy is applied consistently. - Operator documentation and target-provider tests describe supported topologies and residual limitations.

zemion added the area/security codex/ready module/files

labels 2026-07-29 14:15:02 +02:00

zemion commented

2026-07-30 18:56:34 +02:00

A narrowly bounded installer integration has been implemented locally (pending push), without relaxing the fail-closed posture of this issue for arbitrary S3 connectors.

When and only when Core supplies the deployment-owned marker, Files accepts the exact internal endpoint http://garage:3900 and forces path-style addressing. Any other endpoint under that marker is rejected. Without the marker, the existing unpinned-SDK rejection remains unchanged.

Verified with unit tests and a live Garage 2.3 Files backend put/get/exists/stat/delete smoke. This issue remains open for real per-peer SDK pinning, redirects, retries, discovery, TLS authority preservation, and external S3 providers.

A narrowly bounded installer integration has been implemented locally (pending push), without relaxing the fail-closed posture of this issue for arbitrary S3 connectors. When and only when Core supplies the deployment-owned marker, Files accepts the exact internal endpoint http://garage:3900 and forces path-style addressing. Any other endpoint under that marker is rejected. Without the marker, the existing unpinned-SDK rejection remains unchanged. Verified with unit tests and a live Garage 2.3 Files backend put/get/exists/stat/delete smoke. This issue remains open for real per-peer SDK pinning, redirects, retries, discovery, TLS authority preservation, and external S3 providers.

zemion commented

2026-08-01 01:48:29 +02:00

Codex State: note

Summary

Clarified and implemented a separate deployment-storage boundary locally: external durable S3 is allowed only as an operator-selected clean HTTPS origin with FILE_STORAGE_S3_ENDPOINT_TRUSTED=true; Files delegates the backend to Core so independent runtime nodes share one namespace.
This does not relax S3 connector policy. User/configured connector browse/import remains fail-closed until every SDK peer, redirect, retry, and discovery target can be pinned as required by this issue.

Changed Files

src/govoplan_files/backend/storage/backends.py
docs/FILES_HANDBOOK.md

Verification

Files storage backend tests passed (6 tests, 2 subtests); full workspace focused gate passed

Next / Blocked

Keep open for the connector-specific peer-pinning acceptance criteria.

## Codex State: note ### Summary - Clarified and implemented a separate deployment-storage boundary locally: external durable S3 is allowed only as an operator-selected clean HTTPS origin with FILE_STORAGE_S3_ENDPOINT_TRUSTED=true; Files delegates the backend to Core so independent runtime nodes share one namespace. - This does not relax S3 connector policy. User/configured connector browse/import remains fail-closed until every SDK peer, redirect, retry, and discovery target can be pinned as required by this issue. ### Changed Files - `src/govoplan_files/backend/storage/backends.py` - `docs/FILES_HANDBOOK.md` ### Verification - `Files storage backend tests passed (6 tests, 2 subtests); full workspace focused gate passed` ### Next / Blocked - Keep open for the connector-specific peer-pinning acceptance criteria.

zemion commented

2026-08-01 02:04:17 +02:00

Codex State: progress

Summary

Deployment-owned external S3 now has an explicit trusted HTTPS-origin path for shared managed storage; user-configured S3 connectors remain on the fail-closed unpinned-SDK boundary.

Changed Files

src/govoplan_files/backend/storage/backends.py
docs/FILES_HANDBOOK.md

Verification

Files storage tests passed

Next / Blocked

Connector peer pinning remains required before relaxing the user-controlled S3 endpoint boundary.

Suggested status label: status/in-progress

## Codex State: progress ### Summary - Deployment-owned external S3 now has an explicit trusted HTTPS-origin path for shared managed storage; user-configured S3 connectors remain on the fail-closed unpinned-SDK boundary. ### Changed Files - `src/govoplan_files/backend/storage/backends.py` - `docs/FILES_HANDBOOK.md` ### Verification - `Files storage tests passed` ### Next / Blocked - Connector peer pinning remains required before relaxing the user-controlled S3 endpoint boundary. Suggested status label: `status/in-progress`

zemion commented

2026-08-04 10:41:33 +02:00

Codex State: done

Summary

Implemented exact connection-time peer pinning for every S3 SDK socket, including retries, redirects, endpoint discovery, and provider-selected aliases while retaining the original TLS authority.
Disabled outbound proxies and ambient credential discovery; incompatible botocore transport seams now fail closed.

Changed Files

src/govoplan_files/backend/storage/sdk_peer_pinning.py
src/govoplan_files/backend/storage/connector_browse.py
src/govoplan_files/backend/storage/connector_imports.py
docs/FILES_HANDBOOK.md

Verification

Files suite: 117 tests passed
Pinned S3 protocol, mixed-answer, peer-change, redirect/alias, and TLS-authority tests passed
Consolidated focused check passed, including 50 WebUI module permutations

## Codex State: done ### Summary - Implemented exact connection-time peer pinning for every S3 SDK socket, including retries, redirects, endpoint discovery, and provider-selected aliases while retaining the original TLS authority. - Disabled outbound proxies and ambient credential discovery; incompatible botocore transport seams now fail closed. ### Changed Files - `src/govoplan_files/backend/storage/sdk_peer_pinning.py` - `src/govoplan_files/backend/storage/connector_browse.py` - `src/govoplan_files/backend/storage/connector_imports.py` - `docs/FILES_HANDBOOK.md` ### Verification - `Files suite: 117 tests passed` - `Pinned S3 protocol, mixed-answer, peer-change, redirect/alias, and TLS-authority tests passed` - `Consolidated focused check passed, including 50 WebUI module permutations`

zemion closed this issue

2026-08-04 10:41:33 +02:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: GovOPlaN/govoplan-files#34