altium-render-engine/docs/S3.md

Private S3 object storage

Schematic and PCB uploads, comparison manifests, normalized JSON and SVG artifacts use the same storage adapter. Configure it once for both API and worker. No API endpoint, viewer URL, token or database schema changes are required.

Configure .env

STORAGE_BACKEND=s3
S3_BUCKET=your-private-bucket
S3_PREFIX=altium/production
S3_ENDPOINT_URL=https://your-s3-service.example
S3_REGION=us-east-1
S3_ADDRESSING_STYLE=path
AWS_ACCESS_KEY_ID=your-access-key
AWS_SECRET_ACCESS_KEY=your-secret-key
S3_CONNECT_TIMEOUT=5
S3_READ_TIMEOUT=60
S3_MAX_ATTEMPTS=3
S3_SERVER_SIDE_ENCRYPTION=
S3_KMS_KEY_ID=

Replace the example values with your provider's actual bucket, service endpoint, credentials and region. The bucket must already exist. The application never creates a bucket, changes its ACL/policy or makes objects public. Keep the bucket private. API-controlled access and expiry continue to apply; the browser does not connect directly to S3 or receive storage credentials.

For AWS S3, leave S3_ENDPOINT_URL empty and choose the bucket's region; auto addressing is normally suitable. For MinIO and S3-compatible providers use the provider's endpoint and addressing style (often path). Do not put the bucket name, access credentials, query parameters or a public asset URL into S3_ENDPOINT_URL.

Variable	Default / behavior
STORAGE_BACKEND	auto: S3 if S3_BUCKET is nonempty, local otherwise. s3 requires a bucket and never falls back to disk. local explicitly selects local files.
S3_BUCKET	Existing private bucket name, not a URL
S3_PREFIX	Empty for backward compatibility; optional namespace, e.g. altium/production
S3_ENDPOINT_URL	Empty uses the AWS endpoint selected by the SDK
S3_REGION	Falls back to AWS_DEFAULT_REGION, then us-east-1
S3_ADDRESSING_STYLE	auto, path or virtual; default auto
S3_CONNECT_TIMEOUT	Positive integer seconds, default 5
S3_READ_TIMEOUT	Positive integer seconds, default 60
S3_MAX_ATTEMPTS	Positive integer, default 3 total attempts including initial request; standard SDK retry mode
AWS_ACCESS_KEY_ID	Standard SDK credential chain; no credential is embedded in source
AWS_SECRET_ACCESS_KEY	Standard SDK credential chain
AWS_SESSION_TOKEN	Optional for temporary credentials
AWS_CA_BUNDLE	Optional trusted custom CA PEM path accessible to API and worker; certificate verification remains enabled
S3_SERVER_SIDE_ENCRYPTION	Empty uses provider defaults; optional AES256 or aws:kms
S3_KMS_KEY_ID	Optional KMS key/alias, valid only with aws:kms; requires provider support and key permissions

IAM/workload credentials can be used instead of static keys. Do not set empty static credentials when using an IAM role. Keep .env private and ignored by Git; do not put credentials in frontend code or VITE_* variables. The parser subprocess inherits neither AWS credentials nor storage configuration.

Some compatible providers do not implement newer optional AWS checksum behavior. Where the provider documents this requirement, the SDK's standard AWS_REQUEST_CHECKSUM_CALCULATION=when_required / AWS_RESPONSE_CHECKSUM_VALIDATION=when_required environment settings can be used without changing application code.

Prefix rules: leading/trailing / is stripped; internal empty segments, ., .., backslashes, controls and prefixes exceeding 512 UTF-8 bytes are rejected. The prefix is a server-controlled namespace, never supplied by a request. Generated suffixes retain the existing strict allowlist. For example:

s3://your-private-bucket/altium/production/{id}/input.SchDoc
s3://your-private-bucket/altium/production/{id}/input.PcbDoc
s3://your-private-bucket/altium/production/{id}/comparison.json
s3://your-private-bucket/altium/production/{id}/pcb-comparison.json
s3://your-private-bucket/altium/production/{id}/old.PcbDoc
s3://your-private-bucket/altium/production/{id}/new.PcbDoc
s3://your-private-bucket/altium/production/{id}/attempt-1/data.json
s3://your-private-bucket/altium/production/{id}/attempt-1/drawing.svg
s3://your-private-bucket/altium/production/{id}/attempt-1/old.svg
s3://your-private-bucket/altium/production/{id}/attempt-1/new.svg

The database continues storing logical keys such as {id}/input.PcbDoc. Prefix application happens exactly once, at the S3 put/get/delete boundary. Expiry cleanup uses the same mapping and never lists or bulk-deletes a bucket or another prefix.

What stays outside S3

S3 replaces persistent document object storage, not the database. PostgreSQL still stores jobs, authorization hashes, expiry, queue leases and status. SQLite development databases remain local. Keep the same database when switching backends so existing IDs/tokens remain valid.

Multipart spooling and resource-limited parsing still use temporary local files, normally under /tmp, and remove them after processing. They are not durable artifact storage. In S3 mode no STORAGE_DIR/objects files are created/read as fallback. With PostgreSQL the adapter does not require a local storage directory. Existing Compose data volumes are retained for safe migration/rollback; they no longer receive document objects once S3 is selected. Do not delete a previous volume until its objects have been migrated and backups verified.

The bucket/prefix must be identical on all API and worker replicas. Bucket errors propagate through the existing storage error responses or durable failed-job status. An S3 outage does not switch new uploads to local storage. Object retention/lifecycle policies must not remove data earlier than the application link lifetime; database rows and S3 objects should be backed up together.

Grant the service object permissions scoped to the chosen namespace: s3:PutObject, s3:GetObject and s3:DeleteObject. The application does not require public ACLs or bucket-wide listing for normal operations. Encryption/KMS may require additional provider permissions. Administrative backup and lifecycle tooling can have separate permissions.

Existing files and migration

Changing backend/bucket/prefix does not move existing objects automatically. Existing S3 deployments keep working with STORAGE_BACKEND=auto and empty S3_PREFIX. If introducing a prefix for previously unprefixed S3 objects, first copy the existing keys to that namespace using your provider's tools while retaining the original suffixes.

For local-to-S3 migration, scripts/migrate-to-s3.py:

• Reads only allowlisted application objects from the original STORAGE_DIR/objects tree.
• Copies into the configured private S3 namespace and reads each copy back to verify its SHA-256.
• Skips an already identical destination; refuses to overwrite different bytes.
• Rejects symlinks/unrecognized paths and stops on provider errors.
• Never deletes local files or changes database rows. Completed copies remain if a later object fails, allowing a safe rerun.

Stop admission of uploads and all workers/cleanup while copying so neither side changes under migration. Keep the original database and object volume.

For local execution, load the edited .env and point to the old storage directory:

set -a
source .env
set +a
PYTHONPATH=backend .venv/bin/python scripts/migrate-to-s3.py --source-dir ./data

For the existing Compose deployment (/data is the old named volume):

docker compose stop api worker
docker compose build api worker
docker compose run --rm --no-deps \
  -v "$PWD/scripts:/tools:ro" \
  api python /tools/migrate-to-s3.py --source-dir /data
docker compose up -d api worker

Only proceed to the final startup after the copy command succeeds. Do not run production workers against new S3 settings before old objects have been copied; otherwise already queued jobs cannot find their sources.

For a new installation with no old local objects, migration is unnecessary:

docker compose up -d --build api worker

Compose already passes .env to both services. ./scripts/dev.sh also exports .env for both local processes. Direct uvicorn/worker commands need the variables exported explicitly. Editing .env requires recreating Compose containers (a plain docker compose restart does not load new environment values).

After switching, upload a SchDoc and PcbDoc, check their status and private SVGs, and open an existing pre-migration link. Confirm objects appear only under the configured bucket/prefix. No real provider was configured during implementation; protocol tests use botocore stubs and the end-to-end API/worker tests use a fake S3 transport with the real parsers/renderers.

SDK references

Configuration, credential discovery, and put_object/encryption parameters. The application retains its existing pinned boto3 runtime dependency.