125 خطوط
8.9 KiB
Markdown
125 خطوط
8.9 KiB
Markdown
# Private S3 object storage
|
|
|
|
Schematic and PCB uploads, comparison manifests, normalized JSON and SVG artifacts use the same storage adapter. Configure it once for both API and worker. No API endpoint, viewer URL, token or database schema changes are required.
|
|
|
|
## Configure `.env`
|
|
|
|
```dotenv
|
|
STORAGE_BACKEND=s3
|
|
S3_BUCKET=your-private-bucket
|
|
S3_PREFIX=altium/production
|
|
S3_ENDPOINT_URL=https://your-s3-service.example
|
|
S3_REGION=us-east-1
|
|
S3_ADDRESSING_STYLE=path
|
|
AWS_ACCESS_KEY_ID=your-access-key
|
|
AWS_SECRET_ACCESS_KEY=your-secret-key
|
|
S3_CONNECT_TIMEOUT=5
|
|
S3_READ_TIMEOUT=60
|
|
S3_MAX_ATTEMPTS=3
|
|
S3_SERVER_SIDE_ENCRYPTION=
|
|
S3_KMS_KEY_ID=
|
|
```
|
|
|
|
Replace the example values with your provider's actual bucket, service endpoint, credentials and region. The bucket must already exist. The application never creates a bucket, changes its ACL/policy or makes objects public. Keep the bucket private. API-controlled access and expiry continue to apply; the browser does not connect directly to S3 or receive storage credentials.
|
|
|
|
For AWS S3, leave `S3_ENDPOINT_URL` empty and choose the bucket's region; `auto` addressing is normally suitable. For MinIO and S3-compatible providers use the provider's endpoint and addressing style (often `path`). Do not put the bucket name, access credentials, query parameters or a public asset URL into `S3_ENDPOINT_URL`.
|
|
|
|
| Variable | Default / behavior |
|
|
| --- | --- |
|
|
| `STORAGE_BACKEND` | `auto`: S3 if `S3_BUCKET` is nonempty, local otherwise. `s3` requires a bucket and never falls back to disk. `local` explicitly selects local files. |
|
|
| `S3_BUCKET` | Existing private bucket name, not a URL |
|
|
| `S3_PREFIX` | Empty for backward compatibility; optional namespace, e.g. `altium/production` |
|
|
| `S3_ENDPOINT_URL` | Empty uses the AWS endpoint selected by the SDK |
|
|
| `S3_REGION` | Falls back to `AWS_DEFAULT_REGION`, then `us-east-1` |
|
|
| `S3_ADDRESSING_STYLE` | `auto`, `path` or `virtual`; default `auto` |
|
|
| `S3_CONNECT_TIMEOUT` | Positive integer seconds, default 5 |
|
|
| `S3_READ_TIMEOUT` | Positive integer seconds, default 60 |
|
|
| `S3_MAX_ATTEMPTS` | Positive integer, default 3 total attempts including initial request; standard SDK retry mode |
|
|
| `AWS_ACCESS_KEY_ID` | Standard SDK credential chain; no credential is embedded in source |
|
|
| `AWS_SECRET_ACCESS_KEY` | Standard SDK credential chain |
|
|
| `AWS_SESSION_TOKEN` | Optional for temporary credentials |
|
|
| `AWS_CA_BUNDLE` | Optional trusted custom CA PEM path accessible to API and worker; certificate verification remains enabled |
|
|
| `S3_SERVER_SIDE_ENCRYPTION` | Empty uses provider defaults; optional `AES256` or `aws:kms` |
|
|
| `S3_KMS_KEY_ID` | Optional KMS key/alias, valid only with `aws:kms`; requires provider support and key permissions |
|
|
|
|
IAM/workload credentials can be used instead of static keys. Do not set empty static credentials when using an IAM role. Keep `.env` private and ignored by Git; do not put credentials in frontend code or `VITE_*` variables. The parser subprocess inherits neither AWS credentials nor storage configuration.
|
|
|
|
Some compatible providers do not implement newer optional AWS checksum behavior. Where the provider documents this requirement, the SDK's standard `AWS_REQUEST_CHECKSUM_CALCULATION=when_required` / `AWS_RESPONSE_CHECKSUM_VALIDATION=when_required` environment settings can be used without changing application code.
|
|
|
|
Prefix rules: leading/trailing `/` is stripped; internal empty segments, `.`, `..`, backslashes, controls and prefixes exceeding 512 UTF-8 bytes are rejected. The prefix is a server-controlled namespace, never supplied by a request. Generated suffixes retain the existing strict allowlist. For example:
|
|
|
|
```text
|
|
s3://your-private-bucket/altium/production/{id}/input.SchDoc
|
|
s3://your-private-bucket/altium/production/{id}/input.PcbDoc
|
|
s3://your-private-bucket/altium/production/{id}/comparison.json
|
|
s3://your-private-bucket/altium/production/{id}/pcb-comparison.json
|
|
s3://your-private-bucket/altium/production/{id}/old.PcbDoc
|
|
s3://your-private-bucket/altium/production/{id}/new.PcbDoc
|
|
s3://your-private-bucket/altium/production/{id}/attempt-1/data.json
|
|
s3://your-private-bucket/altium/production/{id}/attempt-1/drawing.svg
|
|
s3://your-private-bucket/altium/production/{id}/attempt-1/old.svg
|
|
s3://your-private-bucket/altium/production/{id}/attempt-1/new.svg
|
|
```
|
|
|
|
The database continues storing logical keys such as `{id}/input.PcbDoc`. Prefix application happens exactly once, at the S3 put/get/delete boundary. Expiry cleanup uses the same mapping and never lists or bulk-deletes a bucket or another prefix.
|
|
|
|
## What stays outside S3
|
|
|
|
S3 replaces persistent **document object storage**, not the database. PostgreSQL still stores jobs, authorization hashes, expiry, queue leases and status. SQLite development databases remain local. Keep the same database when switching backends so existing IDs/tokens remain valid.
|
|
|
|
Multipart spooling and resource-limited parsing still use temporary local files, normally under `/tmp`, and remove them after processing. They are not durable artifact storage. In S3 mode no `STORAGE_DIR/objects` files are created/read as fallback. With PostgreSQL the adapter does not require a local storage directory. Existing Compose data volumes are retained for safe migration/rollback; they no longer receive document objects once S3 is selected. Do not delete a previous volume until its objects have been migrated and backups verified.
|
|
|
|
The bucket/prefix must be identical on all API and worker replicas. Bucket errors propagate through the existing storage error responses or durable failed-job status. An S3 outage does not switch new uploads to local storage. Object retention/lifecycle policies must not remove data earlier than the application link lifetime; database rows and S3 objects should be backed up together.
|
|
|
|
Grant the service object permissions scoped to the chosen namespace: `s3:PutObject`, `s3:GetObject` and `s3:DeleteObject`. The application does not require public ACLs or bucket-wide listing for normal operations. Encryption/KMS may require additional provider permissions. Administrative backup and lifecycle tooling can have separate permissions.
|
|
|
|
## Existing files and migration
|
|
|
|
Changing backend/bucket/prefix does **not** move existing objects automatically. Existing S3 deployments keep working with `STORAGE_BACKEND=auto` and empty `S3_PREFIX`. If introducing a prefix for previously unprefixed S3 objects, first copy the existing keys to that namespace using your provider's tools while retaining the original suffixes.
|
|
|
|
For local-to-S3 migration, `scripts/migrate-to-s3.py`:
|
|
|
|
- Reads only allowlisted application objects from the original `STORAGE_DIR/objects` tree.
|
|
- Copies into the configured private S3 namespace and reads each copy back to verify its SHA-256.
|
|
- Skips an already identical destination; refuses to overwrite different bytes.
|
|
- Rejects symlinks/unrecognized paths and stops on provider errors.
|
|
- Never deletes local files or changes database rows. Completed copies remain if a later object fails, allowing a safe rerun.
|
|
|
|
Stop admission of uploads and all workers/cleanup while copying so neither side changes under migration. Keep the original database and object volume.
|
|
|
|
For local execution, load the edited `.env` and point to the **old** storage directory:
|
|
|
|
```bash
|
|
set -a
|
|
source .env
|
|
set +a
|
|
PYTHONPATH=backend .venv/bin/python scripts/migrate-to-s3.py --source-dir ./data
|
|
```
|
|
|
|
For the existing Compose deployment (`/data` is the old named volume):
|
|
|
|
```bash
|
|
docker compose stop api worker
|
|
docker compose build api worker
|
|
docker compose run --rm --no-deps \
|
|
-v "$PWD/scripts:/tools:ro" \
|
|
api python /tools/migrate-to-s3.py --source-dir /data
|
|
docker compose up -d api worker
|
|
```
|
|
|
|
Only proceed to the final startup after the copy command succeeds. Do not run production workers against new S3 settings before old objects have been copied; otherwise already queued jobs cannot find their sources.
|
|
|
|
For a new installation with no old local objects, migration is unnecessary:
|
|
|
|
```bash
|
|
docker compose up -d --build api worker
|
|
```
|
|
|
|
Compose already passes `.env` to both services. `./scripts/dev.sh` also exports `.env` for both local processes. Direct `uvicorn`/worker commands need the variables exported explicitly. Editing `.env` requires recreating Compose containers (a plain `docker compose restart` does not load new environment values).
|
|
|
|
After switching, upload a SchDoc and PcbDoc, check their status and private SVGs, and open an existing pre-migration link. Confirm objects appear only under the configured bucket/prefix. No real provider was configured during implementation; protocol tests use botocore stubs and the end-to-end API/worker tests use a fake S3 transport with the real parsers/renderers.
|
|
|
|
## SDK references
|
|
|
|
[Configuration](https://docs.aws.amazon.com/boto3/latest/guide/configuration.html), [credential discovery](https://docs.aws.amazon.com/boto3/latest/guide/credentials.html), and [put_object/encryption parameters](https://docs.aws.amazon.com/boto3/latest/reference/services/s3/client/put_object.html). The application retains its existing pinned boto3 runtime dependency.
|