diff --git a/CHANGELOG.md b/CHANGELOG.md index 47e3983..ae06034 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,14 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] ### Added +- Added presigned URL support for direct S3 downloads (#48) +- Added `ORCHARD_DOWNLOAD_MODE` config option (`presigned`, `redirect`, `proxy`) (#48) +- Added `ORCHARD_PRESIGNED_URL_EXPIRY` config option (default: 3600 seconds) (#48) +- Added `?mode=` query parameter to override download mode per-request (#48) +- Added `/api/v1/project/{project}/{package}/+/{ref}/url` endpoint for getting presigned URLs (#48) +- Added `PresignedUrlResponse` schema with URL, expiry, checksums, and artifact metadata (#48) +- Added MinIO ingress support in Helm chart for presigned URL access (#48) +- Added `orchard.download.mode` and `orchard.download.presignedUrlExpiry` Helm values (#48) - Added integrity verification workflow design document (#24) - Added `sha256` field to API responses for clarity (alias of `id`) (#25) - Added `checksum_sha1` field to artifacts table for compatibility (#25) @@ -14,6 +22,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - Compute and store MD5, SHA1, and S3 ETag alongside SHA256 during upload (#25) - Added `Dockerfile.local` and `docker-compose.local.yml` for local development (#25) - Added migration script `003_checksum_fields.sql` for existing databases (#25) +### Changed +- Changed default download mode from `proxy` to `presigned` for better performance (#48) ## [0.2.0] - 2025-12-15 ### Changed diff --git a/README.md b/README.md index d1f06c2..2448b4e 100644 --- a/README.md +++ b/README.md @@ -60,7 +60,8 @@ Orchard is a centralized binary artifact storage system that provides content-ad | `GET` | `/api/v1/project/:project/packages/:package` | Get single package with metadata | | `POST` | `/api/v1/project/:project/packages` | Create a new package | | `POST` | `/api/v1/project/:project/:package/upload` | Upload an artifact | -| `GET` | `/api/v1/project/:project/:package/+/:ref` | Download an artifact (supports Range header) | +| `GET` | `/api/v1/project/:project/:package/+/:ref` | Download an artifact (supports Range header, mode param) | +| `GET` | `/api/v1/project/:project/:package/+/:ref/url` | Get presigned URL for direct S3 download | | `HEAD` | `/api/v1/project/:project/:package/+/:ref` | Get artifact metadata without downloading | | `GET` | `/api/v1/project/:project/:package/tags` | List tags (with pagination, search, sorting, artifact metadata) | | `POST` | `/api/v1/project/:project/:package/tags` | Create a tag | @@ -292,6 +293,12 @@ curl -H "Range: bytes=0-1023" http://localhost:8080/api/v1/project/my-project/re # Check file info without downloading (HEAD request) curl -I http://localhost:8080/api/v1/project/my-project/releases/+/v1.0.0 + +# Download with specific mode (presigned, redirect, or proxy) +curl "http://localhost:8080/api/v1/project/my-project/releases/+/v1.0.0?mode=proxy" + +# Get presigned URL for direct S3 download +curl http://localhost:8080/api/v1/project/my-project/releases/+/v1.0.0/url ``` > **Note on curl flags:** @@ -300,6 +307,33 @@ curl -I http://localhost:8080/api/v1/project/my-project/releases/+/v1.0.0 > - `-OJ` combines both: download to a file using the server-provided filename > - `-o ` saves to a specific filename you choose +#### Download Modes + +Orchard supports three download modes, configurable via `ORCHARD_DOWNLOAD_MODE` or per-request with `?mode=`: + +| Mode | Description | Use Case | +|------|-------------|----------| +| `presigned` (default) | Returns JSON with a presigned S3 URL | Clients that handle redirects themselves, web UIs | +| `redirect` | Returns HTTP 302 redirect to presigned S3 URL | Simple clients, browsers, wget | +| `proxy` | Streams content through the backend | When S3 isn't directly accessible to clients | + +**Presigned URL Response:** +```json +{ + "url": "https://minio.example.com/bucket/...", + "expires_at": "2025-01-01T01:00:00Z", + "method": "GET", + "artifact_id": "a3f5d8e...", + "size": 1048576, + "content_type": "application/gzip", + "original_name": "app-v1.0.0.tar.gz", + "checksum_sha256": "a3f5d8e...", + "checksum_md5": "d41d8cd..." +} +``` + +> **Note:** For presigned URLs to work, clients must be able to reach the S3 endpoint directly. In Kubernetes, this requires exposing MinIO via ingress (see Helm configuration below). + ### Create a Tag ```bash @@ -485,6 +519,8 @@ Configuration is provided via environment variables prefixed with `ORCHARD_`: | `ORCHARD_S3_BUCKET` | S3 bucket name | `orchard-artifacts` | | `ORCHARD_S3_ACCESS_KEY_ID` | S3 access key | - | | `ORCHARD_S3_SECRET_ACCESS_KEY` | S3 secret key | - | +| `ORCHARD_DOWNLOAD_MODE` | Download mode: `presigned`, `redirect`, or `proxy` | `presigned` | +| `ORCHARD_PRESIGNED_URL_EXPIRY` | Presigned URL expiry in seconds | `3600` | ## Kubernetes Deployment @@ -505,6 +541,32 @@ helm install orchard ./helm/orchard -n orchard --create-namespace helm install orchard ./helm/orchard -f my-values.yaml ``` +### Helm Configuration + +Key configuration options in `values.yaml`: + +```yaml +orchard: + # Download configuration + download: + mode: "presigned" # presigned, redirect, or proxy + presignedUrlExpiry: 3600 + +# MinIO ingress (required for presigned URL downloads) +minio: + ingress: + enabled: true + className: "nginx" + annotations: + cert-manager.io/cluster-issuer: "letsencrypt" + host: "minio.your-domain.com" + tls: + enabled: true + secretName: minio-tls +``` + +When `minio.ingress.enabled` is `true`, the S3 endpoint automatically uses the external URL (`https://minio.your-domain.com`), making presigned URLs accessible to external clients. + See `helm/orchard/values.yaml` for all configuration options. ## Database Schema diff --git a/backend/app/config.py b/backend/app/config.py index a90dceb..db396fb 100644 --- a/backend/app/config.py +++ b/backend/app/config.py @@ -32,6 +32,10 @@ class Settings(BaseSettings): s3_secret_access_key: str = "" s3_use_path_style: bool = True + # Download settings + download_mode: str = "presigned" # "presigned", "redirect", or "proxy" + presigned_url_expiry: int = 3600 # Presigned URL expiry in seconds (default: 1 hour) + @property def database_url(self) -> str: sslmode = f"?sslmode={self.database_sslmode}" if self.database_sslmode else "" diff --git a/backend/app/routes.py b/backend/app/routes.py index cc6a908..73f2cf3 100644 --- a/backend/app/routes.py +++ b/backend/app/routes.py @@ -1,9 +1,9 @@ -from datetime import datetime +from datetime import datetime, timedelta, timezone from fastapi import APIRouter, Depends, HTTPException, UploadFile, File, Form, Request, Query, Header, Response -from fastapi.responses import StreamingResponse +from fastapi.responses import StreamingResponse, RedirectResponse from sqlalchemy.orm import Session from sqlalchemy import or_, func -from typing import List, Optional +from typing import List, Optional, Literal import math import re import io @@ -29,8 +29,10 @@ from .schemas import ( ResumableUploadCompleteResponse, ResumableUploadStatusResponse, GlobalSearchResponse, SearchResultProject, SearchResultPackage, SearchResultArtifact, + PresignedUrlResponse, ) from .metadata import extract_metadata +from .config import get_settings router = APIRouter() @@ -844,27 +846,13 @@ def get_upload_status( raise HTTPException(status_code=404, detail=str(e)) -# Download artifact with range request support -@router.get("/api/v1/project/{project_name}/{package_name}/+/{ref}") -def download_artifact( - project_name: str, - package_name: str, +# Helper function to resolve artifact reference +def _resolve_artifact_ref( ref: str, - request: Request, - db: Session = Depends(get_db), - storage: S3Storage = Depends(get_storage), - range: Optional[str] = Header(None), -): - # Get project and package - project = db.query(Project).filter(Project.name == project_name).first() - if not project: - raise HTTPException(status_code=404, detail="Project not found") - - package = db.query(Package).filter(Package.project_id == project.id, Package.name == package_name).first() - if not package: - raise HTTPException(status_code=404, detail="Package not found") - - # Resolve reference to artifact + package: Package, + db: Session, +) -> Optional[Artifact]: + """Resolve a reference (tag name, artifact:hash, tag:name) to an artifact""" artifact = None # Check for explicit prefixes @@ -885,11 +873,76 @@ def download_artifact( # Try as direct artifact ID artifact = db.query(Artifact).filter(Artifact.id == ref).first() + return artifact + + +# Download artifact with range request support and download modes +@router.get("/api/v1/project/{project_name}/{package_name}/+/{ref}") +def download_artifact( + project_name: str, + package_name: str, + ref: str, + request: Request, + db: Session = Depends(get_db), + storage: S3Storage = Depends(get_storage), + range: Optional[str] = Header(None), + mode: Optional[Literal["proxy", "redirect", "presigned"]] = Query( + default=None, + description="Download mode: proxy (stream through backend), redirect (302 to presigned URL), presigned (return JSON with URL)" + ), +): + settings = get_settings() + + # Get project and package + project = db.query(Project).filter(Project.name == project_name).first() + if not project: + raise HTTPException(status_code=404, detail="Project not found") + + package = db.query(Package).filter(Package.project_id == project.id, Package.name == package_name).first() + if not package: + raise HTTPException(status_code=404, detail="Package not found") + + # Resolve reference to artifact + artifact = _resolve_artifact_ref(ref, package, db) if not artifact: raise HTTPException(status_code=404, detail="Artifact not found") filename = artifact.original_name or f"{artifact.id}" + # Determine download mode (query param overrides server default) + download_mode = mode or settings.download_mode + + # Handle presigned mode - return JSON with presigned URL + if download_mode == "presigned": + presigned_url = storage.generate_presigned_url( + artifact.s3_key, + response_content_type=artifact.content_type, + response_content_disposition=f'attachment; filename="{filename}"', + ) + expires_at = datetime.now(timezone.utc) + timedelta(seconds=settings.presigned_url_expiry) + + return PresignedUrlResponse( + url=presigned_url, + expires_at=expires_at, + method="GET", + artifact_id=artifact.id, + size=artifact.size, + content_type=artifact.content_type, + original_name=artifact.original_name, + checksum_sha256=artifact.id, + checksum_md5=artifact.checksum_md5, + ) + + # Handle redirect mode - return 302 redirect to presigned URL + if download_mode == "redirect": + presigned_url = storage.generate_presigned_url( + artifact.s3_key, + response_content_type=artifact.content_type, + response_content_disposition=f'attachment; filename="{filename}"', + ) + return RedirectResponse(url=presigned_url, status_code=302) + + # Proxy mode (default fallback) - stream through backend # Handle range requests if range: stream, content_length, content_range = storage.get_stream(artifact.s3_key, range) @@ -923,6 +976,63 @@ def download_artifact( ) +# Get presigned URL endpoint (explicit endpoint for getting URL without redirect) +@router.get("/api/v1/project/{project_name}/{package_name}/+/{ref}/url", response_model=PresignedUrlResponse) +def get_artifact_url( + project_name: str, + package_name: str, + ref: str, + db: Session = Depends(get_db), + storage: S3Storage = Depends(get_storage), + expiry: Optional[int] = Query( + default=None, + description="Custom expiry time in seconds (defaults to server setting)" + ), +): + """ + Get a presigned URL for direct S3 download. + This endpoint always returns a presigned URL regardless of server download mode. + """ + settings = get_settings() + + # Get project and package + project = db.query(Project).filter(Project.name == project_name).first() + if not project: + raise HTTPException(status_code=404, detail="Project not found") + + package = db.query(Package).filter(Package.project_id == project.id, Package.name == package_name).first() + if not package: + raise HTTPException(status_code=404, detail="Package not found") + + # Resolve reference to artifact + artifact = _resolve_artifact_ref(ref, package, db) + if not artifact: + raise HTTPException(status_code=404, detail="Artifact not found") + + filename = artifact.original_name or f"{artifact.id}" + url_expiry = expiry or settings.presigned_url_expiry + + presigned_url = storage.generate_presigned_url( + artifact.s3_key, + expiry=url_expiry, + response_content_type=artifact.content_type, + response_content_disposition=f'attachment; filename="{filename}"', + ) + expires_at = datetime.now(timezone.utc) + timedelta(seconds=url_expiry) + + return PresignedUrlResponse( + url=presigned_url, + expires_at=expires_at, + method="GET", + artifact_id=artifact.id, + size=artifact.size, + content_type=artifact.content_type, + original_name=artifact.original_name, + checksum_sha256=artifact.id, + checksum_md5=artifact.checksum_md5, + ) + + # HEAD request for download (to check file info without downloading) @router.head("/api/v1/project/{project_name}/{package_name}/+/{ref}") def head_artifact( @@ -941,23 +1051,8 @@ def head_artifact( if not package: raise HTTPException(status_code=404, detail="Package not found") - # Resolve reference to artifact (same logic as download) - artifact = None - if ref.startswith("artifact:"): - artifact_id = ref[9:] - artifact = db.query(Artifact).filter(Artifact.id == artifact_id).first() - elif ref.startswith("tag:") or ref.startswith("version:"): - tag_name = ref.split(":", 1)[1] - tag = db.query(Tag).filter(Tag.package_id == package.id, Tag.name == tag_name).first() - if tag: - artifact = db.query(Artifact).filter(Artifact.id == tag.artifact_id).first() - else: - tag = db.query(Tag).filter(Tag.package_id == package.id, Tag.name == ref).first() - if tag: - artifact = db.query(Artifact).filter(Artifact.id == tag.artifact_id).first() - else: - artifact = db.query(Artifact).filter(Artifact.id == ref).first() - + # Resolve reference to artifact + artifact = _resolve_artifact_ref(ref, package, db) if not artifact: raise HTTPException(status_code=404, detail="Artifact not found") diff --git a/backend/app/schemas.py b/backend/app/schemas.py index 48ee2dd..dcc7470 100644 --- a/backend/app/schemas.py +++ b/backend/app/schemas.py @@ -330,6 +330,20 @@ class GlobalSearchResponse(BaseModel): counts: Dict[str, int] # Total counts for each type +# Presigned URL response +class PresignedUrlResponse(BaseModel): + """Response containing a presigned URL for direct S3 download""" + url: str + expires_at: datetime + method: str = "GET" + artifact_id: str + size: int + content_type: Optional[str] = None + original_name: Optional[str] = None + checksum_sha256: Optional[str] = None + checksum_md5: Optional[str] = None + + # Health check class HealthResponse(BaseModel): status: str diff --git a/backend/app/storage.py b/backend/app/storage.py index e314cb7..ef0c510 100644 --- a/backend/app/storage.py +++ b/backend/app/storage.py @@ -450,6 +450,46 @@ class S3Storage: except ClientError: return False + def generate_presigned_url( + self, + s3_key: str, + expiry: Optional[int] = None, + response_content_type: Optional[str] = None, + response_content_disposition: Optional[str] = None, + ) -> str: + """ + Generate a presigned URL for downloading an object. + + Args: + s3_key: The S3 key of the object + expiry: URL expiry in seconds (defaults to settings.presigned_url_expiry) + response_content_type: Override Content-Type header in response + response_content_disposition: Override Content-Disposition header in response + + Returns: + Presigned URL string + """ + if expiry is None: + expiry = settings.presigned_url_expiry + + params = { + "Bucket": self.bucket, + "Key": s3_key, + } + + # Add response header overrides if specified + if response_content_type: + params["ResponseContentType"] = response_content_type + if response_content_disposition: + params["ResponseContentDisposition"] = response_content_disposition + + url = self.client.generate_presigned_url( + "get_object", + Params=params, + ExpiresIn=expiry, + ) + return url + # Singleton instance _storage = None diff --git a/helm/orchard/templates/_helpers.tpl b/helm/orchard/templates/_helpers.tpl index 847ed56..ba58ae7 100644 --- a/helm/orchard/templates/_helpers.tpl +++ b/helm/orchard/templates/_helpers.tpl @@ -97,10 +97,27 @@ password {{- end }} {{/* -MinIO host +MinIO internal host (for server-side operations) +*/}} +{{- define "orchard.minio.internalHost" -}} +{{- if .Values.minio.enabled }} +{{- printf "http://%s-minio:9000" .Release.Name }} +{{- else }} +{{- .Values.orchard.s3.endpoint }} +{{- end }} +{{- end }} + +{{/* +MinIO host (uses external URL if ingress enabled, for presigned URLs) */}} {{- define "orchard.minio.host" -}} -{{- if .Values.minio.enabled }} +{{- if and .Values.minio.enabled .Values.minio.ingress.enabled .Values.minio.ingress.host }} +{{- if .Values.minio.ingress.tls.enabled }} +{{- printf "https://%s" .Values.minio.ingress.host }} +{{- else }} +{{- printf "http://%s" .Values.minio.ingress.host }} +{{- end }} +{{- else if .Values.minio.enabled }} {{- printf "http://%s-minio:9000" .Release.Name }} {{- else }} {{- .Values.orchard.s3.endpoint }} diff --git a/helm/orchard/templates/deployment.yaml b/helm/orchard/templates/deployment.yaml index c24b6f1..3a8c97b 100644 --- a/helm/orchard/templates/deployment.yaml +++ b/helm/orchard/templates/deployment.yaml @@ -92,6 +92,10 @@ spec: secretKeyRef: name: {{ include "orchard.minio.secretName" . }} key: {{ if .Values.minio.enabled }}root-password{{ else }}{{ .Values.orchard.s3.existingSecretSecretKeyKey }}{{ end }} + - name: ORCHARD_DOWNLOAD_MODE + value: {{ .Values.orchard.download.mode | quote }} + - name: ORCHARD_PRESIGNED_URL_EXPIRY + value: {{ .Values.orchard.download.presignedUrlExpiry | quote }} livenessProbe: {{- toYaml .Values.livenessProbe | nindent 12 }} readinessProbe: diff --git a/helm/orchard/templates/minio-ingress.yaml b/helm/orchard/templates/minio-ingress.yaml new file mode 100644 index 0000000..84e40b2 --- /dev/null +++ b/helm/orchard/templates/minio-ingress.yaml @@ -0,0 +1,34 @@ +{{- if and .Values.minio.enabled .Values.minio.ingress.enabled -}} +apiVersion: networking.k8s.io/v1 +kind: Ingress +metadata: + name: {{ include "orchard.fullname" . }}-minio + labels: + {{- include "orchard.labels" . | nindent 4 }} + app.kubernetes.io/component: minio + {{- with .Values.minio.ingress.annotations }} + annotations: + {{- toYaml . | nindent 4 }} + {{- end }} +spec: + {{- if .Values.minio.ingress.className }} + ingressClassName: {{ .Values.minio.ingress.className }} + {{- end }} + {{- if .Values.minio.ingress.tls.enabled }} + tls: + - hosts: + - {{ .Values.minio.ingress.host | quote }} + secretName: {{ .Values.minio.ingress.tls.secretName }} + {{- end }} + rules: + - host: {{ .Values.minio.ingress.host | quote }} + http: + paths: + - path: / + pathType: Prefix + backend: + service: + name: {{ .Release.Name }}-minio + port: + number: 9000 +{{- end }} diff --git a/helm/orchard/values.yaml b/helm/orchard/values.yaml index 1565be3..abfc2e0 100644 --- a/helm/orchard/values.yaml +++ b/helm/orchard/values.yaml @@ -115,6 +115,11 @@ orchard: existingSecretAccessKeyKey: "access-key-id" existingSecretSecretKeyKey: "secret-access-key" + # Download configuration + download: + mode: "presigned" # presigned, redirect, or proxy + presignedUrlExpiry: 3600 # Presigned URL expiry in seconds + # PostgreSQL subchart configuration postgresql: enabled: true @@ -147,6 +152,17 @@ minio: persistence: enabled: false size: 50Gi + # MinIO ingress for presigned URL access + ingress: + enabled: false + className: "nginx" + annotations: + cert-manager.io/cluster-issuer: "letsencrypt" + nginx.ingress.kubernetes.io/proxy-body-size: "0" # Disable body size limit for uploads + host: "" # e.g., minio.your-domain.com + tls: + enabled: true + secretName: minio-tls # Redis subchart configuration (for future caching) redis: