release new image with presigned URL support for direct s3 downloads as default download mode

Add presigned URL support for direct S3 downloads (#48)

Closes #48

See merge request esv/bsf/bsf-integration/orchard/orchard-mvp!17

CHANGELOG.md

@@ -6,7 +6,19 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
 ## [Unreleased]
+
+## [0.3.0] - 2025-12-15
+### Changed
+- Changed default download mode from `proxy` to `presigned` for better performance (#48)
 ### Added
+- Added presigned URL support for direct S3 downloads (#48)
+- Added `ORCHARD_DOWNLOAD_MODE` config option (`presigned`, `redirect`, `proxy`) (#48)
+- Added `ORCHARD_PRESIGNED_URL_EXPIRY` config option (default: 3600 seconds) (#48)
+- Added `?mode=` query parameter to override download mode per-request (#48)
+- Added `/api/v1/project/{project}/{package}/+/{ref}/url` endpoint for getting presigned URLs (#48)
+- Added `PresignedUrlResponse` schema with URL, expiry, checksums, and artifact metadata (#48)
+- Added MinIO ingress support in Helm chart for presigned URL access (#48)
+- Added `orchard.download.mode` and `orchard.download.presignedUrlExpiry` Helm values (#48)
 - Added integrity verification workflow design document (#24)
 - Added `sha256` field to API responses for clarity (alias of `id`) (#25)
 - Added `checksum_sha1` field to artifacts table for compatibility (#25)
@@ -15,6 +27,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - Added `Dockerfile.local` and `docker-compose.local.yml` for local development (#25)
 - Added migration script `003_checksum_fields.sql` for existing databases (#25)
 
+
 ## [0.2.0] - 2025-12-15
 ### Changed
 - Updated images to use internal container BSF proxy (#46)

README.md

@@ -60,7 +60,8 @@ Orchard is a centralized binary artifact storage system that provides content-ad
 | `GET` | `/api/v1/project/:project/packages/:package` | Get single package with metadata |
 | `POST` | `/api/v1/project/:project/packages` | Create a new package |
 | `POST` | `/api/v1/project/:project/:package/upload` | Upload an artifact |
-| `GET` | `/api/v1/project/:project/:package/+/:ref` | Download an artifact (supports Range header) |
+| `GET` | `/api/v1/project/:project/:package/+/:ref` | Download an artifact (supports Range header, mode param) |
+| `GET` | `/api/v1/project/:project/:package/+/:ref/url` | Get presigned URL for direct S3 download |
 | `HEAD` | `/api/v1/project/:project/:package/+/:ref` | Get artifact metadata without downloading |
 | `GET` | `/api/v1/project/:project/:package/tags` | List tags (with pagination, search, sorting, artifact metadata) |
 | `POST` | `/api/v1/project/:project/:package/tags` | Create a tag |
@@ -292,6 +293,12 @@ curl -H "Range: bytes=0-1023" http://localhost:8080/api/v1/project/my-project/re
 
 # Check file info without downloading (HEAD request)
 curl -I http://localhost:8080/api/v1/project/my-project/releases/+/v1.0.0
+
+# Download with specific mode (presigned, redirect, or proxy)
+curl "http://localhost:8080/api/v1/project/my-project/releases/+/v1.0.0?mode=proxy"
+
+# Get presigned URL for direct S3 download
+curl http://localhost:8080/api/v1/project/my-project/releases/+/v1.0.0/url
 ```
 
 > **Note on curl flags:**
@@ -300,6 +307,33 @@ curl -I http://localhost:8080/api/v1/project/my-project/releases/+/v1.0.0
 > - `-OJ` combines both: download to a file using the server-provided filename
 > - `-o <filename>` saves to a specific filename you choose
 
+#### Download Modes
+
+Orchard supports three download modes, configurable via `ORCHARD_DOWNLOAD_MODE` or per-request with `?mode=`:
+
+| Mode | Description | Use Case |
+|------|-------------|----------|
+| `presigned` (default) | Returns JSON with a presigned S3 URL | Clients that handle redirects themselves, web UIs |
+| `redirect` | Returns HTTP 302 redirect to presigned S3 URL | Simple clients, browsers, wget |
+| `proxy` | Streams content through the backend | When S3 isn't directly accessible to clients |
+
+**Presigned URL Response:**
+```json
+{
+  "url": "https://minio.example.com/bucket/...",
+  "expires_at": "2025-01-01T01:00:00Z",
+  "method": "GET",
+  "artifact_id": "a3f5d8e...",
+  "size": 1048576,
+  "content_type": "application/gzip",
+  "original_name": "app-v1.0.0.tar.gz",
+  "checksum_sha256": "a3f5d8e...",
+  "checksum_md5": "d41d8cd..."
+}
+```
+
+> **Note:** For presigned URLs to work, clients must be able to reach the S3 endpoint directly. In Kubernetes, this requires exposing MinIO via ingress (see Helm configuration below).
+
 ### Create a Tag
 
 ```bash
@@ -485,6 +519,8 @@ Configuration is provided via environment variables prefixed with `ORCHARD_`:
 | `ORCHARD_S3_BUCKET` | S3 bucket name | `orchard-artifacts` |
 | `ORCHARD_S3_ACCESS_KEY_ID` | S3 access key | - |
 | `ORCHARD_S3_SECRET_ACCESS_KEY` | S3 secret key | - |
+| `ORCHARD_DOWNLOAD_MODE` | Download mode: `presigned`, `redirect`, or `proxy` | `presigned` |
+| `ORCHARD_PRESIGNED_URL_EXPIRY` | Presigned URL expiry in seconds | `3600` |
 
 ## Kubernetes Deployment
 
@@ -505,6 +541,32 @@ helm install orchard ./helm/orchard -n orchard --create-namespace
 helm install orchard ./helm/orchard -f my-values.yaml
 ```
 
+### Helm Configuration
+
+Key configuration options in `values.yaml`:
+
+```yaml
+orchard:
+  # Download configuration
+  download:
+    mode: "presigned"       # presigned, redirect, or proxy
+    presignedUrlExpiry: 3600
+
+# MinIO ingress (required for presigned URL downloads)
+minio:
+  ingress:
+    enabled: true
+    className: "nginx"
+    annotations:
+      cert-manager.io/cluster-issuer: "letsencrypt"
+    host: "minio.your-domain.com"
+    tls:
+      enabled: true
+      secretName: minio-tls
+```
+
+When `minio.ingress.enabled` is `true`, the S3 endpoint automatically uses the external URL (`https://minio.your-domain.com`), making presigned URLs accessible to external clients.
+
 See `helm/orchard/values.yaml` for all configuration options.
 
 ## Database Schema

backend/app/config.py

@@ -32,6 +32,10 @@ class Settings(BaseSettings):
     s3_secret_access_key: str = ""
     s3_use_path_style: bool = True
 
+    # Download settings
+    download_mode: str = "presigned"  # "presigned", "redirect", or "proxy"
+    presigned_url_expiry: int = 3600  # Presigned URL expiry in seconds (default: 1 hour)
+
     @property
     def database_url(self) -> str:
         sslmode = f"?sslmode={self.database_sslmode}" if self.database_sslmode else ""

backend/app/routes.py

@@ -1,9 +1,9 @@
-from datetime import datetime
+from datetime import datetime, timedelta, timezone
 from fastapi import APIRouter, Depends, HTTPException, UploadFile, File, Form, Request, Query, Header, Response
-from fastapi.responses import StreamingResponse
+from fastapi.responses import StreamingResponse, RedirectResponse
 from sqlalchemy.orm import Session
 from sqlalchemy import or_, func
-from typing import List, Optional
+from typing import List, Optional, Literal
 import math
 import re
 import io
@@ -29,8 +29,10 @@ from .schemas import (
     ResumableUploadCompleteResponse,
     ResumableUploadStatusResponse,
     GlobalSearchResponse, SearchResultProject, SearchResultPackage, SearchResultArtifact,
+    PresignedUrlResponse,
 )
 from .metadata import extract_metadata
+from .config import get_settings
 
 router = APIRouter()
 
@@ -844,27 +846,13 @@ def get_upload_status(
         raise HTTPException(status_code=404, detail=str(e))
 
 
-# Download artifact with range request support
+# Helper function to resolve artifact reference
-@router.get("/api/v1/project/{project_name}/{package_name}/+/{ref}")
+def _resolve_artifact_ref(
-def download_artifact(
-    project_name: str,
-    package_name: str,
     ref: str,
-    request: Request,
+    package: Package,
-    db: Session = Depends(get_db),
+    db: Session,
-    storage: S3Storage = Depends(get_storage),
+) -> Optional[Artifact]:
-    range: Optional[str] = Header(None),
+    """Resolve a reference (tag name, artifact:hash, tag:name) to an artifact"""
-):
-    # Get project and package
-    project = db.query(Project).filter(Project.name == project_name).first()
-    if not project:
-        raise HTTPException(status_code=404, detail="Project not found")
-
-    package = db.query(Package).filter(Package.project_id == project.id, Package.name == package_name).first()
-    if not package:
-        raise HTTPException(status_code=404, detail="Package not found")
-
-    # Resolve reference to artifact
     artifact = None
 
     # Check for explicit prefixes
@@ -885,11 +873,76 @@ def download_artifact(
             # Try as direct artifact ID
             artifact = db.query(Artifact).filter(Artifact.id == ref).first()
 
+    return artifact
+
+
+# Download artifact with range request support and download modes
+@router.get("/api/v1/project/{project_name}/{package_name}/+/{ref}")
+def download_artifact(
+    project_name: str,
+    package_name: str,
+    ref: str,
+    request: Request,
+    db: Session = Depends(get_db),
+    storage: S3Storage = Depends(get_storage),
+    range: Optional[str] = Header(None),
+    mode: Optional[Literal["proxy", "redirect", "presigned"]] = Query(
+        default=None,
+        description="Download mode: proxy (stream through backend), redirect (302 to presigned URL), presigned (return JSON with URL)"
+    ),
+):
+    settings = get_settings()
+
+    # Get project and package
+    project = db.query(Project).filter(Project.name == project_name).first()
+    if not project:
+        raise HTTPException(status_code=404, detail="Project not found")
+
+    package = db.query(Package).filter(Package.project_id == project.id, Package.name == package_name).first()
+    if not package:
+        raise HTTPException(status_code=404, detail="Package not found")
+
+    # Resolve reference to artifact
+    artifact = _resolve_artifact_ref(ref, package, db)
     if not artifact:
         raise HTTPException(status_code=404, detail="Artifact not found")
 
     filename = artifact.original_name or f"{artifact.id}"
 
+    # Determine download mode (query param overrides server default)
+    download_mode = mode or settings.download_mode
+
+    # Handle presigned mode - return JSON with presigned URL
+    if download_mode == "presigned":
+        presigned_url = storage.generate_presigned_url(
+            artifact.s3_key,
+            response_content_type=artifact.content_type,
+            response_content_disposition=f'attachment; filename="{filename}"',
+        )
+        expires_at = datetime.now(timezone.utc) + timedelta(seconds=settings.presigned_url_expiry)
+
+        return PresignedUrlResponse(
+            url=presigned_url,
+            expires_at=expires_at,
+            method="GET",
+            artifact_id=artifact.id,
+            size=artifact.size,
+            content_type=artifact.content_type,
+            original_name=artifact.original_name,
+            checksum_sha256=artifact.id,
+            checksum_md5=artifact.checksum_md5,
+        )
+
+    # Handle redirect mode - return 302 redirect to presigned URL
+    if download_mode == "redirect":
+        presigned_url = storage.generate_presigned_url(
+            artifact.s3_key,
+            response_content_type=artifact.content_type,
+            response_content_disposition=f'attachment; filename="{filename}"',
+        )
+        return RedirectResponse(url=presigned_url, status_code=302)
+
+    # Proxy mode (default fallback) - stream through backend
     # Handle range requests
     if range:
         stream, content_length, content_range = storage.get_stream(artifact.s3_key, range)
@@ -923,6 +976,63 @@ def download_artifact(
     )
 
 
+# Get presigned URL endpoint (explicit endpoint for getting URL without redirect)
+@router.get("/api/v1/project/{project_name}/{package_name}/+/{ref}/url", response_model=PresignedUrlResponse)
+def get_artifact_url(
+    project_name: str,
+    package_name: str,
+    ref: str,
+    db: Session = Depends(get_db),
+    storage: S3Storage = Depends(get_storage),
+    expiry: Optional[int] = Query(
+        default=None,
+        description="Custom expiry time in seconds (defaults to server setting)"
+    ),
+):
+    """
+    Get a presigned URL for direct S3 download.
+    This endpoint always returns a presigned URL regardless of server download mode.
+    """
+    settings = get_settings()
+
+    # Get project and package
+    project = db.query(Project).filter(Project.name == project_name).first()
+    if not project:
+        raise HTTPException(status_code=404, detail="Project not found")
+
+    package = db.query(Package).filter(Package.project_id == project.id, Package.name == package_name).first()
+    if not package:
+        raise HTTPException(status_code=404, detail="Package not found")
+
+    # Resolve reference to artifact
+    artifact = _resolve_artifact_ref(ref, package, db)
+    if not artifact:
+        raise HTTPException(status_code=404, detail="Artifact not found")
+
+    filename = artifact.original_name or f"{artifact.id}"
+    url_expiry = expiry or settings.presigned_url_expiry
+
+    presigned_url = storage.generate_presigned_url(
+        artifact.s3_key,
+        expiry=url_expiry,
+        response_content_type=artifact.content_type,
+        response_content_disposition=f'attachment; filename="{filename}"',
+    )
+    expires_at = datetime.now(timezone.utc) + timedelta(seconds=url_expiry)
+
+    return PresignedUrlResponse(
+        url=presigned_url,
+        expires_at=expires_at,
+        method="GET",
+        artifact_id=artifact.id,
+        size=artifact.size,
+        content_type=artifact.content_type,
+        original_name=artifact.original_name,
+        checksum_sha256=artifact.id,
+        checksum_md5=artifact.checksum_md5,
+    )
+
+
 # HEAD request for download (to check file info without downloading)
 @router.head("/api/v1/project/{project_name}/{package_name}/+/{ref}")
 def head_artifact(
@@ -941,23 +1051,8 @@ def head_artifact(
     if not package:
         raise HTTPException(status_code=404, detail="Package not found")
 
-    # Resolve reference to artifact (same logic as download)
+    # Resolve reference to artifact
-    artifact = None
+    artifact = _resolve_artifact_ref(ref, package, db)
-    if ref.startswith("artifact:"):
-        artifact_id = ref[9:]
-        artifact = db.query(Artifact).filter(Artifact.id == artifact_id).first()
-    elif ref.startswith("tag:") or ref.startswith("version:"):
-        tag_name = ref.split(":", 1)[1]
-        tag = db.query(Tag).filter(Tag.package_id == package.id, Tag.name == tag_name).first()
-        if tag:
-            artifact = db.query(Artifact).filter(Artifact.id == tag.artifact_id).first()
-    else:
-        tag = db.query(Tag).filter(Tag.package_id == package.id, Tag.name == ref).first()
-        if tag:
-            artifact = db.query(Artifact).filter(Artifact.id == tag.artifact_id).first()
-        else:
-            artifact = db.query(Artifact).filter(Artifact.id == ref).first()
-
     if not artifact:
         raise HTTPException(status_code=404, detail="Artifact not found")
 

backend/app/schemas.py

@@ -330,6 +330,20 @@ class GlobalSearchResponse(BaseModel):
     counts: Dict[str, int]  # Total counts for each type
 
 
+# Presigned URL response
+class PresignedUrlResponse(BaseModel):
+    """Response containing a presigned URL for direct S3 download"""
+    url: str
+    expires_at: datetime
+    method: str = "GET"
+    artifact_id: str
+    size: int
+    content_type: Optional[str] = None
+    original_name: Optional[str] = None
+    checksum_sha256: Optional[str] = None
+    checksum_md5: Optional[str] = None
+
+
 # Health check
 class HealthResponse(BaseModel):
     status: str

backend/app/storage.py

@@ -450,6 +450,46 @@ class S3Storage:
         except ClientError:
             return False
 
+    def generate_presigned_url(
+        self,
+        s3_key: str,
+        expiry: Optional[int] = None,
+        response_content_type: Optional[str] = None,
+        response_content_disposition: Optional[str] = None,
+    ) -> str:
+        """
+        Generate a presigned URL for downloading an object.
+
+        Args:
+            s3_key: The S3 key of the object
+            expiry: URL expiry in seconds (defaults to settings.presigned_url_expiry)
+            response_content_type: Override Content-Type header in response
+            response_content_disposition: Override Content-Disposition header in response
+
+        Returns:
+            Presigned URL string
+        """
+        if expiry is None:
+            expiry = settings.presigned_url_expiry
+
+        params = {
+            "Bucket": self.bucket,
+            "Key": s3_key,
+        }
+
+        # Add response header overrides if specified
+        if response_content_type:
+            params["ResponseContentType"] = response_content_type
+        if response_content_disposition:
+            params["ResponseContentDisposition"] = response_content_disposition
+
+        url = self.client.generate_presigned_url(
+            "get_object",
+            Params=params,
+            ExpiresIn=expiry,
+        )
+        return url
+
 
 # Singleton instance
 _storage = None

helm/orchard/templates/_helpers.tpl

@@ -97,10 +97,27 @@ password
 {{- end }}
 
 {{/*
-MinIO host
+MinIO internal host (for server-side operations)
+*/}}
+{{- define "orchard.minio.internalHost" -}}
+{{- if .Values.minio.enabled }}
+{{- printf "http://%s-minio:9000" .Release.Name }}
+{{- else }}
+{{- .Values.orchard.s3.endpoint }}
+{{- end }}
+{{- end }}
+
+{{/*
+MinIO host (uses external URL if ingress enabled, for presigned URLs)
 */}}
 {{- define "orchard.minio.host" -}}
-{{- if .Values.minio.enabled }}
+{{- if and .Values.minio.enabled .Values.minio.ingress.enabled .Values.minio.ingress.host }}
+{{- if .Values.minio.ingress.tls.enabled }}
+{{- printf "https://%s" .Values.minio.ingress.host }}
+{{- else }}
+{{- printf "http://%s" .Values.minio.ingress.host }}
+{{- end }}
+{{- else if .Values.minio.enabled }}
 {{- printf "http://%s-minio:9000" .Release.Name }}
 {{- else }}
 {{- .Values.orchard.s3.endpoint }}

helm/orchard/templates/deployment.yaml

@@ -92,6 +92,10 @@ spec:
                 secretKeyRef:
                   name: {{ include "orchard.minio.secretName" . }}
                   key: {{ if .Values.minio.enabled }}root-password{{ else }}{{ .Values.orchard.s3.existingSecretSecretKeyKey }}{{ end }}
+            - name: ORCHARD_DOWNLOAD_MODE
+              value: {{ .Values.orchard.download.mode | quote }}
+            - name: ORCHARD_PRESIGNED_URL_EXPIRY
+              value: {{ .Values.orchard.download.presignedUrlExpiry | quote }}
           livenessProbe:
             {{- toYaml .Values.livenessProbe | nindent 12 }}
           readinessProbe:

helm/orchard/templates/minio-ingress.yaml

@@ -0,0 +1,34 @@
+{{- if and .Values.minio.enabled .Values.minio.ingress.enabled -}}
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+  name: {{ include "orchard.fullname" . }}-minio
+  labels:
+    {{- include "orchard.labels" . | nindent 4 }}
+    app.kubernetes.io/component: minio
+  {{- with .Values.minio.ingress.annotations }}
+  annotations:
+    {{- toYaml . | nindent 4 }}
+  {{- end }}
+spec:
+  {{- if .Values.minio.ingress.className }}
+  ingressClassName: {{ .Values.minio.ingress.className }}
+  {{- end }}
+  {{- if .Values.minio.ingress.tls.enabled }}
+  tls:
+    - hosts:
+        - {{ .Values.minio.ingress.host | quote }}
+      secretName: {{ .Values.minio.ingress.tls.secretName }}
+  {{- end }}
+  rules:
+    - host: {{ .Values.minio.ingress.host | quote }}
+      http:
+        paths:
+          - path: /
+            pathType: Prefix
+            backend:
+              service:
+                name: {{ .Release.Name }}-minio
+                port:
+                  number: 9000
+{{- end }}

helm/orchard/values.yaml

@@ -115,6 +115,11 @@ orchard:
     existingSecretAccessKeyKey: "access-key-id"
     existingSecretSecretKeyKey: "secret-access-key"
 
+  # Download configuration
+  download:
+    mode: "presigned"  # presigned, redirect, or proxy
+    presignedUrlExpiry: 3600  # Presigned URL expiry in seconds
+
 # PostgreSQL subchart configuration
 postgresql:
   enabled: true
@@ -147,6 +152,17 @@ minio:
   persistence:
     enabled: false
     size: 50Gi
+  # MinIO ingress for presigned URL access
+  ingress:
+    enabled: false
+    className: "nginx"
+    annotations:
+      cert-manager.io/cluster-issuer: "letsencrypt"
+      nginx.ingress.kubernetes.io/proxy-body-size: "0"  # Disable body size limit for uploads
+    host: ""  # e.g., minio.your-domain.com
+    tls:
+      enabled: true
+      secretName: minio-tls
 
 # Redis subchart configuration (for future caching)
 redis: