32 Commits

Author SHA1 Message Date
Mondo Diaz
996e3ee4ce Fix concurrent upload race conditions and bump dev resources
- Add IntegrityError handling for concurrent artifact creation
- Add IntegrityError handling for concurrent tag creation with retry
- Add with_for_update() lock on tag lookup to prevent races
- Add database pool config env vars to Helm deployment template
- Bump dev environment resources (256Mi -> 512Mi memory for all services)
- Increase database pool settings for dev (10 connections, 20 overflow)
2026-01-27 21:14:38 +00:00
Mondo Diaz
aa853b5b32 Use CI variable for stage admin password
- Remove Secrets Manager config from values-stage.yaml
- Pass STAGE_ADMIN_PASSWORD via --set in deploy_stage
- Consistent with feature branch approach

Single source of truth: STAGE_ADMIN_PASSWORD CI variable is used by
deploy, reset, and integration test jobs.
2026-01-27 20:36:21 +00:00
Mondo Diaz
fe07638485 Merge branch 'feature/admin-password-env-var' into 'main'
Add configurable admin password via environment variable

Closes #87

See merge request esv/bsf/bsf-integration/orchard/orchard-mvp!46
2026-01-27 14:23:41 -06:00
Mondo Diaz
7120cf64f1 Add configurable admin password via environment variable 2026-01-27 14:23:40 -06:00
Mondo Diaz
718e6e7193 Merge branch 'feature/package-dependencies' into 'main'
Add package dependencies system and project settings page

Closes #76, #77, #78, #79, #80, and #81

See merge request esv/bsf/bsf-integration/orchard/orchard-mvp!45
2026-01-27 10:11:04 -06:00
Mondo Diaz
abba90ebac Add package dependencies system and project settings page 2026-01-27 10:11:04 -06:00
Mondo Diaz
6c8b922818 Merge branch 'fix/reset-stage-before-tests' into 'main'
Add pre-test stage reset to ensure known environment state

See merge request esv/bsf/bsf-integration/orchard/orchard-mvp!44
2026-01-26 09:13:04 -06:00
Mondo Diaz
99d28cf9c6 Add pre-test stage reset to ensure known environment state 2026-01-26 09:13:03 -06:00
Dane Moss
b5579f1643 Merge branch 'release_0.5.1' into 'main'
add CL entry to bump version

See merge request esv/bsf/bsf-integration/orchard/orchard-mvp!43
2026-01-23 15:37:09 -07:00
Dane Moss
fafa03e4ce add CL entry to bump version 2026-01-23 15:37:09 -07:00
Mondo Diaz
d4b2da3232 Merge branch 'fix/release-wait-for-stage-tests' into 'main'
Add gitleaks fingerprint for test file false positive

See merge request esv/bsf/bsf-integration/orchard/orchard-mvp!42
2026-01-23 16:16:03 -06:00
Mondo Diaz
7b04bbdf05 Add gitleaks fingerprint for test file false positive 2026-01-23 16:16:02 -06:00
Mondo Diaz
3a807870a3 Merge branch 'fix/ci-prod-namespace' into 'main'
Fix production CI deployment and simplify tag pipeline

See merge request esv/bsf/bsf-integration/orchard/orchard-mvp!41
2026-01-23 15:50:24 -06:00
Mondo Diaz
f966fde7df Fix production CI deployment and simplify tag pipeline 2026-01-23 15:50:24 -06:00
Mondo Diaz
133d9cbfd6 Merge branch 'bump_version' into 'main'
add changelog entry to cut a new release

See merge request esv/bsf/bsf-integration/orchard/orchard-mvp!40
2026-01-23 13:00:51 -06:00
Dane Moss
276b4f2743 add changelog entry to cut a new release 2026-01-23 10:46:20 -07:00
Mondo Diaz
67ac6bb3f8 Merge branch 'fix/factory-reset-admin-user' into 'main'
Update CHANGELOG with factory reset fixes (#60)

See merge request esv/bsf/bsf-integration/orchard/orchard-mvp!39
2026-01-23 09:33:01 -06:00
Mondo Diaz
b0bb3ed569 Update CHANGELOG with factory reset fixes (#60) 2026-01-21 23:44:45 +00:00
Mondo Diaz
1ac75e1017 Fix factory reset and improve reset_stage CI job
- Add create_default_admin() call to factory reset (admin user wasn't being
  created after reset, only on server restart)
- Add retry logic to reset_stage CI job (3 attempts with 5s delay)
- Use proper context manager for httpx client
- Increase timeout to 120s for reset operation
- Add retry: 1 at job level for transient failures
2026-01-21 23:20:48 +00:00
Mondo Diaz
693613f111 Fix factory reset - capture username before dropping tables 2026-01-21 23:18:29 +00:00
Mondo Diaz
9da4ae8c0d Add gitleaks fingerprint for test file false positive 2026-01-21 22:59:08 +00:00
Mondo Diaz
7ffdc64364 Fix seed_database call in factory reset - pass fresh db session 2026-01-21 22:51:03 +00:00
Mondo Diaz
6abc0c88b0 Merge branch 'feature/stage-reset-job' into 'main'
Fix reset_stage job to read STAGE_URL from environment

See merge request esv/bsf/bsf-integration/orchard/orchard-mvp!38
2026-01-21 16:39:39 -06:00
Mondo Diaz
e96dc5cde8 Fix reset_stage job to read STAGE_URL from environment 2026-01-21 22:25:04 +00:00
Mondo Diaz
cba5bac383 Merge branch 'feature/stage-reset-job' into 'main'
Add factory reset endpoint for stage environment cleanup (#54)

See merge request esv/bsf/bsf-integration/orchard/orchard-mvp!37
2026-01-21 16:00:02 -06:00
Mondo Diaz
535280a783 Add factory reset endpoint for stage environment cleanup (#54) 2026-01-21 16:00:02 -06:00
Dane Moss
c9026e1950 Merge branch 'fix/s3-irsa-credentials' into 'main'
Fix S3 client to support IRSA credentials (#54)

See merge request esv/bsf/bsf-integration/orchard/orchard-mvp!36
2026-01-21 13:42:53 -07:00
Mondo Diaz
fedbd95cf4 Fix S3 client to support IRSA credentials (#54) 2026-01-21 13:42:53 -07:00
Dane Moss
255e25d66d Merge branch 'fix/prod-values-config' into 'main'
Configure prod and stage for AWS services (RDS, S3, Secrets Manager)

See merge request esv/bsf/bsf-integration/orchard/orchard-mvp!35
2026-01-21 12:32:44 -07:00
Mondo Diaz
427d2fec70 Configure prod and stage for AWS services (RDS, S3, Secrets Manager) 2026-01-21 12:32:44 -07:00
Mondo Diaz
199821b34d Merge branch 'feature/upload-download-tests' into 'main'
Add comprehensive upload/download tests and streaming enhancements (#38, #40, #42, #43)

Closes #38, #40, #42, and #43

See merge request esv/bsf/bsf-integration/orchard/orchard-mvp!34
2026-01-21 09:35:12 -06:00
Mondo Diaz
584acd1e90 Add comprehensive upload/download tests and streaming enhancements (#38, #40, #42, #43) 2026-01-21 09:35:12 -06:00
51 changed files with 11125 additions and 714 deletions

7
.env.example Normal file
View File

@@ -0,0 +1,7 @@
# Orchard Local Development Environment
# Copy this file to .env and customize as needed
# Note: .env is gitignored and will not be committed
# Admin account password (required for local development)
# This sets the initial admin password when the database is first created
ORCHARD_ADMIN_PASSWORD=changeme123

View File

@@ -8,6 +8,18 @@ variables:
PROSPER_VERSION: v0.64.1
# Use internal PyPI proxy instead of public internet
PIP_INDEX_URL: https://deps.global.bsf.tools/artifactory/api/pypi/pypi.org/simple
# Environment URLs (used by deploy and test jobs)
STAGE_URL: https://orchard-stage.common.global.bsf.tools
PROD_URL: https://orchard.common.global.bsf.tools
# Stage environment AWS resources (used by reset job)
STAGE_RDS_HOST: orchard-stage.cluster-cvw3jzjkozoc.us-gov-west-1.rds.amazonaws.com
STAGE_RDS_DBNAME: postgres
STAGE_SECRET_ARN: "arn:aws-us-gov:secretsmanager:us-gov-west-1:052673043337:secret:rds!cluster-a573672b-1a38-4665-a654-1b7df37b5297-IaeFQL"
STAGE_AUTH_SECRET_ARN: "arn:aws-us-gov:secretsmanager:us-gov-west-1:052673043337:secret:orchard-stage-creds-SMqvQx"
STAGE_S3_BUCKET: orchard-artifacts-stage
AWS_REGION: us-gov-west-1
# Shared pip cache directory
PIP_CACHE_DIR: "$CI_PROJECT_DIR/.pip-cache"
# Prevent duplicate pipelines for MRs
workflow:
@@ -25,15 +37,113 @@ stages:
- analyze
- deploy
# Override Prosper template jobs to exclude tag pipelines
# Tags only run deploy_prod and smoke_test_prod (image already built on main)
build_image:
rules:
- if: '$CI_COMMIT_TAG'
when: never
- when: on_success
test_image:
rules:
- if: '$CI_COMMIT_TAG'
when: never
- when: on_success
hadolint:
rules:
- if: '$CI_COMMIT_TAG'
when: never
- when: on_success
kics:
variables:
KICS_CONFIG: kics.config
rules:
- if: '$CI_COMMIT_TAG'
when: never
- when: on_success
# Post-deployment integration tests template
secrets:
rules:
- if: '$CI_COMMIT_TAG'
when: never
- when: on_success
app_deps_scan:
rules:
- if: '$CI_COMMIT_TAG'
when: never
- when: on_success
cve_scan:
rules:
- if: '$CI_COMMIT_TAG'
when: never
- when: on_success
app_sbom_analysis:
rules:
- if: '$CI_COMMIT_TAG'
when: never
- when: on_success
cve_sbom_analysis:
rules:
- if: '$CI_COMMIT_TAG'
when: never
- when: on_success
# Override release job to wait for stage integration tests before creating tag
# This ensures the tag (which triggers prod deploy) is only created after stage passes
release:
needs: [integration_test_stage, changelog]
# Full integration test suite template (for feature/stage deployments)
# Runs the complete pytest integration test suite against the deployed environment
.integration_test_template: &integration_test_template
stage: deploy # Runs in deploy stage, but after deployment due to 'needs'
image: deps.global.bsf.tools/docker/python:3.12-slim
timeout: 10m
timeout: 20m # Full suite takes longer than smoke tests
interruptible: true # Cancel if new pipeline starts
retry: 1 # Retry once on failure (network flakiness)
cache:
key: pip-$CI_COMMIT_REF_SLUG
paths:
- .pip-cache/
policy: pull-push
before_script:
- pip install --index-url "$PIP_INDEX_URL" -r backend/requirements.txt
- pip install --index-url "$PIP_INDEX_URL" pytest pytest-asyncio httpx
script:
- cd backend
# Debug: Print environment variables for test configuration
- echo "ORCHARD_TEST_URL=$ORCHARD_TEST_URL"
- echo "ORCHARD_TEST_PASSWORD is set to '${ORCHARD_TEST_PASSWORD:-NOT SET}'"
# Run full integration test suite, excluding:
# - large/slow tests
# - requires_direct_s3 tests (can't access MinIO from outside K8s cluster)
# ORCHARD_TEST_URL tells the tests which server to connect to
# Note: Auth tests work because dev/stage deployments have relaxed rate limits
- |
python -m pytest tests/integration/ -v \
--junitxml=integration-report.xml \
-m "not large and not slow and not requires_direct_s3" \
--tb=short
artifacts:
when: always
expire_in: 1 week
paths:
- backend/integration-report.xml
reports:
junit: backend/integration-report.xml
# Lightweight smoke test template (for production - no test data creation)
.smoke_test_template: &smoke_test_template
stage: deploy
image: deps.global.bsf.tools/docker/python:3.12-slim
timeout: 5m
before_script:
- pip install --index-url "$PIP_INDEX_URL" httpx
script:
@@ -43,12 +153,12 @@ kics:
import os
import sys
BASE_URL = os.environ.get("BASE_URL")
BASE_URL = os.environ.get("ORCHARD_TEST_URL")
if not BASE_URL:
print("ERROR: BASE_URL not set")
print("ERROR: ORCHARD_TEST_URL not set")
sys.exit(1)
print(f"Running integration tests against {BASE_URL}")
print(f"Running smoke tests against {BASE_URL}")
client = httpx.Client(base_url=BASE_URL, timeout=30.0)
errors = []
@@ -86,38 +196,131 @@ kics:
print(f" FAIL: {e}")
sys.exit(1)
else:
print("SUCCESS: All integration tests passed!")
print("SUCCESS: All smoke tests passed!")
sys.exit(0)
PYTEST_SCRIPT
# Integration tests for stage deployment
integration_test_stage:
<<: *integration_test_template
needs: [deploy_stage]
variables:
BASE_URL: https://orchard-stage.common.global.bsf.tools
# Reset stage template - runs from CI runner, uses CI variable for auth
# Calls the /api/v1/admin/factory-reset endpoint which handles DB and S3 cleanup
.reset_stage_template: &reset_stage_template
stage: deploy
image: deps.global.bsf.tools/docker/python:3.12-slim
timeout: 5m
retry: 1
before_script:
- pip install --index-url "$PIP_INDEX_URL" httpx
script:
- |
python - <<'RESET_SCRIPT'
import httpx
import sys
import os
import time
BASE_URL = os.environ.get("STAGE_URL", "")
ADMIN_USER = "admin"
ADMIN_PASS = os.environ.get("STAGE_ADMIN_PASSWORD", "")
MAX_RETRIES = 3
RETRY_DELAY = 5
if not BASE_URL:
print("ERROR: STAGE_URL not set")
sys.exit(1)
if not ADMIN_PASS:
print("ERROR: STAGE_ADMIN_PASSWORD not set")
sys.exit(1)
print(f"=== Resetting stage environment at {BASE_URL} ===")
def do_reset():
with httpx.Client(base_url=BASE_URL, timeout=120.0) as client:
print("Logging in as admin...")
login_response = client.post(
"/api/v1/auth/login",
json={"username": ADMIN_USER, "password": ADMIN_PASS},
)
if login_response.status_code != 200:
raise Exception(f"Login failed: {login_response.status_code} - {login_response.text}")
print("Login successful")
print("Calling factory reset endpoint...")
reset_response = client.post(
"/api/v1/admin/factory-reset",
headers={"X-Confirm-Reset": "yes-delete-all-data"},
)
if reset_response.status_code == 200:
result = reset_response.json()
print("Factory reset successful!")
print(f" Database tables dropped: {result['results']['database_tables_dropped']}")
print(f" S3 objects deleted: {result['results']['s3_objects_deleted']}")
print(f" Database reinitialized: {result['results']['database_reinitialized']}")
print(f" Seeded: {result['results']['seeded']}")
return True
else:
raise Exception(f"Factory reset failed: {reset_response.status_code} - {reset_response.text}")
for attempt in range(1, MAX_RETRIES + 1):
try:
print(f"Attempt {attempt}/{MAX_RETRIES}")
if do_reset():
sys.exit(0)
except Exception as e:
print(f"Attempt {attempt} failed: {e}")
if attempt < MAX_RETRIES:
print(f"Retrying in {RETRY_DELAY} seconds...")
time.sleep(RETRY_DELAY)
else:
print("All retry attempts failed")
sys.exit(1)
RESET_SCRIPT
rules:
- if: '$CI_COMMIT_BRANCH == "main"'
when: on_success
# Integration tests for feature deployment
# Reset stage BEFORE integration tests (ensure known state)
reset_stage_pre:
<<: *reset_stage_template
needs: [deploy_stage]
# Integration tests for stage deployment
# Uses CI variable STAGE_ADMIN_PASSWORD (set in GitLab CI/CD settings)
integration_test_stage:
<<: *integration_test_template
needs: [reset_stage_pre]
variables:
ORCHARD_TEST_URL: $STAGE_URL
ORCHARD_TEST_PASSWORD: $STAGE_ADMIN_PASSWORD
rules:
- if: '$CI_COMMIT_BRANCH == "main"'
when: on_success
# Reset stage AFTER integration tests (clean slate for next run)
reset_stage:
<<: *reset_stage_template
needs: [integration_test_stage]
allow_failure: true # Don't fail pipeline if reset has issues
# Integration tests for feature deployment (full suite)
# Uses DEV_ADMIN_PASSWORD CI variable (same as deploy_feature)
integration_test_feature:
<<: *integration_test_template
needs: [deploy_feature]
variables:
BASE_URL: https://orchard-$CI_COMMIT_REF_SLUG.common.global.bsf.tools
ORCHARD_TEST_URL: https://orchard-$CI_COMMIT_REF_SLUG.common.global.bsf.tools
ORCHARD_TEST_PASSWORD: $DEV_ADMIN_PASSWORD
rules:
- if: '$CI_COMMIT_BRANCH && $CI_COMMIT_BRANCH != "main"'
when: on_success
# Run Python backend tests
python_tests:
# Run Python backend unit tests
python_unit_tests:
stage: test
needs: [] # Run in parallel with build
image: deps.global.bsf.tools/docker/python:3.12-slim
timeout: 15m
variables:
PIP_CACHE_DIR: "$CI_PROJECT_DIR/.pip-cache"
interruptible: true # Cancel if new pipeline starts
cache:
key: pip-$CI_COMMIT_REF_SLUG
paths:
@@ -128,7 +331,7 @@ python_tests:
- pip install --index-url "$PIP_INDEX_URL" pytest pytest-asyncio pytest-cov httpx
script:
- cd backend
# Only run unit tests - integration tests require Docker Compose services
# Run unit tests (integration tests run post-deployment against live environment)
- python -m pytest tests/unit/ -v --cov=app --cov-report=term --cov-report=xml:coverage.xml --cov-report=html:coverage_html --junitxml=pytest-report.xml
artifacts:
when: always
@@ -143,6 +346,10 @@ python_tests:
coverage_format: cobertura
path: backend/coverage.xml
coverage: '/TOTAL.*\s+(\d+%)/'
rules:
- if: '$CI_COMMIT_TAG'
when: never
- when: on_success
# Run frontend tests
frontend_tests:
@@ -150,6 +357,7 @@ frontend_tests:
needs: [] # Run in parallel with build
image: deps.global.bsf.tools/docker/node:20-alpine
timeout: 15m
interruptible: true # Cancel if new pipeline starts
cache:
key: npm-$CI_COMMIT_REF_SLUG
paths:
@@ -171,11 +379,15 @@ frontend_tests:
coverage_format: cobertura
path: frontend/coverage/cobertura-coverage.xml
coverage: '/All files[^|]*\|[^|]*\s+([\d\.]+)/'
rules:
- if: '$CI_COMMIT_TAG'
when: never
- when: on_success
# Shared deploy configuration
.deploy_template: &deploy_template
stage: deploy
needs: [build_image, test_image, kics, hadolint, python_tests, frontend_tests, secrets, app_deps_scan, cve_scan, cve_sbom_analysis, app_sbom_analysis]
needs: [build_image, test_image, kics, hadolint, python_unit_tests, frontend_tests, secrets, app_deps_scan, cve_scan, cve_sbom_analysis, app_sbom_analysis]
image: deps.global.bsf.tools/registry-1.docker.io/alpine/k8s:1.29.12
.helm_setup: &helm_setup
@@ -184,47 +396,21 @@ frontend_tests:
# OCI-based charts from internal registry - no repo add needed
- helm dependency update
# Simplified deployment verification - just health check
# Full API/frontend checks are done by integration tests post-deployment
.verify_deployment: &verify_deployment |
echo "=== Waiting for health endpoint (certs may take a few minutes) ==="
for i in $(seq 1 30); do
if curl -sf --max-time 10 "$BASE_URL/health" > /dev/null 2>&1; then
echo "Health check passed!"
break
echo "Deployment URL: $BASE_URL"
exit 0
fi
echo "Attempt $i/30 - waiting 10s..."
sleep 10
done
# Verify health endpoint
echo ""
echo "=== Health Check ==="
curl -sf "$BASE_URL/health" || { echo "Health check failed"; exit 1; }
echo ""
# Verify API is responding
echo ""
echo "=== API Check (GET /api/v1/projects) ==="
HTTP_CODE=$(curl -sf -o /dev/null -w "%{http_code}" "$BASE_URL/api/v1/projects")
if [ "$HTTP_CODE" = "200" ]; then
echo "API responding: HTTP $HTTP_CODE"
else
echo "API check failed: HTTP $HTTP_CODE"
exit 1
fi
# Verify frontend is served
echo ""
echo "=== Frontend Check ==="
if curl -sf "$BASE_URL/" | grep -q "</html>"; then
echo "Frontend is being served"
else
echo "Frontend check failed"
exit 1
fi
echo ""
echo "=== All checks passed! ==="
echo "Deployment URL: $BASE_URL"
echo "Health check failed after 30 attempts"
exit 1
# Deploy to stage (main branch)
deploy_stage:
@@ -232,7 +418,7 @@ deploy_stage:
variables:
NAMESPACE: orch-stage-namespace
VALUES_FILE: helm/orchard/values-stage.yaml
BASE_URL: https://orchard-stage.common.global.bsf.tools
BASE_URL: $STAGE_URL
before_script:
- kubectl config use-context esv/bsf/bsf-integration/orchard/orchard-mvp:orchard-stage
- *helm_setup
@@ -244,6 +430,7 @@ deploy_stage:
--namespace $NAMESPACE \
-f $VALUES_FILE \
--set image.tag=git.linux-amd64-$CI_COMMIT_SHA \
--set orchard.auth.adminPassword=$STAGE_ADMIN_PASSWORD \
--wait \
--atomic \
--timeout 10m
@@ -251,7 +438,7 @@ deploy_stage:
- *verify_deployment
environment:
name: stage
url: https://orchard-stage.common.global.bsf.tools
url: $STAGE_URL
kubernetes:
agent: esv/bsf/bsf-integration/orchard/orchard-mvp:orchard-stage
rules:
@@ -275,6 +462,7 @@ deploy_feature:
--namespace $NAMESPACE \
-f $VALUES_FILE \
--set image.tag=git.linux-amd64-$CI_COMMIT_SHA \
--set orchard.auth.adminPassword=$DEV_ADMIN_PASSWORD \
--set ingress.hosts[0].host=orchard-$CI_COMMIT_REF_SLUG.common.global.bsf.tools \
--set ingress.tls[0].hosts[0]=orchard-$CI_COMMIT_REF_SLUG.common.global.bsf.tools \
--set ingress.tls[0].secretName=orchard-$CI_COMMIT_REF_SLUG-tls \
@@ -297,10 +485,12 @@ deploy_feature:
- if: '$CI_COMMIT_BRANCH && $CI_COMMIT_BRANCH != "main"'
when: on_success
# Cleanup feature branch deployment
# Cleanup feature branch deployment (standalone - doesn't need deploy dependencies)
cleanup_feature:
<<: *deploy_template
stage: deploy
needs: []
image: deps.global.bsf.tools/registry-1.docker.io/alpine/k8s:1.29.12
timeout: 5m
variables:
NAMESPACE: orch-dev-namespace
GIT_STRATEGY: none # No source needed, branch may be deleted
@@ -319,17 +509,16 @@ cleanup_feature:
when: manual
allow_failure: true
# Deploy to production (version tags only, manual approval required)
# Deploy to production (version tags only)
deploy_prod:
stage: deploy
# For tag pipelines, most jobs don't run (trusting main was tested)
# We only need build_image to have the image available
needs: [build_image]
# For tag pipelines, no other jobs run - image was already built when commit was on main
needs: []
image: deps.global.bsf.tools/registry-1.docker.io/alpine/k8s:1.29.12
variables:
NAMESPACE: orch-prod-namespace
NAMESPACE: orch-namespace
VALUES_FILE: helm/orchard/values-prod.yaml
BASE_URL: https://orchard.common.global.bsf.tools
BASE_URL: $PROD_URL
before_script:
- kubectl config use-context esv/bsf/bsf-integration/orchard/orchard-mvp:orchard-prod
- *helm_setup
@@ -348,21 +537,21 @@ deploy_prod:
- *verify_deployment
environment:
name: production
url: https://orchard.common.global.bsf.tools
url: $PROD_URL
kubernetes:
agent: esv/bsf/bsf-integration/orchard/orchard-mvp:orchard-prod
rules:
# Only run on semantic version tags (v1.0.0, v1.2.3, etc.)
- if: '$CI_COMMIT_TAG =~ /^v\d+\.\d+\.\d+$/'
when: manual # Require manual approval for prod
when: on_success
allow_failure: false
# Integration tests for production deployment
integration_test_prod:
<<: *integration_test_template
# Smoke tests for production deployment (read-only, no test data creation)
smoke_test_prod:
<<: *smoke_test_template
needs: [deploy_prod]
variables:
BASE_URL: https://orchard.common.global.bsf.tools
ORCHARD_TEST_URL: $PROD_URL
rules:
- if: '$CI_COMMIT_TAG =~ /^v\d+\.\d+\.\d+$/'
when: on_success

8
.gitleaks.toml Normal file
View File

@@ -0,0 +1,8 @@
# Gitleaks configuration
# https://github.com/gitleaks/gitleaks#configuration
[allowlist]
# Test files that contain variable names matching secret patterns (e.g., s3_key)
paths = [
'''backend/tests/.*\.py''',
]

View File

@@ -4,6 +4,7 @@
# False positive: s3_key is an attribute name in test assertions, not a secret
# These are historical commits - files have since been deleted or updated with inline comments
7e68baed0886a3c928644cd01aa3b39f92d4f976:backend/tests/test_duplicate_detection.py:generic-api-key:154
81458b3bcb5ace97109ba4c16f4afa6e55b1b8bd:backend/tests/test_duplicate_detection.py:generic-api-key:154
2f1891cf0126ec0e7d4c789d872a2cb2dd3a1745:backend/tests/unit/test_storage.py:generic-api-key:381
10d36947948de796f0bacea3827f4531529c405d:backend/tests/unit/test_storage.py:generic-api-key:381
bccbc71c13570d14b8b26a11335c45f102fe3072:backend/tests/unit/test_storage.py:generic-api-key:381
@@ -11,3 +12,8 @@ bccbc71c13570d14b8b26a11335c45f102fe3072:backend/tests/unit/test_storage.py:gene
90bb2a3a393d2361dc3136ee8d761debb0726d8a:backend/tests/unit/test_storage.py:generic-api-key:381
37666e41a72d2a4f34447c0d1a8728e1d7271d24:backend/tests/unit/test_storage.py:generic-api-key:381
0cc4f253621a9601c5193f6ae1e7ae33f0e7fc9b:backend/tests/unit/test_storage.py:generic-api-key:381
35fda65d381acc5ab59bc592ee3013f75906c197:backend/tests/unit/test_storage.py:generic-api-key:381
08dce6cbb836b687002751fed4159bfc2da61f8b:backend/tests/unit/test_storage.py:generic-api-key:381
617bcbe89cff9a009d77e4f1f1864efed1820e63:backend/tests/unit/test_storage.py:generic-api-key:381
1cbd33544388e0fe6db752fa8886fab33cf9ce7c:backend/tests/unit/test_storage.py:generic-api-key:381
7cfad28f678f5a5b8b927d694a17b9ba446b7138:backend/tests/unit/test_storage.py:generic-api-key:381

View File

@@ -7,6 +7,101 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
## [Unreleased]
### Added
- Added `ORCHARD_ADMIN_PASSWORD` environment variable to configure initial admin password (#87)
- When set, admin user is created with the specified password (no password change required)
- When not set, defaults to `changeme123` and requires password change on first login
- Added Helm chart support for admin password via multiple sources (#87):
- `orchard.auth.adminPassword` - plain value (creates K8s secret)
- `orchard.auth.existingSecret` - reference existing K8s secret
- `orchard.auth.secretsManager` - AWS Secrets Manager integration
- Added `.env.example` template for local development (#87)
- Added `.env` file support in docker-compose.local.yml (#87)
- Added Project Settings page accessible to project admins (#65)
- General settings section for editing description and visibility
- Access Management section (moved from project page)
- Danger Zone section with inline delete confirmation requiring project name
- Settings button (gear icon) on project page header for admins
- Added artifact dependency management system (#76, #77, #78, #79, #80, #81)
- `artifact_dependencies` table with version/tag constraints and check constraints
- `ArtifactDependency` SQLAlchemy model with indexes for fast lookups
- Ensure file parsing (`orchard.ensure` YAML format) during artifact upload
- Circular dependency detection at upload time (rejected with 400)
- Dependency conflict detection at resolution time (409 with conflict details)
- Added dependency API endpoints (#78, #79):
- `GET /api/v1/artifact/{artifact_id}/dependencies` - Get dependencies by artifact ID
- `GET /api/v1/project/{project}/{package}/+/{ref}/dependencies` - Get dependencies by ref
- `GET /api/v1/project/{project}/{package}/reverse-dependencies` - Get reverse dependencies (paginated)
- `GET /api/v1/project/{project}/{package}/+/{ref}/resolve` - Resolve full dependency tree
- Added dependency resolution with topological sorting (#79)
- Returns flat list of all artifacts needed in dependency order
- Includes download URLs, sizes, and version info for each artifact
- Added frontend dependency visualization (#84, #85, #86):
- Dependencies section on package page showing direct dependencies for selected tag
- Tag/version selector to switch between artifacts
- "Used By" section showing reverse dependencies with pagination
- Interactive dependency graph modal with:
- Tree visualization with collapsible nodes
- Zoom (mouse wheel + buttons) and pan (click-drag)
- Click to navigate to package
- Hover tooltip with package details
- Error display for circular dependencies and conflicts
- Added migration `008_artifact_dependencies.sql` for dependency schema
- Added `dependencies.py` module with parsing, validation, and resolution logic
- Added comprehensive integration tests for all dependency features
### Changed
- Added pre-test stage reset to ensure known environment state before integration tests (#54)
- Upload endpoint now accepts optional `ensure` file parameter for declaring dependencies
- Updated upload API documentation with ensure file format and examples
## [0.5.1] - 2026-01-23
### Changed
- Simplified tag pipeline to only run deploy and smoke tests (image already built on main) (#54)
### Fixed
- Fixed production CI deployment namespace to use correct `orch-namespace` (#54)
- Added gitleaks config to allowlist test files from secret scanning (#54)
## [0.5.0] - 2026-01-23
### Added
- Added factory reset endpoint `POST /api/v1/admin/factory-reset` for test environment cleanup (#54)
- Requires admin authentication and `X-Confirm-Reset: yes-delete-all-data` header
- Drops all database tables, clears S3 bucket, reinitializes schema, re-seeds default data
- CI pipeline automatically calls this after integration tests on stage
- Added `delete_all()` method to storage backend for bulk S3 object deletion (#54)
- Added AWS Secrets Manager CSI driver support for database credentials (#54)
- Added SecretProviderClass template for Secrets Manager integration (#54)
- Added IRSA service account annotations for prod and stage environments (#54)
- Added comprehensive upload/download tests for size boundaries (1B to 1GB) (#38)
- Added concurrent upload/download tests (2, 5, 10 parallel operations) (#38)
- Added data integrity tests (binary, text, unicode, compressed content) (#38)
- Added chunk boundary tests for edge cases (#38)
- Added `@pytest.mark.large` and `@pytest.mark.concurrent` test markers (#38)
- Added `generate_content()` and `generate_content_with_hash()` test helpers (#38)
- Added `sized_content` fixture for generating test content of specific sizes (#38)
- Added upload API tests: upload without tag, artifact creation verification, S3 object creation (#38)
- Added download API tests: tag: prefix resolution, 404 for nonexistent project/package/artifact (#38)
- Added download header tests: Content-Type, Content-Length, Content-Disposition, ETag, X-Checksum-SHA256 (#38)
- Added error handling tests: timeout behavior, checksum validation, resource cleanup, graceful error responses (#38)
- Added version API tests: version creation, auto-detection, listing, download by version prefix (#38)
- Added integrity verification tests: round-trip hash verification, client-side verification workflow, size variants (1KB-10MB) (#40)
- Added consistency check endpoint tests with response format validation (#40)
- Added corruption detection tests: bit flip, truncation, appended content, size mismatch, missing S3 objects (#40)
- Added Digest header tests (RFC 3230) and verification mode tests (#40)
- Added integrity verification documentation (`docs/integrity-verification.md`) (#40)
- Added conditional request support for downloads (If-None-Match, If-Modified-Since) returning 304 Not Modified (#42)
- Added caching headers to downloads: Cache-Control (immutable), Last-Modified (#42)
- Added 416 Range Not Satisfiable response for invalid range requests (#42)
- Added download completion logging with bytes transferred and throughput (#42)
- Added client disconnect handling during streaming downloads (#42)
- Added streaming download tests: range requests, conditional requests, caching headers, download resume (#42)
- Added upload duration and throughput metrics (`duration_ms`, `throughput_mbps`) to upload response (#43)
- Added upload progress logging for large files (hash computation and multipart upload phases) (#43)
- Added client disconnect handling during uploads with proper cleanup (#43)
- Added upload progress tracking endpoint `GET /upload/{upload_id}/progress` for resumable uploads (#43)
- Added large file upload tests (10MB, 100MB, 1GB) with multipart upload verification (#43)
- Added upload cancellation and timeout handling tests (#43)
- Added comprehensive API documentation for upload endpoints with curl, Python, and JavaScript examples (#43)
- Added `package_versions` table for immutable version tracking separate from mutable tags (#56)
- Versions are set at upload time via explicit `version` parameter or auto-detected from filename/metadata
- Version detection priority: explicit parameter > package metadata > filename pattern
@@ -31,6 +126,19 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- Added internal proxy configuration for npm, pip, helm, and apt (#51)
### Changed
- Configured stage and prod to use AWS RDS instead of PostgreSQL subchart (#54)
- Configured stage and prod to use AWS S3 instead of MinIO subchart (#54)
- Changed prod deployment from manual to automatic on version tags (#54)
- Updated S3 client to support IRSA credentials when no explicit keys provided (#54)
- Changed prod image pullPolicy to Always (#54)
- Added proxy-body-size annotation to prod ingress for large uploads (#54)
- CI integration tests now run full pytest suite (~350 tests) against deployed environment instead of 3 smoke tests
- CI production deployment uses lightweight smoke tests only (no test data creation in prod)
- CI pipeline improvements: shared pip cache, `interruptible` flag on test jobs, retry on integration tests
- Simplified deploy verification to health check only (full checks done by integration tests)
- Extracted environment URLs to global variables for maintainability
- Made `cleanup_feature` job standalone (no longer inherits deploy template dependencies)
- Renamed `integration_test_prod` to `smoke_test_prod` for clarity
- Updated download ref resolution to check versions before tags (version → tag → artifact ID) (#56)
- Deploy jobs now require all security scans to pass before deployment (added test_image, app_deps_scan, cve_scan, cve_sbom_analysis, app_sbom_analysis to dependencies) (#63)
- Increased deploy job timeout from 5m to 10m (#63)
@@ -44,6 +152,21 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- Improved pod naming: Orchard pods now named `orchard-{env}-server-*` for clarity (#51)
### Fixed
- Fixed factory reset not creating default admin user after reset (#60)
- Admin user was only created at server startup, not after factory reset
- CI reset job would fail to login because admin user didn't exist
- Improved reset_stage CI job reliability (#60)
- Added application-level retry logic (3 attempts with 5s delay)
- Added job-level retry for transient failures
- Fixed httpx client to use proper context manager
- Increased timeout to 120s for reset operations
- Fixed CI integration test rate limiting: added configurable `ORCHARD_LOGIN_RATE_LIMIT` env var, relaxed to 1000/minute for dev/stage
- Fixed duplicate `TestSecurityEdgeCases` class definition in test_auth_api.py
- Fixed integration tests auth: session-scoped client, configurable credentials via env vars, fail-fast on auth errors
- Fixed 413 Request Entity Too Large errors on uploads by adding `proxy-body-size: "0"` nginx annotation to Orchard ingress
- Fixed CI tests that require direct S3 access: added `@pytest.mark.requires_direct_s3` marker and excluded from CI
- Fixed ref_count triggers not being created: added auto-migration for tags ref_count trigger functions
- Fixed Content-Disposition header encoding for non-ASCII filenames using RFC 5987 (#38)
- Fixed deploy jobs running even when tests or security scans fail (changed rules from `when: always` to `when: on_success`) (#63)
- Fixed python_tests job not using internal PyPI proxy (#63)
- Fixed `cleanup_feature` job failing when branch is deleted (`GIT_STRATEGY: none`) (#51)
@@ -53,9 +176,12 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- Fixed deploy jobs running when secrets scan fails (added `secrets` to deploy dependencies)
- Fixed dev environment memory requests to equal limits per cluster Kyverno policy
- Fixed init containers missing resource limits (Kyverno policy compliance)
- Fixed Python SyntaxWarning for invalid escape sequence in database migration regex pattern
### Removed
- Removed unused `store_streaming()` method from storage.py (#51)
- Disabled PostgreSQL subchart for stage and prod environments (#54)
- Disabled MinIO subchart for stage and prod environments (#54)
## [0.4.0] - 2026-01-12
### Added

View File

@@ -360,21 +360,36 @@ def create_default_admin(db: Session) -> Optional[User]:
"""Create the default admin user if no users exist.
Returns the created user, or None if users already exist.
The admin password can be set via ORCHARD_ADMIN_PASSWORD environment variable.
If not set, defaults to 'changeme123' and requires password change on first login.
"""
# Check if any users exist
user_count = db.query(User).count()
if user_count > 0:
return None
settings = get_settings()
# Use configured password or default
password = settings.admin_password if settings.admin_password else "changeme123"
# Only require password change if using the default password
must_change = not settings.admin_password
# Create default admin
auth_service = AuthService(db)
admin = auth_service.create_user(
username="admin",
password="changeme123",
password=password,
is_admin=True,
must_change_password=True,
must_change_password=must_change,
)
if settings.admin_password:
logger.info("Created default admin user with configured password")
else:
logger.info("Created default admin user with default password (changeme123)")
return admin

View File

@@ -53,6 +53,9 @@ class Settings(BaseSettings):
log_level: str = "INFO" # DEBUG, INFO, WARNING, ERROR, CRITICAL
log_format: str = "auto" # "json", "standard", or "auto" (json in production)
# Initial admin user settings
admin_password: str = "" # Initial admin password (if empty, uses 'changeme123')
# JWT Authentication settings (optional, for external identity providers)
jwt_enabled: bool = False # Enable JWT token validation
jwt_secret: str = "" # Secret key for HS256, or leave empty for RS256 with JWKS

View File

@@ -170,6 +170,62 @@ def _run_migrations():
END IF;
END $$;
""",
# Create ref_count trigger functions for tags (ensures triggers exist even if initial migration wasn't run)
"""
CREATE OR REPLACE FUNCTION increment_artifact_ref_count()
RETURNS TRIGGER AS $$
BEGIN
UPDATE artifacts SET ref_count = ref_count + 1 WHERE id = NEW.artifact_id;
RETURN NEW;
END;
$$ LANGUAGE plpgsql;
""",
"""
CREATE OR REPLACE FUNCTION decrement_artifact_ref_count()
RETURNS TRIGGER AS $$
BEGIN
UPDATE artifacts SET ref_count = ref_count - 1 WHERE id = OLD.artifact_id;
RETURN OLD;
END;
$$ LANGUAGE plpgsql;
""",
"""
CREATE OR REPLACE FUNCTION update_artifact_ref_count()
RETURNS TRIGGER AS $$
BEGIN
IF OLD.artifact_id != NEW.artifact_id THEN
UPDATE artifacts SET ref_count = ref_count - 1 WHERE id = OLD.artifact_id;
UPDATE artifacts SET ref_count = ref_count + 1 WHERE id = NEW.artifact_id;
END IF;
RETURN NEW;
END;
$$ LANGUAGE plpgsql;
""",
# Create triggers for tags ref_count management
"""
DO $$
BEGIN
-- Drop and recreate triggers to ensure they're current
DROP TRIGGER IF EXISTS tags_ref_count_insert_trigger ON tags;
CREATE TRIGGER tags_ref_count_insert_trigger
AFTER INSERT ON tags
FOR EACH ROW
EXECUTE FUNCTION increment_artifact_ref_count();
DROP TRIGGER IF EXISTS tags_ref_count_delete_trigger ON tags;
CREATE TRIGGER tags_ref_count_delete_trigger
AFTER DELETE ON tags
FOR EACH ROW
EXECUTE FUNCTION decrement_artifact_ref_count();
DROP TRIGGER IF EXISTS tags_ref_count_update_trigger ON tags;
CREATE TRIGGER tags_ref_count_update_trigger
AFTER UPDATE ON tags
FOR EACH ROW
WHEN (OLD.artifact_id IS DISTINCT FROM NEW.artifact_id)
EXECUTE FUNCTION update_artifact_ref_count();
END $$;
""",
# Create ref_count trigger functions for package_versions
"""
CREATE OR REPLACE FUNCTION increment_version_ref_count()
@@ -210,7 +266,7 @@ def _run_migrations():
END $$;
""",
# Migrate existing semver tags to package_versions
"""
r"""
DO $$
BEGIN
IF EXISTS (SELECT 1 FROM information_schema.tables WHERE table_name = 'package_versions') THEN

723
backend/app/dependencies.py Normal file
View File

@@ -0,0 +1,723 @@
"""
Dependency management module for artifact dependencies.
Handles:
- Parsing orchard.ensure files
- Storing dependencies in the database
- Querying dependencies and reverse dependencies
- Dependency resolution with topological sorting
- Circular dependency detection
- Conflict detection
"""
import yaml
from typing import List, Dict, Any, Optional, Set, Tuple
from sqlalchemy.orm import Session
from sqlalchemy import and_
from .models import (
Project,
Package,
Artifact,
Tag,
ArtifactDependency,
PackageVersion,
)
from .schemas import (
EnsureFileContent,
EnsureFileDependency,
DependencyResponse,
ArtifactDependenciesResponse,
DependentInfo,
ReverseDependenciesResponse,
ResolvedArtifact,
DependencyResolutionResponse,
DependencyConflict,
PaginationMeta,
)
class DependencyError(Exception):
"""Base exception for dependency errors."""
pass
class CircularDependencyError(DependencyError):
"""Raised when a circular dependency is detected."""
def __init__(self, cycle: List[str]):
self.cycle = cycle
super().__init__(f"Circular dependency detected: {' -> '.join(cycle)}")
class DependencyConflictError(DependencyError):
"""Raised when conflicting dependency versions are detected."""
def __init__(self, conflicts: List[DependencyConflict]):
self.conflicts = conflicts
super().__init__(f"Dependency conflicts detected: {len(conflicts)} conflict(s)")
class DependencyNotFoundError(DependencyError):
"""Raised when a dependency cannot be resolved."""
def __init__(self, project: str, package: str, constraint: str):
self.project = project
self.package = package
self.constraint = constraint
super().__init__(f"Dependency not found: {project}/{package}@{constraint}")
class InvalidEnsureFileError(DependencyError):
"""Raised when the ensure file is invalid."""
pass
class DependencyDepthExceededError(DependencyError):
"""Raised when dependency resolution exceeds max depth."""
def __init__(self, max_depth: int):
self.max_depth = max_depth
super().__init__(f"Dependency resolution exceeded maximum depth of {max_depth}")
# Safety limits to prevent DoS attacks
MAX_DEPENDENCY_DEPTH = 50 # Maximum levels of nested dependencies
MAX_DEPENDENCIES_PER_ARTIFACT = 200 # Maximum direct dependencies per artifact
def parse_ensure_file(content: bytes) -> EnsureFileContent:
"""
Parse an orchard.ensure file.
Args:
content: Raw bytes of the ensure file
Returns:
Parsed EnsureFileContent
Raises:
InvalidEnsureFileError: If the file is invalid YAML or has wrong structure
"""
try:
data = yaml.safe_load(content.decode('utf-8'))
except yaml.YAMLError as e:
raise InvalidEnsureFileError(f"Invalid YAML: {e}")
except UnicodeDecodeError as e:
raise InvalidEnsureFileError(f"Invalid encoding: {e}")
if data is None:
return EnsureFileContent(dependencies=[])
if not isinstance(data, dict):
raise InvalidEnsureFileError("Ensure file must be a YAML dictionary")
dependencies = []
deps_data = data.get('dependencies', [])
if not isinstance(deps_data, list):
raise InvalidEnsureFileError("'dependencies' must be a list")
# Safety limit: prevent DoS through excessive dependencies
if len(deps_data) > MAX_DEPENDENCIES_PER_ARTIFACT:
raise InvalidEnsureFileError(
f"Too many dependencies: {len(deps_data)} exceeds maximum of {MAX_DEPENDENCIES_PER_ARTIFACT}"
)
for i, dep in enumerate(deps_data):
if not isinstance(dep, dict):
raise InvalidEnsureFileError(f"Dependency {i} must be a dictionary")
project = dep.get('project')
package = dep.get('package')
version = dep.get('version')
tag = dep.get('tag')
if not project:
raise InvalidEnsureFileError(f"Dependency {i} missing 'project'")
if not package:
raise InvalidEnsureFileError(f"Dependency {i} missing 'package'")
if not version and not tag:
raise InvalidEnsureFileError(
f"Dependency {i} must have either 'version' or 'tag'"
)
if version and tag:
raise InvalidEnsureFileError(
f"Dependency {i} cannot have both 'version' and 'tag'"
)
dependencies.append(EnsureFileDependency(
project=project,
package=package,
version=version,
tag=tag,
))
return EnsureFileContent(dependencies=dependencies)
def validate_dependencies(
db: Session,
dependencies: List[EnsureFileDependency],
) -> List[str]:
"""
Validate that all dependency projects exist.
Args:
db: Database session
dependencies: List of dependencies to validate
Returns:
List of error messages (empty if all valid)
"""
errors = []
for dep in dependencies:
project = db.query(Project).filter(Project.name == dep.project).first()
if not project:
errors.append(f"Project '{dep.project}' not found")
return errors
def store_dependencies(
db: Session,
artifact_id: str,
dependencies: List[EnsureFileDependency],
) -> List[ArtifactDependency]:
"""
Store dependencies for an artifact.
Args:
db: Database session
artifact_id: The artifact ID that has these dependencies
dependencies: List of dependencies to store
Returns:
List of created ArtifactDependency objects
"""
created = []
for dep in dependencies:
artifact_dep = ArtifactDependency(
artifact_id=artifact_id,
dependency_project=dep.project,
dependency_package=dep.package,
version_constraint=dep.version,
tag_constraint=dep.tag,
)
db.add(artifact_dep)
created.append(artifact_dep)
return created
def get_artifact_dependencies(
db: Session,
artifact_id: str,
) -> List[DependencyResponse]:
"""
Get all dependencies for an artifact.
Args:
db: Database session
artifact_id: The artifact ID
Returns:
List of DependencyResponse objects
"""
deps = db.query(ArtifactDependency).filter(
ArtifactDependency.artifact_id == artifact_id
).all()
return [DependencyResponse.from_orm_model(dep) for dep in deps]
def get_reverse_dependencies(
db: Session,
project_name: str,
package_name: str,
page: int = 1,
limit: int = 50,
) -> ReverseDependenciesResponse:
"""
Get all artifacts that depend on a given package.
Args:
db: Database session
project_name: Target project name
package_name: Target package name
page: Page number (1-indexed)
limit: Results per page
Returns:
ReverseDependenciesResponse with dependents and pagination
"""
# Query dependencies that point to this project/package
query = db.query(ArtifactDependency).filter(
ArtifactDependency.dependency_project == project_name,
ArtifactDependency.dependency_package == package_name,
)
total = query.count()
offset = (page - 1) * limit
deps = query.offset(offset).limit(limit).all()
dependents = []
for dep in deps:
# Get artifact info to find the project/package/version
artifact = db.query(Artifact).filter(Artifact.id == dep.artifact_id).first()
if not artifact:
continue
# Find which package this artifact belongs to via tags or versions
tag = db.query(Tag).filter(Tag.artifact_id == dep.artifact_id).first()
if tag:
pkg = db.query(Package).filter(Package.id == tag.package_id).first()
if pkg:
proj = db.query(Project).filter(Project.id == pkg.project_id).first()
if proj:
# Get version if available
version_record = db.query(PackageVersion).filter(
PackageVersion.artifact_id == dep.artifact_id,
PackageVersion.package_id == pkg.id,
).first()
dependents.append(DependentInfo(
artifact_id=dep.artifact_id,
project=proj.name,
package=pkg.name,
version=version_record.version if version_record else None,
constraint_type="version" if dep.version_constraint else "tag",
constraint_value=dep.version_constraint or dep.tag_constraint,
))
total_pages = (total + limit - 1) // limit
return ReverseDependenciesResponse(
project=project_name,
package=package_name,
dependents=dependents,
pagination=PaginationMeta(
page=page,
limit=limit,
total=total,
total_pages=total_pages,
has_more=page < total_pages,
),
)
def _resolve_dependency_to_artifact(
db: Session,
project_name: str,
package_name: str,
version: Optional[str],
tag: Optional[str],
) -> Optional[Tuple[str, str, int]]:
"""
Resolve a dependency constraint to an artifact ID.
Args:
db: Database session
project_name: Project name
package_name: Package name
version: Version constraint (exact)
tag: Tag constraint
Returns:
Tuple of (artifact_id, resolved_version_or_tag, size) or None if not found
"""
# Get project and package
project = db.query(Project).filter(Project.name == project_name).first()
if not project:
return None
package = db.query(Package).filter(
Package.project_id == project.id,
Package.name == package_name,
).first()
if not package:
return None
if version:
# Look up by version
pkg_version = db.query(PackageVersion).filter(
PackageVersion.package_id == package.id,
PackageVersion.version == version,
).first()
if pkg_version:
artifact = db.query(Artifact).filter(
Artifact.id == pkg_version.artifact_id
).first()
if artifact:
return (artifact.id, version, artifact.size)
# Also check if there's a tag with this exact name
tag_record = db.query(Tag).filter(
Tag.package_id == package.id,
Tag.name == version,
).first()
if tag_record:
artifact = db.query(Artifact).filter(
Artifact.id == tag_record.artifact_id
).first()
if artifact:
return (artifact.id, version, artifact.size)
if tag:
# Look up by tag
tag_record = db.query(Tag).filter(
Tag.package_id == package.id,
Tag.name == tag,
).first()
if tag_record:
artifact = db.query(Artifact).filter(
Artifact.id == tag_record.artifact_id
).first()
if artifact:
return (artifact.id, tag, artifact.size)
return None
def _detect_package_cycle(
db: Session,
project_name: str,
package_name: str,
target_project: str,
target_package: str,
visiting: Set[str],
visited: Set[str],
path: List[str],
) -> Optional[List[str]]:
"""
Detect cycles at the package level using DFS.
Args:
db: Database session
project_name: Current project being visited
package_name: Current package being visited
target_project: The project we're checking for cycles back to
target_package: The package we're checking for cycles back to
visiting: Set of package keys currently in the recursion stack
visited: Set of fully processed package keys
path: Current path for cycle reporting
Returns:
Cycle path if detected, None otherwise
"""
pkg_key = f"{project_name}/{package_name}"
# Check if we've reached the target package (cycle detected)
if project_name == target_project and package_name == target_package:
return path + [pkg_key]
if pkg_key in visiting:
# Unexpected internal cycle
return None
if pkg_key in visited:
return None
visiting.add(pkg_key)
path.append(pkg_key)
# Get the package and find any artifacts with dependencies
project = db.query(Project).filter(Project.name == project_name).first()
if project:
package = db.query(Package).filter(
Package.project_id == project.id,
Package.name == package_name,
).first()
if package:
# Find all artifacts in this package via tags
tags = db.query(Tag).filter(Tag.package_id == package.id).all()
artifact_ids = {t.artifact_id for t in tags}
# Get dependencies from all artifacts in this package
for artifact_id in artifact_ids:
deps = db.query(ArtifactDependency).filter(
ArtifactDependency.artifact_id == artifact_id
).all()
for dep in deps:
cycle = _detect_package_cycle(
db,
dep.dependency_project,
dep.dependency_package,
target_project,
target_package,
visiting,
visited,
path,
)
if cycle:
return cycle
path.pop()
visiting.remove(pkg_key)
visited.add(pkg_key)
return None
def check_circular_dependencies(
db: Session,
artifact_id: str,
new_dependencies: List[EnsureFileDependency],
project_name: Optional[str] = None,
package_name: Optional[str] = None,
) -> Optional[List[str]]:
"""
Check if adding the new dependencies would create a circular dependency.
Args:
db: Database session
artifact_id: The artifact that will have these dependencies
new_dependencies: Dependencies to be added
project_name: Project name (optional, will try to look up from tag if not provided)
package_name: Package name (optional, will try to look up from tag if not provided)
Returns:
Cycle path if detected, None otherwise
"""
# First, get the package info for this artifact to build path labels
if project_name and package_name:
current_path = f"{project_name}/{package_name}"
else:
# Try to look up from tag
artifact = db.query(Artifact).filter(Artifact.id == artifact_id).first()
if not artifact:
return None
# Find package for this artifact
tag = db.query(Tag).filter(Tag.artifact_id == artifact_id).first()
if not tag:
return None
package = db.query(Package).filter(Package.id == tag.package_id).first()
if not package:
return None
project = db.query(Project).filter(Project.id == package.project_id).first()
if not project:
return None
current_path = f"{project.name}/{package.name}"
# Extract target project and package from current_path
if "/" in current_path:
target_project, target_package = current_path.split("/", 1)
else:
return None
# For each new dependency, check if it would create a cycle back to our package
for dep in new_dependencies:
# Check if this dependency (transitively) depends on us at the package level
visiting: Set[str] = set()
visited: Set[str] = set()
path: List[str] = [current_path]
# Check from the dependency's package
cycle = _detect_package_cycle(
db,
dep.project,
dep.package,
target_project,
target_package,
visiting,
visited,
path,
)
if cycle:
return cycle
return None
def resolve_dependencies(
db: Session,
project_name: str,
package_name: str,
ref: str,
base_url: str,
) -> DependencyResolutionResponse:
"""
Resolve all dependencies for an artifact recursively.
Args:
db: Database session
project_name: Project name
package_name: Package name
ref: Tag or version reference
base_url: Base URL for download URLs
Returns:
DependencyResolutionResponse with all resolved artifacts
Raises:
DependencyNotFoundError: If a dependency cannot be resolved
CircularDependencyError: If circular dependencies are detected
DependencyConflictError: If conflicting versions are required
"""
# Resolve the initial artifact
project = db.query(Project).filter(Project.name == project_name).first()
if not project:
raise DependencyNotFoundError(project_name, package_name, ref)
package = db.query(Package).filter(
Package.project_id == project.id,
Package.name == package_name,
).first()
if not package:
raise DependencyNotFoundError(project_name, package_name, ref)
# Try to find artifact by tag or version
resolved = _resolve_dependency_to_artifact(
db, project_name, package_name, ref, ref
)
if not resolved:
raise DependencyNotFoundError(project_name, package_name, ref)
root_artifact_id, root_version, root_size = resolved
# Track resolved artifacts and their versions
resolved_artifacts: Dict[str, ResolvedArtifact] = {}
# Track version requirements for conflict detection
version_requirements: Dict[str, List[Dict[str, Any]]] = {} # pkg_key -> [(version, required_by)]
# Track visiting/visited for cycle detection
visiting: Set[str] = set()
visited: Set[str] = set()
# Resolution order (topological)
resolution_order: List[str] = []
def _resolve_recursive(
artifact_id: str,
proj_name: str,
pkg_name: str,
version_or_tag: str,
size: int,
required_by: Optional[str],
depth: int = 0,
):
"""Recursively resolve dependencies with cycle/conflict detection."""
# Safety limit: prevent DoS through deeply nested dependencies
if depth > MAX_DEPENDENCY_DEPTH:
raise DependencyDepthExceededError(MAX_DEPENDENCY_DEPTH)
pkg_key = f"{proj_name}/{pkg_name}"
# Cycle detection (at artifact level)
if artifact_id in visiting:
# Build cycle path
raise CircularDependencyError([pkg_key, pkg_key])
# Conflict detection - check if we've seen this package before with a different version
if pkg_key in version_requirements:
existing_versions = {r["version"] for r in version_requirements[pkg_key]}
if version_or_tag not in existing_versions:
# Conflict detected - same package, different version
requirements = version_requirements[pkg_key] + [
{"version": version_or_tag, "required_by": required_by}
]
raise DependencyConflictError([
DependencyConflict(
project=proj_name,
package=pkg_name,
requirements=[
{
"version": r["version"],
"required_by": [{"path": r["required_by"]}] if r["required_by"] else []
}
for r in requirements
],
)
])
# Same version already resolved - skip
if artifact_id in visited:
return
if artifact_id in visited:
return
visiting.add(artifact_id)
# Track version requirement
if pkg_key not in version_requirements:
version_requirements[pkg_key] = []
version_requirements[pkg_key].append({
"version": version_or_tag,
"required_by": required_by,
})
# Get dependencies
deps = db.query(ArtifactDependency).filter(
ArtifactDependency.artifact_id == artifact_id
).all()
# Resolve each dependency first (depth-first)
for dep in deps:
resolved_dep = _resolve_dependency_to_artifact(
db,
dep.dependency_project,
dep.dependency_package,
dep.version_constraint,
dep.tag_constraint,
)
if not resolved_dep:
constraint = dep.version_constraint or dep.tag_constraint
raise DependencyNotFoundError(
dep.dependency_project,
dep.dependency_package,
constraint,
)
dep_artifact_id, dep_version, dep_size = resolved_dep
_resolve_recursive(
dep_artifact_id,
dep.dependency_project,
dep.dependency_package,
dep_version,
dep_size,
pkg_key,
depth + 1,
)
visiting.remove(artifact_id)
visited.add(artifact_id)
# Add to resolution order (dependencies before dependents)
resolution_order.append(artifact_id)
# Store resolved artifact info
resolved_artifacts[artifact_id] = ResolvedArtifact(
artifact_id=artifact_id,
project=proj_name,
package=pkg_name,
version=version_or_tag,
size=size,
download_url=f"{base_url}/api/v1/project/{proj_name}/{pkg_name}/+/{version_or_tag}",
)
# Start resolution from root
_resolve_recursive(
root_artifact_id,
project_name,
package_name,
root_version,
root_size,
None,
)
# Build response in topological order
resolved_list = [resolved_artifacts[aid] for aid in resolution_order]
total_size = sum(r.size for r in resolved_list)
return DependencyResolutionResponse(
requested={
"project": project_name,
"package": package_name,
"ref": ref,
},
resolved=resolved_list,
total_size=total_size,
artifact_count=len(resolved_list),
)

View File

@@ -117,6 +117,9 @@ class Artifact(Base):
tags = relationship("Tag", back_populates="artifact")
uploads = relationship("Upload", back_populates="artifact")
versions = relationship("PackageVersion", back_populates="artifact")
dependencies = relationship(
"ArtifactDependency", back_populates="artifact", cascade="all, delete-orphan"
)
@property
def sha256(self) -> str:
@@ -507,3 +510,54 @@ class PackageHistory(Base):
Index("idx_package_history_changed_at", "changed_at"),
Index("idx_package_history_package_changed_at", "package_id", "changed_at"),
)
class ArtifactDependency(Base):
"""Dependency declared by an artifact on another package.
Each artifact can declare dependencies on other packages, specifying either
an exact version or a tag. This enables recursive dependency resolution.
"""
__tablename__ = "artifact_dependencies"
id = Column(UUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
artifact_id = Column(
String(64),
ForeignKey("artifacts.id", ondelete="CASCADE"),
nullable=False,
)
dependency_project = Column(String(255), nullable=False)
dependency_package = Column(String(255), nullable=False)
version_constraint = Column(String(255), nullable=True)
tag_constraint = Column(String(255), nullable=True)
created_at = Column(DateTime(timezone=True), default=datetime.utcnow)
# Relationship to the artifact that declares this dependency
artifact = relationship("Artifact", back_populates="dependencies")
__table_args__ = (
# Exactly one of version_constraint or tag_constraint must be set
CheckConstraint(
"(version_constraint IS NOT NULL AND tag_constraint IS NULL) OR "
"(version_constraint IS NULL AND tag_constraint IS NOT NULL)",
name="check_constraint_type",
),
# Each artifact can only depend on a specific project/package once
Index(
"idx_artifact_dependencies_artifact_id",
"artifact_id",
),
Index(
"idx_artifact_dependencies_target",
"dependency_project",
"dependency_package",
),
Index(
"idx_artifact_dependencies_unique",
"artifact_id",
"dependency_project",
"dependency_package",
unique=True,
),
)

File diff suppressed because it is too large Load Diff

View File

@@ -412,6 +412,9 @@ class UploadResponse(BaseModel):
content_type: Optional[str] = None
original_name: Optional[str] = None
created_at: Optional[datetime] = None
# Upload metrics (Issue #43)
duration_ms: Optional[int] = None # Upload duration in milliseconds
throughput_mbps: Optional[float] = None # Upload throughput in MB/s
# Resumable upload schemas
@@ -478,6 +481,21 @@ class ResumableUploadStatusResponse(BaseModel):
total_uploaded_bytes: int
class UploadProgressResponse(BaseModel):
"""Progress information for an in-flight upload"""
upload_id: str
status: str # 'in_progress', 'completed', 'failed', 'not_found'
bytes_uploaded: int = 0
bytes_total: Optional[int] = None
percent_complete: Optional[float] = None
parts_uploaded: int = 0
parts_total: Optional[int] = None
started_at: Optional[datetime] = None
elapsed_seconds: Optional[float] = None
throughput_mbps: Optional[float] = None
# Consumer schemas
class ConsumerResponse(BaseModel):
id: UUID
@@ -898,3 +916,140 @@ class ProjectWithAccessResponse(ProjectResponse):
"""Project response with user's access level"""
user_access_level: Optional[str] = None
# Artifact Dependency schemas
class DependencyCreate(BaseModel):
"""Schema for creating a dependency"""
project: str
package: str
version: Optional[str] = None
tag: Optional[str] = None
@field_validator('version', 'tag')
@classmethod
def validate_constraint(cls, v, info):
return v
def model_post_init(self, __context):
"""Validate that exactly one of version or tag is set"""
if self.version is None and self.tag is None:
raise ValueError("Either 'version' or 'tag' must be specified")
if self.version is not None and self.tag is not None:
raise ValueError("Cannot specify both 'version' and 'tag'")
class DependencyResponse(BaseModel):
"""Schema for dependency response"""
id: UUID
artifact_id: str
project: str
package: str
version: Optional[str] = None
tag: Optional[str] = None
created_at: datetime
class Config:
from_attributes = True
@classmethod
def from_orm_model(cls, dep) -> "DependencyResponse":
"""Create from ORM model with field mapping"""
return cls(
id=dep.id,
artifact_id=dep.artifact_id,
project=dep.dependency_project,
package=dep.dependency_package,
version=dep.version_constraint,
tag=dep.tag_constraint,
created_at=dep.created_at,
)
class ArtifactDependenciesResponse(BaseModel):
"""Response containing all dependencies for an artifact"""
artifact_id: str
dependencies: List[DependencyResponse]
class DependentInfo(BaseModel):
"""Information about an artifact that depends on a package"""
artifact_id: str
project: str
package: str
version: Optional[str] = None
constraint_type: str # 'version' or 'tag'
constraint_value: str
class ReverseDependenciesResponse(BaseModel):
"""Response containing packages that depend on a given package"""
project: str
package: str
dependents: List[DependentInfo]
pagination: PaginationMeta
class EnsureFileDependency(BaseModel):
"""Dependency entry from orchard.ensure file"""
project: str
package: str
version: Optional[str] = None
tag: Optional[str] = None
@field_validator('version', 'tag')
@classmethod
def validate_constraint(cls, v, info):
return v
def model_post_init(self, __context):
"""Validate that exactly one of version or tag is set"""
if self.version is None and self.tag is None:
raise ValueError("Either 'version' or 'tag' must be specified")
if self.version is not None and self.tag is not None:
raise ValueError("Cannot specify both 'version' and 'tag'")
class EnsureFileContent(BaseModel):
"""Parsed content of orchard.ensure file"""
dependencies: List[EnsureFileDependency] = []
class ResolvedArtifact(BaseModel):
"""A resolved artifact in the dependency tree"""
artifact_id: str
project: str
package: str
version: Optional[str] = None
tag: Optional[str] = None
size: int
download_url: str
class DependencyResolutionResponse(BaseModel):
"""Response from dependency resolution endpoint"""
requested: Dict[str, str] # project, package, ref
resolved: List[ResolvedArtifact]
total_size: int
artifact_count: int
class DependencyConflict(BaseModel):
"""Details about a dependency conflict"""
project: str
package: str
requirements: List[Dict[str, Any]] # version/tag and required_by info
class DependencyConflictError(BaseModel):
"""Error response for dependency conflicts"""
error: str = "dependency_conflict"
message: str
conflicts: List[DependencyConflict]
class CircularDependencyError(BaseModel):
"""Error response for circular dependencies"""
error: str = "circular_dependency"
message: str
cycle: List[str] # List of "project/package" strings showing the cycle

View File

@@ -5,7 +5,7 @@ import hashlib
import logging
from sqlalchemy.orm import Session
from .models import Project, Package, Artifact, Tag, Upload, PackageVersion
from .models import Project, Package, Artifact, Tag, Upload, PackageVersion, ArtifactDependency
from .storage import get_storage
logger = logging.getLogger(__name__)
@@ -123,6 +123,17 @@ TEST_ARTIFACTS = [
},
]
# Dependencies to create (source artifact -> dependency)
# Format: (source_project, source_package, source_version, dep_project, dep_package, version_constraint, tag_constraint)
TEST_DEPENDENCIES = [
# ui-components v1.1.0 depends on design-tokens v1.0.0
("frontend-libs", "ui-components", "1.1.0", "frontend-libs", "design-tokens", "1.0.0", None),
# auth-lib v1.0.0 depends on common-utils v2.0.0
("backend-services", "auth-lib", "1.0.0", "backend-services", "common-utils", "2.0.0", None),
# auth-lib v1.0.0 also depends on design-tokens (stable tag)
("backend-services", "auth-lib", "1.0.0", "frontend-libs", "design-tokens", None, "latest"),
]
def is_database_empty(db: Session) -> bool:
"""Check if the database has any projects."""
@@ -240,6 +251,40 @@ def seed_database(db: Session) -> None:
db.add(tag)
tag_count += 1
db.flush()
# Create dependencies
dependency_count = 0
for dep_data in TEST_DEPENDENCIES:
src_project, src_package, src_version, dep_project, dep_package, version_constraint, tag_constraint = dep_data
# Find the source artifact by looking up its version
src_pkg = package_map.get((src_project, src_package))
if not src_pkg:
logger.warning(f"Source package not found: {src_project}/{src_package}")
continue
# Find the artifact for this version
src_version_record = db.query(PackageVersion).filter(
PackageVersion.package_id == src_pkg.id,
PackageVersion.version == src_version,
).first()
if not src_version_record:
logger.warning(f"Source version not found: {src_project}/{src_package}@{src_version}")
continue
# Create the dependency
dependency = ArtifactDependency(
artifact_id=src_version_record.artifact_id,
dependency_project=dep_project,
dependency_package=dep_package,
version_constraint=version_constraint,
tag_constraint=tag_constraint,
)
db.add(dependency)
dependency_count += 1
db.commit()
logger.info(f"Created {artifact_count} artifacts, {tag_count} tags, and {version_count} versions")
logger.info(f"Created {artifact_count} artifacts, {tag_count} tags, {version_count} versions, and {dependency_count} dependencies")
logger.info("Database seeding complete")

View File

@@ -242,15 +242,19 @@ class S3Storage:
},
)
self.client = boto3.client(
"s3",
endpoint_url=settings.s3_endpoint if settings.s3_endpoint else None,
region_name=settings.s3_region,
aws_access_key_id=settings.s3_access_key_id,
aws_secret_access_key=settings.s3_secret_access_key,
config=config,
verify=settings.s3_verify_ssl, # SSL/TLS verification
)
# Build client kwargs - only include credentials if explicitly provided
# This allows IRSA/IAM role credentials to be used when no explicit creds are set
client_kwargs = {
"endpoint_url": settings.s3_endpoint if settings.s3_endpoint else None,
"region_name": settings.s3_region,
"config": config,
"verify": settings.s3_verify_ssl,
}
if settings.s3_access_key_id and settings.s3_secret_access_key:
client_kwargs["aws_access_key_id"] = settings.s3_access_key_id
client_kwargs["aws_secret_access_key"] = settings.s3_secret_access_key
self.client = boto3.client("s3", **client_kwargs)
self.bucket = settings.s3_bucket
# Store active multipart uploads for resumable support
self._active_uploads: Dict[str, Dict[str, Any]] = {}
@@ -378,10 +382,16 @@ class S3Storage:
"""
# First pass: compute all hashes by streaming through file
try:
import time
sha256_hasher = hashlib.sha256()
md5_hasher = hashlib.md5()
sha1_hasher = hashlib.sha1()
size = 0
hash_start_time = time.time()
last_log_time = hash_start_time
log_interval_seconds = 5 # Log progress every 5 seconds
logger.info(f"Computing hashes for large file: expected_size={content_length}")
# Read file in chunks to compute hashes
while True:
@@ -393,6 +403,18 @@ class S3Storage:
sha1_hasher.update(chunk)
size += len(chunk)
# Log hash computation progress periodically
current_time = time.time()
if current_time - last_log_time >= log_interval_seconds:
elapsed = current_time - hash_start_time
percent = (size / content_length) * 100 if content_length > 0 else 0
throughput = (size / (1024 * 1024)) / elapsed if elapsed > 0 else 0
logger.info(
f"Hash computation progress: bytes={size}/{content_length} ({percent:.1f}%) "
f"throughput={throughput:.2f}MB/s"
)
last_log_time = current_time
# Enforce file size limit during streaming (protection against spoofing)
if size > settings.max_file_size:
raise FileSizeExceededError(
@@ -405,6 +427,14 @@ class S3Storage:
sha256_hash = sha256_hasher.hexdigest()
md5_hash = md5_hasher.hexdigest()
sha1_hash = sha1_hasher.hexdigest()
# Log hash computation completion
hash_elapsed = time.time() - hash_start_time
hash_throughput = (size / (1024 * 1024)) / hash_elapsed if hash_elapsed > 0 else 0
logger.info(
f"Hash computation completed: hash={sha256_hash[:16]}... "
f"size={size} duration={hash_elapsed:.2f}s throughput={hash_throughput:.2f}MB/s"
)
except (HashComputationError, FileSizeExceededError):
raise
except Exception as e:
@@ -458,8 +488,19 @@ class S3Storage:
upload_id = mpu["UploadId"]
try:
import time
parts = []
part_number = 1
bytes_uploaded = 0
upload_start_time = time.time()
last_log_time = upload_start_time
log_interval_seconds = 5 # Log progress every 5 seconds
total_parts = (content_length + MULTIPART_CHUNK_SIZE - 1) // MULTIPART_CHUNK_SIZE
logger.info(
f"Starting multipart upload: hash={sha256_hash[:16]}... "
f"size={content_length} parts={total_parts}"
)
while True:
chunk = file.read(MULTIPART_CHUNK_SIZE)
@@ -479,8 +520,32 @@ class S3Storage:
"ETag": response["ETag"],
}
)
bytes_uploaded += len(chunk)
# Log progress periodically
current_time = time.time()
if current_time - last_log_time >= log_interval_seconds:
elapsed = current_time - upload_start_time
percent = (bytes_uploaded / content_length) * 100
throughput = (bytes_uploaded / (1024 * 1024)) / elapsed if elapsed > 0 else 0
logger.info(
f"Upload progress: hash={sha256_hash[:16]}... "
f"part={part_number}/{total_parts} "
f"bytes={bytes_uploaded}/{content_length} ({percent:.1f}%) "
f"throughput={throughput:.2f}MB/s"
)
last_log_time = current_time
part_number += 1
# Log completion
total_elapsed = time.time() - upload_start_time
final_throughput = (content_length / (1024 * 1024)) / total_elapsed if total_elapsed > 0 else 0
logger.info(
f"Multipart upload completed: hash={sha256_hash[:16]}... "
f"size={content_length} duration={total_elapsed:.2f}s throughput={final_throughput:.2f}MB/s"
)
# Complete multipart upload
complete_response = self.client.complete_multipart_upload(
Bucket=self.bucket,
@@ -502,12 +567,28 @@ class S3Storage:
except Exception as e:
# Abort multipart upload on failure
logger.error(f"Multipart upload failed: {e}")
self.client.abort_multipart_upload(
Bucket=self.bucket,
Key=s3_key,
UploadId=upload_id,
error_str = str(e).lower()
is_client_disconnect = (
isinstance(e, (ConnectionResetError, BrokenPipeError)) or
"connection" in error_str or "broken pipe" in error_str or "reset" in error_str
)
if is_client_disconnect:
logger.warning(
f"Multipart upload aborted (client disconnect): hash={sha256_hash[:16]}... "
f"parts_uploaded={len(parts)} bytes_uploaded={bytes_uploaded}"
)
else:
logger.error(f"Multipart upload failed: hash={sha256_hash[:16]}... error={e}")
try:
self.client.abort_multipart_upload(
Bucket=self.bucket,
Key=s3_key,
UploadId=upload_id,
)
logger.info(f"Multipart upload aborted and cleaned up: upload_id={upload_id[:16]}...")
except Exception as abort_error:
logger.error(f"Failed to abort multipart upload: {abort_error}")
raise
def initiate_resumable_upload(self, expected_hash: str) -> Dict[str, Any]:
@@ -529,12 +610,17 @@ class S3Storage:
mpu = self.client.create_multipart_upload(Bucket=self.bucket, Key=s3_key)
upload_id = mpu["UploadId"]
import time
session = {
"upload_id": upload_id,
"s3_key": s3_key,
"already_exists": False,
"parts": [],
"expected_hash": expected_hash,
"started_at": time.time(),
"bytes_uploaded": 0,
"expected_size": None, # Set when init provides size
"status": "in_progress",
}
self._active_uploads[upload_id] = session
return session
@@ -561,10 +647,57 @@ class S3Storage:
part_info = {
"PartNumber": part_number,
"ETag": response["ETag"],
"size": len(data),
}
session["parts"].append(part_info)
session["bytes_uploaded"] = session.get("bytes_uploaded", 0) + len(data)
return part_info
def get_upload_progress(self, upload_id: str) -> Optional[Dict[str, Any]]:
"""
Get progress information for a resumable upload.
Returns None if upload not found.
"""
import time
session = self._active_uploads.get(upload_id)
if not session:
return None
bytes_uploaded = session.get("bytes_uploaded", 0)
expected_size = session.get("expected_size")
started_at = session.get("started_at")
progress = {
"upload_id": upload_id,
"status": session.get("status", "in_progress"),
"bytes_uploaded": bytes_uploaded,
"bytes_total": expected_size,
"parts_uploaded": len(session.get("parts", [])),
"parts_total": None,
"started_at": started_at,
"elapsed_seconds": None,
"percent_complete": None,
"throughput_mbps": None,
}
if expected_size and expected_size > 0:
progress["percent_complete"] = round((bytes_uploaded / expected_size) * 100, 2)
progress["parts_total"] = (expected_size + MULTIPART_CHUNK_SIZE - 1) // MULTIPART_CHUNK_SIZE
if started_at:
elapsed = time.time() - started_at
progress["elapsed_seconds"] = round(elapsed, 2)
if elapsed > 0 and bytes_uploaded > 0:
progress["throughput_mbps"] = round((bytes_uploaded / (1024 * 1024)) / elapsed, 2)
return progress
def set_upload_expected_size(self, upload_id: str, size: int):
"""Set the expected size for an upload (for progress tracking)."""
session = self._active_uploads.get(upload_id)
if session:
session["expected_size"] = size
def complete_resumable_upload(self, upload_id: str) -> Tuple[str, str]:
"""
Complete a resumable upload.
@@ -702,6 +835,36 @@ class S3Storage:
except ClientError:
return False
def delete_all(self) -> int:
"""
Delete all objects in the bucket.
Returns:
Number of objects deleted
"""
deleted_count = 0
try:
paginator = self.client.get_paginator("list_objects_v2")
for page in paginator.paginate(Bucket=self.bucket):
objects = page.get("Contents", [])
if not objects:
continue
# Delete objects in batches of 1000 (S3 limit)
delete_keys = [{"Key": obj["Key"]} for obj in objects]
if delete_keys:
self.client.delete_objects(
Bucket=self.bucket, Delete={"Objects": delete_keys}
)
deleted_count += len(delete_keys)
logger.info(f"Deleted {len(delete_keys)} objects from S3")
logger.info(f"Total objects deleted from S3: {deleted_count}")
return deleted_count
except ClientError as e:
logger.error(f"Failed to delete all S3 objects: {e}")
raise
def generate_presigned_url(
self,
s3_key: str,

View File

@@ -12,6 +12,8 @@ markers =
unit: Unit tests (no external dependencies)
integration: Integration tests (require database/storage)
slow: Slow tests (skip with -m "not slow")
large: Large file tests (100MB+, skip with -m "not large")
concurrent: Concurrent operation tests
# Coverage configuration
[coverage:run]

View File

@@ -9,6 +9,37 @@ This module provides:
import os
import pytest
# =============================================================================
# Pytest Markers
# =============================================================================
def pytest_configure(config):
"""Register custom pytest markers."""
config.addinivalue_line(
"markers",
"auth_intensive: marks tests that make many login requests (excluded from CI integration tests due to rate limiting)",
)
config.addinivalue_line(
"markers",
"integration: marks tests as integration tests",
)
config.addinivalue_line(
"markers",
"large: marks tests that handle large files (slow)",
)
config.addinivalue_line(
"markers",
"slow: marks tests as slow running",
)
config.addinivalue_line(
"markers",
"requires_direct_s3: marks tests that require direct S3/MinIO access (skipped in CI where S3 is not directly accessible)",
)
import io
from typing import Generator
from unittest.mock import MagicMock
@@ -25,6 +56,26 @@ os.environ.setdefault("ORCHARD_S3_BUCKET", "test-bucket")
os.environ.setdefault("ORCHARD_S3_ACCESS_KEY_ID", "test")
os.environ.setdefault("ORCHARD_S3_SECRET_ACCESS_KEY", "test")
# =============================================================================
# Admin Credentials Helper
# =============================================================================
def get_admin_password() -> str:
"""Get the admin password for test authentication.
Returns the password from ORCHARD_TEST_PASSWORD environment variable,
or 'changeme123' as the default for local development.
"""
return os.environ.get("ORCHARD_TEST_PASSWORD", "changeme123")
def get_admin_username() -> str:
"""Get the admin username for test authentication."""
return os.environ.get("ORCHARD_TEST_USERNAME", "admin")
# Re-export factory functions for backward compatibility
from tests.factories import (
create_test_file,
@@ -32,6 +83,8 @@ from tests.factories import (
compute_md5,
compute_sha1,
upload_test_file,
generate_content,
generate_content_with_hash,
TEST_CONTENT_HELLO,
TEST_HASH_HELLO,
TEST_MD5_HELLO,
@@ -179,29 +232,64 @@ def test_app():
# =============================================================================
@pytest.fixture
@pytest.fixture(scope="session")
def integration_client():
"""
Create an authenticated test client for integration tests.
Uses the real database and MinIO from docker-compose.local.yml.
Authenticates as admin for write operations.
Uses the real database and MinIO from docker-compose.local.yml or deployed environment.
Authenticates as admin for write operations. Session-scoped to reuse login across tests.
Environment variables:
ORCHARD_TEST_URL: Base URL of the Orchard server (default: http://localhost:8080)
ORCHARD_TEST_USERNAME: Admin username for authentication (default: admin)
ORCHARD_TEST_PASSWORD: Admin password for authentication (default: changeme123)
"""
from httpx import Client
import httpx
# Connect to the running orchard-server container
# Connect to the running orchard-server container or deployed environment
base_url = os.environ.get("ORCHARD_TEST_URL", "http://localhost:8080")
username = os.environ.get("ORCHARD_TEST_USERNAME", "admin")
password = os.environ.get("ORCHARD_TEST_PASSWORD", "changeme123")
with Client(base_url=base_url, timeout=30.0) as client:
with httpx.Client(base_url=base_url, timeout=30.0) as client:
# Login as admin to enable write operations
login_response = client.post(
"/api/v1/auth/login",
json={"username": "admin", "password": "changeme123"},
json={"username": username, "password": password},
)
# If login fails, tests will fail - that's expected if auth is broken
if login_response.status_code != 200:
# Try to continue without auth for backward compatibility
pass
pytest.fail(
f"Authentication failed against {base_url}: {login_response.status_code} - {login_response.text}. "
f"Set ORCHARD_TEST_USERNAME and ORCHARD_TEST_PASSWORD environment variables if using non-default credentials."
)
# Verify cookie was set
if not client.cookies:
pytest.fail(
f"Login succeeded but no session cookie was set. Response headers: {login_response.headers}"
)
yield client
@pytest.fixture
def auth_client():
"""
Create a function-scoped test client for authentication tests.
Unlike integration_client (session-scoped), this creates a fresh client
for each test. Use this for tests that manipulate authentication state
(login, logout, cookie clearing) to avoid polluting other tests.
Environment variables:
ORCHARD_TEST_URL: Base URL of the Orchard server (default: http://localhost:8080)
"""
import httpx
base_url = os.environ.get("ORCHARD_TEST_URL", "http://localhost:8080")
with httpx.Client(base_url=base_url, timeout=30.0) as client:
yield client
@@ -271,3 +359,41 @@ def test_content():
content = f"test-content-{uuid.uuid4().hex}".encode()
sha256 = compute_sha256(content)
return (content, sha256)
@pytest.fixture
def sized_content():
"""
Factory fixture for generating content of specific sizes.
Usage:
def test_example(sized_content):
content, hash = sized_content(1024) # 1KB
content, hash = sized_content(1024 * 1024) # 1MB
"""
def _generate(size: int, seed: int = None):
return generate_content_with_hash(size, seed)
return _generate
# =============================================================================
# Size Constants for Tests
# =============================================================================
# Common file sizes for boundary testing
SIZE_1B = 1
SIZE_1KB = 1024
SIZE_10KB = 10 * 1024
SIZE_100KB = 100 * 1024
SIZE_1MB = 1024 * 1024
SIZE_5MB = 5 * 1024 * 1024
SIZE_10MB = 10 * 1024 * 1024
SIZE_50MB = 50 * 1024 * 1024
SIZE_100MB = 100 * 1024 * 1024
SIZE_250MB = 250 * 1024 * 1024
SIZE_500MB = 500 * 1024 * 1024
SIZE_1GB = 1024 * 1024 * 1024
# Chunk size boundaries (based on typical S3 multipart chunk sizes)
CHUNK_SIZE = 64 * 1024 # 64KB typical chunk
MULTIPART_THRESHOLD = 100 * 1024 * 1024 # 100MB multipart threshold

View File

@@ -130,6 +130,41 @@ def upload_test_file(
return response.json()
def generate_content(size: int, seed: Optional[int] = None) -> bytes:
"""
Generate deterministic or random content of a specified size.
Args:
size: Size of content in bytes
seed: Optional seed for reproducible content (None for random)
Returns:
Bytes of the specified size
"""
if size == 0:
return b""
if seed is not None:
import random
rng = random.Random(seed)
return bytes(rng.randint(0, 255) for _ in range(size))
return os.urandom(size)
def generate_content_with_hash(size: int, seed: Optional[int] = None) -> tuple[bytes, str]:
"""
Generate content of specified size and compute its SHA256 hash.
Args:
size: Size of content in bytes
seed: Optional seed for reproducible content
Returns:
Tuple of (content_bytes, sha256_hash)
"""
content = generate_content(size, seed)
return content, compute_sha256(content)
# =============================================================================
# Project/Package Factories
# =============================================================================

View File

@@ -1,39 +1,50 @@
"""Integration tests for authentication API endpoints."""
"""Integration tests for authentication API endpoints.
Note: These tests are marked as auth_intensive because they make many login
requests. Dev/stage deployments have relaxed rate limits (1000/minute) to
allow these tests to run. Production uses strict rate limits (5/minute).
"""
import pytest
from uuid import uuid4
from tests.conftest import get_admin_password, get_admin_username
# Mark all tests in this module as auth_intensive (informational, not excluded from CI)
pytestmark = pytest.mark.auth_intensive
class TestAuthLogin:
"""Tests for login endpoint."""
@pytest.mark.integration
def test_login_success(self, integration_client):
def test_login_success(self, auth_client):
"""Test successful login with default admin credentials."""
response = integration_client.post(
response = auth_client.post(
"/api/v1/auth/login",
json={"username": "admin", "password": "changeme123"},
json={"username": get_admin_username(), "password": get_admin_password()},
)
assert response.status_code == 200
data = response.json()
assert data["username"] == "admin"
assert data["username"] == get_admin_username()
assert data["is_admin"] is True
assert "orchard_session" in response.cookies
@pytest.mark.integration
def test_login_invalid_password(self, integration_client):
def test_login_invalid_password(self, auth_client):
"""Test login with wrong password."""
response = integration_client.post(
response = auth_client.post(
"/api/v1/auth/login",
json={"username": "admin", "password": "wrongpassword"},
json={"username": get_admin_username(), "password": "wrongpassword"},
)
assert response.status_code == 401
assert "Invalid username or password" in response.json()["detail"]
@pytest.mark.integration
def test_login_nonexistent_user(self, integration_client):
def test_login_nonexistent_user(self, auth_client):
"""Test login with non-existent user."""
response = integration_client.post(
response = auth_client.post(
"/api/v1/auth/login",
json={"username": "nonexistent", "password": "password"},
)
@@ -44,24 +55,24 @@ class TestAuthLogout:
"""Tests for logout endpoint."""
@pytest.mark.integration
def test_logout_success(self, integration_client):
def test_logout_success(self, auth_client):
"""Test successful logout."""
# First login
login_response = integration_client.post(
login_response = auth_client.post(
"/api/v1/auth/login",
json={"username": "admin", "password": "changeme123"},
json={"username": get_admin_username(), "password": get_admin_password()},
)
assert login_response.status_code == 200
# Then logout
logout_response = integration_client.post("/api/v1/auth/logout")
logout_response = auth_client.post("/api/v1/auth/logout")
assert logout_response.status_code == 200
assert "Logged out successfully" in logout_response.json()["message"]
@pytest.mark.integration
def test_logout_without_session(self, integration_client):
def test_logout_without_session(self, auth_client):
"""Test logout without being logged in."""
response = integration_client.post("/api/v1/auth/logout")
response = auth_client.post("/api/v1/auth/logout")
# Should succeed even without session
assert response.status_code == 200
@@ -70,84 +81,105 @@ class TestAuthMe:
"""Tests for get current user endpoint."""
@pytest.mark.integration
def test_get_me_authenticated(self, integration_client):
def test_get_me_authenticated(self, auth_client):
"""Test getting current user when authenticated."""
# Login first
integration_client.post(
auth_client.post(
"/api/v1/auth/login",
json={"username": "admin", "password": "changeme123"},
json={"username": get_admin_username(), "password": get_admin_password()},
)
response = integration_client.get("/api/v1/auth/me")
response = auth_client.get("/api/v1/auth/me")
assert response.status_code == 200
data = response.json()
assert data["username"] == "admin"
assert data["username"] == get_admin_username()
assert data["is_admin"] is True
assert "id" in data
assert "created_at" in data
@pytest.mark.integration
def test_get_me_unauthenticated(self, integration_client):
def test_get_me_unauthenticated(self, auth_client):
"""Test getting current user without authentication."""
# Clear any existing cookies
integration_client.cookies.clear()
auth_client.cookies.clear()
response = integration_client.get("/api/v1/auth/me")
response = auth_client.get("/api/v1/auth/me")
assert response.status_code == 401
assert "Not authenticated" in response.json()["detail"]
class TestAuthChangePassword:
"""Tests for change password endpoint."""
"""Tests for change password endpoint.
Note: These tests use dedicated test users instead of admin to avoid
invalidating the integration_client session (which uses admin).
"""
@pytest.mark.integration
def test_change_password_success(self, integration_client):
def test_change_password_success(self, auth_client):
"""Test successful password change."""
# Login first
integration_client.post(
# Login as admin to create a test user
auth_client.post(
"/api/v1/auth/login",
json={"username": "admin", "password": "changeme123"},
json={"username": get_admin_username(), "password": get_admin_password()},
)
test_username = f"pwchange_{uuid4().hex[:8]}"
auth_client.post(
"/api/v1/admin/users",
json={"username": test_username, "password": "oldpassword123"},
)
# Login as test user
auth_client.cookies.clear()
auth_client.post(
"/api/v1/auth/login",
json={"username": test_username, "password": "oldpassword123"},
)
# Change password
response = integration_client.post(
response = auth_client.post(
"/api/v1/auth/change-password",
json={"current_password": "changeme123", "new_password": "newpassword123"},
json={"current_password": "oldpassword123", "new_password": "newpassword123"},
)
assert response.status_code == 200
# Verify old password no longer works
integration_client.cookies.clear()
response = integration_client.post(
auth_client.cookies.clear()
response = auth_client.post(
"/api/v1/auth/login",
json={"username": "admin", "password": "changeme123"},
json={"username": test_username, "password": "oldpassword123"},
)
assert response.status_code == 401
# Verify new password works
response = integration_client.post(
response = auth_client.post(
"/api/v1/auth/login",
json={"username": "admin", "password": "newpassword123"},
json={"username": test_username, "password": "newpassword123"},
)
assert response.status_code == 200
# Reset password back to original for other tests
reset_response = integration_client.post(
"/api/v1/auth/change-password",
json={"current_password": "newpassword123", "new_password": "changeme123"},
)
assert reset_response.status_code == 200, "Failed to reset admin password back to default"
@pytest.mark.integration
def test_change_password_wrong_current(self, integration_client):
def test_change_password_wrong_current(self, auth_client):
"""Test password change with wrong current password."""
# Login first
integration_client.post(
# Login as admin to create a test user
auth_client.post(
"/api/v1/auth/login",
json={"username": "admin", "password": "changeme123"},
json={"username": get_admin_username(), "password": get_admin_password()},
)
test_username = f"pwwrong_{uuid4().hex[:8]}"
auth_client.post(
"/api/v1/admin/users",
json={"username": test_username, "password": "password123"},
)
response = integration_client.post(
# Login as test user
auth_client.cookies.clear()
auth_client.post(
"/api/v1/auth/login",
json={"username": test_username, "password": "password123"},
)
response = auth_client.post(
"/api/v1/auth/change-password",
json={"current_password": "wrongpassword", "new_password": "newpassword"},
)
@@ -159,16 +191,16 @@ class TestAPIKeys:
"""Tests for API key management endpoints."""
@pytest.mark.integration
def test_create_and_list_api_key(self, integration_client):
def test_create_and_list_api_key(self, auth_client):
"""Test creating and listing API keys."""
# Login first
integration_client.post(
auth_client.post(
"/api/v1/auth/login",
json={"username": "admin", "password": "changeme123"},
json={"username": get_admin_username(), "password": get_admin_password()},
)
# Create API key
create_response = integration_client.post(
create_response = auth_client.post(
"/api/v1/auth/keys",
json={"name": "test-key", "description": "Test API key"},
)
@@ -182,23 +214,23 @@ class TestAPIKeys:
api_key = data["key"]
# List API keys
list_response = integration_client.get("/api/v1/auth/keys")
list_response = auth_client.get("/api/v1/auth/keys")
assert list_response.status_code == 200
keys = list_response.json()
assert any(k["id"] == key_id for k in keys)
# Clean up - delete the key
integration_client.delete(f"/api/v1/auth/keys/{key_id}")
auth_client.delete(f"/api/v1/auth/keys/{key_id}")
@pytest.mark.integration
def test_use_api_key_for_auth(self, integration_client):
def test_use_api_key_for_auth(self, auth_client):
"""Test using API key for authentication."""
# Login and create API key
integration_client.post(
auth_client.post(
"/api/v1/auth/login",
json={"username": "admin", "password": "changeme123"},
json={"username": get_admin_username(), "password": get_admin_password()},
)
create_response = integration_client.post(
create_response = auth_client.post(
"/api/v1/auth/keys",
json={"name": "auth-test-key"},
)
@@ -206,30 +238,30 @@ class TestAPIKeys:
key_id = create_response.json()["id"]
# Clear cookies and use API key
integration_client.cookies.clear()
response = integration_client.get(
auth_client.cookies.clear()
response = auth_client.get(
"/api/v1/auth/me",
headers={"Authorization": f"Bearer {api_key}"},
)
assert response.status_code == 200
assert response.json()["username"] == "admin"
assert response.json()["username"] == get_admin_username()
# Clean up
integration_client.post(
auth_client.post(
"/api/v1/auth/login",
json={"username": "admin", "password": "changeme123"},
json={"username": get_admin_username(), "password": get_admin_password()},
)
integration_client.delete(f"/api/v1/auth/keys/{key_id}")
auth_client.delete(f"/api/v1/auth/keys/{key_id}")
@pytest.mark.integration
def test_delete_api_key(self, integration_client):
def test_delete_api_key(self, auth_client):
"""Test revoking an API key."""
# Login and create API key
integration_client.post(
auth_client.post(
"/api/v1/auth/login",
json={"username": "admin", "password": "changeme123"},
json={"username": get_admin_username(), "password": get_admin_password()},
)
create_response = integration_client.post(
create_response = auth_client.post(
"/api/v1/auth/keys",
json={"name": "delete-test-key"},
)
@@ -237,12 +269,12 @@ class TestAPIKeys:
api_key = create_response.json()["key"]
# Delete the key
delete_response = integration_client.delete(f"/api/v1/auth/keys/{key_id}")
delete_response = auth_client.delete(f"/api/v1/auth/keys/{key_id}")
assert delete_response.status_code == 200
# Verify key no longer works
integration_client.cookies.clear()
response = integration_client.get(
auth_client.cookies.clear()
response = auth_client.get(
"/api/v1/auth/me",
headers={"Authorization": f"Bearer {api_key}"},
)
@@ -253,32 +285,32 @@ class TestAdminUserManagement:
"""Tests for admin user management endpoints."""
@pytest.mark.integration
def test_list_users(self, integration_client):
def test_list_users(self, auth_client):
"""Test listing users as admin."""
# Login as admin
integration_client.post(
auth_client.post(
"/api/v1/auth/login",
json={"username": "admin", "password": "changeme123"},
json={"username": get_admin_username(), "password": get_admin_password()},
)
response = integration_client.get("/api/v1/admin/users")
response = auth_client.get("/api/v1/admin/users")
assert response.status_code == 200
users = response.json()
assert len(users) >= 1
assert any(u["username"] == "admin" for u in users)
assert any(u["username"] == get_admin_username() for u in users)
@pytest.mark.integration
def test_create_user(self, integration_client):
def test_create_user(self, auth_client):
"""Test creating a new user as admin."""
# Login as admin
integration_client.post(
auth_client.post(
"/api/v1/auth/login",
json={"username": "admin", "password": "changeme123"},
json={"username": get_admin_username(), "password": get_admin_password()},
)
# Create new user
test_username = f"testuser_{uuid4().hex[:8]}"
response = integration_client.post(
response = auth_client.post(
"/api/v1/admin/users",
json={
"username": test_username,
@@ -293,31 +325,31 @@ class TestAdminUserManagement:
assert data["is_admin"] is False
# Verify new user can login
integration_client.cookies.clear()
login_response = integration_client.post(
auth_client.cookies.clear()
login_response = auth_client.post(
"/api/v1/auth/login",
json={"username": test_username, "password": "testpassword"},
)
assert login_response.status_code == 200
@pytest.mark.integration
def test_update_user(self, integration_client):
def test_update_user(self, auth_client):
"""Test updating a user as admin."""
# Login as admin
integration_client.post(
auth_client.post(
"/api/v1/auth/login",
json={"username": "admin", "password": "changeme123"},
json={"username": get_admin_username(), "password": get_admin_password()},
)
# Create a test user
test_username = f"updateuser_{uuid4().hex[:8]}"
integration_client.post(
auth_client.post(
"/api/v1/admin/users",
json={"username": test_username, "password": "password"},
)
# Update the user
response = integration_client.put(
response = auth_client.put(
f"/api/v1/admin/users/{test_username}",
json={"email": "updated@example.com", "is_admin": True},
)
@@ -327,59 +359,59 @@ class TestAdminUserManagement:
assert data["is_admin"] is True
@pytest.mark.integration
def test_reset_user_password(self, integration_client):
def test_reset_user_password(self, auth_client):
"""Test resetting a user's password as admin."""
# Login as admin
integration_client.post(
auth_client.post(
"/api/v1/auth/login",
json={"username": "admin", "password": "changeme123"},
json={"username": get_admin_username(), "password": get_admin_password()},
)
# Create a test user
test_username = f"resetuser_{uuid4().hex[:8]}"
integration_client.post(
auth_client.post(
"/api/v1/admin/users",
json={"username": test_username, "password": "oldpassword"},
)
# Reset password
response = integration_client.post(
response = auth_client.post(
f"/api/v1/admin/users/{test_username}/reset-password",
json={"new_password": "newpassword"},
)
assert response.status_code == 200
# Verify new password works
integration_client.cookies.clear()
login_response = integration_client.post(
auth_client.cookies.clear()
login_response = auth_client.post(
"/api/v1/auth/login",
json={"username": test_username, "password": "newpassword"},
)
assert login_response.status_code == 200
@pytest.mark.integration
def test_non_admin_cannot_access_admin_endpoints(self, integration_client):
def test_non_admin_cannot_access_admin_endpoints(self, auth_client):
"""Test that non-admin users cannot access admin endpoints."""
# Login as admin and create non-admin user
integration_client.post(
auth_client.post(
"/api/v1/auth/login",
json={"username": "admin", "password": "changeme123"},
json={"username": get_admin_username(), "password": get_admin_password()},
)
test_username = f"nonadmin_{uuid4().hex[:8]}"
integration_client.post(
auth_client.post(
"/api/v1/admin/users",
json={"username": test_username, "password": "password", "is_admin": False},
)
# Login as non-admin
integration_client.cookies.clear()
integration_client.post(
auth_client.cookies.clear()
auth_client.post(
"/api/v1/auth/login",
json={"username": test_username, "password": "password"},
)
# Try to access admin endpoints
response = integration_client.get("/api/v1/admin/users")
response = auth_client.get("/api/v1/admin/users")
assert response.status_code == 403
assert "Admin privileges required" in response.json()["detail"]
@@ -388,28 +420,28 @@ class TestSecurityEdgeCases:
"""Tests for security edge cases and validation."""
@pytest.mark.integration
def test_login_inactive_user(self, integration_client):
def test_login_inactive_user(self, auth_client):
"""Test that inactive users cannot login."""
# Login as admin and create a user
integration_client.post(
auth_client.post(
"/api/v1/auth/login",
json={"username": "admin", "password": "changeme123"},
json={"username": get_admin_username(), "password": get_admin_password()},
)
test_username = f"inactive_{uuid4().hex[:8]}"
integration_client.post(
auth_client.post(
"/api/v1/admin/users",
json={"username": test_username, "password": "password123"},
)
# Deactivate the user
integration_client.put(
auth_client.put(
f"/api/v1/admin/users/{test_username}",
json={"is_active": False},
)
# Try to login as inactive user
integration_client.cookies.clear()
response = integration_client.post(
auth_client.cookies.clear()
response = auth_client.post(
"/api/v1/auth/login",
json={"username": test_username, "password": "password123"},
)
@@ -417,14 +449,14 @@ class TestSecurityEdgeCases:
assert "Invalid username or password" in response.json()["detail"]
@pytest.mark.integration
def test_password_too_short_on_create(self, integration_client):
def test_password_too_short_on_create(self, auth_client):
"""Test that short passwords are rejected when creating users."""
integration_client.post(
auth_client.post(
"/api/v1/auth/login",
json={"username": "admin", "password": "changeme123"},
json={"username": get_admin_username(), "password": get_admin_password()},
)
response = integration_client.post(
response = auth_client.post(
"/api/v1/admin/users",
json={"username": f"shortpw_{uuid4().hex[:8]}", "password": "short"},
)
@@ -432,36 +464,49 @@ class TestSecurityEdgeCases:
assert "at least 8 characters" in response.json()["detail"]
@pytest.mark.integration
def test_password_too_short_on_change(self, integration_client):
def test_password_too_short_on_change(self, auth_client):
"""Test that short passwords are rejected when changing password."""
integration_client.post(
# Create test user
auth_client.post(
"/api/v1/auth/login",
json={"username": "admin", "password": "changeme123"},
json={"username": get_admin_username(), "password": get_admin_password()},
)
test_username = f"shortchange_{uuid4().hex[:8]}"
auth_client.post(
"/api/v1/admin/users",
json={"username": test_username, "password": "password123"},
)
response = integration_client.post(
# Login as test user
auth_client.cookies.clear()
auth_client.post(
"/api/v1/auth/login",
json={"username": test_username, "password": "password123"},
)
response = auth_client.post(
"/api/v1/auth/change-password",
json={"current_password": "changeme123", "new_password": "short"},
json={"current_password": "password123", "new_password": "short"},
)
assert response.status_code == 400
assert "at least 8 characters" in response.json()["detail"]
@pytest.mark.integration
def test_password_too_short_on_reset(self, integration_client):
def test_password_too_short_on_reset(self, auth_client):
"""Test that short passwords are rejected when resetting password."""
integration_client.post(
auth_client.post(
"/api/v1/auth/login",
json={"username": "admin", "password": "changeme123"},
json={"username": get_admin_username(), "password": get_admin_password()},
)
# Create a test user first
test_username = f"resetshort_{uuid4().hex[:8]}"
integration_client.post(
auth_client.post(
"/api/v1/admin/users",
json={"username": test_username, "password": "password123"},
)
response = integration_client.post(
response = auth_client.post(
f"/api/v1/admin/users/{test_username}/reset-password",
json={"new_password": "short"},
)
@@ -469,23 +514,23 @@ class TestSecurityEdgeCases:
assert "at least 8 characters" in response.json()["detail"]
@pytest.mark.integration
def test_duplicate_username_rejected(self, integration_client):
def test_duplicate_username_rejected(self, auth_client):
"""Test that duplicate usernames are rejected."""
integration_client.post(
auth_client.post(
"/api/v1/auth/login",
json={"username": "admin", "password": "changeme123"},
json={"username": get_admin_username(), "password": get_admin_password()},
)
test_username = f"duplicate_{uuid4().hex[:8]}"
# Create user first time
response1 = integration_client.post(
response1 = auth_client.post(
"/api/v1/admin/users",
json={"username": test_username, "password": "password123"},
)
assert response1.status_code == 200
# Try to create same username again
response2 = integration_client.post(
response2 = auth_client.post(
"/api/v1/admin/users",
json={"username": test_username, "password": "password456"},
)
@@ -493,14 +538,14 @@ class TestSecurityEdgeCases:
assert "already exists" in response2.json()["detail"]
@pytest.mark.integration
def test_cannot_delete_other_users_api_key(self, integration_client):
def test_cannot_delete_other_users_api_key(self, auth_client):
"""Test that users cannot delete API keys owned by other users."""
# Login as admin and create an API key
integration_client.post(
auth_client.post(
"/api/v1/auth/login",
json={"username": "admin", "password": "changeme123"},
json={"username": get_admin_username(), "password": get_admin_password()},
)
create_response = integration_client.post(
create_response = auth_client.post(
"/api/v1/auth/keys",
json={"name": "admin-key"},
)
@@ -508,253 +553,65 @@ class TestSecurityEdgeCases:
# Create a non-admin user
test_username = f"nonadmin_{uuid4().hex[:8]}"
integration_client.post(
auth_client.post(
"/api/v1/admin/users",
json={"username": test_username, "password": "password123"},
)
# Login as non-admin
integration_client.cookies.clear()
integration_client.post(
auth_client.cookies.clear()
auth_client.post(
"/api/v1/auth/login",
json={"username": test_username, "password": "password123"},
)
# Try to delete admin's API key
response = integration_client.delete(f"/api/v1/auth/keys/{admin_key_id}")
response = auth_client.delete(f"/api/v1/auth/keys/{admin_key_id}")
assert response.status_code == 403
assert "Cannot delete another user's API key" in response.json()["detail"]
# Cleanup: login as admin and delete the key
integration_client.cookies.clear()
integration_client.post(
auth_client.cookies.clear()
auth_client.post(
"/api/v1/auth/login",
json={"username": "admin", "password": "changeme123"},
json={"username": get_admin_username(), "password": get_admin_password()},
)
integration_client.delete(f"/api/v1/auth/keys/{admin_key_id}")
auth_client.delete(f"/api/v1/auth/keys/{admin_key_id}")
@pytest.mark.integration
def test_sessions_invalidated_on_password_change(self, integration_client):
def test_sessions_invalidated_on_password_change(self, auth_client):
"""Test that all sessions are invalidated when password is changed."""
# Create a test user
integration_client.post(
auth_client.post(
"/api/v1/auth/login",
json={"username": "admin", "password": "changeme123"},
json={"username": get_admin_username(), "password": get_admin_password()},
)
test_username = f"sessiontest_{uuid4().hex[:8]}"
integration_client.post(
auth_client.post(
"/api/v1/admin/users",
json={"username": test_username, "password": "password123"},
)
# Login as test user
integration_client.cookies.clear()
login_response = integration_client.post(
auth_client.cookies.clear()
login_response = auth_client.post(
"/api/v1/auth/login",
json={"username": test_username, "password": "password123"},
)
assert login_response.status_code == 200
# Verify session works
me_response = integration_client.get("/api/v1/auth/me")
me_response = auth_client.get("/api/v1/auth/me")
assert me_response.status_code == 200
# Change password
integration_client.post(
auth_client.post(
"/api/v1/auth/change-password",
json={"current_password": "password123", "new_password": "newpassword123"},
)
# Old session should be invalidated - try to access /me
# (note: the change-password call itself may have cleared the session cookie)
me_response2 = integration_client.get("/api/v1/auth/me")
# This should fail because all sessions were invalidated
assert me_response2.status_code == 401
class TestSecurityEdgeCases:
"""Tests for security edge cases and validation."""
@pytest.mark.integration
def test_login_inactive_user(self, integration_client):
"""Test that inactive users cannot login."""
# Login as admin and create a user
integration_client.post(
"/api/v1/auth/login",
json={"username": "admin", "password": "changeme123"},
)
test_username = f"inactive_{uuid4().hex[:8]}"
integration_client.post(
"/api/v1/admin/users",
json={"username": test_username, "password": "password123"},
)
# Deactivate the user
integration_client.put(
f"/api/v1/admin/users/{test_username}",
json={"is_active": False},
)
# Try to login as inactive user
integration_client.cookies.clear()
response = integration_client.post(
"/api/v1/auth/login",
json={"username": test_username, "password": "password123"},
)
assert response.status_code == 401
assert "Invalid username or password" in response.json()["detail"]
@pytest.mark.integration
def test_password_too_short_on_create(self, integration_client):
"""Test that short passwords are rejected when creating users."""
integration_client.post(
"/api/v1/auth/login",
json={"username": "admin", "password": "changeme123"},
)
response = integration_client.post(
"/api/v1/admin/users",
json={"username": f"shortpw_{uuid4().hex[:8]}", "password": "short"},
)
assert response.status_code == 400
assert "at least 8 characters" in response.json()["detail"]
@pytest.mark.integration
def test_password_too_short_on_change(self, integration_client):
"""Test that short passwords are rejected when changing password."""
integration_client.post(
"/api/v1/auth/login",
json={"username": "admin", "password": "changeme123"},
)
response = integration_client.post(
"/api/v1/auth/change-password",
json={"current_password": "changeme123", "new_password": "short"},
)
assert response.status_code == 400
assert "at least 8 characters" in response.json()["detail"]
@pytest.mark.integration
def test_password_too_short_on_reset(self, integration_client):
"""Test that short passwords are rejected when resetting password."""
integration_client.post(
"/api/v1/auth/login",
json={"username": "admin", "password": "changeme123"},
)
# Create a test user first
test_username = f"resetshort_{uuid4().hex[:8]}"
integration_client.post(
"/api/v1/admin/users",
json={"username": test_username, "password": "password123"},
)
response = integration_client.post(
f"/api/v1/admin/users/{test_username}/reset-password",
json={"new_password": "short"},
)
assert response.status_code == 400
assert "at least 8 characters" in response.json()["detail"]
@pytest.mark.integration
def test_duplicate_username_rejected(self, integration_client):
"""Test that duplicate usernames are rejected."""
integration_client.post(
"/api/v1/auth/login",
json={"username": "admin", "password": "changeme123"},
)
test_username = f"duplicate_{uuid4().hex[:8]}"
# Create user first time
response1 = integration_client.post(
"/api/v1/admin/users",
json={"username": test_username, "password": "password123"},
)
assert response1.status_code == 200
# Try to create same username again
response2 = integration_client.post(
"/api/v1/admin/users",
json={"username": test_username, "password": "password456"},
)
assert response2.status_code == 409
assert "already exists" in response2.json()["detail"]
@pytest.mark.integration
def test_cannot_delete_other_users_api_key(self, integration_client):
"""Test that users cannot delete API keys owned by other users."""
# Login as admin and create an API key
integration_client.post(
"/api/v1/auth/login",
json={"username": "admin", "password": "changeme123"},
)
create_response = integration_client.post(
"/api/v1/auth/keys",
json={"name": "admin-key"},
)
admin_key_id = create_response.json()["id"]
# Create a non-admin user
test_username = f"nonadmin_{uuid4().hex[:8]}"
integration_client.post(
"/api/v1/admin/users",
json={"username": test_username, "password": "password123"},
)
# Login as non-admin
integration_client.cookies.clear()
integration_client.post(
"/api/v1/auth/login",
json={"username": test_username, "password": "password123"},
)
# Try to delete admin's API key
response = integration_client.delete(f"/api/v1/auth/keys/{admin_key_id}")
assert response.status_code == 403
assert "Cannot delete another user's API key" in response.json()["detail"]
# Cleanup: login as admin and delete the key
integration_client.cookies.clear()
integration_client.post(
"/api/v1/auth/login",
json={"username": "admin", "password": "changeme123"},
)
integration_client.delete(f"/api/v1/auth/keys/{admin_key_id}")
@pytest.mark.integration
def test_sessions_invalidated_on_password_change(self, integration_client):
"""Test that all sessions are invalidated when password is changed."""
# Create a test user
integration_client.post(
"/api/v1/auth/login",
json={"username": "admin", "password": "changeme123"},
)
test_username = f"sessiontest_{uuid4().hex[:8]}"
integration_client.post(
"/api/v1/admin/users",
json={"username": test_username, "password": "password123"},
)
# Login as test user
integration_client.cookies.clear()
login_response = integration_client.post(
"/api/v1/auth/login",
json={"username": test_username, "password": "password123"},
)
assert login_response.status_code == 200
# Verify session works
me_response = integration_client.get("/api/v1/auth/me")
assert me_response.status_code == 200
# Change password
integration_client.post(
"/api/v1/auth/change-password",
json={"current_password": "password123", "new_password": "newpassword123"},
)
# Old session should be invalidated - try to access /me
# (note: the change-password call itself may have cleared the session cookie)
me_response2 = integration_client.get("/api/v1/auth/me")
me_response2 = auth_client.get("/api/v1/auth/me")
# This should fail because all sessions were invalidated
assert me_response2.status_code == 401

View File

@@ -0,0 +1,737 @@
"""
Integration tests for concurrent upload and download operations.
Tests cover:
- Concurrent uploads of different files
- Concurrent uploads of same file (deduplication race)
- Concurrent downloads of same artifact
- Concurrent downloads of different artifacts
- Mixed concurrent uploads and downloads
- Data corruption prevention under concurrency
"""
import pytest
import io
import os
from concurrent.futures import ThreadPoolExecutor, as_completed
from tests.factories import (
compute_sha256,
upload_test_file,
generate_content_with_hash,
)
def get_api_key(integration_client):
"""Create an API key for concurrent test workers."""
import uuid
response = integration_client.post(
"/api/v1/auth/keys",
json={"name": f"concurrent-test-{uuid.uuid4().hex[:8]}"},
)
if response.status_code == 200:
return response.json()["key"]
return None
class TestConcurrentUploads:
"""Tests for concurrent upload operations."""
@pytest.mark.integration
@pytest.mark.concurrent
def test_2_concurrent_uploads_different_files(self, integration_client, test_package):
"""Test 2 concurrent uploads of different files."""
project, package = test_package
api_key = get_api_key(integration_client)
assert api_key, "Failed to create API key"
files_data = [
generate_content_with_hash(1024, seed=i) for i in range(2)
]
results = []
errors = []
def upload_worker(idx, content, expected_hash):
try:
from httpx import Client
base_url = os.environ.get("ORCHARD_TEST_URL", "http://localhost:8080")
with Client(base_url=base_url, timeout=60.0) as client:
files = {
"file": (f"file-{idx}.bin", io.BytesIO(content), "application/octet-stream")
}
response = client.post(
f"/api/v1/project/{project}/{package}/upload",
files=files,
data={"tag": f"concurrent-{idx}"},
headers={"Authorization": f"Bearer {api_key}"},
)
if response.status_code == 200:
result = response.json()
results.append((idx, result, expected_hash))
else:
errors.append(f"Worker {idx}: Status {response.status_code}: {response.text}")
except Exception as e:
errors.append(f"Worker {idx}: {str(e)}")
with ThreadPoolExecutor(max_workers=2) as executor:
futures = [
executor.submit(upload_worker, i, content, hash)
for i, (content, hash) in enumerate(files_data)
]
for future in as_completed(futures):
pass
assert len(errors) == 0, f"Errors: {errors}"
assert len(results) == 2
# Verify each upload returned correct artifact_id
for idx, result, expected_hash in results:
assert result["artifact_id"] == expected_hash
@pytest.mark.integration
@pytest.mark.concurrent
def test_5_concurrent_uploads_different_files(self, integration_client, test_package):
"""Test 5 concurrent uploads of different files."""
project, package = test_package
api_key = get_api_key(integration_client)
assert api_key, "Failed to create API key"
num_files = 5
files_data = [
generate_content_with_hash(2048, seed=100 + i) for i in range(num_files)
]
results = []
errors = []
def upload_worker(idx, content, expected_hash):
try:
from httpx import Client
base_url = os.environ.get("ORCHARD_TEST_URL", "http://localhost:8080")
with Client(base_url=base_url, timeout=60.0) as client:
files = {
"file": (f"file-{idx}.bin", io.BytesIO(content), "application/octet-stream")
}
response = client.post(
f"/api/v1/project/{project}/{package}/upload",
files=files,
data={"tag": f"concurrent5-{idx}"},
headers={"Authorization": f"Bearer {api_key}"},
)
if response.status_code == 200:
result = response.json()
results.append((idx, result, expected_hash))
else:
errors.append(f"Worker {idx}: Status {response.status_code}")
except Exception as e:
errors.append(f"Worker {idx}: {str(e)}")
with ThreadPoolExecutor(max_workers=num_files) as executor:
futures = [
executor.submit(upload_worker, i, content, hash)
for i, (content, hash) in enumerate(files_data)
]
for future in as_completed(futures):
pass
assert len(errors) == 0, f"Errors: {errors}"
assert len(results) == num_files
# Verify all uploads have unique artifact_ids
artifact_ids = set(r[1]["artifact_id"] for r in results)
assert len(artifact_ids) == num_files
@pytest.mark.integration
@pytest.mark.concurrent
def test_10_concurrent_uploads_different_files(self, integration_client, test_package):
"""Test 10 concurrent uploads of different files."""
project, package = test_package
api_key = get_api_key(integration_client)
assert api_key, "Failed to create API key"
num_files = 10
files_data = [
generate_content_with_hash(1024, seed=200 + i) for i in range(num_files)
]
results = []
errors = []
def upload_worker(idx, content, expected_hash):
try:
from httpx import Client
base_url = os.environ.get("ORCHARD_TEST_URL", "http://localhost:8080")
with Client(base_url=base_url, timeout=60.0) as client:
files = {
"file": (f"file-{idx}.bin", io.BytesIO(content), "application/octet-stream")
}
response = client.post(
f"/api/v1/project/{project}/{package}/upload",
files=files,
data={"tag": f"concurrent10-{idx}"},
headers={"Authorization": f"Bearer {api_key}"},
)
if response.status_code == 200:
result = response.json()
results.append((idx, result, expected_hash))
else:
errors.append(f"Worker {idx}: Status {response.status_code}")
except Exception as e:
errors.append(f"Worker {idx}: {str(e)}")
with ThreadPoolExecutor(max_workers=num_files) as executor:
futures = [
executor.submit(upload_worker, i, content, hash)
for i, (content, hash) in enumerate(files_data)
]
for future in as_completed(futures):
pass
assert len(errors) == 0, f"Errors: {errors}"
assert len(results) == num_files
@pytest.mark.integration
@pytest.mark.concurrent
def test_concurrent_uploads_same_file_deduplication(self, integration_client, test_package):
"""Test concurrent uploads of same file handle deduplication correctly."""
project, package = test_package
api_key = get_api_key(integration_client)
assert api_key, "Failed to create API key"
content, expected_hash = generate_content_with_hash(4096, seed=999)
num_concurrent = 5
results = []
errors = []
def upload_worker(idx):
try:
from httpx import Client
base_url = os.environ.get("ORCHARD_TEST_URL", "http://localhost:8080")
with Client(base_url=base_url, timeout=60.0) as client:
files = {
"file": (f"same-{idx}.bin", io.BytesIO(content), "application/octet-stream")
}
response = client.post(
f"/api/v1/project/{project}/{package}/upload",
files=files,
data={"tag": f"dedup-{idx}"},
headers={"Authorization": f"Bearer {api_key}"},
)
if response.status_code == 200:
results.append(response.json())
else:
errors.append(f"Worker {idx}: Status {response.status_code}")
except Exception as e:
errors.append(f"Worker {idx}: {str(e)}")
with ThreadPoolExecutor(max_workers=num_concurrent) as executor:
futures = [executor.submit(upload_worker, i) for i in range(num_concurrent)]
for future in as_completed(futures):
pass
assert len(errors) == 0, f"Errors: {errors}"
assert len(results) == num_concurrent
# All should have same artifact_id
artifact_ids = set(r["artifact_id"] for r in results)
assert len(artifact_ids) == 1
assert expected_hash in artifact_ids
# Verify final ref_count equals number of uploads
response = integration_client.get(f"/api/v1/artifact/{expected_hash}")
assert response.status_code == 200
assert response.json()["ref_count"] == num_concurrent
@pytest.mark.integration
@pytest.mark.concurrent
def test_concurrent_uploads_to_different_packages(self, integration_client, test_project, unique_test_id):
"""Test concurrent uploads to different packages."""
project = test_project
api_key = get_api_key(integration_client)
assert api_key, "Failed to create API key"
num_packages = 3
package_names = []
# Create multiple packages
for i in range(num_packages):
pkg_name = f"pkg-{unique_test_id}-{i}"
response = integration_client.post(
f"/api/v1/project/{project}/packages",
json={"name": pkg_name, "description": f"Package {i}"},
)
assert response.status_code == 200
package_names.append(pkg_name)
files_data = [
generate_content_with_hash(1024, seed=300 + i) for i in range(num_packages)
]
results = []
errors = []
def upload_worker(idx, package, content, expected_hash):
try:
from httpx import Client
base_url = os.environ.get("ORCHARD_TEST_URL", "http://localhost:8080")
with Client(base_url=base_url, timeout=60.0) as client:
files = {
"file": (f"file-{idx}.bin", io.BytesIO(content), "application/octet-stream")
}
response = client.post(
f"/api/v1/project/{project}/{package}/upload",
files=files,
data={"tag": "latest"},
headers={"Authorization": f"Bearer {api_key}"},
)
if response.status_code == 200:
result = response.json()
results.append((package, result, expected_hash))
else:
errors.append(f"Worker {idx}: Status {response.status_code}")
except Exception as e:
errors.append(f"Worker {idx}: {str(e)}")
with ThreadPoolExecutor(max_workers=num_packages) as executor:
futures = [
executor.submit(upload_worker, i, package_names[i], content, hash)
for i, (content, hash) in enumerate(files_data)
]
for future in as_completed(futures):
pass
assert len(errors) == 0, f"Errors: {errors}"
assert len(results) == num_packages
class TestConcurrentDownloads:
"""Tests for concurrent download operations."""
@pytest.mark.integration
@pytest.mark.concurrent
def test_2_concurrent_downloads_same_artifact(self, integration_client, test_package):
"""Test 2 concurrent downloads of same artifact."""
project, package = test_package
content, expected_hash = generate_content_with_hash(2048, seed=400)
# Upload first
upload_test_file(integration_client, project, package, content, tag="download-test")
results = []
errors = []
def download_worker(idx):
try:
from httpx import Client
base_url = os.environ.get("ORCHARD_TEST_URL", "http://localhost:8080")
with Client(base_url=base_url, timeout=60.0) as client:
response = client.get(
f"/api/v1/project/{project}/{package}/+/download-test",
params={"mode": "proxy"},
)
if response.status_code == 200:
results.append((idx, response.content))
else:
errors.append(f"Worker {idx}: Status {response.status_code}")
except Exception as e:
errors.append(f"Worker {idx}: {str(e)}")
with ThreadPoolExecutor(max_workers=2) as executor:
futures = [executor.submit(download_worker, i) for i in range(2)]
for future in as_completed(futures):
pass
assert len(errors) == 0, f"Errors: {errors}"
assert len(results) == 2
# All downloads should match original
for idx, downloaded in results:
assert downloaded == content
@pytest.mark.integration
@pytest.mark.concurrent
def test_5_concurrent_downloads_same_artifact(self, integration_client, test_package):
"""Test 5 concurrent downloads of same artifact."""
project, package = test_package
content, expected_hash = generate_content_with_hash(4096, seed=500)
upload_test_file(integration_client, project, package, content, tag="download5-test")
num_downloads = 5
results = []
errors = []
def download_worker(idx):
try:
from httpx import Client
base_url = os.environ.get("ORCHARD_TEST_URL", "http://localhost:8080")
with Client(base_url=base_url, timeout=60.0) as client:
response = client.get(
f"/api/v1/project/{project}/{package}/+/download5-test",
params={"mode": "proxy"},
)
if response.status_code == 200:
results.append((idx, response.content))
else:
errors.append(f"Worker {idx}: Status {response.status_code}")
except Exception as e:
errors.append(f"Worker {idx}: {str(e)}")
with ThreadPoolExecutor(max_workers=num_downloads) as executor:
futures = [executor.submit(download_worker, i) for i in range(num_downloads)]
for future in as_completed(futures):
pass
assert len(errors) == 0, f"Errors: {errors}"
assert len(results) == num_downloads
for idx, downloaded in results:
assert downloaded == content
@pytest.mark.integration
@pytest.mark.concurrent
def test_10_concurrent_downloads_same_artifact(self, integration_client, test_package):
"""Test 10 concurrent downloads of same artifact."""
project, package = test_package
content, expected_hash = generate_content_with_hash(8192, seed=600)
upload_test_file(integration_client, project, package, content, tag="download10-test")
num_downloads = 10
results = []
errors = []
def download_worker(idx):
try:
from httpx import Client
base_url = os.environ.get("ORCHARD_TEST_URL", "http://localhost:8080")
with Client(base_url=base_url, timeout=60.0) as client:
response = client.get(
f"/api/v1/project/{project}/{package}/+/download10-test",
params={"mode": "proxy"},
)
if response.status_code == 200:
results.append((idx, response.content))
else:
errors.append(f"Worker {idx}: Status {response.status_code}")
except Exception as e:
errors.append(f"Worker {idx}: {str(e)}")
with ThreadPoolExecutor(max_workers=num_downloads) as executor:
futures = [executor.submit(download_worker, i) for i in range(num_downloads)]
for future in as_completed(futures):
pass
assert len(errors) == 0, f"Errors: {errors}"
assert len(results) == num_downloads
for idx, downloaded in results:
assert downloaded == content
@pytest.mark.integration
@pytest.mark.concurrent
def test_concurrent_downloads_different_artifacts(self, integration_client, test_package):
"""Test concurrent downloads of different artifacts."""
project, package = test_package
# Upload multiple files
num_files = 5
uploads = []
for i in range(num_files):
content, expected_hash = generate_content_with_hash(1024, seed=700 + i)
upload_test_file(
integration_client, project, package, content,
tag=f"multi-download-{i}"
)
uploads.append((f"multi-download-{i}", content))
results = []
errors = []
def download_worker(tag, expected_content):
try:
from httpx import Client
base_url = os.environ.get("ORCHARD_TEST_URL", "http://localhost:8080")
with Client(base_url=base_url, timeout=60.0) as client:
response = client.get(
f"/api/v1/project/{project}/{package}/+/{tag}",
params={"mode": "proxy"},
)
if response.status_code == 200:
results.append((tag, response.content, expected_content))
else:
errors.append(f"Tag {tag}: Status {response.status_code}")
except Exception as e:
errors.append(f"Tag {tag}: {str(e)}")
with ThreadPoolExecutor(max_workers=num_files) as executor:
futures = [
executor.submit(download_worker, tag, content)
for tag, content in uploads
]
for future in as_completed(futures):
pass
assert len(errors) == 0, f"Errors: {errors}"
assert len(results) == num_files
for tag, downloaded, expected in results:
assert downloaded == expected, f"Content mismatch for {tag}"
class TestMixedConcurrentOperations:
"""Tests for mixed concurrent upload and download operations."""
@pytest.mark.integration
@pytest.mark.concurrent
def test_upload_while_download_in_progress(self, integration_client, test_package):
"""Test uploading while a download is in progress."""
project, package = test_package
api_key = get_api_key(integration_client)
assert api_key, "Failed to create API key"
# Upload initial content
content1, hash1 = generate_content_with_hash(10240, seed=800) # 10KB
upload_test_file(integration_client, project, package, content1, tag="initial")
# New content for upload during download
content2, hash2 = generate_content_with_hash(10240, seed=801)
results = {"downloads": [], "uploads": []}
errors = []
def download_worker():
try:
from httpx import Client
base_url = os.environ.get("ORCHARD_TEST_URL", "http://localhost:8080")
with Client(base_url=base_url, timeout=60.0) as client:
response = client.get(
f"/api/v1/project/{project}/{package}/+/initial",
params={"mode": "proxy"},
)
if response.status_code == 200:
results["downloads"].append(response.content)
else:
errors.append(f"Download: Status {response.status_code}")
except Exception as e:
errors.append(f"Download: {str(e)}")
def upload_worker():
try:
from httpx import Client
base_url = os.environ.get("ORCHARD_TEST_URL", "http://localhost:8080")
with Client(base_url=base_url, timeout=60.0) as client:
files = {
"file": ("new.bin", io.BytesIO(content2), "application/octet-stream")
}
response = client.post(
f"/api/v1/project/{project}/{package}/upload",
files=files,
data={"tag": "during-download"},
headers={"Authorization": f"Bearer {api_key}"},
)
if response.status_code == 200:
results["uploads"].append(response.json())
else:
errors.append(f"Upload: Status {response.status_code}")
except Exception as e:
errors.append(f"Upload: {str(e)}")
with ThreadPoolExecutor(max_workers=2) as executor:
futures = [
executor.submit(download_worker),
executor.submit(upload_worker),
]
for future in as_completed(futures):
pass
assert len(errors) == 0, f"Errors: {errors}"
assert len(results["downloads"]) == 1
assert len(results["uploads"]) == 1
# Verify download got correct content
assert results["downloads"][0] == content1
# Verify upload succeeded
assert results["uploads"][0]["artifact_id"] == hash2
@pytest.mark.integration
@pytest.mark.concurrent
def test_multiple_uploads_and_downloads_simultaneously(self, integration_client, test_package):
"""Test multiple uploads and downloads running simultaneously."""
project, package = test_package
api_key = get_api_key(integration_client)
assert api_key, "Failed to create API key"
# Pre-upload some files for downloading
existing_files = []
for i in range(3):
content, hash = generate_content_with_hash(2048, seed=900 + i)
upload_test_file(integration_client, project, package, content, tag=f"existing-{i}")
existing_files.append((f"existing-{i}", content))
# New files for uploading
new_files = [
generate_content_with_hash(2048, seed=910 + i) for i in range(3)
]
results = {"downloads": [], "uploads": []}
errors = []
def download_worker(tag, expected):
try:
from httpx import Client
base_url = os.environ.get("ORCHARD_TEST_URL", "http://localhost:8080")
with Client(base_url=base_url, timeout=60.0) as client:
response = client.get(
f"/api/v1/project/{project}/{package}/+/{tag}",
params={"mode": "proxy"},
)
if response.status_code == 200:
results["downloads"].append((tag, response.content, expected))
else:
errors.append(f"Download {tag}: Status {response.status_code}")
except Exception as e:
errors.append(f"Download {tag}: {str(e)}")
def upload_worker(idx, content, expected_hash):
try:
from httpx import Client
base_url = os.environ.get("ORCHARD_TEST_URL", "http://localhost:8080")
with Client(base_url=base_url, timeout=60.0) as client:
files = {
"file": (f"new-{idx}.bin", io.BytesIO(content), "application/octet-stream")
}
response = client.post(
f"/api/v1/project/{project}/{package}/upload",
files=files,
data={"tag": f"new-{idx}"},
headers={"Authorization": f"Bearer {api_key}"},
)
if response.status_code == 200:
results["uploads"].append((idx, response.json(), expected_hash))
else:
errors.append(f"Upload {idx}: Status {response.status_code}")
except Exception as e:
errors.append(f"Upload {idx}: {str(e)}")
with ThreadPoolExecutor(max_workers=6) as executor:
futures = []
# Submit downloads
for tag, content in existing_files:
futures.append(executor.submit(download_worker, tag, content))
# Submit uploads
for i, (content, hash) in enumerate(new_files):
futures.append(executor.submit(upload_worker, i, content, hash))
for future in as_completed(futures):
pass
assert len(errors) == 0, f"Errors: {errors}"
assert len(results["downloads"]) == 3
assert len(results["uploads"]) == 3
# Verify downloads
for tag, downloaded, expected in results["downloads"]:
assert downloaded == expected, f"Download mismatch for {tag}"
# Verify uploads
for idx, result, expected_hash in results["uploads"]:
assert result["artifact_id"] == expected_hash
@pytest.mark.integration
@pytest.mark.concurrent
def test_no_data_corruption_under_concurrency(self, integration_client, test_package):
"""Test that no data corruption occurs under concurrent operations."""
project, package = test_package
api_key = get_api_key(integration_client)
assert api_key, "Failed to create API key"
# Create content with recognizable patterns
num_files = 5
files_data = []
for i in range(num_files):
# Each file has unique repeating pattern for easy corruption detection
pattern = bytes([i] * 256)
content = pattern * 40 # 10KB each
hash = compute_sha256(content)
files_data.append((content, hash))
results = []
errors = []
def upload_and_verify(idx, content, expected_hash):
try:
from httpx import Client
base_url = os.environ.get("ORCHARD_TEST_URL", "http://localhost:8080")
with Client(base_url=base_url, timeout=60.0) as client:
# Upload
files = {
"file": (f"pattern-{idx}.bin", io.BytesIO(content), "application/octet-stream")
}
upload_resp = client.post(
f"/api/v1/project/{project}/{package}/upload",
files=files,
data={"tag": f"pattern-{idx}"},
headers={"Authorization": f"Bearer {api_key}"},
)
if upload_resp.status_code != 200:
errors.append(f"Upload {idx}: Status {upload_resp.status_code}")
return
upload_result = upload_resp.json()
if upload_result["artifact_id"] != expected_hash:
errors.append(f"Upload {idx}: Hash mismatch")
return
# Immediately download and verify
download_resp = client.get(
f"/api/v1/project/{project}/{package}/+/pattern-{idx}",
params={"mode": "proxy"},
)
if download_resp.status_code != 200:
errors.append(f"Download {idx}: Status {download_resp.status_code}")
return
if download_resp.content != content:
errors.append(f"Worker {idx}: DATA CORRUPTION DETECTED")
return
# Verify the downloaded content hash
downloaded_hash = compute_sha256(download_resp.content)
if downloaded_hash != expected_hash:
errors.append(f"Worker {idx}: Hash verification failed")
return
results.append(idx)
except Exception as e:
errors.append(f"Worker {idx}: {str(e)}")
with ThreadPoolExecutor(max_workers=num_files) as executor:
futures = [
executor.submit(upload_and_verify, i, content, hash)
for i, (content, hash) in enumerate(files_data)
]
for future in as_completed(futures):
pass
assert len(errors) == 0, f"Errors: {errors}"
assert len(results) == num_files

View File

@@ -0,0 +1,322 @@
"""
Integration tests for error handling in upload and download operations.
Tests cover:
- Timeout handling
- Invalid request handling
- Resource cleanup on failures
- Graceful error responses
"""
import pytest
import io
import time
from tests.factories import (
compute_sha256,
upload_test_file,
generate_content_with_hash,
)
class TestUploadErrorHandling:
"""Tests for upload error handling."""
@pytest.mark.integration
def test_upload_to_nonexistent_project_returns_404(
self, integration_client, unique_test_id
):
"""Test upload to nonexistent project returns 404."""
content = b"test content for nonexistent project"
files = {"file": ("test.bin", io.BytesIO(content), "application/octet-stream")}
response = integration_client.post(
f"/api/v1/project/nonexistent-project-{unique_test_id}/nonexistent-pkg/upload",
files=files,
)
assert response.status_code == 404
@pytest.mark.integration
def test_upload_to_nonexistent_package_returns_404(
self, integration_client, test_project, unique_test_id
):
"""Test upload to nonexistent package returns 404."""
content = b"test content for nonexistent package"
files = {"file": ("test.bin", io.BytesIO(content), "application/octet-stream")}
response = integration_client.post(
f"/api/v1/project/{test_project}/nonexistent-package-{unique_test_id}/upload",
files=files,
)
assert response.status_code == 404
@pytest.mark.integration
def test_upload_empty_file_rejected(self, integration_client, test_package):
"""Test empty file upload is rejected."""
project, package = test_package
files = {"file": ("empty.bin", io.BytesIO(b""), "application/octet-stream")}
response = integration_client.post(
f"/api/v1/project/{project}/{package}/upload",
files=files,
)
assert response.status_code in [400, 422]
@pytest.mark.integration
def test_upload_missing_file_returns_422(self, integration_client, test_package):
"""Test upload without file field returns 422."""
project, package = test_package
response = integration_client.post(
f"/api/v1/project/{project}/{package}/upload",
data={"tag": "no-file-provided"},
)
assert response.status_code == 422
@pytest.mark.integration
def test_upload_invalid_checksum_format_returns_400(
self, integration_client, test_package
):
"""Test upload with invalid checksum format returns 400."""
project, package = test_package
content = b"checksum format test"
files = {"file": ("test.bin", io.BytesIO(content), "application/octet-stream")}
response = integration_client.post(
f"/api/v1/project/{project}/{package}/upload",
files=files,
headers={"X-Checksum-SHA256": "invalid-hash-format"},
)
assert response.status_code == 400
@pytest.mark.integration
def test_upload_checksum_mismatch_returns_422(
self, integration_client, test_package
):
"""Test upload with mismatched checksum returns 422."""
project, package = test_package
content = b"checksum mismatch test"
wrong_hash = "0" * 64 # Valid format but wrong hash
files = {"file": ("test.bin", io.BytesIO(content), "application/octet-stream")}
response = integration_client.post(
f"/api/v1/project/{project}/{package}/upload",
files=files,
headers={"X-Checksum-SHA256": wrong_hash},
)
assert response.status_code == 422
@pytest.mark.integration
def test_upload_with_correct_checksum_succeeds(
self, integration_client, test_package
):
"""Test upload with correct checksum succeeds."""
project, package = test_package
content = b"correct checksum test"
correct_hash = compute_sha256(content)
files = {"file": ("test.bin", io.BytesIO(content), "application/octet-stream")}
response = integration_client.post(
f"/api/v1/project/{project}/{package}/upload",
files=files,
headers={"X-Checksum-SHA256": correct_hash},
)
assert response.status_code == 200
assert response.json()["artifact_id"] == correct_hash
class TestDownloadErrorHandling:
"""Tests for download error handling."""
@pytest.mark.integration
def test_download_nonexistent_tag_returns_404(
self, integration_client, test_package
):
"""Test download of nonexistent tag returns 404."""
project, package = test_package
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/nonexistent-tag-xyz"
)
assert response.status_code == 404
@pytest.mark.integration
def test_download_nonexistent_artifact_returns_404(
self, integration_client, test_package
):
"""Test download of nonexistent artifact ID returns 404."""
project, package = test_package
fake_hash = "a" * 64
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/artifact:{fake_hash}"
)
assert response.status_code == 404
@pytest.mark.integration
def test_download_invalid_artifact_id_format(
self, integration_client, test_package
):
"""Test download with invalid artifact ID format."""
project, package = test_package
# Too short
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/artifact:abc123"
)
assert response.status_code == 404
@pytest.mark.integration
def test_download_from_nonexistent_project_returns_404(
self, integration_client, unique_test_id
):
"""Test download from nonexistent project returns 404."""
response = integration_client.get(
f"/api/v1/project/nonexistent-{unique_test_id}/pkg/+/tag"
)
assert response.status_code == 404
@pytest.mark.integration
def test_download_from_nonexistent_package_returns_404(
self, integration_client, test_project, unique_test_id
):
"""Test download from nonexistent package returns 404."""
response = integration_client.get(
f"/api/v1/project/{test_project}/nonexistent-{unique_test_id}/+/tag"
)
assert response.status_code == 404
class TestTimeoutBehavior:
"""Tests for timeout behavior (integration level)."""
@pytest.mark.integration
@pytest.mark.slow
def test_large_upload_completes_within_reasonable_time(
self, integration_client, test_package, sized_content
):
"""Test that a 10MB upload completes within reasonable time."""
project, package = test_package
content, expected_hash = sized_content(10 * 1024 * 1024, seed=999) # 10MB
start_time = time.time()
result = upload_test_file(
integration_client, project, package, content, tag="timeout-test"
)
elapsed = time.time() - start_time
assert result["artifact_id"] == expected_hash
# Should complete within 60 seconds for 10MB on local docker
assert elapsed < 60, f"Upload took too long: {elapsed:.2f}s"
@pytest.mark.integration
@pytest.mark.slow
def test_large_download_completes_within_reasonable_time(
self, integration_client, test_package, sized_content
):
"""Test that a 10MB download completes within reasonable time."""
project, package = test_package
content, expected_hash = sized_content(10 * 1024 * 1024, seed=998) # 10MB
# First upload
upload_test_file(
integration_client, project, package, content, tag="download-timeout-test"
)
# Then download and time it
start_time = time.time()
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/download-timeout-test",
params={"mode": "proxy"},
)
elapsed = time.time() - start_time
assert response.status_code == 200
assert len(response.content) == len(content)
# Should complete within 60 seconds for 10MB on local docker
assert elapsed < 60, f"Download took too long: {elapsed:.2f}s"
class TestResourceCleanup:
"""Tests for proper resource cleanup on failures.
Note: More comprehensive cleanup tests are in test_upload_download_api.py
(TestUploadFailureCleanup class) including S3 object cleanup verification.
"""
@pytest.mark.integration
def test_checksum_mismatch_no_orphaned_artifact(
self, integration_client, test_package, unique_test_id
):
"""Test checksum mismatch doesn't leave orphaned artifact."""
project, package = test_package
# Use unique content to ensure artifact doesn't exist from prior tests
content = f"checksum mismatch orphan test {unique_test_id}".encode()
wrong_hash = "0" * 64
actual_hash = compute_sha256(content)
# Verify artifact doesn't exist before test
pre_check = integration_client.get(f"/api/v1/artifact/{actual_hash}")
assert pre_check.status_code == 404, "Artifact should not exist before test"
files = {"file": ("test.bin", io.BytesIO(content), "application/octet-stream")}
response = integration_client.post(
f"/api/v1/project/{project}/{package}/upload",
files=files,
headers={"X-Checksum-SHA256": wrong_hash},
)
assert response.status_code == 422
# Verify no artifact was created with either hash
response1 = integration_client.get(f"/api/v1/artifact/{wrong_hash}")
response2 = integration_client.get(f"/api/v1/artifact/{actual_hash}")
assert response1.status_code == 404
assert response2.status_code == 404
class TestGracefulErrorResponses:
"""Tests for graceful and informative error responses."""
@pytest.mark.integration
def test_404_response_has_detail_message(
self, integration_client, test_package
):
"""Test 404 responses include a detail message."""
project, package = test_package
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/nonexistent-tag"
)
assert response.status_code == 404
data = response.json()
assert "detail" in data
assert len(data["detail"]) > 0
@pytest.mark.integration
def test_422_response_has_detail_message(self, integration_client, test_package):
"""Test 422 responses include a detail message."""
project, package = test_package
# Upload with mismatched checksum
content = b"detail message test"
wrong_hash = "0" * 64
files = {"file": ("test.bin", io.BytesIO(content), "application/octet-stream")}
response = integration_client.post(
f"/api/v1/project/{project}/{package}/upload",
files=files,
headers={"X-Checksum-SHA256": wrong_hash},
)
assert response.status_code == 422
data = response.json()
assert "detail" in data
@pytest.mark.integration
def test_error_response_is_json(self, integration_client, unique_test_id):
"""Test error responses are valid JSON."""
response = integration_client.get(
f"/api/v1/project/nonexistent-{unique_test_id}/pkg/+/tag"
)
assert response.status_code == 404
# Should not raise exception - valid JSON
data = response.json()
assert isinstance(data, dict)

View File

@@ -0,0 +1,768 @@
"""
Integration tests for artifact integrity verification.
Tests cover:
- Round-trip verification (upload -> download -> verify hash)
- Consistency check endpoint
- Header-based verification
- Integrity verification across file sizes
- Client-side verification workflow
"""
import pytest
import io
import hashlib
from tests.factories import (
compute_sha256,
upload_test_file,
generate_content_with_hash,
s3_object_exists,
get_s3_client,
get_s3_bucket,
)
from tests.conftest import (
SIZE_1KB,
SIZE_10KB,
SIZE_100KB,
SIZE_1MB,
SIZE_10MB,
)
class TestRoundTripVerification:
"""Tests for complete round-trip integrity verification."""
@pytest.mark.integration
def test_upload_download_hash_matches(self, integration_client, test_package):
"""Test that upload -> download round trip preserves content integrity."""
project, package = test_package
content = b"Round trip integrity test content"
expected_hash = compute_sha256(content)
# Upload and capture returned hash
result = upload_test_file(
integration_client, project, package, content, tag="roundtrip"
)
uploaded_hash = result["artifact_id"]
# Verify upload returned correct hash
assert uploaded_hash == expected_hash
# Download artifact
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/roundtrip",
params={"mode": "proxy"},
)
assert response.status_code == 200
# Compute hash of downloaded content
downloaded_hash = compute_sha256(response.content)
# All three hashes should match
assert downloaded_hash == expected_hash
assert downloaded_hash == uploaded_hash
@pytest.mark.integration
def test_upload_response_contains_hash(self, integration_client, test_package):
"""Test upload response contains artifact_id which is the SHA256 hash."""
project, package = test_package
content = b"Upload response hash test"
expected_hash = compute_sha256(content)
result = upload_test_file(integration_client, project, package, content)
assert "artifact_id" in result
assert result["artifact_id"] == expected_hash
assert len(result["artifact_id"]) == 64
assert all(c in "0123456789abcdef" for c in result["artifact_id"])
@pytest.mark.integration
def test_download_header_matches_artifact_id(self, integration_client, test_package):
"""Test X-Checksum-SHA256 header matches artifact ID."""
project, package = test_package
content = b"Header verification test"
expected_hash = compute_sha256(content)
upload_test_file(
integration_client, project, package, content, tag="header-check"
)
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/header-check",
params={"mode": "proxy"},
)
assert response.status_code == 200
assert response.headers.get("X-Checksum-SHA256") == expected_hash
@pytest.mark.integration
def test_etag_matches_artifact_id(self, integration_client, test_package):
"""Test ETag header matches artifact ID."""
project, package = test_package
content = b"ETag verification test"
expected_hash = compute_sha256(content)
upload_test_file(
integration_client, project, package, content, tag="etag-check"
)
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/etag-check",
params={"mode": "proxy"},
)
assert response.status_code == 200
etag = response.headers.get("ETag", "").strip('"')
assert etag == expected_hash
@pytest.mark.integration
def test_artifact_endpoint_returns_correct_hash(self, integration_client, test_package):
"""Test artifact endpoint returns correct hash/ID."""
project, package = test_package
content = b"Artifact endpoint hash test"
expected_hash = compute_sha256(content)
upload_test_file(integration_client, project, package, content)
# Query artifact directly
response = integration_client.get(f"/api/v1/artifact/{expected_hash}")
assert response.status_code == 200
data = response.json()
assert data["id"] == expected_hash
assert data.get("sha256") == expected_hash
class TestClientSideVerificationWorkflow:
"""Tests for client-side verification workflow."""
@pytest.mark.integration
def test_client_can_verify_before_upload(self, integration_client, test_package):
"""Test client can compute hash before upload and verify response matches."""
project, package = test_package
content = b"Client pre-upload verification test"
# Client computes hash locally before upload
client_hash = compute_sha256(content)
# Upload
result = upload_test_file(integration_client, project, package, content)
# Client verifies server returned the same hash
assert result["artifact_id"] == client_hash
@pytest.mark.integration
def test_client_can_provide_checksum_header(self, integration_client, test_package):
"""Test client can provide X-Checksum-SHA256 header for verification."""
project, package = test_package
content = b"Client checksum header test"
client_hash = compute_sha256(content)
files = {"file": ("test.bin", io.BytesIO(content), "application/octet-stream")}
response = integration_client.post(
f"/api/v1/project/{project}/{package}/upload",
files=files,
headers={"X-Checksum-SHA256": client_hash},
)
assert response.status_code == 200
assert response.json()["artifact_id"] == client_hash
@pytest.mark.integration
def test_checksum_mismatch_rejected(self, integration_client, test_package):
"""Test upload with wrong client checksum is rejected."""
project, package = test_package
content = b"Checksum mismatch test"
wrong_hash = "0" * 64
files = {"file": ("test.bin", io.BytesIO(content), "application/octet-stream")}
response = integration_client.post(
f"/api/v1/project/{project}/{package}/upload",
files=files,
headers={"X-Checksum-SHA256": wrong_hash},
)
assert response.status_code == 422
@pytest.mark.integration
def test_client_can_verify_after_download(self, integration_client, test_package):
"""Test client can verify downloaded content matches header hash."""
project, package = test_package
content = b"Client post-download verification"
upload_test_file(
integration_client, project, package, content, tag="verify-after"
)
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/verify-after",
params={"mode": "proxy"},
)
assert response.status_code == 200
# Client gets hash from header
header_hash = response.headers.get("X-Checksum-SHA256")
# Client computes hash of downloaded content
downloaded_hash = compute_sha256(response.content)
# Client verifies they match
assert downloaded_hash == header_hash
class TestIntegritySizeVariants:
"""Tests for integrity verification across different file sizes."""
@pytest.mark.integration
def test_integrity_1kb(self, integration_client, test_package, sized_content):
"""Test integrity verification for 1KB file."""
project, package = test_package
content, expected_hash = sized_content(SIZE_1KB, seed=100)
result = upload_test_file(
integration_client, project, package, content, tag="int-1kb"
)
assert result["artifact_id"] == expected_hash
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/int-1kb",
params={"mode": "proxy"},
)
assert response.status_code == 200
assert compute_sha256(response.content) == expected_hash
assert response.headers.get("X-Checksum-SHA256") == expected_hash
@pytest.mark.integration
def test_integrity_100kb(self, integration_client, test_package, sized_content):
"""Test integrity verification for 100KB file."""
project, package = test_package
content, expected_hash = sized_content(SIZE_100KB, seed=101)
result = upload_test_file(
integration_client, project, package, content, tag="int-100kb"
)
assert result["artifact_id"] == expected_hash
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/int-100kb",
params={"mode": "proxy"},
)
assert response.status_code == 200
assert compute_sha256(response.content) == expected_hash
assert response.headers.get("X-Checksum-SHA256") == expected_hash
@pytest.mark.integration
def test_integrity_1mb(self, integration_client, test_package, sized_content):
"""Test integrity verification for 1MB file."""
project, package = test_package
content, expected_hash = sized_content(SIZE_1MB, seed=102)
result = upload_test_file(
integration_client, project, package, content, tag="int-1mb"
)
assert result["artifact_id"] == expected_hash
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/int-1mb",
params={"mode": "proxy"},
)
assert response.status_code == 200
assert compute_sha256(response.content) == expected_hash
assert response.headers.get("X-Checksum-SHA256") == expected_hash
@pytest.mark.integration
@pytest.mark.slow
def test_integrity_10mb(self, integration_client, test_package, sized_content):
"""Test integrity verification for 10MB file."""
project, package = test_package
content, expected_hash = sized_content(SIZE_10MB, seed=103)
result = upload_test_file(
integration_client, project, package, content, tag="int-10mb"
)
assert result["artifact_id"] == expected_hash
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/int-10mb",
params={"mode": "proxy"},
)
assert response.status_code == 200
assert compute_sha256(response.content) == expected_hash
assert response.headers.get("X-Checksum-SHA256") == expected_hash
class TestConsistencyCheck:
"""Tests for the admin consistency check endpoint."""
@pytest.mark.integration
def test_consistency_check_returns_200(self, integration_client):
"""Test consistency check endpoint returns 200."""
response = integration_client.get("/api/v1/admin/consistency-check")
assert response.status_code == 200
@pytest.mark.integration
def test_consistency_check_response_format(self, integration_client):
"""Test consistency check returns expected response format."""
response = integration_client.get("/api/v1/admin/consistency-check")
assert response.status_code == 200
data = response.json()
# Check expected fields
assert "total_artifacts_checked" in data
assert "orphaned_s3_objects" in data
assert "missing_s3_objects" in data
assert "size_mismatches" in data
assert "healthy" in data
assert "orphaned_s3_keys" in data
assert "missing_s3_keys" in data
assert "size_mismatch_artifacts" in data
# Verify types
assert isinstance(data["total_artifacts_checked"], int)
assert isinstance(data["orphaned_s3_objects"], int)
assert isinstance(data["missing_s3_objects"], int)
assert isinstance(data["size_mismatches"], int)
assert isinstance(data["healthy"], bool)
assert isinstance(data["orphaned_s3_keys"], list)
assert isinstance(data["missing_s3_keys"], list)
assert isinstance(data["size_mismatch_artifacts"], list)
@pytest.mark.integration
def test_consistency_check_after_upload(self, integration_client, test_package):
"""Test consistency check passes after valid upload."""
project, package = test_package
content = b"Consistency check test content"
# Upload artifact
upload_test_file(integration_client, project, package, content)
# Run consistency check
response = integration_client.get("/api/v1/admin/consistency-check")
assert response.status_code == 200
data = response.json()
# Verify check ran and no issues
assert data["total_artifacts_checked"] >= 1
assert data["healthy"] is True
@pytest.mark.integration
def test_consistency_check_limit_parameter(self, integration_client):
"""Test consistency check respects limit parameter."""
response = integration_client.get(
"/api/v1/admin/consistency-check",
params={"limit": 10}
)
assert response.status_code == 200
data = response.json()
# Lists should not exceed limit
assert len(data["orphaned_s3_keys"]) <= 10
assert len(data["missing_s3_keys"]) <= 10
assert len(data["size_mismatch_artifacts"]) <= 10
class TestDigestHeader:
"""Tests for RFC 3230 Digest header."""
@pytest.mark.integration
def test_download_includes_digest_header(self, integration_client, test_package):
"""Test download includes Digest header in RFC 3230 format."""
project, package = test_package
content = b"Digest header test"
expected_hash = compute_sha256(content)
upload_test_file(
integration_client, project, package, content, tag="digest-test"
)
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/digest-test",
params={"mode": "proxy"},
)
assert response.status_code == 200
assert "Digest" in response.headers
# Verify Digest format (sha-256=base64hash)
digest = response.headers["Digest"]
assert digest.startswith("sha-256=")
@pytest.mark.integration
def test_digest_header_base64_valid(self, integration_client, test_package):
"""Test Digest header contains valid base64 encoding."""
import base64
project, package = test_package
content = b"Digest base64 test"
expected_hash = compute_sha256(content)
upload_test_file(
integration_client, project, package, content, tag="digest-b64"
)
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/digest-b64",
params={"mode": "proxy"},
)
assert response.status_code == 200
digest = response.headers["Digest"]
base64_part = digest.split("=", 1)[1]
# Should be valid base64
try:
decoded = base64.b64decode(base64_part)
assert len(decoded) == 32 # SHA256 is 32 bytes
except Exception as e:
pytest.fail(f"Invalid base64 in Digest header: {e}")
class TestVerificationModes:
"""Tests for download verification modes."""
@pytest.mark.integration
def test_pre_verification_mode(self, integration_client, test_package):
"""Test pre-verification mode verifies before streaming."""
project, package = test_package
content = b"Pre-verification mode test"
upload_test_file(
integration_client, project, package, content, tag="pre-verify"
)
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/pre-verify",
params={"mode": "proxy", "verify": "true", "verify_mode": "pre"},
)
assert response.status_code == 200
assert response.content == content
# X-Verified header should be true
assert response.headers.get("X-Verified") == "true"
@pytest.mark.integration
def test_stream_verification_mode(self, integration_client, test_package):
"""Test streaming verification mode."""
project, package = test_package
content = b"Stream verification mode test"
upload_test_file(
integration_client, project, package, content, tag="stream-verify"
)
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/stream-verify",
params={"mode": "proxy", "verify": "true", "verify_mode": "stream"},
)
assert response.status_code == 200
assert response.content == content
class TestArtifactIntegrityEndpoint:
"""Tests for artifact-specific integrity operations."""
@pytest.mark.integration
def test_artifact_size_matches(self, integration_client, test_package):
"""Test artifact endpoint returns correct size."""
project, package = test_package
content = b"Artifact size test content"
expected_size = len(content)
result = upload_test_file(integration_client, project, package, content)
artifact_id = result["artifact_id"]
response = integration_client.get(f"/api/v1/artifact/{artifact_id}")
assert response.status_code == 200
data = response.json()
assert data["size"] == expected_size
@pytest.mark.integration
def test_content_length_header_matches_size(self, integration_client, test_package):
"""Test Content-Length header matches artifact size."""
project, package = test_package
content = b"Content-Length header test"
expected_size = len(content)
upload_test_file(
integration_client, project, package, content, tag="content-len"
)
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/content-len",
params={"mode": "proxy"},
)
assert response.status_code == 200
assert int(response.headers.get("Content-Length", 0)) == expected_size
assert len(response.content) == expected_size
@pytest.mark.requires_direct_s3
class TestCorruptionDetection:
"""Tests for detecting corrupted S3 objects.
These tests directly manipulate S3 objects to simulate corruption
and verify that the system can detect hash mismatches.
Note: These tests require direct S3/MinIO access and are skipped in CI
where S3 is not directly accessible from the test runner.
"""
@pytest.mark.integration
def test_detection_of_corrupted_content(self, integration_client, test_package):
"""Test that corrupted S3 content is detected via hash mismatch.
Uploads content, then directly modifies the S3 object, then
verifies that the downloaded content hash doesn't match.
"""
project, package = test_package
content = b"Original content for corruption test"
expected_hash = compute_sha256(content)
# Upload original content
result = upload_test_file(
integration_client, project, package, content, tag="corrupt-test"
)
assert result["artifact_id"] == expected_hash
# Get the S3 object and corrupt it
s3_client = get_s3_client()
bucket = get_s3_bucket()
s3_key = f"fruits/{expected_hash[:2]}/{expected_hash[2:4]}/{expected_hash}"
# Replace with corrupted content
corrupted_content = b"Corrupted content - different from original!"
s3_client.put_object(Bucket=bucket, Key=s3_key, Body=corrupted_content)
# Download via proxy (bypasses hash verification)
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/corrupt-test",
params={"mode": "proxy"},
)
assert response.status_code == 200
# Verify the downloaded content doesn't match original hash
downloaded_hash = compute_sha256(response.content)
assert downloaded_hash != expected_hash, "Corruption was not detected - hashes match"
assert response.content == corrupted_content
# The X-Checksum-SHA256 header should still show the original hash (from DB)
# but the actual content hash is different
header_hash = response.headers.get("X-Checksum-SHA256")
assert header_hash == expected_hash # Header shows expected hash
assert downloaded_hash != header_hash # But content is corrupted
# Restore original content for cleanup
s3_client.put_object(Bucket=bucket, Key=s3_key, Body=content)
@pytest.mark.integration
def test_detection_of_single_bit_flip(self, integration_client, test_package):
"""Test detection of a single bit flip in S3 object content."""
project, package = test_package
content = b"Content for single bit flip detection test"
expected_hash = compute_sha256(content)
result = upload_test_file(
integration_client, project, package, content, tag="bitflip-test"
)
assert result["artifact_id"] == expected_hash
# Get S3 object and flip a single bit
s3_client = get_s3_client()
bucket = get_s3_bucket()
s3_key = f"fruits/{expected_hash[:2]}/{expected_hash[2:4]}/{expected_hash}"
# Flip the first bit of the first byte
corrupted_content = bytearray(content)
corrupted_content[0] ^= 0x01
corrupted_content = bytes(corrupted_content)
s3_client.put_object(Bucket=bucket, Key=s3_key, Body=corrupted_content)
# Download and verify hash mismatch
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/bitflip-test",
params={"mode": "proxy"},
)
assert response.status_code == 200
downloaded_hash = compute_sha256(response.content)
assert downloaded_hash != expected_hash, "Single bit flip not detected"
# Restore original
s3_client.put_object(Bucket=bucket, Key=s3_key, Body=content)
@pytest.mark.integration
def test_detection_of_truncated_content(self, integration_client, test_package):
"""Test detection of truncated S3 object."""
project, package = test_package
content = b"This is content that will be truncated for testing purposes"
expected_hash = compute_sha256(content)
result = upload_test_file(
integration_client, project, package, content, tag="truncate-test"
)
assert result["artifact_id"] == expected_hash
# Get S3 object and truncate it
s3_client = get_s3_client()
bucket = get_s3_bucket()
s3_key = f"fruits/{expected_hash[:2]}/{expected_hash[2:4]}/{expected_hash}"
# Truncate to half the original size
truncated_content = content[: len(content) // 2]
s3_client.put_object(Bucket=bucket, Key=s3_key, Body=truncated_content)
# Download and verify hash mismatch
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/truncate-test",
params={"mode": "proxy"},
)
assert response.status_code == 200
downloaded_hash = compute_sha256(response.content)
assert downloaded_hash != expected_hash, "Truncation not detected"
assert len(response.content) < len(content), "Content was not truncated"
# Restore original
s3_client.put_object(Bucket=bucket, Key=s3_key, Body=content)
@pytest.mark.integration
def test_detection_of_appended_content(self, integration_client, test_package):
"""Test detection of content with extra bytes appended."""
project, package = test_package
content = b"Original content"
expected_hash = compute_sha256(content)
result = upload_test_file(
integration_client, project, package, content, tag="append-test"
)
assert result["artifact_id"] == expected_hash
# Get S3 object and append extra bytes
s3_client = get_s3_client()
bucket = get_s3_bucket()
s3_key = f"fruits/{expected_hash[:2]}/{expected_hash[2:4]}/{expected_hash}"
appended_content = content + b" - extra bytes appended"
s3_client.put_object(Bucket=bucket, Key=s3_key, Body=appended_content)
# Download and verify hash mismatch
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/append-test",
params={"mode": "proxy"},
)
assert response.status_code == 200
downloaded_hash = compute_sha256(response.content)
assert downloaded_hash != expected_hash, "Appended content not detected"
assert len(response.content) > len(content), "Content was not extended"
# Restore original
s3_client.put_object(Bucket=bucket, Key=s3_key, Body=content)
@pytest.mark.integration
def test_client_detects_hash_mismatch_post_download(
self, integration_client, test_package
):
"""Test that a client can detect hash mismatch after downloading corrupted content.
This simulates the full client verification workflow:
1. Download content
2. Get expected hash from header
3. Compute actual hash of content
4. Verify they match (or detect corruption)
"""
project, package = test_package
content = b"Content for client-side corruption detection"
expected_hash = compute_sha256(content)
result = upload_test_file(
integration_client, project, package, content, tag="client-detect"
)
# Corrupt the S3 object
s3_client = get_s3_client()
bucket = get_s3_bucket()
s3_key = f"fruits/{expected_hash[:2]}/{expected_hash[2:4]}/{expected_hash}"
corrupted = b"This is completely different content"
s3_client.put_object(Bucket=bucket, Key=s3_key, Body=corrupted)
# Simulate client download and verification
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/client-detect",
params={"mode": "proxy"},
)
assert response.status_code == 200
# Client gets expected hash from header
header_hash = response.headers.get("X-Checksum-SHA256")
# Client computes hash of downloaded content
actual_hash = compute_sha256(response.content)
# Client detects the mismatch
corruption_detected = actual_hash != header_hash
assert corruption_detected, "Client should detect hash mismatch"
# Restore original
s3_client.put_object(Bucket=bucket, Key=s3_key, Body=content)
@pytest.mark.integration
def test_consistency_check_detects_size_mismatch(
self, integration_client, test_package, unique_test_id
):
"""Test that consistency check detects size mismatches.
Uploads content, modifies S3 object size, then runs consistency check.
"""
project, package = test_package
content = b"Content for size mismatch consistency check test " + unique_test_id.encode()
expected_hash = compute_sha256(content)
result = upload_test_file(
integration_client, project, package, content, tag="size-mismatch"
)
# Modify S3 object to have different size
s3_client = get_s3_client()
bucket = get_s3_bucket()
s3_key = f"fruits/{expected_hash[:2]}/{expected_hash[2:4]}/{expected_hash}"
different_size_content = content + b"extra extra extra"
s3_client.put_object(Bucket=bucket, Key=s3_key, Body=different_size_content)
# Run consistency check
response = integration_client.get("/api/v1/admin/consistency-check")
assert response.status_code == 200
data = response.json()
# Should detect the size mismatch
assert data["size_mismatches"] >= 1 or len(data["size_mismatch_artifacts"]) >= 1
# Restore original
s3_client.put_object(Bucket=bucket, Key=s3_key, Body=content)
@pytest.mark.integration
def test_consistency_check_detects_missing_s3_object(
self, integration_client, test_package, unique_test_id
):
"""Test that consistency check detects missing S3 objects.
Uploads content, deletes S3 object, then runs consistency check.
"""
project, package = test_package
content = b"Content for missing S3 object test " + unique_test_id.encode()
expected_hash = compute_sha256(content)
result = upload_test_file(
integration_client, project, package, content, tag="missing-s3"
)
# Delete the S3 object
s3_client = get_s3_client()
bucket = get_s3_bucket()
s3_key = f"fruits/{expected_hash[:2]}/{expected_hash[2:4]}/{expected_hash}"
s3_client.delete_object(Bucket=bucket, Key=s3_key)
# Run consistency check
response = integration_client.get("/api/v1/admin/consistency-check")
assert response.status_code == 200
data = response.json()
# Should detect the missing S3 object
assert data["missing_s3_objects"] >= 1 or len(data["missing_s3_keys"]) >= 1
# Restore the object for cleanup
s3_client.put_object(Bucket=bucket, Key=s3_key, Body=content)

View File

@@ -0,0 +1,552 @@
"""
Integration tests for large file upload functionality.
Tests cover:
- Large file uploads (100MB, 1GB)
- Multipart upload behavior
- Upload metrics (duration, throughput)
- Memory efficiency during uploads
- Upload progress tracking
Note: Large tests are marked with @pytest.mark.slow and will be skipped
by default. Run with `pytest --run-slow` to include them.
"""
import os
import pytest
import io
import time
from tests.factories import (
compute_sha256,
upload_test_file,
s3_object_exists,
)
from tests.conftest import (
SIZE_1KB,
SIZE_100KB,
SIZE_1MB,
SIZE_10MB,
SIZE_100MB,
SIZE_1GB,
)
class TestUploadMetrics:
"""Tests for upload duration and throughput metrics."""
@pytest.mark.integration
def test_upload_response_includes_duration_ms(self, integration_client, test_package):
"""Test upload response includes duration_ms field."""
project, package = test_package
content = b"duration test content"
result = upload_test_file(
integration_client, project, package, content, tag="duration-test"
)
assert "duration_ms" in result
assert result["duration_ms"] is not None
assert result["duration_ms"] >= 0
@pytest.mark.integration
def test_upload_response_includes_throughput(self, integration_client, test_package):
"""Test upload response includes throughput_mbps field."""
project, package = test_package
content = b"throughput test content"
result = upload_test_file(
integration_client, project, package, content, tag="throughput-test"
)
assert "throughput_mbps" in result
# For small files throughput may be very high or None
# Just verify the field exists
@pytest.mark.integration
def test_upload_duration_reasonable(
self, integration_client, test_package, sized_content
):
"""Test upload duration is reasonable for file size."""
project, package = test_package
content, _ = sized_content(SIZE_1MB, seed=100)
start = time.time()
result = upload_test_file(
integration_client, project, package, content, tag="duration-check"
)
actual_duration = (time.time() - start) * 1000 # ms
# Reported duration should be close to actual
assert result["duration_ms"] is not None
# Allow some variance (network overhead)
assert result["duration_ms"] <= actual_duration + 1000 # Within 1s
class TestLargeFileUploads:
"""Tests for large file uploads using multipart."""
@pytest.mark.integration
def test_upload_10mb_file(self, integration_client, test_package, sized_content):
"""Test uploading a 10MB file."""
project, package = test_package
content, expected_hash = sized_content(SIZE_10MB, seed=200)
result = upload_test_file(
integration_client, project, package, content, tag="large-10mb"
)
assert result["artifact_id"] == expected_hash
assert result["size"] == SIZE_10MB
assert result["duration_ms"] is not None
assert result["throughput_mbps"] is not None
@pytest.mark.integration
@pytest.mark.slow
@pytest.mark.requires_direct_s3
def test_upload_100mb_file(self, integration_client, test_package, sized_content):
"""Test uploading a 100MB file (triggers multipart upload)."""
project, package = test_package
content, expected_hash = sized_content(SIZE_100MB, seed=300)
result = upload_test_file(
integration_client, project, package, content, tag="large-100mb"
)
assert result["artifact_id"] == expected_hash
assert result["size"] == SIZE_100MB
# Verify S3 object exists
assert s3_object_exists(expected_hash)
@pytest.mark.integration
@pytest.mark.slow
@pytest.mark.large
def test_upload_1gb_file(self, integration_client, test_package, sized_content):
"""Test uploading a 1GB file."""
project, package = test_package
content, expected_hash = sized_content(SIZE_1GB, seed=400)
result = upload_test_file(
integration_client, project, package, content, tag="large-1gb"
)
assert result["artifact_id"] == expected_hash
assert result["size"] == SIZE_1GB
# Should have measurable throughput
assert result["throughput_mbps"] is not None
assert result["throughput_mbps"] > 0
@pytest.mark.integration
def test_large_file_deduplication(
self, integration_client, test_package, sized_content, unique_test_id
):
"""Test deduplication works for large files."""
project, package = test_package
# Use unique_test_id to ensure unique content per test run
seed = hash(unique_test_id) % 10000
content, expected_hash = sized_content(SIZE_10MB, seed=seed)
# First upload
result1 = upload_test_file(
integration_client, project, package, content, tag=f"dedup-{unique_test_id}-1"
)
# Note: may be True if previous test uploaded same content
first_dedupe = result1["deduplicated"]
# Second upload of same content
result2 = upload_test_file(
integration_client, project, package, content, tag=f"dedup-{unique_test_id}-2"
)
assert result2["artifact_id"] == expected_hash
# Second upload MUST be deduplicated
assert result2["deduplicated"] is True
class TestUploadProgress:
"""Tests for upload progress tracking endpoint."""
@pytest.mark.integration
def test_progress_endpoint_returns_not_found_for_invalid_id(
self, integration_client, test_package
):
"""Test progress endpoint returns not_found status for invalid upload ID."""
project, package = test_package
response = integration_client.get(
f"/api/v1/project/{project}/{package}/upload/invalid-upload-id/progress"
)
assert response.status_code == 200
data = response.json()
assert data["status"] == "not_found"
assert data["upload_id"] == "invalid-upload-id"
@pytest.mark.integration
def test_progress_endpoint_requires_valid_project(
self, integration_client, unique_test_id
):
"""Test progress endpoint validates project exists."""
response = integration_client.get(
f"/api/v1/project/nonexistent-{unique_test_id}/pkg/upload/upload-id/progress"
)
assert response.status_code == 404
@pytest.mark.integration
def test_progress_endpoint_requires_valid_package(
self, integration_client, test_project, unique_test_id
):
"""Test progress endpoint validates package exists."""
response = integration_client.get(
f"/api/v1/project/{test_project}/nonexistent-{unique_test_id}/upload/upload-id/progress"
)
assert response.status_code == 404
class TestResumableUploadProgress:
"""Tests for progress tracking during resumable uploads."""
@pytest.mark.integration
def test_resumable_upload_init_and_progress(
self, integration_client, test_package, sized_content
):
"""Test initializing resumable upload and checking progress."""
project, package = test_package
content, expected_hash = sized_content(SIZE_100KB, seed=600)
# Get API key for auth
api_key_response = integration_client.post(
"/api/v1/auth/keys",
json={"name": "progress-test-key"},
)
assert api_key_response.status_code == 200
api_key = api_key_response.json()["key"]
# Initialize resumable upload
init_response = integration_client.post(
f"/api/v1/project/{project}/{package}/upload/init",
json={
"expected_hash": expected_hash,
"filename": "progress-test.bin",
"size": SIZE_100KB,
},
headers={"Authorization": f"Bearer {api_key}"},
)
assert init_response.status_code == 200
upload_id = init_response.json().get("upload_id")
if upload_id:
# Check initial progress
progress_response = integration_client.get(
f"/api/v1/project/{project}/{package}/upload/{upload_id}/progress",
headers={"Authorization": f"Bearer {api_key}"},
)
assert progress_response.status_code == 200
progress = progress_response.json()
assert progress["status"] == "in_progress"
assert progress["bytes_uploaded"] == 0
assert progress["bytes_total"] == SIZE_100KB
# Abort to clean up
integration_client.delete(
f"/api/v1/project/{project}/{package}/upload/{upload_id}",
headers={"Authorization": f"Bearer {api_key}"},
)
class TestUploadSizeLimits:
"""Tests for upload size limit enforcement."""
@pytest.mark.integration
def test_empty_file_rejected(self, integration_client, test_package):
"""Test empty files are rejected."""
project, package = test_package
files = {"file": ("empty.txt", io.BytesIO(b""), "application/octet-stream")}
response = integration_client.post(
f"/api/v1/project/{project}/{package}/upload",
files=files,
)
assert response.status_code in [400, 422]
@pytest.mark.integration
def test_minimum_size_accepted(self, integration_client, test_package):
"""Test 1-byte file is accepted."""
project, package = test_package
content = b"X"
result = upload_test_file(
integration_client, project, package, content, tag="min-size"
)
assert result["size"] == 1
@pytest.mark.integration
def test_content_length_header_used_in_response(self, integration_client, test_package):
"""Test that upload response size matches Content-Length."""
project, package = test_package
content = b"content length verification test"
result = upload_test_file(
integration_client, project, package, content, tag="content-length-test"
)
# Size in response should match actual content length
assert result["size"] == len(content)
class TestUploadErrorHandling:
"""Tests for upload error handling."""
@pytest.mark.integration
def test_upload_to_nonexistent_project_returns_404(
self, integration_client, unique_test_id
):
"""Test upload to nonexistent project returns 404."""
content = b"test content"
files = {"file": ("test.bin", io.BytesIO(content), "application/octet-stream")}
response = integration_client.post(
f"/api/v1/project/nonexistent-{unique_test_id}/pkg/upload",
files=files,
)
assert response.status_code == 404
@pytest.mark.integration
def test_upload_to_nonexistent_package_returns_404(
self, integration_client, test_project, unique_test_id
):
"""Test upload to nonexistent package returns 404."""
content = b"test content"
files = {"file": ("test.bin", io.BytesIO(content), "application/octet-stream")}
response = integration_client.post(
f"/api/v1/project/{test_project}/nonexistent-{unique_test_id}/upload",
files=files,
)
assert response.status_code == 404
@pytest.mark.integration
def test_upload_without_file_returns_422(self, integration_client, test_package):
"""Test upload without file field returns 422."""
project, package = test_package
response = integration_client.post(
f"/api/v1/project/{project}/{package}/upload",
data={"tag": "no-file"},
)
assert response.status_code == 422
@pytest.mark.integration
def test_upload_with_invalid_checksum_rejected(
self, integration_client, test_package
):
"""Test upload with invalid checksum header format is rejected."""
project, package = test_package
content = b"checksum test"
files = {"file": ("test.bin", io.BytesIO(content), "application/octet-stream")}
response = integration_client.post(
f"/api/v1/project/{project}/{package}/upload",
files=files,
headers={"X-Checksum-SHA256": "invalid-checksum"},
)
assert response.status_code == 400
@pytest.mark.integration
def test_upload_with_mismatched_checksum_rejected(
self, integration_client, test_package
):
"""Test upload with wrong checksum is rejected."""
project, package = test_package
content = b"mismatch test"
wrong_hash = "0" * 64
files = {"file": ("test.bin", io.BytesIO(content), "application/octet-stream")}
response = integration_client.post(
f"/api/v1/project/{project}/{package}/upload",
files=files,
headers={"X-Checksum-SHA256": wrong_hash},
)
assert response.status_code == 422
assert "verification failed" in response.json().get("detail", "").lower()
class TestResumableUploadCancellation:
"""Tests for resumable upload cancellation."""
@pytest.mark.integration
def test_abort_resumable_upload(self, integration_client, test_package, sized_content):
"""Test aborting a resumable upload cleans up properly."""
project, package = test_package
content, expected_hash = sized_content(SIZE_100KB, seed=700)
# Get API key for auth
api_key_response = integration_client.post(
"/api/v1/auth/keys",
json={"name": "abort-test-key"},
)
assert api_key_response.status_code == 200
api_key = api_key_response.json()["key"]
# Initialize resumable upload
init_response = integration_client.post(
f"/api/v1/project/{project}/{package}/upload/init",
json={
"expected_hash": expected_hash,
"filename": "abort-test.bin",
"size": SIZE_100KB,
},
headers={"Authorization": f"Bearer {api_key}"},
)
assert init_response.status_code == 200
upload_id = init_response.json().get("upload_id")
if upload_id:
# Abort the upload (without uploading any parts)
abort_response = integration_client.delete(
f"/api/v1/project/{project}/{package}/upload/{upload_id}",
headers={"Authorization": f"Bearer {api_key}"},
)
assert abort_response.status_code in [200, 204]
# Verify progress shows not_found after abort
progress_response = integration_client.get(
f"/api/v1/project/{project}/{package}/upload/{upload_id}/progress",
headers={"Authorization": f"Bearer {api_key}"},
)
assert progress_response.status_code == 200
assert progress_response.json()["status"] == "not_found"
@pytest.mark.integration
def test_abort_nonexistent_upload(self, integration_client, test_package):
"""Test aborting nonexistent upload returns appropriate error."""
project, package = test_package
# Get API key for auth
api_key_response = integration_client.post(
"/api/v1/auth/keys",
json={"name": "abort-nonexistent-key"},
)
assert api_key_response.status_code == 200
api_key = api_key_response.json()["key"]
response = integration_client.delete(
f"/api/v1/project/{project}/{package}/upload/nonexistent-upload-id",
headers={"Authorization": f"Bearer {api_key}"},
)
# Should return 404 or 200 (idempotent delete)
assert response.status_code in [200, 204, 404]
class TestUploadTimeout:
"""Tests for upload timeout handling."""
@pytest.mark.integration
def test_upload_with_short_timeout_succeeds_for_small_file(
self, integration_client, test_package
):
"""Test small file upload succeeds with reasonable timeout."""
project, package = test_package
content = b"small timeout test"
# httpx client should handle this quickly
result = upload_test_file(
integration_client, project, package, content, tag="timeout-small"
)
assert result["artifact_id"] is not None
@pytest.mark.integration
def test_upload_response_duration_under_timeout(
self, integration_client, test_package, sized_content
):
"""Test upload completes within reasonable time."""
project, package = test_package
content, _ = sized_content(SIZE_1MB, seed=800)
start = time.time()
result = upload_test_file(
integration_client, project, package, content, tag="timeout-check"
)
duration = time.time() - start
# 1MB should upload in well under 60 seconds on local
assert duration < 60
assert result["artifact_id"] is not None
class TestConcurrentUploads:
"""Tests for concurrent upload handling."""
@pytest.mark.integration
def test_concurrent_different_files(
self, integration_client, test_package, sized_content
):
"""Test concurrent uploads of different files succeed."""
from concurrent.futures import ThreadPoolExecutor, as_completed
project, package = test_package
# Get API key for auth
api_key_response = integration_client.post(
"/api/v1/auth/keys",
json={"name": "concurrent-diff-key"},
)
assert api_key_response.status_code == 200
api_key = api_key_response.json()["key"]
num_uploads = 3
results = []
errors = []
def upload_unique_file(idx):
try:
from httpx import Client
content, expected_hash = sized_content(SIZE_100KB, seed=900 + idx)
base_url = os.environ.get("ORCHARD_TEST_URL", "http://localhost:8080")
with Client(base_url=base_url, timeout=30.0) as client:
files = {
"file": (
f"concurrent-{idx}.bin",
io.BytesIO(content),
"application/octet-stream",
)
}
response = client.post(
f"/api/v1/project/{project}/{package}/upload",
files=files,
data={"tag": f"concurrent-diff-{idx}"},
headers={"Authorization": f"Bearer {api_key}"},
)
if response.status_code == 200:
results.append((idx, response.json(), expected_hash))
else:
errors.append(f"Upload {idx}: {response.status_code} - {response.text}")
except Exception as e:
errors.append(f"Upload {idx}: {str(e)}")
with ThreadPoolExecutor(max_workers=num_uploads) as executor:
futures = [executor.submit(upload_unique_file, i) for i in range(num_uploads)]
for future in as_completed(futures):
pass
assert len(errors) == 0, f"Concurrent upload errors: {errors}"
assert len(results) == num_uploads
# Each upload should have unique artifact ID
artifact_ids = set(r[1]["artifact_id"] for r in results)
assert len(artifact_ids) == num_uploads
# Each should match expected hash
for idx, result, expected_hash in results:
assert result["artifact_id"] == expected_hash

View File

@@ -0,0 +1,583 @@
"""
Integration tests for upload/download with various file sizes.
Tests cover:
- Small files (0B - 100KB)
- Medium files (1MB - 50MB)
- Large files (100MB - 1GB) - marked as slow/large
- Exact chunk boundaries
- Data integrity verification across all sizes
"""
import pytest
import io
import time
from concurrent.futures import ThreadPoolExecutor, as_completed
from tests.factories import (
compute_sha256,
upload_test_file,
generate_content,
generate_content_with_hash,
)
from tests.conftest import (
SIZE_1B,
SIZE_1KB,
SIZE_10KB,
SIZE_100KB,
SIZE_1MB,
SIZE_5MB,
SIZE_10MB,
SIZE_50MB,
SIZE_100MB,
SIZE_250MB,
SIZE_500MB,
SIZE_1GB,
CHUNK_SIZE,
MULTIPART_THRESHOLD,
)
class TestSmallFileSizes:
"""Tests for small file uploads/downloads (0B - 100KB)."""
@pytest.mark.integration
def test_upload_download_1_byte(self, integration_client, test_package, sized_content):
"""Test upload/download of 1 byte file."""
project, package = test_package
content, expected_hash = sized_content(SIZE_1B, seed=1)
result = upload_test_file(
integration_client, project, package, content,
filename="1byte.bin", tag="1byte"
)
assert result["artifact_id"] == expected_hash
assert result["size"] == SIZE_1B
# Download and verify
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/1byte",
params={"mode": "proxy"},
)
assert response.status_code == 200
assert response.content == content
assert len(response.content) == SIZE_1B
@pytest.mark.integration
def test_upload_download_1kb(self, integration_client, test_package, sized_content):
"""Test upload/download of 1KB file."""
project, package = test_package
content, expected_hash = sized_content(SIZE_1KB, seed=2)
result = upload_test_file(
integration_client, project, package, content,
filename="1kb.bin", tag="1kb"
)
assert result["artifact_id"] == expected_hash
assert result["size"] == SIZE_1KB
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/1kb",
params={"mode": "proxy"},
)
assert response.status_code == 200
assert response.content == content
@pytest.mark.integration
def test_upload_download_10kb(self, integration_client, test_package, sized_content):
"""Test upload/download of 10KB file."""
project, package = test_package
content, expected_hash = sized_content(SIZE_10KB, seed=3)
result = upload_test_file(
integration_client, project, package, content,
filename="10kb.bin", tag="10kb"
)
assert result["artifact_id"] == expected_hash
assert result["size"] == SIZE_10KB
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/10kb",
params={"mode": "proxy"},
)
assert response.status_code == 200
assert response.content == content
@pytest.mark.integration
def test_upload_download_100kb(self, integration_client, test_package, sized_content):
"""Test upload/download of 100KB file."""
project, package = test_package
content, expected_hash = sized_content(SIZE_100KB, seed=4)
result = upload_test_file(
integration_client, project, package, content,
filename="100kb.bin", tag="100kb"
)
assert result["artifact_id"] == expected_hash
assert result["size"] == SIZE_100KB
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/100kb",
params={"mode": "proxy"},
)
assert response.status_code == 200
assert response.content == content
class TestMediumFileSizes:
"""Tests for medium file uploads/downloads (1MB - 50MB)."""
@pytest.mark.integration
def test_upload_download_1mb(self, integration_client, test_package, sized_content):
"""Test upload/download of 1MB file."""
project, package = test_package
content, expected_hash = sized_content(SIZE_1MB, seed=10)
result = upload_test_file(
integration_client, project, package, content,
filename="1mb.bin", tag="1mb"
)
assert result["artifact_id"] == expected_hash
assert result["size"] == SIZE_1MB
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/1mb",
params={"mode": "proxy"},
)
assert response.status_code == 200
assert len(response.content) == SIZE_1MB
assert compute_sha256(response.content) == expected_hash
@pytest.mark.integration
def test_upload_download_5mb(self, integration_client, test_package, sized_content):
"""Test upload/download of 5MB file (multipart threshold boundary area)."""
project, package = test_package
content, expected_hash = sized_content(SIZE_5MB, seed=11)
result = upload_test_file(
integration_client, project, package, content,
filename="5mb.bin", tag="5mb"
)
assert result["artifact_id"] == expected_hash
assert result["size"] == SIZE_5MB
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/5mb",
params={"mode": "proxy"},
)
assert response.status_code == 200
assert len(response.content) == SIZE_5MB
assert compute_sha256(response.content) == expected_hash
@pytest.mark.integration
@pytest.mark.slow
def test_upload_download_10mb(self, integration_client, test_package, sized_content):
"""Test upload/download of 10MB file."""
project, package = test_package
content, expected_hash = sized_content(SIZE_10MB, seed=12)
result = upload_test_file(
integration_client, project, package, content,
filename="10mb.bin", tag="10mb"
)
assert result["artifact_id"] == expected_hash
assert result["size"] == SIZE_10MB
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/10mb",
params={"mode": "proxy"},
)
assert response.status_code == 200
assert len(response.content) == SIZE_10MB
assert compute_sha256(response.content) == expected_hash
@pytest.mark.integration
@pytest.mark.slow
def test_upload_download_50mb(self, integration_client, test_package, sized_content):
"""Test upload/download of 50MB file."""
project, package = test_package
content, expected_hash = sized_content(SIZE_50MB, seed=13)
start_time = time.time()
result = upload_test_file(
integration_client, project, package, content,
filename="50mb.bin", tag="50mb"
)
upload_time = time.time() - start_time
assert result["artifact_id"] == expected_hash
assert result["size"] == SIZE_50MB
start_time = time.time()
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/50mb",
params={"mode": "proxy"},
)
download_time = time.time() - start_time
assert response.status_code == 200
assert len(response.content) == SIZE_50MB
assert compute_sha256(response.content) == expected_hash
# Log timing for performance tracking
print(f"\n50MB upload: {upload_time:.2f}s, download: {download_time:.2f}s")
class TestLargeFileSizes:
"""Tests for large file uploads/downloads (100MB - 1GB).
These tests are marked as slow and large, skipped by default.
Run with: pytest -m "large" to include these tests.
"""
@pytest.mark.integration
@pytest.mark.slow
@pytest.mark.large
def test_upload_download_100mb(self, integration_client, test_package, sized_content):
"""Test upload/download of 100MB file (multipart threshold)."""
project, package = test_package
content, expected_hash = sized_content(SIZE_100MB, seed=100)
start_time = time.time()
result = upload_test_file(
integration_client, project, package, content,
filename="100mb.bin", tag="100mb"
)
upload_time = time.time() - start_time
assert result["artifact_id"] == expected_hash
assert result["size"] == SIZE_100MB
start_time = time.time()
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/100mb",
params={"mode": "proxy"},
)
download_time = time.time() - start_time
assert response.status_code == 200
assert len(response.content) == SIZE_100MB
assert compute_sha256(response.content) == expected_hash
print(f"\n100MB upload: {upload_time:.2f}s, download: {download_time:.2f}s")
@pytest.mark.integration
@pytest.mark.slow
@pytest.mark.large
def test_upload_download_250mb(self, integration_client, test_package, sized_content):
"""Test upload/download of 250MB file."""
project, package = test_package
content, expected_hash = sized_content(SIZE_250MB, seed=250)
start_time = time.time()
result = upload_test_file(
integration_client, project, package, content,
filename="250mb.bin", tag="250mb"
)
upload_time = time.time() - start_time
assert result["artifact_id"] == expected_hash
assert result["size"] == SIZE_250MB
start_time = time.time()
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/250mb",
params={"mode": "proxy"},
)
download_time = time.time() - start_time
assert response.status_code == 200
assert len(response.content) == SIZE_250MB
assert compute_sha256(response.content) == expected_hash
print(f"\n250MB upload: {upload_time:.2f}s, download: {download_time:.2f}s")
@pytest.mark.integration
@pytest.mark.slow
@pytest.mark.large
def test_upload_download_500mb(self, integration_client, test_package, sized_content):
"""Test upload/download of 500MB file."""
project, package = test_package
content, expected_hash = sized_content(SIZE_500MB, seed=500)
start_time = time.time()
result = upload_test_file(
integration_client, project, package, content,
filename="500mb.bin", tag="500mb"
)
upload_time = time.time() - start_time
assert result["artifact_id"] == expected_hash
assert result["size"] == SIZE_500MB
start_time = time.time()
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/500mb",
params={"mode": "proxy"},
)
download_time = time.time() - start_time
assert response.status_code == 200
assert len(response.content) == SIZE_500MB
assert compute_sha256(response.content) == expected_hash
print(f"\n500MB upload: {upload_time:.2f}s, download: {download_time:.2f}s")
@pytest.mark.integration
@pytest.mark.slow
@pytest.mark.large
def test_upload_download_1gb(self, integration_client, test_package, sized_content):
"""Test upload/download of 1GB file.
This test may take several minutes depending on network/disk speed.
"""
project, package = test_package
content, expected_hash = sized_content(SIZE_1GB, seed=1024)
start_time = time.time()
result = upload_test_file(
integration_client, project, package, content,
filename="1gb.bin", tag="1gb"
)
upload_time = time.time() - start_time
assert result["artifact_id"] == expected_hash
assert result["size"] == SIZE_1GB
start_time = time.time()
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/1gb",
params={"mode": "proxy"},
)
download_time = time.time() - start_time
assert response.status_code == 200
assert len(response.content) == SIZE_1GB
assert compute_sha256(response.content) == expected_hash
print(f"\n1GB upload: {upload_time:.2f}s, download: {download_time:.2f}s")
class TestChunkBoundaries:
"""Tests for exact chunk size boundaries."""
@pytest.mark.integration
def test_upload_download_at_chunk_size(self, integration_client, test_package, sized_content):
"""Test upload/download at exact chunk size (64KB)."""
project, package = test_package
content, expected_hash = sized_content(CHUNK_SIZE, seed=64)
result = upload_test_file(
integration_client, project, package, content,
filename="chunk.bin", tag="chunk-exact"
)
assert result["artifact_id"] == expected_hash
assert result["size"] == CHUNK_SIZE
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/chunk-exact",
params={"mode": "proxy"},
)
assert response.status_code == 200
assert response.content == content
@pytest.mark.integration
def test_upload_download_chunk_size_plus_1(self, integration_client, test_package, sized_content):
"""Test upload/download at chunk size + 1 byte."""
project, package = test_package
size = CHUNK_SIZE + 1
content, expected_hash = sized_content(size, seed=65)
result = upload_test_file(
integration_client, project, package, content,
filename="chunk_plus.bin", tag="chunk-plus"
)
assert result["artifact_id"] == expected_hash
assert result["size"] == size
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/chunk-plus",
params={"mode": "proxy"},
)
assert response.status_code == 200
assert response.content == content
@pytest.mark.integration
def test_upload_download_chunk_size_minus_1(self, integration_client, test_package, sized_content):
"""Test upload/download at chunk size - 1 byte."""
project, package = test_package
size = CHUNK_SIZE - 1
content, expected_hash = sized_content(size, seed=63)
result = upload_test_file(
integration_client, project, package, content,
filename="chunk_minus.bin", tag="chunk-minus"
)
assert result["artifact_id"] == expected_hash
assert result["size"] == size
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/chunk-minus",
params={"mode": "proxy"},
)
assert response.status_code == 200
assert response.content == content
@pytest.mark.integration
def test_upload_download_multiple_chunks(self, integration_client, test_package, sized_content):
"""Test upload/download spanning multiple chunks."""
project, package = test_package
size = CHUNK_SIZE * 3 + 1000 # 3 full chunks + partial
content, expected_hash = sized_content(size, seed=300)
result = upload_test_file(
integration_client, project, package, content,
filename="multi_chunk.bin", tag="multi-chunk"
)
assert result["artifact_id"] == expected_hash
assert result["size"] == size
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/multi-chunk",
params={"mode": "proxy"},
)
assert response.status_code == 200
assert response.content == content
class TestDataIntegrity:
"""Tests for data integrity with various content types."""
@pytest.mark.integration
def test_binary_content_integrity(self, integration_client, test_package):
"""Test binary content (all byte values 0-255) integrity."""
project, package = test_package
# Content with all 256 possible byte values
content = bytes(range(256)) * 100 # 25.6KB
expected_hash = compute_sha256(content)
result = upload_test_file(
integration_client, project, package, content,
filename="binary.bin", tag="binary"
)
assert result["artifact_id"] == expected_hash
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/binary",
params={"mode": "proxy"},
)
assert response.status_code == 200
assert response.content == content
@pytest.mark.integration
def test_text_content_integrity(self, integration_client, test_package):
"""Test UTF-8 text content integrity."""
project, package = test_package
content = "Hello, World! 你好世界 🌍 مرحبا العالم".encode("utf-8")
expected_hash = compute_sha256(content)
result = upload_test_file(
integration_client, project, package, content,
filename="text.txt", tag="text"
)
assert result["artifact_id"] == expected_hash
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/text",
params={"mode": "proxy"},
)
assert response.status_code == 200
assert response.content == content
assert response.content.decode("utf-8") == "Hello, World! 你好世界 🌍 مرحبا العالم"
@pytest.mark.integration
def test_null_bytes_content_integrity(self, integration_client, test_package):
"""Test content with null bytes."""
project, package = test_package
content = b"before\x00null\x00bytes\x00after"
expected_hash = compute_sha256(content)
result = upload_test_file(
integration_client, project, package, content,
filename="nulls.bin", tag="nulls"
)
assert result["artifact_id"] == expected_hash
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/nulls",
params={"mode": "proxy"},
)
assert response.status_code == 200
assert response.content == content
assert b"\x00" in response.content
@pytest.mark.integration
def test_unicode_filename_integrity(self, integration_client, test_package):
"""Test file with unicode filename."""
project, package = test_package
content = b"unicode filename test"
expected_hash = compute_sha256(content)
result = upload_test_file(
integration_client, project, package, content,
filename="文件名.txt", tag="unicode-name"
)
assert result["artifact_id"] == expected_hash
assert result["original_name"] == "文件名.txt"
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/unicode-name",
params={"mode": "proxy"},
)
assert response.status_code == 200
assert response.content == content
@pytest.mark.integration
def test_compressed_content_integrity(self, integration_client, test_package):
"""Test gzip-compressed content integrity."""
import gzip
project, package = test_package
original = b"This is some text that will be compressed " * 100
content = gzip.compress(original)
expected_hash = compute_sha256(content)
result = upload_test_file(
integration_client, project, package, content,
filename="data.gz", tag="compressed"
)
assert result["artifact_id"] == expected_hash
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/compressed",
params={"mode": "proxy"},
)
assert response.status_code == 200
assert response.content == content
# Verify we can decompress
assert gzip.decompress(response.content) == original
@pytest.mark.integration
def test_hash_verification_matches(self, integration_client, test_package, sized_content):
"""Test that computed hash matches artifact_id for various sizes."""
project, package = test_package
sizes = [SIZE_1B, SIZE_1KB, SIZE_10KB, SIZE_100KB, SIZE_1MB]
for i, size in enumerate(sizes):
content, expected_hash = sized_content(size, seed=1000 + i)
result = upload_test_file(
integration_client, project, package, content,
filename=f"hash_test_{size}.bin", tag=f"hash-{size}"
)
# Verify artifact_id matches expected hash
assert result["artifact_id"] == expected_hash
# Download and verify hash of downloaded content
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/hash-{size}",
params={"mode": "proxy"},
)
downloaded_hash = compute_sha256(response.content)
assert downloaded_hash == expected_hash

View File

@@ -0,0 +1,535 @@
"""
Integration tests for streaming download functionality.
Tests cover:
- HTTP Range requests (partial downloads, resume)
- Conditional requests (If-None-Match, If-Modified-Since)
- Caching headers (Cache-Control, Last-Modified, Accept-Ranges)
- Large file streaming
- Download modes (proxy, redirect, presigned)
"""
import pytest
import io
import time
from email.utils import formatdate
from tests.factories import (
compute_sha256,
upload_test_file,
)
from tests.conftest import (
SIZE_1KB,
SIZE_100KB,
SIZE_1MB,
)
class TestRangeRequests:
"""Tests for HTTP Range request support (partial downloads)."""
@pytest.mark.integration
def test_range_request_first_bytes(self, integration_client, test_package):
"""Test range request for first N bytes."""
project, package = test_package
content = b"0123456789" * 100 # 1000 bytes
upload_test_file(integration_client, project, package, content, tag="range-test")
# Request first 10 bytes
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/range-test",
params={"mode": "proxy"},
headers={"Range": "bytes=0-9"},
)
assert response.status_code == 206 # Partial Content
assert response.content == b"0123456789"
assert "Content-Range" in response.headers
assert response.headers["Content-Range"].startswith("bytes 0-9/")
@pytest.mark.integration
def test_range_request_middle_bytes(self, integration_client, test_package):
"""Test range request for bytes in the middle."""
project, package = test_package
content = b"ABCDEFGHIJKLMNOPQRSTUVWXYZ"
upload_test_file(integration_client, project, package, content, tag="range-mid")
# Request bytes 10-19 (KLMNOPQRST)
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/range-mid",
params={"mode": "proxy"},
headers={"Range": "bytes=10-19"},
)
assert response.status_code == 206
assert response.content == b"KLMNOPQRST"
@pytest.mark.integration
def test_range_request_suffix_bytes(self, integration_client, test_package):
"""Test range request for last N bytes (suffix range)."""
project, package = test_package
content = b"0123456789ABCDEF" # 16 bytes
upload_test_file(integration_client, project, package, content, tag="range-suffix")
# Request last 4 bytes
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/range-suffix",
params={"mode": "proxy"},
headers={"Range": "bytes=-4"},
)
assert response.status_code == 206
assert response.content == b"CDEF"
@pytest.mark.integration
def test_range_request_open_ended(self, integration_client, test_package):
"""Test range request from offset to end."""
project, package = test_package
content = b"0123456789"
upload_test_file(integration_client, project, package, content, tag="range-open")
# Request from byte 5 to end
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/range-open",
params={"mode": "proxy"},
headers={"Range": "bytes=5-"},
)
assert response.status_code == 206
assert response.content == b"56789"
@pytest.mark.integration
def test_range_request_includes_accept_ranges_header(
self, integration_client, test_package
):
"""Test that range requests include Accept-Ranges header."""
project, package = test_package
content = b"test content"
upload_test_file(integration_client, project, package, content, tag="accept-ranges")
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/accept-ranges",
params={"mode": "proxy"},
headers={"Range": "bytes=0-4"},
)
assert response.status_code == 206
assert response.headers.get("Accept-Ranges") == "bytes"
@pytest.mark.integration
def test_full_download_advertises_accept_ranges(
self, integration_client, test_package
):
"""Test that full downloads advertise range support."""
project, package = test_package
content = b"test content"
upload_test_file(integration_client, project, package, content, tag="full-accept")
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/full-accept",
params={"mode": "proxy"},
)
assert response.status_code == 200
assert response.headers.get("Accept-Ranges") == "bytes"
class TestConditionalRequests:
"""Tests for conditional request handling (304 Not Modified)."""
@pytest.mark.integration
def test_if_none_match_returns_304(self, integration_client, test_package):
"""Test If-None-Match with matching ETag returns 304."""
project, package = test_package
content = b"conditional request test content"
expected_hash = compute_sha256(content)
upload_test_file(integration_client, project, package, content, tag="cond-etag")
# Request with matching ETag
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/cond-etag",
params={"mode": "proxy"},
headers={"If-None-Match": f'"{expected_hash}"'},
)
assert response.status_code == 304
assert response.content == b"" # No body for 304
@pytest.mark.integration
def test_if_none_match_without_quotes(self, integration_client, test_package):
"""Test If-None-Match works with or without quotes."""
project, package = test_package
content = b"etag no quotes test"
expected_hash = compute_sha256(content)
upload_test_file(integration_client, project, package, content, tag="cond-noquote")
# Request with ETag without quotes
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/cond-noquote",
params={"mode": "proxy"},
headers={"If-None-Match": expected_hash},
)
assert response.status_code == 304
@pytest.mark.integration
def test_if_none_match_mismatch_returns_200(self, integration_client, test_package):
"""Test If-None-Match with non-matching ETag returns 200."""
project, package = test_package
content = b"etag mismatch test"
upload_test_file(integration_client, project, package, content, tag="cond-mismatch")
# Request with different ETag
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/cond-mismatch",
params={"mode": "proxy"},
headers={"If-None-Match": '"different-etag-value"'},
)
assert response.status_code == 200
assert response.content == content
@pytest.mark.integration
def test_if_modified_since_returns_304(self, integration_client, test_package):
"""Test If-Modified-Since with future date returns 304."""
project, package = test_package
content = b"modified since test"
upload_test_file(integration_client, project, package, content, tag="cond-modified")
# Request with future date (artifact was definitely created before this)
future_date = formatdate(time.time() + 86400, usegmt=True) # Tomorrow
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/cond-modified",
params={"mode": "proxy"},
headers={"If-Modified-Since": future_date},
)
assert response.status_code == 304
@pytest.mark.integration
def test_if_modified_since_old_date_returns_200(
self, integration_client, test_package
):
"""Test If-Modified-Since with old date returns 200."""
project, package = test_package
content = b"old date test"
upload_test_file(integration_client, project, package, content, tag="cond-old")
# Request with old date (2020-01-01)
old_date = "Wed, 01 Jan 2020 00:00:00 GMT"
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/cond-old",
params={"mode": "proxy"},
headers={"If-Modified-Since": old_date},
)
assert response.status_code == 200
assert response.content == content
@pytest.mark.integration
def test_304_includes_etag(self, integration_client, test_package):
"""Test 304 response includes ETag header."""
project, package = test_package
content = b"304 etag test"
expected_hash = compute_sha256(content)
upload_test_file(integration_client, project, package, content, tag="304-etag")
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/304-etag",
params={"mode": "proxy"},
headers={"If-None-Match": f'"{expected_hash}"'},
)
assert response.status_code == 304
assert response.headers.get("ETag") == f'"{expected_hash}"'
@pytest.mark.integration
def test_304_includes_cache_control(self, integration_client, test_package):
"""Test 304 response includes Cache-Control header."""
project, package = test_package
content = b"304 cache test"
expected_hash = compute_sha256(content)
upload_test_file(integration_client, project, package, content, tag="304-cache")
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/304-cache",
params={"mode": "proxy"},
headers={"If-None-Match": f'"{expected_hash}"'},
)
assert response.status_code == 304
assert "immutable" in response.headers.get("Cache-Control", "")
class TestCachingHeaders:
"""Tests for caching headers on download responses."""
@pytest.mark.integration
def test_download_includes_cache_control(self, integration_client, test_package):
"""Test download response includes Cache-Control header."""
project, package = test_package
content = b"cache control test"
upload_test_file(integration_client, project, package, content, tag="cache-ctl")
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/cache-ctl",
params={"mode": "proxy"},
)
assert response.status_code == 200
cache_control = response.headers.get("Cache-Control", "")
assert "public" in cache_control
assert "immutable" in cache_control
assert "max-age" in cache_control
@pytest.mark.integration
def test_download_includes_last_modified(self, integration_client, test_package):
"""Test download response includes Last-Modified header."""
project, package = test_package
content = b"last modified test"
upload_test_file(integration_client, project, package, content, tag="last-mod")
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/last-mod",
params={"mode": "proxy"},
)
assert response.status_code == 200
assert "Last-Modified" in response.headers
# Should be in RFC 7231 format
last_modified = response.headers["Last-Modified"]
assert "GMT" in last_modified
@pytest.mark.integration
def test_download_includes_etag(self, integration_client, test_package):
"""Test download response includes ETag header."""
project, package = test_package
content = b"etag header test"
expected_hash = compute_sha256(content)
upload_test_file(integration_client, project, package, content, tag="etag-hdr")
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/etag-hdr",
params={"mode": "proxy"},
)
assert response.status_code == 200
assert response.headers.get("ETag") == f'"{expected_hash}"'
class TestDownloadResume:
"""Tests for download resume functionality using range requests."""
@pytest.mark.integration
def test_resume_download_after_partial(self, integration_client, test_package):
"""Test resuming download from where it left off."""
project, package = test_package
content = b"ABCDEFGHIJ" * 100 # 1000 bytes
upload_test_file(integration_client, project, package, content, tag="resume-test")
# Simulate partial download (first 500 bytes)
response1 = integration_client.get(
f"/api/v1/project/{project}/{package}/+/resume-test",
params={"mode": "proxy"},
headers={"Range": "bytes=0-499"},
)
assert response1.status_code == 206
first_half = response1.content
assert len(first_half) == 500
# Resume from byte 500
response2 = integration_client.get(
f"/api/v1/project/{project}/{package}/+/resume-test",
params={"mode": "proxy"},
headers={"Range": "bytes=500-"},
)
assert response2.status_code == 206
second_half = response2.content
assert len(second_half) == 500
# Combine and verify
combined = first_half + second_half
assert combined == content
@pytest.mark.integration
def test_resume_with_etag_verification(self, integration_client, test_package):
"""Test that resumed download can verify content hasn't changed."""
project, package = test_package
content = b"resume etag verification test content"
expected_hash = compute_sha256(content)
upload_test_file(integration_client, project, package, content, tag="resume-etag")
# Get ETag from first request
response1 = integration_client.get(
f"/api/v1/project/{project}/{package}/+/resume-etag",
params={"mode": "proxy"},
headers={"Range": "bytes=0-9"},
)
assert response1.status_code == 206
etag = response1.headers.get("ETag")
assert etag == f'"{expected_hash}"'
# Resume with If-Match to ensure content hasn't changed
# (Note: If-Match would fail and return 412 if content changed)
response2 = integration_client.get(
f"/api/v1/project/{project}/{package}/+/resume-etag",
params={"mode": "proxy"},
headers={"Range": "bytes=10-"},
)
assert response2.status_code == 206
# ETag should be the same
assert response2.headers.get("ETag") == etag
class TestLargeFileStreaming:
"""Tests for streaming large files."""
@pytest.mark.integration
def test_stream_1mb_file(self, integration_client, test_package, sized_content):
"""Test streaming a 1MB file."""
project, package = test_package
content, expected_hash = sized_content(SIZE_1MB, seed=500)
upload_test_file(integration_client, project, package, content, tag="stream-1mb")
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/stream-1mb",
params={"mode": "proxy"},
)
assert response.status_code == 200
assert len(response.content) == SIZE_1MB
assert compute_sha256(response.content) == expected_hash
@pytest.mark.integration
def test_stream_large_file_has_correct_headers(
self, integration_client, test_package, sized_content
):
"""Test that large file streaming has correct headers."""
project, package = test_package
content, expected_hash = sized_content(SIZE_100KB, seed=501)
upload_test_file(integration_client, project, package, content, tag="stream-hdr")
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/stream-hdr",
params={"mode": "proxy"},
)
assert response.status_code == 200
assert int(response.headers.get("Content-Length", 0)) == SIZE_100KB
assert response.headers.get("X-Checksum-SHA256") == expected_hash
assert response.headers.get("Accept-Ranges") == "bytes"
@pytest.mark.integration
def test_range_request_on_large_file(
self, integration_client, test_package, sized_content
):
"""Test range request on a larger file."""
project, package = test_package
content, _ = sized_content(SIZE_100KB, seed=502)
upload_test_file(integration_client, project, package, content, tag="range-large")
# Request a slice from the middle
start = 50000
end = 50999
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/range-large",
params={"mode": "proxy"},
headers={"Range": f"bytes={start}-{end}"},
)
assert response.status_code == 206
assert len(response.content) == 1000
assert response.content == content[start : end + 1]
class TestDownloadModes:
"""Tests for different download modes."""
@pytest.mark.integration
def test_proxy_mode_streams_content(self, integration_client, test_package):
"""Test proxy mode streams content through backend."""
project, package = test_package
content = b"proxy mode test content"
upload_test_file(integration_client, project, package, content, tag="mode-proxy")
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/mode-proxy",
params={"mode": "proxy"},
)
assert response.status_code == 200
assert response.content == content
@pytest.mark.integration
def test_presigned_mode_returns_url(self, integration_client, test_package):
"""Test presigned mode returns JSON with URL."""
project, package = test_package
content = b"presigned mode test"
upload_test_file(integration_client, project, package, content, tag="mode-presign")
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/mode-presign",
params={"mode": "presigned"},
)
assert response.status_code == 200
data = response.json()
assert "url" in data
assert "expires_at" in data
assert data["url"].startswith("http")
@pytest.mark.integration
def test_redirect_mode_returns_302(self, integration_client, test_package):
"""Test redirect mode returns 302 to presigned URL."""
project, package = test_package
content = b"redirect mode test"
upload_test_file(integration_client, project, package, content, tag="mode-redir")
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/mode-redir",
params={"mode": "redirect"},
follow_redirects=False,
)
assert response.status_code == 302
assert "Location" in response.headers
class TestIntegrityDuringStreaming:
"""Tests for data integrity during streaming downloads."""
@pytest.mark.integration
def test_checksum_header_matches_content(self, integration_client, test_package):
"""Test X-Checksum-SHA256 header matches actual downloaded content."""
project, package = test_package
content = b"integrity check content"
expected_hash = compute_sha256(content)
upload_test_file(integration_client, project, package, content, tag="integrity")
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/integrity",
params={"mode": "proxy"},
)
assert response.status_code == 200
header_hash = response.headers.get("X-Checksum-SHA256")
actual_hash = compute_sha256(response.content)
assert header_hash == expected_hash
assert actual_hash == expected_hash
assert header_hash == actual_hash
@pytest.mark.integration
def test_etag_matches_content_hash(self, integration_client, test_package):
"""Test ETag header matches content hash."""
project, package = test_package
content = b"etag integrity test"
expected_hash = compute_sha256(content)
upload_test_file(integration_client, project, package, content, tag="etag-int")
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/etag-int",
params={"mode": "proxy"},
)
assert response.status_code == 200
etag = response.headers.get("ETag", "").strip('"')
actual_hash = compute_sha256(response.content)
assert etag == expected_hash
assert actual_hash == expected_hash
@pytest.mark.integration
def test_digest_header_present(self, integration_client, test_package):
"""Test Digest header is present in RFC 3230 format."""
project, package = test_package
content = b"digest header test"
upload_test_file(integration_client, project, package, content, tag="digest")
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/digest",
params={"mode": "proxy"},
)
assert response.status_code == 200
assert "Digest" in response.headers
assert response.headers["Digest"].startswith("sha-256=")

View File

@@ -10,6 +10,7 @@ Tests cover:
- S3 storage verification
"""
import os
import pytest
import io
import threading
@@ -25,6 +26,19 @@ from tests.factories import (
class TestUploadBasics:
"""Tests for basic upload functionality."""
@pytest.mark.integration
def test_upload_returns_200(self, integration_client, test_package):
"""Test upload with valid file returns 200."""
project, package = test_package
content = b"valid file upload test"
files = {"file": ("test.bin", io.BytesIO(content), "application/octet-stream")}
response = integration_client.post(
f"/api/v1/project/{project}/{package}/upload",
files=files,
)
assert response.status_code == 200
@pytest.mark.integration
def test_upload_returns_artifact_id(self, integration_client, test_package):
"""Test upload returns the artifact ID (SHA256 hash)."""
@@ -101,6 +115,83 @@ class TestUploadBasics:
assert "created_at" in result
assert result["created_at"] is not None
@pytest.mark.integration
def test_upload_without_tag_succeeds(self, integration_client, test_package):
"""Test upload without tag succeeds (no tag created)."""
project, package = test_package
content = b"upload without tag test"
expected_hash = compute_sha256(content)
files = {"file": ("no_tag.bin", io.BytesIO(content), "application/octet-stream")}
response = integration_client.post(
f"/api/v1/project/{project}/{package}/upload",
files=files,
# No tag parameter
)
assert response.status_code == 200
result = response.json()
assert result["artifact_id"] == expected_hash
# Verify no tag was created - list tags and check
tags_response = integration_client.get(
f"/api/v1/project/{project}/{package}/tags"
)
assert tags_response.status_code == 200
tags = tags_response.json()
# Filter for tags pointing to this artifact
artifact_tags = [t for t in tags.get("items", tags) if t.get("artifact_id") == expected_hash]
assert len(artifact_tags) == 0, "Tag should not be created when not specified"
@pytest.mark.integration
def test_upload_creates_artifact_in_database(self, integration_client, test_package):
"""Test upload creates artifact record in database."""
project, package = test_package
content = b"database artifact test"
expected_hash = compute_sha256(content)
upload_test_file(integration_client, project, package, content)
# Verify artifact exists via API
response = integration_client.get(f"/api/v1/artifact/{expected_hash}")
assert response.status_code == 200
artifact = response.json()
assert artifact["id"] == expected_hash
assert artifact["size"] == len(content)
@pytest.mark.integration
@pytest.mark.requires_direct_s3
def test_upload_creates_object_in_s3(self, integration_client, test_package):
"""Test upload creates object in S3 storage."""
project, package = test_package
content = b"s3 object creation test"
expected_hash = compute_sha256(content)
upload_test_file(integration_client, project, package, content)
# Verify S3 object exists
assert s3_object_exists(expected_hash), "S3 object should exist after upload"
@pytest.mark.integration
def test_upload_with_tag_creates_tag_record(self, integration_client, test_package):
"""Test upload with tag creates tag record."""
project, package = test_package
content = b"tag creation test"
expected_hash = compute_sha256(content)
tag_name = "my-tag-v1"
upload_test_file(
integration_client, project, package, content, tag=tag_name
)
# Verify tag exists
tags_response = integration_client.get(
f"/api/v1/project/{project}/{package}/tags"
)
assert tags_response.status_code == 200
tags = tags_response.json()
tag_names = [t["name"] for t in tags.get("items", tags)]
assert tag_name in tag_names
class TestDuplicateUploads:
"""Tests for duplicate upload deduplication behavior."""
@@ -248,6 +339,23 @@ class TestDownload:
assert response.status_code == 200
assert response.content == original_content
@pytest.mark.integration
def test_download_by_tag_prefix(self, integration_client, test_package):
"""Test downloading artifact using tag: prefix."""
project, package = test_package
original_content = b"download by tag prefix test"
upload_test_file(
integration_client, project, package, original_content, tag="prefix-tag"
)
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/tag:prefix-tag",
params={"mode": "proxy"},
)
assert response.status_code == 200
assert response.content == original_content
@pytest.mark.integration
def test_download_nonexistent_tag(self, integration_client, test_package):
"""Test downloading nonexistent tag returns 404."""
@@ -258,6 +366,33 @@ class TestDownload:
)
assert response.status_code == 404
@pytest.mark.integration
def test_download_nonexistent_artifact(self, integration_client, test_package):
"""Test downloading nonexistent artifact ID returns 404."""
project, package = test_package
fake_hash = "0" * 64
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/artifact:{fake_hash}"
)
assert response.status_code == 404
@pytest.mark.integration
def test_download_from_nonexistent_project(self, integration_client, unique_test_id):
"""Test downloading from nonexistent project returns 404."""
response = integration_client.get(
f"/api/v1/project/nonexistent-project-{unique_test_id}/somepackage/+/sometag"
)
assert response.status_code == 404
@pytest.mark.integration
def test_download_from_nonexistent_package(self, integration_client, test_project, unique_test_id):
"""Test downloading from nonexistent package returns 404."""
response = integration_client.get(
f"/api/v1/project/{test_project}/nonexistent-package-{unique_test_id}/+/sometag"
)
assert response.status_code == 404
@pytest.mark.integration
def test_content_matches_original(self, integration_client, test_package):
"""Test downloaded content matches original exactly."""
@@ -275,6 +410,111 @@ class TestDownload:
assert response.content == original_content
class TestDownloadHeaders:
"""Tests for download response headers."""
@pytest.mark.integration
def test_download_content_type_header(self, integration_client, test_package):
"""Test download returns correct Content-Type header."""
project, package = test_package
content = b"content type header test"
upload_test_file(
integration_client, project, package, content,
filename="test.txt", tag="content-type-test"
)
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/content-type-test",
params={"mode": "proxy"},
)
assert response.status_code == 200
# Content-Type should be set (either text/plain or application/octet-stream)
assert "content-type" in response.headers
@pytest.mark.integration
def test_download_content_length_header(self, integration_client, test_package):
"""Test download returns correct Content-Length header."""
project, package = test_package
content = b"content length header test - exactly 41 bytes!"
expected_length = len(content)
upload_test_file(
integration_client, project, package, content, tag="content-length-test"
)
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/content-length-test",
params={"mode": "proxy"},
)
assert response.status_code == 200
assert "content-length" in response.headers
assert int(response.headers["content-length"]) == expected_length
@pytest.mark.integration
def test_download_content_disposition_header(self, integration_client, test_package):
"""Test download returns correct Content-Disposition header."""
project, package = test_package
content = b"content disposition test"
filename = "my-test-file.bin"
upload_test_file(
integration_client, project, package, content,
filename=filename, tag="disposition-test"
)
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/disposition-test",
params={"mode": "proxy"},
)
assert response.status_code == 200
assert "content-disposition" in response.headers
disposition = response.headers["content-disposition"]
assert "attachment" in disposition
assert filename in disposition
@pytest.mark.integration
def test_download_checksum_headers(self, integration_client, test_package):
"""Test download returns checksum headers."""
project, package = test_package
content = b"checksum header test content"
expected_hash = compute_sha256(content)
upload_test_file(
integration_client, project, package, content, tag="checksum-headers"
)
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/checksum-headers",
params={"mode": "proxy"},
)
assert response.status_code == 200
# Check for checksum headers
assert "x-checksum-sha256" in response.headers
assert response.headers["x-checksum-sha256"] == expected_hash
@pytest.mark.integration
def test_download_etag_header(self, integration_client, test_package):
"""Test download returns ETag header (artifact ID)."""
project, package = test_package
content = b"etag header test"
expected_hash = compute_sha256(content)
upload_test_file(
integration_client, project, package, content, tag="etag-test"
)
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/etag-test",
params={"mode": "proxy"},
)
assert response.status_code == 200
assert "etag" in response.headers
# ETag should contain the artifact ID (hash)
etag = response.headers["etag"].strip('"')
assert etag == expected_hash
class TestConcurrentUploads:
"""Tests for concurrent upload handling."""
@@ -301,7 +541,7 @@ class TestConcurrentUploads:
try:
from httpx import Client
base_url = "http://localhost:8080"
base_url = os.environ.get("ORCHARD_TEST_URL", "http://localhost:8080")
with Client(base_url=base_url, timeout=30.0) as client:
files = {
"file": (
@@ -397,6 +637,7 @@ class TestUploadFailureCleanup:
"""Tests for cleanup when uploads fail."""
@pytest.mark.integration
@pytest.mark.requires_direct_s3
def test_upload_failure_invalid_project_no_orphaned_s3(
self, integration_client, unique_test_id
):
@@ -419,6 +660,7 @@ class TestUploadFailureCleanup:
)
@pytest.mark.integration
@pytest.mark.requires_direct_s3
def test_upload_failure_invalid_package_no_orphaned_s3(
self, integration_client, test_project, unique_test_id
):
@@ -466,6 +708,7 @@ class TestS3StorageVerification:
"""Tests to verify S3 storage behavior."""
@pytest.mark.integration
@pytest.mark.requires_direct_s3
def test_s3_single_object_after_duplicates(
self, integration_client, test_package, unique_test_id
):
@@ -521,6 +764,7 @@ class TestSecurityPathTraversal:
"""
@pytest.mark.integration
@pytest.mark.requires_direct_s3
def test_path_traversal_in_filename_stored_safely(
self, integration_client, test_package
):

View File

@@ -0,0 +1,347 @@
"""
Integration tests for package version API endpoints.
Tests cover:
- Version creation via upload
- Version auto-detection from filename
- Version listing and retrieval
- Download by version prefix
- Version deletion
"""
import pytest
import io
from tests.factories import (
compute_sha256,
upload_test_file,
)
class TestVersionCreation:
"""Tests for creating versions via upload."""
@pytest.mark.integration
def test_upload_with_explicit_version(self, integration_client, test_package):
"""Test upload with explicit version parameter creates version record."""
project, package = test_package
content = b"version creation test"
expected_hash = compute_sha256(content)
files = {"file": ("app.tar.gz", io.BytesIO(content), "application/octet-stream")}
response = integration_client.post(
f"/api/v1/project/{project}/{package}/upload",
files=files,
data={"version": "1.0.0"},
)
assert response.status_code == 200
result = response.json()
assert result["artifact_id"] == expected_hash
assert result.get("version") == "1.0.0"
assert result.get("version_source") == "explicit"
@pytest.mark.integration
def test_upload_with_version_and_tag(self, integration_client, test_package):
"""Test upload with both version and tag creates both records."""
project, package = test_package
content = b"version and tag test"
files = {"file": ("app.tar.gz", io.BytesIO(content), "application/octet-stream")}
response = integration_client.post(
f"/api/v1/project/{project}/{package}/upload",
files=files,
data={"version": "2.0.0", "tag": "latest"},
)
assert response.status_code == 200
result = response.json()
assert result.get("version") == "2.0.0"
# Verify tag was also created
tags_response = integration_client.get(
f"/api/v1/project/{project}/{package}/tags"
)
assert tags_response.status_code == 200
tags = tags_response.json()
tag_names = [t["name"] for t in tags.get("items", tags)]
assert "latest" in tag_names
@pytest.mark.integration
def test_duplicate_version_same_content_succeeds(self, integration_client, test_package):
"""Test uploading same version with same content succeeds (deduplication)."""
project, package = test_package
content = b"version dedup test"
# First upload with version
files1 = {"file": ("app1.tar.gz", io.BytesIO(content), "application/octet-stream")}
response1 = integration_client.post(
f"/api/v1/project/{project}/{package}/upload",
files=files1,
data={"version": "3.0.0"},
)
assert response1.status_code == 200
# Second upload with same version and same content succeeds
files2 = {"file": ("app2.tar.gz", io.BytesIO(content), "application/octet-stream")}
response2 = integration_client.post(
f"/api/v1/project/{project}/{package}/upload",
files=files2,
data={"version": "3.0.0"},
)
# This succeeds because it's the same artifact (deduplication)
assert response2.status_code == 200
class TestVersionAutoDetection:
"""Tests for automatic version detection from filename."""
@pytest.mark.integration
def test_version_detected_from_filename_tarball(self, integration_client, test_package):
"""Test version is auto-detected from tarball filename or metadata."""
project, package = test_package
content = b"auto detect version tarball"
files = {"file": ("myapp-1.2.3.tar.gz", io.BytesIO(content), "application/octet-stream")}
response = integration_client.post(
f"/api/v1/project/{project}/{package}/upload",
files=files,
)
assert response.status_code == 200
result = response.json()
assert result.get("version") == "1.2.3"
# Version source can be 'filename' or 'metadata' depending on detection order
assert result.get("version_source") in ["filename", "metadata"]
@pytest.mark.integration
def test_version_detected_from_filename_zip(self, integration_client, test_package):
"""Test version is auto-detected from zip filename."""
project, package = test_package
content = b"auto detect version zip"
files = {"file": ("package-2.0.0.zip", io.BytesIO(content), "application/octet-stream")}
response = integration_client.post(
f"/api/v1/project/{project}/{package}/upload",
files=files,
)
assert response.status_code == 200
result = response.json()
assert result.get("version") == "2.0.0"
assert result.get("version_source") == "filename"
@pytest.mark.integration
def test_explicit_version_overrides_filename(self, integration_client, test_package):
"""Test explicit version parameter overrides filename detection."""
project, package = test_package
content = b"explicit override test"
files = {"file": ("myapp-1.0.0.tar.gz", io.BytesIO(content), "application/octet-stream")}
response = integration_client.post(
f"/api/v1/project/{project}/{package}/upload",
files=files,
data={"version": "9.9.9"},
)
assert response.status_code == 200
result = response.json()
assert result.get("version") == "9.9.9"
assert result.get("version_source") == "explicit"
@pytest.mark.integration
def test_no_version_detected_from_plain_filename(self, integration_client, test_package):
"""Test no version is created for filenames without version pattern."""
project, package = test_package
content = b"no version in filename"
files = {"file": ("plain-file.bin", io.BytesIO(content), "application/octet-stream")}
response = integration_client.post(
f"/api/v1/project/{project}/{package}/upload",
files=files,
)
assert response.status_code == 200
result = response.json()
# Version should be None or not present
assert result.get("version") is None
class TestVersionListing:
"""Tests for listing and retrieving versions."""
@pytest.mark.integration
def test_list_versions(self, integration_client, test_package):
"""Test listing all versions for a package."""
project, package = test_package
# Create multiple versions
for ver in ["1.0.0", "1.1.0", "2.0.0"]:
content = f"version {ver} content".encode()
files = {"file": (f"app-{ver}.tar.gz", io.BytesIO(content), "application/octet-stream")}
response = integration_client.post(
f"/api/v1/project/{project}/{package}/upload",
files=files,
data={"version": ver},
)
assert response.status_code == 200
# List versions
response = integration_client.get(
f"/api/v1/project/{project}/{package}/versions"
)
assert response.status_code == 200
data = response.json()
versions = [v["version"] for v in data.get("items", data)]
assert "1.0.0" in versions
assert "1.1.0" in versions
assert "2.0.0" in versions
@pytest.mark.integration
def test_get_specific_version(self, integration_client, test_package):
"""Test getting details for a specific version."""
project, package = test_package
content = b"specific version test"
expected_hash = compute_sha256(content)
# Create version
files = {"file": ("app-4.0.0.tar.gz", io.BytesIO(content), "application/octet-stream")}
integration_client.post(
f"/api/v1/project/{project}/{package}/upload",
files=files,
data={"version": "4.0.0"},
)
# Get version details
response = integration_client.get(
f"/api/v1/project/{project}/{package}/versions/4.0.0"
)
assert response.status_code == 200
data = response.json()
assert data["version"] == "4.0.0"
assert data["artifact_id"] == expected_hash
@pytest.mark.integration
def test_get_nonexistent_version_returns_404(self, integration_client, test_package):
"""Test getting nonexistent version returns 404."""
project, package = test_package
response = integration_client.get(
f"/api/v1/project/{project}/{package}/versions/99.99.99"
)
assert response.status_code == 404
class TestDownloadByVersion:
"""Tests for downloading artifacts by version."""
@pytest.mark.integration
def test_download_by_version_prefix(self, integration_client, test_package):
"""Test downloading artifact using version: prefix."""
project, package = test_package
content = b"download by version test"
expected_hash = compute_sha256(content)
# Upload with version
files = {"file": ("app.tar.gz", io.BytesIO(content), "application/octet-stream")}
integration_client.post(
f"/api/v1/project/{project}/{package}/upload",
files=files,
data={"version": "5.0.0"},
)
# Download by version prefix
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/version:5.0.0",
params={"mode": "proxy"},
)
assert response.status_code == 200
assert response.content == content
@pytest.mark.integration
def test_download_nonexistent_version_returns_404(self, integration_client, test_package):
"""Test downloading nonexistent version returns 404."""
project, package = test_package
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/version:99.0.0"
)
assert response.status_code == 404
@pytest.mark.integration
def test_version_resolution_priority(self, integration_client, test_package):
"""Test that version: prefix explicitly resolves to version, not tag."""
project, package = test_package
version_content = b"this is the version content"
tag_content = b"this is the tag content"
# Create a version 6.0.0
files1 = {"file": ("app-v.tar.gz", io.BytesIO(version_content), "application/octet-stream")}
integration_client.post(
f"/api/v1/project/{project}/{package}/upload",
files=files1,
data={"version": "6.0.0"},
)
# Create a tag named "6.0.0" pointing to different content
files2 = {"file": ("app-t.tar.gz", io.BytesIO(tag_content), "application/octet-stream")}
integration_client.post(
f"/api/v1/project/{project}/{package}/upload",
files=files2,
data={"tag": "6.0.0"},
)
# Download with version: prefix should get version content
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/version:6.0.0",
params={"mode": "proxy"},
)
assert response.status_code == 200
assert response.content == version_content
# Download with tag: prefix should get tag content
response2 = integration_client.get(
f"/api/v1/project/{project}/{package}/+/tag:6.0.0",
params={"mode": "proxy"},
)
assert response2.status_code == 200
assert response2.content == tag_content
class TestVersionDeletion:
"""Tests for deleting versions."""
@pytest.mark.integration
def test_delete_version(self, integration_client, test_package):
"""Test deleting a version."""
project, package = test_package
content = b"delete version test"
# Create version
files = {"file": ("app.tar.gz", io.BytesIO(content), "application/octet-stream")}
integration_client.post(
f"/api/v1/project/{project}/{package}/upload",
files=files,
data={"version": "7.0.0"},
)
# Verify version exists
response = integration_client.get(
f"/api/v1/project/{project}/{package}/versions/7.0.0"
)
assert response.status_code == 200
# Delete version - returns 204 No Content on success
delete_response = integration_client.delete(
f"/api/v1/project/{project}/{package}/versions/7.0.0"
)
assert delete_response.status_code == 204
# Verify version no longer exists
response2 = integration_client.get(
f"/api/v1/project/{project}/{package}/versions/7.0.0"
)
assert response2.status_code == 404
@pytest.mark.integration
def test_delete_nonexistent_version_returns_404(self, integration_client, test_package):
"""Test deleting nonexistent version returns 404."""
project, package = test_package
response = integration_client.delete(
f"/api/v1/project/{project}/{package}/versions/99.0.0"
)
assert response.status_code == 404

File diff suppressed because it is too large Load Diff

View File

@@ -0,0 +1,95 @@
"""Unit tests for authentication module."""
import pytest
from unittest.mock import patch, MagicMock
class TestCreateDefaultAdmin:
"""Tests for the create_default_admin function."""
def test_create_default_admin_with_env_password(self):
"""Test that ORCHARD_ADMIN_PASSWORD env var sets admin password."""
from app.auth import create_default_admin, verify_password
# Create mock settings with custom password
mock_settings = MagicMock()
mock_settings.admin_password = "my-custom-password-123"
# Mock database session
mock_db = MagicMock()
mock_db.query.return_value.count.return_value = 0 # No existing users
# Track the user that gets created
created_user = None
def capture_user(user):
nonlocal created_user
created_user = user
mock_db.add.side_effect = capture_user
with patch("app.auth.get_settings", return_value=mock_settings):
admin = create_default_admin(mock_db)
# Verify the user was created
assert mock_db.add.called
assert created_user is not None
assert created_user.username == "admin"
assert created_user.is_admin is True
# Password should NOT require change when set via env var
assert created_user.must_change_password is False
# Verify password was hashed correctly
assert verify_password("my-custom-password-123", created_user.password_hash)
def test_create_default_admin_with_default_password(self):
"""Test that default password 'changeme123' is used when env var not set."""
from app.auth import create_default_admin, verify_password
# Create mock settings with empty password (default)
mock_settings = MagicMock()
mock_settings.admin_password = ""
# Mock database session
mock_db = MagicMock()
mock_db.query.return_value.count.return_value = 0 # No existing users
# Track the user that gets created
created_user = None
def capture_user(user):
nonlocal created_user
created_user = user
mock_db.add.side_effect = capture_user
with patch("app.auth.get_settings", return_value=mock_settings):
admin = create_default_admin(mock_db)
# Verify the user was created
assert mock_db.add.called
assert created_user is not None
assert created_user.username == "admin"
assert created_user.is_admin is True
# Password SHOULD require change when using default
assert created_user.must_change_password is True
# Verify default password was used
assert verify_password("changeme123", created_user.password_hash)
def test_create_default_admin_skips_when_users_exist(self):
"""Test that no admin is created when users already exist."""
from app.auth import create_default_admin
# Create mock settings
mock_settings = MagicMock()
mock_settings.admin_password = "some-password"
# Mock database session with existing users
mock_db = MagicMock()
mock_db.query.return_value.count.return_value = 1 # Users exist
with patch("app.auth.get_settings", return_value=mock_settings):
result = create_default_admin(mock_db)
# Should return None and not create any user
assert result is None
assert not mock_db.add.called

View File

@@ -26,6 +26,8 @@ services:
- ORCHARD_REDIS_PORT=6379
# Higher rate limit for local development/testing
- ORCHARD_LOGIN_RATE_LIMIT=1000/minute
# Admin password - set in .env file or environment (see .env.example)
- ORCHARD_ADMIN_PASSWORD=${ORCHARD_ADMIN_PASSWORD:-}
depends_on:
postgres:
condition: service_healthy

View File

@@ -0,0 +1,294 @@
# Integrity Verification
Orchard uses content-addressable storage with SHA256 hashing to ensure artifact integrity. This document describes how integrity verification works and how to use it.
## How It Works
### Content-Addressable Storage
Orchard stores artifacts using their SHA256 hash as the unique identifier. This provides several benefits:
1. **Automatic deduplication**: Identical content is stored only once
2. **Built-in integrity**: The artifact ID *is* the content hash
3. **Tamper detection**: Any modification changes the hash, making corruption detectable
When you upload a file:
1. Orchard computes the SHA256 hash of the content
2. The hash becomes the artifact ID (64-character hex string)
3. The file is stored in S3 at `fruits/{hash[0:2]}/{hash[2:4]}/{hash}`
4. The hash and metadata are recorded in the database
### Hash Format
- Algorithm: SHA256
- Format: 64-character lowercase hexadecimal string
- Example: `dffd6021bb2bd5b0af676290809ec3a53191dd81c7f70a4b28688a362182986f`
## Client-Side Verification
### Before Upload
Compute the hash locally before uploading to verify the server received your content correctly:
```python
import hashlib
def compute_sha256(content: bytes) -> str:
return hashlib.sha256(content).hexdigest()
# Compute hash before upload
content = open("myfile.tar.gz", "rb").read()
local_hash = compute_sha256(content)
# Upload the file
response = requests.post(
f"{base_url}/api/v1/project/{project}/{package}/upload",
files={"file": ("myfile.tar.gz", content)},
)
result = response.json()
# Verify server computed the same hash
assert result["artifact_id"] == local_hash, "Hash mismatch!"
```
### Providing Expected Hash on Upload
You can provide the expected hash in the upload request. The server will reject the upload if the computed hash doesn't match:
```python
response = requests.post(
f"{base_url}/api/v1/project/{project}/{package}/upload",
files={"file": ("myfile.tar.gz", content)},
headers={"X-Checksum-SHA256": local_hash},
)
# Returns 422 if hash doesn't match
if response.status_code == 422:
print("Checksum mismatch - upload rejected")
```
### After Download
Verify downloaded content matches the expected hash using response headers:
```python
response = requests.get(
f"{base_url}/api/v1/project/{project}/{package}/+/{tag}",
params={"mode": "proxy"},
)
# Get expected hash from header
expected_hash = response.headers.get("X-Checksum-SHA256")
# Compute hash of downloaded content
actual_hash = compute_sha256(response.content)
# Verify
if actual_hash != expected_hash:
raise Exception(f"Integrity check failed! Expected {expected_hash}, got {actual_hash}")
```
### Response Headers for Verification
Download responses include multiple headers for verification:
| Header | Format | Description |
|--------|--------|-------------|
| `X-Checksum-SHA256` | Hex string | SHA256 hash (64 chars) |
| `ETag` | `"<hash>"` | SHA256 hash in quotes |
| `Digest` | `sha-256=<base64>` | RFC 3230 format (base64-encoded) |
| `Content-Length` | Integer | File size in bytes |
### Server-Side Verification on Download
Request server-side verification during download:
```bash
# Pre-verification: Server verifies before streaming (returns 500 if corrupt)
curl "${base_url}/api/v1/project/${project}/${package}/+/${tag}?mode=proxy&verify=true&verify_mode=pre"
# Stream verification: Server verifies while streaming (logs error if corrupt)
curl "${base_url}/api/v1/project/${project}/${package}/+/${tag}?mode=proxy&verify=true&verify_mode=stream"
```
The `X-Verified` header indicates whether server-side verification was performed:
- `X-Verified: true` - Content was verified by the server
## Server-Side Consistency Check
### Consistency Check Endpoint
Administrators can run a consistency check to verify all stored artifacts:
```bash
curl "${base_url}/api/v1/admin/consistency-check"
```
Response:
```json
{
"total_artifacts_checked": 1234,
"healthy": true,
"orphaned_s3_objects": 0,
"missing_s3_objects": 0,
"size_mismatches": 0,
"orphaned_s3_keys": [],
"missing_s3_keys": [],
"size_mismatch_artifacts": []
}
```
### What the Check Verifies
1. **Missing S3 objects**: Database records with no corresponding S3 object
2. **Orphaned S3 objects**: S3 objects with no database record
3. **Size mismatches**: S3 object size doesn't match database record
### Running Consistency Checks
**Manual check:**
```bash
# Check all artifacts
curl "${base_url}/api/v1/admin/consistency-check"
# Limit results (for large deployments)
curl "${base_url}/api/v1/admin/consistency-check?limit=100"
```
**Scheduled checks (recommended):**
Set up a cron job or Kubernetes CronJob to run periodic checks:
```yaml
# Kubernetes CronJob example
apiVersion: batch/v1
kind: CronJob
metadata:
name: orchard-consistency-check
spec:
schedule: "0 2 * * *" # Daily at 2 AM
jobTemplate:
spec:
template:
spec:
containers:
- name: check
image: curlimages/curl
command:
- /bin/sh
- -c
- |
response=$(curl -s "${ORCHARD_URL}/api/v1/admin/consistency-check")
healthy=$(echo "$response" | jq -r '.healthy')
if [ "$healthy" != "true" ]; then
echo "ALERT: Consistency check failed!"
echo "$response"
exit 1
fi
echo "Consistency check passed"
restartPolicy: OnFailure
```
## Recovery Procedures
### Corrupted Artifact (Size Mismatch)
If the consistency check reports size mismatches:
1. **Identify affected artifacts:**
```bash
curl "${base_url}/api/v1/admin/consistency-check" | jq '.size_mismatch_artifacts'
```
2. **Check if artifact can be re-uploaded:**
- If the original content is available, delete the corrupted artifact and re-upload
- The same content will produce the same artifact ID
3. **If original content is lost:**
- The artifact data is corrupted and cannot be recovered
- Delete the artifact record and notify affected users
- Consider restoring from backup if available
### Missing S3 Object
If database records exist but S3 objects are missing:
1. **Identify affected artifacts:**
```bash
curl "${base_url}/api/v1/admin/consistency-check" | jq '.missing_s3_keys'
```
2. **Check S3 bucket:**
- Verify the S3 bucket exists and is accessible
- Check S3 access logs for deletion events
- Check if objects were moved or lifecycle-deleted
3. **Recovery options:**
- Restore from S3 versioning (if enabled)
- Restore from backup
- Re-upload original content (if available)
- Delete orphaned database records
### Orphaned S3 Objects
If S3 objects exist without database records:
1. **Identify orphaned objects:**
```bash
curl "${base_url}/api/v1/admin/consistency-check" | jq '.orphaned_s3_keys'
```
2. **Investigate cause:**
- Upload interrupted before database commit?
- Database record deleted but S3 cleanup failed?
3. **Resolution:**
- If content is needed, create database record manually
- If content is not needed, delete the S3 object to reclaim storage
### Preventive Measures
1. **Enable S3 versioning** to recover from accidental deletions
2. **Regular backups** of both database and S3 bucket
3. **Scheduled consistency checks** to detect issues early
4. **Monitoring and alerting** on consistency check failures
5. **Audit logging** to track all artifact operations
## Verification in CI/CD
### Verifying Artifacts in Pipelines
```bash
#!/bin/bash
# Download and verify artifact in CI pipeline
ARTIFACT_URL="${ORCHARD_URL}/api/v1/project/${PROJECT}/${PACKAGE}/+/${TAG}"
# Download with verification headers
response=$(curl -s -D - "${ARTIFACT_URL}?mode=proxy" -o artifact.tar.gz)
expected_hash=$(echo "$response" | grep -i "X-Checksum-SHA256" | cut -d: -f2 | tr -d ' \r')
# Compute actual hash
actual_hash=$(sha256sum artifact.tar.gz | cut -d' ' -f1)
# Verify
if [ "$actual_hash" != "$expected_hash" ]; then
echo "ERROR: Integrity check failed!"
echo "Expected: $expected_hash"
echo "Actual: $actual_hash"
exit 1
fi
echo "Integrity verified: $actual_hash"
```
### Using Server-Side Verification
For critical deployments, use server-side pre-verification:
```bash
# Server verifies before streaming - returns 500 if corrupt
curl -f "${ARTIFACT_URL}?mode=proxy&verify=true&verify_mode=pre" -o artifact.tar.gz
```
This ensures the artifact is verified before any bytes are streamed to your pipeline.

View File

@@ -10,6 +10,7 @@ import ChangePasswordPage from './pages/ChangePasswordPage';
import APIKeysPage from './pages/APIKeysPage';
import AdminUsersPage from './pages/AdminUsersPage';
import AdminOIDCPage from './pages/AdminOIDCPage';
import ProjectSettingsPage from './pages/ProjectSettingsPage';
// Component that checks if user must change password
function RequirePasswordChange({ children }: { children: React.ReactNode }) {
@@ -45,6 +46,7 @@ function AppRoutes() {
<Route path="/admin/users" element={<AdminUsersPage />} />
<Route path="/admin/oidc" element={<AdminOIDCPage />} />
<Route path="/project/:projectName" element={<ProjectPage />} />
<Route path="/project/:projectName/settings" element={<ProjectSettingsPage />} />
<Route path="/project/:projectName/:packageName" element={<PackagePage />} />
</Routes>
</Layout>

View File

@@ -33,6 +33,9 @@ import {
OIDCConfigUpdate,
OIDCStatus,
PackageVersion,
ArtifactDependenciesResponse,
ReverseDependenciesResponse,
DependencyResolutionResponse,
} from './types';
const API_BASE = '/api/v1';
@@ -171,6 +174,30 @@ export async function getProject(name: string): Promise<Project> {
return handleResponse<Project>(response);
}
export async function updateProject(
projectName: string,
data: { description?: string; is_public?: boolean }
): Promise<Project> {
const response = await fetch(`${API_BASE}/projects/${projectName}`, {
method: 'PUT',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify(data),
credentials: 'include',
});
return handleResponse<Project>(response);
}
export async function deleteProject(projectName: string): Promise<void> {
const response = await fetch(`${API_BASE}/projects/${projectName}`, {
method: 'DELETE',
credentials: 'include',
});
if (!response.ok) {
const error = await response.json().catch(() => ({ detail: 'Unknown error' }));
throw new Error(error.detail || `HTTP ${response.status}`);
}
}
// Package API
export async function listPackages(projectName: string, params: PackageListParams = {}): Promise<PaginatedResponse<Package>> {
const query = buildQueryString(params as Record<string, unknown>);
@@ -488,3 +515,50 @@ export async function deleteVersion(
throw new Error(error.detail || `HTTP ${response.status}`);
}
}
// Dependency API
export async function getArtifactDependencies(artifactId: string): Promise<ArtifactDependenciesResponse> {
const response = await fetch(`${API_BASE}/artifact/${artifactId}/dependencies`);
return handleResponse<ArtifactDependenciesResponse>(response);
}
export async function getDependenciesByRef(
projectName: string,
packageName: string,
ref: string
): Promise<ArtifactDependenciesResponse> {
const response = await fetch(`${API_BASE}/project/${projectName}/${packageName}/+/${ref}/dependencies`);
return handleResponse<ArtifactDependenciesResponse>(response);
}
export async function getReverseDependencies(
projectName: string,
packageName: string,
params: { page?: number; limit?: number } = {}
): Promise<ReverseDependenciesResponse> {
const query = buildQueryString(params as Record<string, unknown>);
const response = await fetch(`${API_BASE}/project/${projectName}/${packageName}/reverse-dependencies${query}`);
return handleResponse<ReverseDependenciesResponse>(response);
}
export async function resolveDependencies(
projectName: string,
packageName: string,
ref: string
): Promise<DependencyResolutionResponse> {
const response = await fetch(`${API_BASE}/project/${projectName}/${packageName}/+/${ref}/resolve`);
return handleResponse<DependencyResolutionResponse>(response);
}
export async function getEnsureFile(
projectName: string,
packageName: string,
ref: string
): Promise<string> {
const response = await fetch(`${API_BASE}/project/${projectName}/${packageName}/+/${ref}/ensure`);
if (!response.ok) {
const error = await response.json().catch(() => ({ detail: 'Unknown error' }));
throw new ApiError(error.detail || `HTTP ${response.status}`, response.status);
}
return response.text();
}

View File

@@ -0,0 +1,338 @@
/* Dependency Graph Modal */
.dependency-graph-modal {
position: fixed;
top: 0;
left: 0;
right: 0;
bottom: 0;
background: rgba(0, 0, 0, 0.8);
display: flex;
align-items: center;
justify-content: center;
z-index: 1000;
padding: 24px;
}
.dependency-graph-content {
background: var(--bg-secondary);
border: 1px solid var(--border-primary);
border-radius: var(--radius-lg);
width: 100%;
max-width: 1200px;
height: 80vh;
display: flex;
flex-direction: column;
overflow: hidden;
}
.dependency-graph-header {
display: flex;
align-items: center;
gap: 16px;
padding: 16px 20px;
border-bottom: 1px solid var(--border-primary);
background: var(--bg-tertiary);
}
.dependency-graph-header h2 {
margin: 0;
font-size: 1.125rem;
font-weight: 600;
color: var(--text-primary);
}
.dependency-graph-info {
display: flex;
align-items: center;
gap: 12px;
flex: 1;
font-size: 0.875rem;
color: var(--text-secondary);
}
.graph-stats {
color: var(--text-muted);
font-size: 0.8125rem;
}
.close-btn {
background: transparent;
border: none;
color: var(--text-secondary);
cursor: pointer;
padding: 4px;
border-radius: var(--radius-sm);
display: flex;
align-items: center;
justify-content: center;
}
.close-btn:hover {
background: var(--bg-hover);
color: var(--text-primary);
}
.dependency-graph-toolbar {
display: flex;
align-items: center;
gap: 8px;
padding: 12px 20px;
border-bottom: 1px solid var(--border-primary);
background: var(--bg-secondary);
}
.zoom-level {
margin-left: auto;
font-size: 0.8125rem;
color: var(--text-muted);
font-family: 'JetBrains Mono', monospace;
}
.dependency-graph-container {
flex: 1;
overflow: hidden;
position: relative;
background:
linear-gradient(90deg, var(--border-primary) 1px, transparent 1px),
linear-gradient(var(--border-primary) 1px, transparent 1px);
background-size: 20px 20px;
background-position: center center;
}
.graph-canvas {
padding: 40px;
min-width: 100%;
min-height: 100%;
transform-origin: center center;
transition: transform 0.1s ease-out;
}
/* Graph Nodes */
.graph-node-container {
display: flex;
flex-direction: column;
align-items: flex-start;
}
.graph-node {
background: var(--bg-tertiary);
border: 2px solid var(--border-primary);
border-radius: var(--radius-md);
padding: 12px 16px;
min-width: 200px;
cursor: pointer;
transition: all var(--transition-fast);
position: relative;
}
.graph-node:hover {
border-color: var(--accent-primary);
box-shadow: 0 4px 12px rgba(16, 185, 129, 0.2);
}
.graph-node--root {
background: linear-gradient(135deg, rgba(16, 185, 129, 0.15) 0%, rgba(5, 150, 105, 0.15) 100%);
border-color: var(--accent-primary);
}
.graph-node--hovered {
transform: scale(1.02);
}
.graph-node__header {
display: flex;
align-items: center;
gap: 8px;
margin-bottom: 4px;
}
.graph-node__name {
font-weight: 600;
color: var(--accent-primary);
font-family: 'JetBrains Mono', monospace;
font-size: 0.875rem;
}
.graph-node__toggle {
background: var(--bg-hover);
border: 1px solid var(--border-primary);
border-radius: 4px;
width: 20px;
height: 20px;
display: flex;
align-items: center;
justify-content: center;
cursor: pointer;
font-size: 0.875rem;
color: var(--text-secondary);
font-weight: 600;
margin-left: auto;
}
.graph-node__toggle:hover {
background: var(--bg-tertiary);
color: var(--text-primary);
}
.graph-node__details {
display: flex;
align-items: center;
gap: 12px;
font-size: 0.75rem;
color: var(--text-muted);
}
.graph-node__version {
font-family: 'JetBrains Mono', monospace;
color: var(--text-secondary);
}
.graph-node__size {
color: var(--text-muted);
}
/* Graph Children / Tree Structure */
.graph-children {
display: flex;
padding-left: 24px;
margin-top: 8px;
position: relative;
}
.graph-connector {
position: absolute;
left: 12px;
top: 0;
bottom: 50%;
width: 12px;
border-left: 2px solid var(--border-primary);
border-bottom: 2px solid var(--border-primary);
border-bottom-left-radius: 8px;
}
.graph-children-list {
display: flex;
flex-direction: column;
gap: 8px;
position: relative;
}
.graph-children-list::before {
content: '';
position: absolute;
left: -12px;
top: 20px;
bottom: 20px;
border-left: 2px solid var(--border-primary);
}
.graph-children-list > .graph-node-container {
position: relative;
}
.graph-children-list > .graph-node-container::before {
content: '';
position: absolute;
left: -12px;
top: 20px;
width: 12px;
border-top: 2px solid var(--border-primary);
}
/* Loading, Error, Empty States */
.graph-loading,
.graph-error,
.graph-empty {
display: flex;
flex-direction: column;
align-items: center;
justify-content: center;
height: 100%;
gap: 16px;
color: var(--text-muted);
}
.graph-loading .spinner {
width: 32px;
height: 32px;
border: 3px solid var(--border-primary);
border-top-color: var(--accent-primary);
border-radius: 50%;
animation: spin 1s linear infinite;
}
@keyframes spin {
to { transform: rotate(360deg); }
}
.graph-error {
color: var(--error-color, #ef4444);
}
.graph-error svg {
opacity: 0.6;
}
.graph-error p {
max-width: 400px;
text-align: center;
line-height: 1.5;
}
/* Tooltip */
.graph-tooltip {
position: fixed;
bottom: 24px;
left: 50%;
transform: translateX(-50%);
background: var(--bg-tertiary);
border: 1px solid var(--border-primary);
border-radius: var(--radius-md);
padding: 12px 16px;
font-size: 0.8125rem;
box-shadow: 0 8px 24px rgba(0, 0, 0, 0.4);
z-index: 1001;
}
.graph-tooltip strong {
display: block;
color: var(--accent-primary);
font-family: 'JetBrains Mono', monospace;
margin-bottom: 4px;
}
.graph-tooltip div {
color: var(--text-secondary);
margin-top: 2px;
}
.tooltip-hint {
margin-top: 8px;
padding-top: 8px;
border-top: 1px solid var(--border-primary);
color: var(--text-muted);
font-size: 0.75rem;
}
/* Responsive */
@media (max-width: 768px) {
.dependency-graph-modal {
padding: 0;
}
.dependency-graph-content {
height: 100vh;
border-radius: 0;
max-width: none;
}
.dependency-graph-header {
flex-wrap: wrap;
}
.dependency-graph-info {
flex-basis: 100%;
order: 3;
margin-top: 8px;
}
}

View File

@@ -0,0 +1,323 @@
import { useState, useEffect, useCallback, useRef } from 'react';
import { useNavigate } from 'react-router-dom';
import { ResolvedArtifact, DependencyResolutionResponse, Dependency } from '../types';
import { resolveDependencies, getArtifactDependencies } from '../api';
import './DependencyGraph.css';
interface DependencyGraphProps {
projectName: string;
packageName: string;
tagName: string;
onClose: () => void;
}
interface GraphNode {
id: string;
project: string;
package: string;
version: string | null;
size: number;
depth: number;
children: GraphNode[];
isRoot?: boolean;
}
function formatBytes(bytes: number): string {
if (bytes === 0) return '0 B';
const k = 1024;
const sizes = ['B', 'KB', 'MB', 'GB'];
const i = Math.floor(Math.log(bytes) / Math.log(k));
return parseFloat((bytes / Math.pow(k, i)).toFixed(1)) + ' ' + sizes[i];
}
function DependencyGraph({ projectName, packageName, tagName, onClose }: DependencyGraphProps) {
const navigate = useNavigate();
const containerRef = useRef<HTMLDivElement>(null);
const [loading, setLoading] = useState(true);
const [error, setError] = useState<string | null>(null);
const [resolution, setResolution] = useState<DependencyResolutionResponse | null>(null);
const [graphRoot, setGraphRoot] = useState<GraphNode | null>(null);
const [hoveredNode, setHoveredNode] = useState<GraphNode | null>(null);
const [zoom, setZoom] = useState(1);
const [pan, setPan] = useState({ x: 0, y: 0 });
const [isDragging, setIsDragging] = useState(false);
const [dragStart, setDragStart] = useState({ x: 0, y: 0 });
const [collapsedNodes, setCollapsedNodes] = useState<Set<string>>(new Set());
// Build graph structure from resolution data
const buildGraph = useCallback(async (resolutionData: DependencyResolutionResponse) => {
const artifactMap = new Map<string, ResolvedArtifact>();
resolutionData.resolved.forEach(artifact => {
artifactMap.set(artifact.artifact_id, artifact);
});
// Fetch dependencies for each artifact to build the tree
const depsMap = new Map<string, Dependency[]>();
for (const artifact of resolutionData.resolved) {
try {
const deps = await getArtifactDependencies(artifact.artifact_id);
depsMap.set(artifact.artifact_id, deps.dependencies);
} catch {
depsMap.set(artifact.artifact_id, []);
}
}
// Find the root artifact (the requested one)
const rootArtifact = resolutionData.resolved.find(
a => a.project === resolutionData.requested.project &&
a.package === resolutionData.requested.package
);
if (!rootArtifact) {
return null;
}
// Build tree recursively
const visited = new Set<string>();
const buildNode = (artifact: ResolvedArtifact, depth: number): GraphNode => {
const nodeId = `${artifact.project}/${artifact.package}`;
visited.add(artifact.artifact_id);
const deps = depsMap.get(artifact.artifact_id) || [];
const children: GraphNode[] = [];
for (const dep of deps) {
// Find the resolved artifact for this dependency
const childArtifact = resolutionData.resolved.find(
a => a.project === dep.project && a.package === dep.package
);
if (childArtifact && !visited.has(childArtifact.artifact_id)) {
children.push(buildNode(childArtifact, depth + 1));
}
}
return {
id: nodeId,
project: artifact.project,
package: artifact.package,
version: artifact.version || artifact.tag,
size: artifact.size,
depth,
children,
isRoot: depth === 0,
};
};
return buildNode(rootArtifact, 0);
}, []);
useEffect(() => {
async function loadData() {
setLoading(true);
setError(null);
try {
const result = await resolveDependencies(projectName, packageName, tagName);
setResolution(result);
const graph = await buildGraph(result);
setGraphRoot(graph);
} catch (err) {
if (err instanceof Error) {
// Check if it's a resolution error
try {
const errorData = JSON.parse(err.message);
if (errorData.error === 'circular_dependency') {
setError(`Circular dependency detected: ${errorData.cycle?.join(' → ')}`);
} else if (errorData.error === 'dependency_conflict') {
setError(`Dependency conflict: ${errorData.message}`);
} else {
setError(err.message);
}
} catch {
setError(err.message);
}
} else {
setError('Failed to load dependency graph');
}
} finally {
setLoading(false);
}
}
loadData();
}, [projectName, packageName, tagName, buildGraph]);
const handleNodeClick = (node: GraphNode) => {
navigate(`/project/${node.project}/${node.package}`);
onClose();
};
const handleNodeToggle = (node: GraphNode, e: React.MouseEvent) => {
e.stopPropagation();
setCollapsedNodes(prev => {
const next = new Set(prev);
if (next.has(node.id)) {
next.delete(node.id);
} else {
next.add(node.id);
}
return next;
});
};
const handleWheel = (e: React.WheelEvent) => {
e.preventDefault();
const delta = e.deltaY > 0 ? -0.1 : 0.1;
setZoom(z => Math.max(0.25, Math.min(2, z + delta)));
};
const handleMouseDown = (e: React.MouseEvent) => {
if (e.target === containerRef.current || (e.target as HTMLElement).classList.contains('graph-canvas')) {
setIsDragging(true);
setDragStart({ x: e.clientX - pan.x, y: e.clientY - pan.y });
}
};
const handleMouseMove = (e: React.MouseEvent) => {
if (isDragging) {
setPan({ x: e.clientX - dragStart.x, y: e.clientY - dragStart.y });
}
};
const handleMouseUp = () => {
setIsDragging(false);
};
const resetView = () => {
setZoom(1);
setPan({ x: 0, y: 0 });
};
const renderNode = (node: GraphNode, index: number = 0): JSX.Element => {
const isCollapsed = collapsedNodes.has(node.id);
const hasChildren = node.children.length > 0;
return (
<div key={`${node.id}-${index}`} className="graph-node-container">
<div
className={`graph-node ${node.isRoot ? 'graph-node--root' : ''} ${hoveredNode?.id === node.id ? 'graph-node--hovered' : ''}`}
onClick={() => handleNodeClick(node)}
onMouseEnter={() => setHoveredNode(node)}
onMouseLeave={() => setHoveredNode(null)}
>
<div className="graph-node__header">
<span className="graph-node__name">{node.project}/{node.package}</span>
{hasChildren && (
<button
className="graph-node__toggle"
onClick={(e) => handleNodeToggle(node, e)}
title={isCollapsed ? 'Expand' : 'Collapse'}
>
{isCollapsed ? '+' : '-'}
</button>
)}
</div>
<div className="graph-node__details">
{node.version && <span className="graph-node__version">@ {node.version}</span>}
<span className="graph-node__size">{formatBytes(node.size)}</span>
</div>
</div>
{hasChildren && !isCollapsed && (
<div className="graph-children">
<div className="graph-connector"></div>
<div className="graph-children-list">
{node.children.map((child, i) => renderNode(child, i))}
</div>
</div>
)}
</div>
);
};
return (
<div className="dependency-graph-modal" onClick={onClose}>
<div className="dependency-graph-content" onClick={e => e.stopPropagation()}>
<div className="dependency-graph-header">
<h2>Dependency Graph</h2>
<div className="dependency-graph-info">
<span>{projectName}/{packageName} @ {tagName}</span>
{resolution && (
<span className="graph-stats">
{resolution.artifact_count} packages {formatBytes(resolution.total_size)} total
</span>
)}
</div>
<button className="close-btn" onClick={onClose} title="Close">
<svg width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" strokeWidth="2">
<line x1="18" y1="6" x2="6" y2="18"></line>
<line x1="6" y1="6" x2="18" y2="18"></line>
</svg>
</button>
</div>
<div className="dependency-graph-toolbar">
<button className="btn btn-secondary btn-small" onClick={() => setZoom(z => Math.min(2, z + 0.25))}>
Zoom In
</button>
<button className="btn btn-secondary btn-small" onClick={() => setZoom(z => Math.max(0.25, z - 0.25))}>
Zoom Out
</button>
<button className="btn btn-secondary btn-small" onClick={resetView}>
Reset View
</button>
<span className="zoom-level">{Math.round(zoom * 100)}%</span>
</div>
<div
ref={containerRef}
className="dependency-graph-container"
onWheel={handleWheel}
onMouseDown={handleMouseDown}
onMouseMove={handleMouseMove}
onMouseUp={handleMouseUp}
onMouseLeave={handleMouseUp}
>
{loading ? (
<div className="graph-loading">
<div className="spinner"></div>
<span>Resolving dependencies...</span>
</div>
) : error ? (
<div className="graph-error">
<svg width="48" height="48" viewBox="0 0 24 24" fill="none" stroke="currentColor" strokeWidth="2">
<circle cx="12" cy="12" r="10"></circle>
<line x1="12" y1="8" x2="12" y2="12"></line>
<line x1="12" y1="16" x2="12.01" y2="16"></line>
</svg>
<p>{error}</p>
</div>
) : graphRoot ? (
<div
className="graph-canvas"
style={{
transform: `translate(${pan.x}px, ${pan.y}px) scale(${zoom})`,
cursor: isDragging ? 'grabbing' : 'grab',
}}
>
{renderNode(graphRoot)}
</div>
) : (
<div className="graph-empty">No dependencies to display</div>
)}
</div>
{hoveredNode && (
<div className="graph-tooltip">
<strong>{hoveredNode.project}/{hoveredNode.package}</strong>
{hoveredNode.version && <div>Version: {hoveredNode.version}</div>}
<div>Size: {formatBytes(hoveredNode.size)}</div>
<div className="tooltip-hint">Click to navigate</div>
</div>
)}
</div>
</div>
);
}
export default DependencyGraph;

View File

@@ -193,7 +193,6 @@ function Layout({ children }: LayoutProps) {
</div>
<div className="footer-links">
<a href="/docs">Documentation</a>
<a href="/api/v1">API</a>
</div>
</div>
</footer>

View File

@@ -358,6 +358,12 @@
gap: 4px;
}
.page-header__actions {
display: flex;
align-items: center;
gap: 12px;
}
/* Package card styles */
.package-card__header {
display: flex;

View File

@@ -127,6 +127,12 @@ h2 {
font-size: 0.75rem;
}
/* Action buttons in table */
.action-buttons {
display: flex;
gap: 8px;
}
/* Download by Artifact ID Section */
.download-by-id-section {
margin-top: 32px;
@@ -424,6 +430,340 @@ tr:hover .copy-btn {
white-space: nowrap;
}
/* Dependencies Section */
.dependencies-section {
margin-top: 32px;
background: var(--bg-secondary);
}
.dependencies-header {
display: flex;
align-items: center;
justify-content: space-between;
margin-bottom: 12px;
}
.dependencies-header h3 {
margin: 0;
color: var(--text-primary);
font-size: 1rem;
font-weight: 600;
}
.dependencies-controls {
display: flex;
align-items: center;
gap: 8px;
}
.dependencies-controls .btn {
display: inline-flex;
align-items: center;
}
.dependencies-tag-select {
margin-bottom: 16px;
}
.tag-selector {
padding: 8px 12px;
background: var(--bg-tertiary);
border: 1px solid var(--border-primary);
border-radius: var(--radius-md);
color: var(--text-primary);
font-size: 0.875rem;
cursor: pointer;
min-width: 200px;
}
.tag-selector:focus {
outline: none;
border-color: var(--accent-primary);
}
.deps-loading {
color: var(--text-muted);
font-size: 0.875rem;
padding: 16px 0;
}
.deps-error {
color: var(--error-color, #ef4444);
font-size: 0.875rem;
padding: 12px 16px;
background: rgba(239, 68, 68, 0.1);
border-radius: var(--radius-md);
}
.deps-empty {
color: var(--text-muted);
font-size: 0.875rem;
padding: 16px 0;
}
.deps-summary {
color: var(--text-secondary);
font-size: 0.875rem;
margin-bottom: 12px;
}
.deps-summary strong {
color: var(--accent-primary);
}
.deps-items {
list-style: none;
margin: 0;
padding: 0;
display: flex;
flex-direction: column;
gap: 8px;
}
.dep-item {
display: flex;
align-items: center;
gap: 12px;
padding: 12px 16px;
background: var(--bg-tertiary);
border-radius: var(--radius-md);
border: 1px solid var(--border-primary);
}
.dep-link {
color: var(--accent-primary);
font-weight: 500;
text-decoration: none;
font-family: 'JetBrains Mono', 'Fira Code', 'Consolas', monospace;
font-size: 0.875rem;
}
.dep-link:hover {
text-decoration: underline;
}
.dep-constraint {
color: var(--text-muted);
font-size: 0.8125rem;
font-family: 'JetBrains Mono', 'Fira Code', 'Consolas', monospace;
}
.dep-status {
margin-left: auto;
font-size: 0.875rem;
font-weight: 600;
}
.dep-status--ok {
color: var(--success-color, #10b981);
}
.dep-status--missing {
color: var(--warning-color, #f59e0b);
}
/* Tag name link in table */
.tag-name-link {
color: var(--accent-primary);
transition: opacity var(--transition-fast);
}
.tag-name-link:hover {
opacity: 0.8;
}
.tag-name-link.selected {
text-decoration: underline;
}
/* Used By (Reverse Dependencies) Section */
.used-by-section {
margin-top: 32px;
background: var(--bg-secondary);
}
.used-by-section h3 {
margin-bottom: 16px;
color: var(--text-primary);
font-size: 1rem;
font-weight: 600;
}
.reverse-dep-item {
display: flex;
align-items: center;
gap: 12px;
flex-wrap: wrap;
}
.dep-version {
color: var(--accent-primary);
font-family: 'JetBrains Mono', 'Fira Code', 'Consolas', monospace;
font-size: 0.8125rem;
background: rgba(16, 185, 129, 0.1);
padding: 2px 8px;
border-radius: var(--radius-sm);
}
.dep-requires {
color: var(--text-muted);
font-size: 0.8125rem;
font-family: 'JetBrains Mono', 'Fira Code', 'Consolas', monospace;
margin-left: auto;
}
.reverse-deps-pagination {
display: flex;
align-items: center;
justify-content: center;
gap: 16px;
margin-top: 16px;
padding-top: 16px;
border-top: 1px solid var(--border-primary);
}
.pagination-info {
color: var(--text-secondary);
font-size: 0.875rem;
}
/* Ensure File Modal */
.modal-overlay {
position: fixed;
top: 0;
left: 0;
right: 0;
bottom: 0;
background: rgba(0, 0, 0, 0.7);
display: flex;
align-items: center;
justify-content: center;
z-index: 1000;
padding: 20px;
}
.ensure-file-modal {
background: var(--bg-secondary);
border: 1px solid var(--border-primary);
border-radius: var(--radius-lg);
max-width: 700px;
width: 100%;
max-height: 80vh;
display: flex;
flex-direction: column;
box-shadow: 0 20px 50px rgba(0, 0, 0, 0.5);
}
.ensure-file-header {
display: flex;
align-items: center;
justify-content: space-between;
padding: 16px 20px;
border-bottom: 1px solid var(--border-primary);
}
.ensure-file-header h3 {
margin: 0;
color: var(--text-primary);
font-size: 1rem;
font-weight: 600;
}
.ensure-file-actions {
display: flex;
align-items: center;
gap: 8px;
}
.ensure-file-actions .copy-btn {
opacity: 1;
width: 32px;
height: 32px;
}
.modal-close {
display: flex;
align-items: center;
justify-content: center;
width: 32px;
height: 32px;
padding: 0;
background: transparent;
border: none;
border-radius: var(--radius-sm);
color: var(--text-muted);
cursor: pointer;
transition: all var(--transition-fast);
}
.modal-close:hover {
background: var(--bg-hover);
color: var(--text-primary);
}
.ensure-file-content {
flex: 1;
overflow: auto;
padding: 20px;
}
.ensure-file-loading {
color: var(--text-muted);
text-align: center;
padding: 40px 20px;
}
.ensure-file-error {
color: var(--error-color, #ef4444);
padding: 16px;
background: rgba(239, 68, 68, 0.1);
border-radius: var(--radius-md);
}
.ensure-file-empty {
color: var(--text-muted);
text-align: center;
padding: 40px 20px;
font-style: italic;
}
.ensure-file-yaml {
margin: 0;
padding: 16px;
background: #0d0d0f;
border: 1px solid var(--border-primary);
border-radius: var(--radius-md);
overflow-x: auto;
}
.ensure-file-yaml code {
font-family: 'JetBrains Mono', 'Fira Code', 'Consolas', monospace;
font-size: 0.8125rem;
color: #e2e8f0;
white-space: pre;
}
.ensure-file-footer {
padding: 16px 20px;
border-top: 1px solid var(--border-primary);
background: var(--bg-tertiary);
border-radius: 0 0 var(--radius-lg) var(--radius-lg);
}
.ensure-file-hint {
margin: 0;
color: var(--text-muted);
font-size: 0.8125rem;
}
.ensure-file-hint code {
background: rgba(0, 0, 0, 0.2);
padding: 2px 6px;
border-radius: var(--radius-sm);
font-family: 'JetBrains Mono', 'Fira Code', 'Consolas', monospace;
color: var(--accent-primary);
}
/* Responsive adjustments */
@media (max-width: 768px) {
.upload-form {
@@ -439,4 +779,18 @@ tr:hover .copy-btn {
flex-wrap: wrap;
gap: 12px;
}
.dependencies-header {
flex-direction: column;
align-items: flex-start;
gap: 12px;
}
.tag-selector {
width: 100%;
}
.ensure-file-modal {
max-height: 90vh;
}
}

View File

@@ -1,7 +1,7 @@
import { useState, useEffect, useCallback } from 'react';
import { useParams, useSearchParams, useNavigate, useLocation } from 'react-router-dom';
import { TagDetail, Package, PaginatedResponse, AccessLevel } from '../types';
import { listTags, getDownloadUrl, getPackage, getMyProjectAccess, createTag, UnauthorizedError, ForbiddenError } from '../api';
import { useParams, useSearchParams, useNavigate, useLocation, Link } from 'react-router-dom';
import { TagDetail, Package, PaginatedResponse, AccessLevel, Dependency, DependentInfo } from '../types';
import { listTags, getDownloadUrl, getPackage, getMyProjectAccess, createTag, getArtifactDependencies, getReverseDependencies, getEnsureFile, UnauthorizedError, ForbiddenError } from '../api';
import { Breadcrumb } from '../components/Breadcrumb';
import { Badge } from '../components/Badge';
import { SearchInput } from '../components/SearchInput';
@@ -10,6 +10,7 @@ import { DataTable } from '../components/DataTable';
import { Pagination } from '../components/Pagination';
import { DragDropUpload, UploadResult } from '../components/DragDropUpload';
import { useAuth } from '../contexts/AuthContext';
import DependencyGraph from '../components/DependencyGraph';
import './Home.css';
import './PackagePage.css';
@@ -68,6 +69,30 @@ function PackagePage() {
const [createTagArtifactId, setCreateTagArtifactId] = useState('');
const [createTagLoading, setCreateTagLoading] = useState(false);
// Dependencies state
const [selectedTag, setSelectedTag] = useState<TagDetail | null>(null);
const [dependencies, setDependencies] = useState<Dependency[]>([]);
const [depsLoading, setDepsLoading] = useState(false);
const [depsError, setDepsError] = useState<string | null>(null);
// Reverse dependencies state
const [reverseDeps, setReverseDeps] = useState<DependentInfo[]>([]);
const [reverseDepsLoading, setReverseDepsLoading] = useState(false);
const [reverseDepsError, setReverseDepsError] = useState<string | null>(null);
const [reverseDepsPage, setReverseDepsPage] = useState(1);
const [reverseDepsTotal, setReverseDepsTotal] = useState(0);
const [reverseDepsHasMore, setReverseDepsHasMore] = useState(false);
// Dependency graph modal state
const [showGraph, setShowGraph] = useState(false);
// Ensure file modal state
const [showEnsureFile, setShowEnsureFile] = useState(false);
const [ensureFileContent, setEnsureFileContent] = useState<string | null>(null);
const [ensureFileLoading, setEnsureFileLoading] = useState(false);
const [ensureFileError, setEnsureFileError] = useState<string | null>(null);
const [ensureFileTagName, setEnsureFileTagName] = useState<string | null>(null);
// Derived permissions
const canWrite = accessLevel === 'write' || accessLevel === 'admin';
@@ -128,6 +153,98 @@ function PackagePage() {
loadData();
}, [loadData]);
// Auto-select tag when tags are loaded (prefer version from URL, then first tag)
// Re-run when package changes to pick up new tags
useEffect(() => {
if (tagsData?.items && tagsData.items.length > 0) {
const versionParam = searchParams.get('version');
if (versionParam) {
// Find tag matching the version parameter
const matchingTag = tagsData.items.find(t => t.version === versionParam);
if (matchingTag) {
setSelectedTag(matchingTag);
setDependencies([]);
return;
}
}
// Fall back to first tag
setSelectedTag(tagsData.items[0]);
setDependencies([]);
}
}, [tagsData, searchParams, projectName, packageName]);
// Fetch dependencies when selected tag changes
const fetchDependencies = useCallback(async (artifactId: string) => {
setDepsLoading(true);
setDepsError(null);
try {
const result = await getArtifactDependencies(artifactId);
setDependencies(result.dependencies);
} catch (err) {
setDepsError(err instanceof Error ? err.message : 'Failed to load dependencies');
setDependencies([]);
} finally {
setDepsLoading(false);
}
}, []);
useEffect(() => {
if (selectedTag) {
fetchDependencies(selectedTag.artifact_id);
}
}, [selectedTag, fetchDependencies]);
// Fetch reverse dependencies
const fetchReverseDeps = useCallback(async (pageNum: number = 1) => {
if (!projectName || !packageName) return;
setReverseDepsLoading(true);
setReverseDepsError(null);
try {
const result = await getReverseDependencies(projectName, packageName, { page: pageNum, limit: 10 });
setReverseDeps(result.dependents);
setReverseDepsTotal(result.pagination.total);
setReverseDepsHasMore(result.pagination.has_more);
setReverseDepsPage(pageNum);
} catch (err) {
setReverseDepsError(err instanceof Error ? err.message : 'Failed to load reverse dependencies');
setReverseDeps([]);
} finally {
setReverseDepsLoading(false);
}
}, [projectName, packageName]);
useEffect(() => {
if (projectName && packageName && !loading) {
fetchReverseDeps(1);
}
}, [projectName, packageName, loading, fetchReverseDeps]);
// Fetch ensure file for a specific tag
const fetchEnsureFileForTag = useCallback(async (tagName: string) => {
if (!projectName || !packageName) return;
setEnsureFileTagName(tagName);
setEnsureFileLoading(true);
setEnsureFileError(null);
try {
const content = await getEnsureFile(projectName, packageName, tagName);
setEnsureFileContent(content);
setShowEnsureFile(true);
} catch (err) {
setEnsureFileError(err instanceof Error ? err.message : 'Failed to load ensure file');
setShowEnsureFile(true);
} finally {
setEnsureFileLoading(false);
}
}, [projectName, packageName]);
// Fetch ensure file for selected tag
const fetchEnsureFile = useCallback(async () => {
if (!selectedTag) return;
fetchEnsureFileForTag(selectedTag.name);
}, [selectedTag, fetchEnsureFileForTag]);
// Keyboard navigation - go back with backspace
useEffect(() => {
const handleKeyDown = (e: KeyboardEvent) => {
@@ -202,12 +319,24 @@ function PackagePage() {
const tags = tagsData?.items || [];
const pagination = tagsData?.pagination;
const handleTagSelect = (tag: TagDetail) => {
setSelectedTag(tag);
};
const columns = [
{
key: 'name',
header: 'Tag',
sortable: true,
render: (t: TagDetail) => <strong>{t.name}</strong>,
render: (t: TagDetail) => (
<strong
className={`tag-name-link ${selectedTag?.id === t.id ? 'selected' : ''}`}
onClick={() => handleTagSelect(t)}
style={{ cursor: 'pointer' }}
>
{t.name}
</strong>
),
},
{
key: 'version',
@@ -261,13 +390,22 @@ function PackagePage() {
key: 'actions',
header: 'Actions',
render: (t: TagDetail) => (
<a
href={getDownloadUrl(projectName!, packageName!, t.name)}
className="btn btn-secondary btn-small"
download
>
Download
</a>
<div className="action-buttons">
<button
className="btn btn-secondary btn-small"
onClick={() => fetchEnsureFileForTag(t.name)}
title="View orchard.ensure file"
>
Ensure
</button>
<a
href={getDownloadUrl(projectName!, packageName!, t.name)}
className="btn btn-secondary btn-small"
download
>
Download
</a>
</div>
),
},
];
@@ -439,6 +577,166 @@ function PackagePage() {
/>
)}
{/* Dependencies Section */}
{tags.length > 0 && (
<div className="dependencies-section card">
<div className="dependencies-header">
<h3>Dependencies</h3>
<div className="dependencies-controls">
{selectedTag && (
<>
<button
className="btn btn-secondary btn-small"
onClick={fetchEnsureFile}
disabled={ensureFileLoading}
title="View orchard.ensure file"
>
<svg width="14" height="14" viewBox="0 0 24 24" fill="none" stroke="currentColor" strokeWidth="2" style={{ marginRight: '6px' }}>
<path d="M14 2H6a2 2 0 0 0-2 2v16a2 2 0 0 0 2 2h12a2 2 0 0 0 2-2V8z"></path>
<polyline points="14 2 14 8 20 8"></polyline>
<line x1="16" y1="13" x2="8" y2="13"></line>
<line x1="16" y1="17" x2="8" y2="17"></line>
<polyline points="10 9 9 9 8 9"></polyline>
</svg>
{ensureFileLoading ? 'Loading...' : 'View Ensure File'}
</button>
<button
className="btn btn-secondary btn-small"
onClick={() => setShowGraph(true)}
title="View full dependency tree"
>
<svg width="14" height="14" viewBox="0 0 24 24" fill="none" stroke="currentColor" strokeWidth="2" style={{ marginRight: '6px' }}>
<circle cx="12" cy="12" r="3"></circle>
<circle cx="4" cy="4" r="2"></circle>
<circle cx="20" cy="4" r="2"></circle>
<circle cx="4" cy="20" r="2"></circle>
<circle cx="20" cy="20" r="2"></circle>
<line x1="9.5" y1="9.5" x2="5.5" y2="5.5"></line>
<line x1="14.5" y1="9.5" x2="18.5" y2="5.5"></line>
<line x1="9.5" y1="14.5" x2="5.5" y2="18.5"></line>
<line x1="14.5" y1="14.5" x2="18.5" y2="18.5"></line>
</svg>
View Graph
</button>
</>
)}
</div>
</div>
<div className="dependencies-tag-select">
{selectedTag && (
<select
className="tag-selector"
value={selectedTag.id}
onChange={(e) => {
const tag = tags.find(t => t.id === e.target.value);
if (tag) setSelectedTag(tag);
}}
>
{tags.map(t => (
<option key={t.id} value={t.id}>
{t.name}{t.version ? ` (${t.version})` : ''}
</option>
))}
</select>
)}
</div>
{depsLoading ? (
<div className="deps-loading">Loading dependencies...</div>
) : depsError ? (
<div className="deps-error">{depsError}</div>
) : dependencies.length === 0 ? (
<div className="deps-empty">
{selectedTag ? (
<span><strong>{selectedTag.name}</strong> has no dependencies</span>
) : (
<span>No dependencies</span>
)}
</div>
) : (
<div className="deps-list">
<div className="deps-summary">
<strong>{selectedTag?.name}</strong> has {dependencies.length} {dependencies.length === 1 ? 'dependency' : 'dependencies'}:
</div>
<ul className="deps-items">
{dependencies.map((dep) => (
<li key={dep.id} className="dep-item">
<Link
to={`/project/${dep.project}/${dep.package}`}
className="dep-link"
>
{dep.project}/{dep.package}
</Link>
<span className="dep-constraint">
@ {dep.version || dep.tag}
</span>
<span className="dep-status dep-status--ok" title="Package exists">
&#10003;
</span>
</li>
))}
</ul>
</div>
)}
</div>
)}
{/* Used By (Reverse Dependencies) Section */}
<div className="used-by-section card">
<h3>Used By</h3>
{reverseDepsLoading ? (
<div className="deps-loading">Loading reverse dependencies...</div>
) : reverseDepsError ? (
<div className="deps-error">{reverseDepsError}</div>
) : reverseDeps.length === 0 ? (
<div className="deps-empty">No packages depend on this package</div>
) : (
<div className="reverse-deps-list">
<div className="deps-summary">
{reverseDepsTotal} {reverseDepsTotal === 1 ? 'package depends' : 'packages depend'} on this:
</div>
<ul className="deps-items">
{reverseDeps.map((dep) => (
<li key={dep.artifact_id} className="dep-item reverse-dep-item">
<Link
to={`/project/${dep.project}/${dep.package}${dep.version ? `?version=${dep.version}` : ''}`}
className="dep-link"
>
{dep.project}/{dep.package}
{dep.version && (
<span className="dep-version">v{dep.version}</span>
)}
</Link>
<span className="dep-requires">
requires @ {dep.constraint_value}
</span>
</li>
))}
</ul>
{(reverseDepsHasMore || reverseDepsPage > 1) && (
<div className="reverse-deps-pagination">
<button
className="btn btn-secondary btn-small"
onClick={() => fetchReverseDeps(reverseDepsPage - 1)}
disabled={reverseDepsPage <= 1 || reverseDepsLoading}
>
Previous
</button>
<span className="pagination-info">Page {reverseDepsPage}</span>
<button
className="btn btn-secondary btn-small"
onClick={() => fetchReverseDeps(reverseDepsPage + 1)}
disabled={!reverseDepsHasMore || reverseDepsLoading}
>
Next
</button>
</div>
)}
</div>
)}
</div>
<div className="download-by-id-section card">
<h3>Download by Artifact ID</h3>
<div className="download-by-id-form">
@@ -522,6 +820,58 @@ function PackagePage() {
<code>curl -O {window.location.origin}/api/v1/project/{projectName}/{packageName}/+/v1.0.0</code>
</pre>
</div>
{/* Dependency Graph Modal */}
{showGraph && selectedTag && (
<DependencyGraph
projectName={projectName!}
packageName={packageName!}
tagName={selectedTag.name}
onClose={() => setShowGraph(false)}
/>
)}
{/* Ensure File Modal */}
{showEnsureFile && (
<div className="modal-overlay" onClick={() => setShowEnsureFile(false)}>
<div className="ensure-file-modal" onClick={(e) => e.stopPropagation()}>
<div className="ensure-file-header">
<h3>orchard.ensure for {ensureFileTagName}</h3>
<div className="ensure-file-actions">
{ensureFileContent && (
<CopyButton text={ensureFileContent} />
)}
<button
className="modal-close"
onClick={() => setShowEnsureFile(false)}
title="Close"
>
<svg width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" strokeWidth="2">
<line x1="18" y1="6" x2="6" y2="18"></line>
<line x1="6" y1="6" x2="18" y2="18"></line>
</svg>
</button>
</div>
</div>
<div className="ensure-file-content">
{ensureFileLoading ? (
<div className="ensure-file-loading">Loading...</div>
) : ensureFileError ? (
<div className="ensure-file-error">{ensureFileError}</div>
) : ensureFileContent ? (
<pre className="ensure-file-yaml"><code>{ensureFileContent}</code></pre>
) : (
<div className="ensure-file-empty">No dependencies defined for this artifact.</div>
)}
</div>
<div className="ensure-file-footer">
<p className="ensure-file-hint">
Save this as <code>orchard.ensure</code> in your project root to declare dependencies.
</p>
</div>
</div>
</div>
)}
</div>
);
}

View File

@@ -8,7 +8,6 @@ import { DataTable } from '../components/DataTable';
import { SearchInput } from '../components/SearchInput';
import { FilterChip, FilterChipGroup } from '../components/FilterChip';
import { Pagination } from '../components/Pagination';
import { AccessManagement } from '../components/AccessManagement';
import { useAuth } from '../contexts/AuthContext';
import './Home.css';
@@ -211,15 +210,30 @@ function ProjectPage() {
<span className="meta-item">by {project.created_by}</span>
</div>
</div>
{canWrite ? (
<button className="btn btn-primary" onClick={() => setShowForm(!showForm)}>
{showForm ? 'Cancel' : '+ New Package'}
</button>
) : user ? (
<span className="text-muted" title="You have read-only access to this project">
Read-only access
</span>
) : null}
<div className="page-header__actions">
{canAdmin && (
<button
className="btn btn-secondary"
onClick={() => navigate(`/project/${projectName}/settings`)}
title="Project Settings"
>
<svg width="16" height="16" viewBox="0 0 24 24" fill="none" stroke="currentColor" strokeWidth="2" strokeLinecap="round" strokeLinejoin="round">
<circle cx="12" cy="12" r="3" />
<path d="M19.4 15a1.65 1.65 0 0 0 .33 1.82l.06.06a2 2 0 0 1 0 2.83 2 2 0 0 1-2.83 0l-.06-.06a1.65 1.65 0 0 0-1.82-.33 1.65 1.65 0 0 0-1 1.51V21a2 2 0 0 1-2 2 2 2 0 0 1-2-2v-.09A1.65 1.65 0 0 0 9 19.4a1.65 1.65 0 0 0-1.82.33l-.06.06a2 2 0 0 1-2.83 0 2 2 0 0 1 0-2.83l.06-.06a1.65 1.65 0 0 0 .33-1.82 1.65 1.65 0 0 0-1.51-1H3a2 2 0 0 1-2-2 2 2 0 0 1 2-2h.09A1.65 1.65 0 0 0 4.6 9a1.65 1.65 0 0 0-.33-1.82l-.06-.06a2 2 0 0 1 0-2.83 2 2 0 0 1 2.83 0l.06.06a1.65 1.65 0 0 0 1.82.33H9a1.65 1.65 0 0 0 1-1.51V3a2 2 0 0 1 2-2 2 2 0 0 1 2 2v.09a1.65 1.65 0 0 0 1 1.51 1.65 1.65 0 0 0 1.82-.33l.06-.06a2 2 0 0 1 2.83 0 2 2 0 0 1 0 2.83l-.06.06a1.65 1.65 0 0 0-.33 1.82V9a1.65 1.65 0 0 0 1.51 1H21a2 2 0 0 1 2 2 2 2 0 0 1-2 2h-.09a1.65 1.65 0 0 0-1.51 1z" />
</svg>
Settings
</button>
)}
{canWrite ? (
<button className="btn btn-primary" onClick={() => setShowForm(!showForm)}>
{showForm ? 'Cancel' : '+ New Package'}
</button>
) : user ? (
<span className="text-muted" title="You have read-only access to this project">
Read-only access
</span>
) : null}
</div>
</div>
{error && <div className="error-message">{error}</div>}
@@ -371,10 +385,6 @@ function ProjectPage() {
onPageChange={handlePageChange}
/>
)}
{canAdmin && projectName && (
<AccessManagement projectName={projectName} />
)}
</div>
);
}

View File

@@ -0,0 +1,476 @@
.project-settings-page {
max-width: 900px;
margin: 0 auto;
}
.project-settings-header {
margin-bottom: 32px;
}
.project-settings-header h1 {
font-size: 1.75rem;
font-weight: 600;
color: var(--text-primary);
margin-bottom: 8px;
letter-spacing: -0.02em;
}
.project-settings-subtitle {
color: var(--text-tertiary);
font-size: 0.9375rem;
}
.project-settings-loading {
display: flex;
align-items: center;
justify-content: center;
gap: 12px;
padding: 64px 24px;
color: var(--text-tertiary);
font-size: 0.9375rem;
}
.project-settings-spinner {
width: 20px;
height: 20px;
border: 2px solid var(--border-secondary);
border-top-color: var(--accent-primary);
border-radius: 50%;
animation: project-settings-spin 0.6s linear infinite;
}
@keyframes project-settings-spin {
to {
transform: rotate(360deg);
}
}
.project-settings-error {
display: flex;
align-items: center;
gap: 10px;
background: var(--error-bg);
border: 1px solid rgba(239, 68, 68, 0.2);
color: var(--error);
padding: 12px 16px;
border-radius: var(--radius-md);
margin-bottom: 24px;
font-size: 0.875rem;
}
.project-settings-success {
display: flex;
align-items: center;
gap: 10px;
background: var(--success-bg);
border: 1px solid rgba(34, 197, 94, 0.2);
color: var(--success);
padding: 12px 16px;
border-radius: var(--radius-md);
margin-bottom: 24px;
font-size: 0.875rem;
animation: project-settings-fade-in 0.2s ease;
}
@keyframes project-settings-fade-in {
from {
opacity: 0;
transform: translateY(-8px);
}
to {
opacity: 1;
transform: translateY(0);
}
}
.project-settings-section {
background: var(--bg-secondary);
border: 1px solid var(--border-primary);
border-radius: var(--radius-lg);
padding: 24px;
margin-bottom: 24px;
}
.project-settings-section h2 {
font-size: 1.125rem;
font-weight: 600;
color: var(--text-primary);
margin-bottom: 20px;
padding-bottom: 16px;
border-bottom: 1px solid var(--border-primary);
}
.project-settings-form {
display: flex;
flex-direction: column;
gap: 16px;
}
.project-settings-form-group {
display: flex;
flex-direction: column;
gap: 6px;
}
.project-settings-form-group label {
font-size: 0.8125rem;
font-weight: 500;
color: var(--text-secondary);
}
.project-settings-form-group textarea,
.project-settings-form-group input[type="text"] {
padding: 12px 14px;
background: var(--bg-tertiary);
border: 1px solid var(--border-primary);
border-radius: var(--radius-md);
font-size: 0.875rem;
color: var(--text-primary);
transition: all var(--transition-fast);
font-family: inherit;
resize: vertical;
}
.project-settings-form-group textarea {
min-height: 100px;
}
.project-settings-form-group textarea::placeholder,
.project-settings-form-group input::placeholder {
color: var(--text-muted);
}
.project-settings-form-group textarea:hover:not(:disabled),
.project-settings-form-group input:hover:not(:disabled) {
border-color: var(--border-secondary);
background: var(--bg-elevated);
}
.project-settings-form-group textarea:focus,
.project-settings-form-group input:focus {
outline: none;
border-color: var(--accent-primary);
box-shadow: 0 0 0 3px rgba(16, 185, 129, 0.15);
background: var(--bg-elevated);
}
.project-settings-form-group textarea:disabled,
.project-settings-form-group input:disabled {
opacity: 0.6;
cursor: not-allowed;
}
.project-settings-checkbox-group {
flex-direction: row;
align-items: center;
}
.project-settings-checkbox-label {
display: flex;
align-items: center;
gap: 10px;
cursor: pointer;
font-size: 0.875rem;
font-weight: 400;
color: var(--text-secondary);
user-select: none;
}
.project-settings-checkbox-label input[type="checkbox"] {
position: absolute;
opacity: 0;
width: 0;
height: 0;
}
.project-settings-checkbox-custom {
width: 18px;
height: 18px;
background: var(--bg-tertiary);
border: 1px solid var(--border-secondary);
border-radius: var(--radius-sm);
transition: all var(--transition-fast);
position: relative;
flex-shrink: 0;
}
.project-settings-checkbox-label input[type="checkbox"]:checked + .project-settings-checkbox-custom {
background: var(--accent-primary);
border-color: var(--accent-primary);
}
.project-settings-checkbox-label input[type="checkbox"]:checked + .project-settings-checkbox-custom::after {
content: '';
position: absolute;
left: 5px;
top: 2px;
width: 5px;
height: 9px;
border: solid white;
border-width: 0 2px 2px 0;
transform: rotate(45deg);
}
.project-settings-checkbox-label input[type="checkbox"]:focus + .project-settings-checkbox-custom {
box-shadow: 0 0 0 3px rgba(16, 185, 129, 0.15);
}
.project-settings-checkbox-label:hover .project-settings-checkbox-custom {
border-color: var(--accent-primary);
}
.project-settings-form-actions {
display: flex;
justify-content: flex-end;
gap: 12px;
margin-top: 8px;
}
.project-settings-save-button {
display: flex;
align-items: center;
justify-content: center;
gap: 8px;
padding: 10px 18px;
background: var(--accent-gradient);
border: none;
border-radius: var(--radius-md);
font-size: 0.875rem;
font-weight: 500;
color: white;
cursor: pointer;
transition: all var(--transition-fast);
min-width: 120px;
}
.project-settings-save-button:hover:not(:disabled) {
transform: translateY(-1px);
box-shadow: var(--shadow-sm), 0 0 20px rgba(16, 185, 129, 0.2);
}
.project-settings-save-button:disabled {
opacity: 0.5;
cursor: not-allowed;
transform: none;
}
.project-settings-button-spinner {
width: 14px;
height: 14px;
border: 2px solid rgba(255, 255, 255, 0.3);
border-top-color: white;
border-radius: 50%;
animation: project-settings-spin 0.6s linear infinite;
}
/* Danger Zone */
.project-settings-danger-zone {
background: var(--bg-secondary);
border: 1px solid rgba(239, 68, 68, 0.3);
border-radius: var(--radius-lg);
padding: 24px;
margin-bottom: 24px;
}
.project-settings-danger-zone h2 {
font-size: 1.125rem;
font-weight: 600;
color: var(--error);
margin-bottom: 20px;
padding-bottom: 16px;
border-bottom: 1px solid rgba(239, 68, 68, 0.2);
}
.project-settings-danger-item {
display: flex;
justify-content: space-between;
align-items: flex-start;
gap: 24px;
}
.project-settings-danger-info h3 {
font-size: 0.9375rem;
font-weight: 600;
color: var(--text-primary);
margin-bottom: 4px;
}
.project-settings-danger-info p {
color: var(--text-tertiary);
font-size: 0.8125rem;
max-width: 400px;
}
.project-settings-delete-button {
padding: 10px 18px;
background: transparent;
border: 1px solid rgba(239, 68, 68, 0.3);
border-radius: var(--radius-md);
font-size: 0.875rem;
font-weight: 500;
color: var(--error);
cursor: pointer;
transition: all var(--transition-fast);
flex-shrink: 0;
}
.project-settings-delete-button:hover:not(:disabled) {
background: var(--error-bg);
border-color: rgba(239, 68, 68, 0.5);
}
.project-settings-delete-button:disabled {
opacity: 0.5;
cursor: not-allowed;
}
/* Delete Confirmation */
.project-settings-delete-confirm {
margin-top: 20px;
padding-top: 20px;
border-top: 1px solid rgba(239, 68, 68, 0.2);
animation: project-settings-fade-in 0.2s ease;
}
.project-settings-delete-confirm p {
color: var(--text-secondary);
font-size: 0.875rem;
margin-bottom: 12px;
}
.project-settings-delete-confirm strong {
color: var(--text-primary);
font-family: 'JetBrains Mono', 'Fira Code', 'SF Mono', Monaco, monospace;
background: var(--bg-tertiary);
padding: 2px 6px;
border-radius: var(--radius-sm);
}
.project-settings-delete-confirm-input {
width: 100%;
padding: 12px 14px;
background: var(--bg-tertiary);
border: 1px solid var(--border-primary);
border-radius: var(--radius-md);
font-size: 0.875rem;
color: var(--text-primary);
transition: all var(--transition-fast);
margin-bottom: 16px;
}
.project-settings-delete-confirm-input:focus {
outline: none;
border-color: var(--error);
box-shadow: 0 0 0 3px rgba(239, 68, 68, 0.15);
}
.project-settings-delete-confirm-input::placeholder {
color: var(--text-muted);
}
.project-settings-delete-confirm-actions {
display: flex;
gap: 12px;
}
.project-settings-confirm-delete-button {
display: flex;
align-items: center;
justify-content: center;
gap: 8px;
padding: 10px 18px;
background: var(--error);
border: none;
border-radius: var(--radius-md);
font-size: 0.875rem;
font-weight: 500;
color: white;
cursor: pointer;
transition: all var(--transition-fast);
min-width: 120px;
}
.project-settings-confirm-delete-button:hover:not(:disabled) {
opacity: 0.9;
}
.project-settings-confirm-delete-button:disabled {
opacity: 0.5;
cursor: not-allowed;
}
.project-settings-cancel-button {
padding: 10px 18px;
background: transparent;
border: 1px solid var(--border-primary);
border-radius: var(--radius-md);
font-size: 0.875rem;
font-weight: 500;
color: var(--text-secondary);
cursor: pointer;
transition: all var(--transition-fast);
}
.project-settings-cancel-button:hover:not(:disabled) {
background: var(--bg-hover);
border-color: var(--border-secondary);
color: var(--text-primary);
}
.project-settings-cancel-button:disabled {
opacity: 0.5;
cursor: not-allowed;
}
.project-settings-delete-spinner {
width: 14px;
height: 14px;
border: 2px solid rgba(255, 255, 255, 0.3);
border-top-color: white;
border-radius: 50%;
animation: project-settings-spin 0.6s linear infinite;
}
/* Access denied */
.project-settings-access-denied {
display: flex;
flex-direction: column;
align-items: center;
justify-content: center;
padding: 80px 24px;
text-align: center;
}
.project-settings-access-denied h2 {
font-size: 1.5rem;
font-weight: 600;
color: var(--text-primary);
margin-bottom: 12px;
}
.project-settings-access-denied p {
color: var(--text-tertiary);
font-size: 0.9375rem;
max-width: 400px;
}
/* Responsive */
@media (max-width: 768px) {
.project-settings-danger-item {
flex-direction: column;
gap: 16px;
}
.project-settings-delete-button {
align-self: flex-start;
}
.project-settings-delete-confirm-actions {
flex-direction: column;
}
.project-settings-confirm-delete-button,
.project-settings-cancel-button {
width: 100%;
}
}

View File

@@ -0,0 +1,308 @@
import { useState, useEffect, useCallback } from 'react';
import { useParams, useNavigate } from 'react-router-dom';
import { Project } from '../types';
import {
getProject,
updateProject,
deleteProject,
getMyProjectAccess,
UnauthorizedError,
ForbiddenError,
} from '../api';
import { Breadcrumb } from '../components/Breadcrumb';
import { AccessManagement } from '../components/AccessManagement';
import { useAuth } from '../contexts/AuthContext';
import './ProjectSettingsPage.css';
function ProjectSettingsPage() {
const { projectName } = useParams<{ projectName: string }>();
const navigate = useNavigate();
const { user } = useAuth();
const [project, setProject] = useState<Project | null>(null);
const [loading, setLoading] = useState(true);
const [error, setError] = useState<string | null>(null);
const [success, setSuccess] = useState<string | null>(null);
const [accessDenied, setAccessDenied] = useState(false);
const [canAdmin, setCanAdmin] = useState(false);
// General settings form state
const [description, setDescription] = useState('');
const [isPublic, setIsPublic] = useState(false);
const [saving, setSaving] = useState(false);
// Delete confirmation state
const [showDeleteConfirm, setShowDeleteConfirm] = useState(false);
const [deleteConfirmText, setDeleteConfirmText] = useState('');
const [deleting, setDeleting] = useState(false);
const loadData = useCallback(async () => {
if (!projectName) return;
try {
setLoading(true);
setAccessDenied(false);
const [projectData, accessResult] = await Promise.all([
getProject(projectName),
getMyProjectAccess(projectName),
]);
setProject(projectData);
setDescription(projectData.description || '');
setIsPublic(projectData.is_public);
const hasAdminAccess = accessResult.access_level === 'admin';
setCanAdmin(hasAdminAccess);
if (!hasAdminAccess) {
setAccessDenied(true);
}
setError(null);
} catch (err) {
if (err instanceof UnauthorizedError) {
navigate('/login', { state: { from: `/project/${projectName}/settings` } });
return;
}
if (err instanceof ForbiddenError) {
setAccessDenied(true);
setLoading(false);
return;
}
setError(err instanceof Error ? err.message : 'Failed to load project');
} finally {
setLoading(false);
}
}, [projectName, navigate]);
useEffect(() => {
loadData();
}, [loadData]);
const handleSaveSettings = async (e: React.FormEvent) => {
e.preventDefault();
if (!projectName) return;
try {
setSaving(true);
setError(null);
const updatedProject = await updateProject(projectName, {
description: description || undefined,
is_public: isPublic,
});
setProject(updatedProject);
setSuccess('Settings saved successfully');
setTimeout(() => setSuccess(null), 3000);
} catch (err) {
setError(err instanceof Error ? err.message : 'Failed to save settings');
} finally {
setSaving(false);
}
};
const handleDeleteProject = async () => {
if (!projectName || deleteConfirmText !== projectName) return;
try {
setDeleting(true);
setError(null);
await deleteProject(projectName);
navigate('/');
} catch (err) {
setError(err instanceof Error ? err.message : 'Failed to delete project');
setDeleting(false);
}
};
const handleCancelDelete = () => {
setShowDeleteConfirm(false);
setDeleteConfirmText('');
};
if (loading) {
return (
<div className="project-settings-page">
<Breadcrumb
items={[
{ label: 'Projects', href: '/' },
{ label: projectName || '', href: `/project/${projectName}` },
{ label: 'Settings' },
]}
/>
<div className="project-settings-loading">
<div className="project-settings-spinner" />
<span>Loading...</span>
</div>
</div>
);
}
if (accessDenied || !canAdmin) {
return (
<div className="project-settings-page">
<Breadcrumb
items={[
{ label: 'Projects', href: '/' },
{ label: projectName || '', href: `/project/${projectName}` },
{ label: 'Settings' },
]}
/>
<div className="project-settings-access-denied">
<h2>Access Denied</h2>
<p>You must be a project admin to access settings.</p>
{!user && (
<p style={{ marginTop: '16px' }}>
<a href="/login" className="btn btn-primary">
Sign in
</a>
</p>
)}
</div>
</div>
);
}
if (!project) {
return (
<div className="project-settings-page">
<Breadcrumb
items={[
{ label: 'Projects', href: '/' },
{ label: projectName || '' },
]}
/>
<div className="project-settings-error">Project not found</div>
</div>
);
}
return (
<div className="project-settings-page">
<Breadcrumb
items={[
{ label: 'Projects', href: '/' },
{ label: project.name, href: `/project/${project.name}` },
{ label: 'Settings' },
]}
/>
<div className="project-settings-header">
<h1>Project Settings</h1>
<p className="project-settings-subtitle">Manage settings for {project.name}</p>
</div>
{error && <div className="project-settings-error">{error}</div>}
{success && <div className="project-settings-success">{success}</div>}
{/* General Settings Section */}
<div className="project-settings-section">
<h2>General</h2>
<form className="project-settings-form" onSubmit={handleSaveSettings}>
<div className="project-settings-form-group">
<label htmlFor="description">Description</label>
<textarea
id="description"
value={description}
onChange={(e) => setDescription(e.target.value)}
placeholder="Describe your project..."
disabled={saving}
/>
</div>
<div className="project-settings-form-group project-settings-checkbox-group">
<label className="project-settings-checkbox-label">
<input
type="checkbox"
checked={isPublic}
onChange={(e) => setIsPublic(e.target.checked)}
disabled={saving}
/>
<span className="project-settings-checkbox-custom" />
<span>Public project (visible to everyone)</span>
</label>
</div>
<div className="project-settings-form-actions">
<button type="submit" className="project-settings-save-button" disabled={saving}>
{saving ? (
<>
<span className="project-settings-button-spinner" />
Saving...
</>
) : (
'Save Changes'
)}
</button>
</div>
</form>
</div>
{/* Access Management Section */}
<AccessManagement projectName={projectName!} />
{/* Danger Zone Section */}
<div className="project-settings-danger-zone">
<h2>Danger Zone</h2>
<div className="project-settings-danger-item">
<div className="project-settings-danger-info">
<h3>Delete this project</h3>
<p>
Once you delete a project, there is no going back. This will permanently delete the
project, all packages, artifacts, and tags.
</p>
</div>
{!showDeleteConfirm && (
<button
className="project-settings-delete-button"
onClick={() => setShowDeleteConfirm(true)}
disabled={deleting}
>
Delete Project
</button>
)}
</div>
{showDeleteConfirm && (
<div className="project-settings-delete-confirm">
<p>
Type <strong>{project.name}</strong> to confirm deletion:
</p>
<input
type="text"
className="project-settings-delete-confirm-input"
value={deleteConfirmText}
onChange={(e) => setDeleteConfirmText(e.target.value)}
placeholder={project.name}
disabled={deleting}
autoFocus
/>
<div className="project-settings-delete-confirm-actions">
<button
className="project-settings-confirm-delete-button"
onClick={handleDeleteProject}
disabled={deleting || deleteConfirmText !== project.name}
>
{deleting ? (
<>
<span className="project-settings-delete-spinner" />
Deleting...
</>
) : (
'Yes, delete this project'
)}
</button>
<button
className="project-settings-cancel-button"
onClick={handleCancelDelete}
disabled={deleting}
>
Cancel
</button>
</div>
</div>
)}
</div>
</div>
);
}
export default ProjectSettingsPage;

View File

@@ -373,3 +373,77 @@ export interface OIDCStatus {
enabled: boolean;
issuer_url?: string;
}
// Dependency types
export interface Dependency {
id: string;
artifact_id: string;
project: string;
package: string;
version: string | null;
tag: string | null;
created_at: string;
}
export interface ArtifactDependenciesResponse {
artifact_id: string;
dependencies: Dependency[];
}
export interface DependentInfo {
artifact_id: string;
project: string;
package: string;
version: string | null;
constraint_type: 'version' | 'tag';
constraint_value: string;
}
export interface ReverseDependenciesResponse {
project: string;
package: string;
dependents: DependentInfo[];
pagination: {
page: number;
limit: number;
total: number;
total_pages: number;
has_more: boolean;
};
}
// Dependency Resolution types
export interface ResolvedArtifact {
artifact_id: string;
project: string;
package: string;
version: string | null;
tag: string | null;
size: number;
download_url: string;
}
export interface DependencyResolutionResponse {
requested: {
project: string;
package: string;
ref: string;
};
resolved: ResolvedArtifact[];
total_size: number;
artifact_count: number;
}
export interface DependencyResolutionError {
error: 'circular_dependency' | 'dependency_conflict' | 'not_found';
message: string;
cycle?: string[];
conflicts?: Array<{
project: string;
package: string;
requirements: Array<{
version: string;
required_by: Array<{ path: string }>;
}>;
}>;
}

View File

@@ -77,6 +77,8 @@ PostgreSQL secret name
{{- define "orchard.postgresql.secretName" -}}
{{- if .Values.orchard.database.existingSecret }}
{{- .Values.orchard.database.existingSecret }}
{{- else if and .Values.orchard.database.secretsManager .Values.orchard.database.secretsManager.enabled }}
{{- printf "%s-db-credentials" (include "orchard.fullname" .) }}
{{- else if .Values.postgresql.enabled }}
{{- printf "%s-postgresql" .Release.Name }}
{{- else }}
@@ -90,6 +92,8 @@ PostgreSQL password key in secret
{{- define "orchard.postgresql.passwordKey" -}}
{{- if .Values.orchard.database.existingSecret -}}
{{- .Values.orchard.database.existingSecretPasswordKey -}}
{{- else if and .Values.orchard.database.secretsManager .Values.orchard.database.secretsManager.enabled -}}
password
{{- else if .Values.postgresql.enabled -}}
password
{{- else -}}
@@ -137,3 +141,16 @@ MinIO secret name
{{- printf "%s-s3-secret" (include "orchard.fullname" .) }}
{{- end }}
{{- end }}
{{/*
Auth secret name (for admin password)
*/}}
{{- define "orchard.auth.secretName" -}}
{{- if and .Values.orchard.auth .Values.orchard.auth.existingSecret }}
{{- .Values.orchard.auth.existingSecret }}
{{- else if and .Values.orchard.auth .Values.orchard.auth.secretsManager .Values.orchard.auth.secretsManager.enabled }}
{{- printf "%s-auth-credentials" (include "orchard.fullname" .) }}
{{- else }}
{{- printf "%s-auth-secret" (include "orchard.fullname" .) }}
{{- end }}
{{- end }}

View File

@@ -69,6 +69,8 @@ spec:
containerPort: {{ .Values.orchard.server.port }}
protocol: TCP
env:
- name: ORCHARD_ENV
value: {{ .Values.orchard.env | default "development" | quote }}
- name: ORCHARD_SERVER_HOST
value: {{ .Values.orchard.server.host | quote }}
- name: ORCHARD_SERVER_PORT
@@ -77,8 +79,16 @@ spec:
value: {{ include "orchard.postgresql.host" . | quote }}
- name: ORCHARD_DATABASE_PORT
value: {{ .Values.orchard.database.port | quote }}
{{- if and .Values.orchard.database.secretsManager .Values.orchard.database.secretsManager.enabled }}
- name: ORCHARD_DATABASE_USER
valueFrom:
secretKeyRef:
name: {{ include "orchard.postgresql.secretName" . }}
key: username
{{- else }}
- name: ORCHARD_DATABASE_USER
value: {{ .Values.orchard.database.user | default .Values.postgresql.auth.username | quote }}
{{- end }}
- name: ORCHARD_DATABASE_DBNAME
value: {{ .Values.orchard.database.dbname | default .Values.postgresql.auth.database | quote }}
- name: ORCHARD_DATABASE_SSLMODE
@@ -96,6 +106,7 @@ spec:
value: {{ .Values.orchard.s3.bucket | quote }}
- name: ORCHARD_S3_USE_PATH_STYLE
value: {{ .Values.orchard.s3.usePathStyle | quote }}
{{- if or .Values.minio.enabled .Values.orchard.s3.existingSecret .Values.orchard.s3.accessKeyId }}
- name: ORCHARD_S3_ACCESS_KEY_ID
valueFrom:
secretKeyRef:
@@ -106,16 +117,76 @@ spec:
secretKeyRef:
name: {{ include "orchard.minio.secretName" . }}
key: {{ if .Values.minio.enabled }}root-password{{ else }}{{ .Values.orchard.s3.existingSecretSecretKeyKey }}{{ end }}
{{- end }}
- name: ORCHARD_DOWNLOAD_MODE
value: {{ .Values.orchard.download.mode | quote }}
- name: ORCHARD_PRESIGNED_URL_EXPIRY
value: {{ .Values.orchard.download.presignedUrlExpiry | quote }}
{{- if .Values.orchard.rateLimit }}
{{- if .Values.orchard.rateLimit.login }}
- name: ORCHARD_LOGIN_RATE_LIMIT
value: {{ .Values.orchard.rateLimit.login | quote }}
{{- end }}
{{- end }}
{{- if .Values.orchard.database.poolSize }}
- name: ORCHARD_DATABASE_POOL_SIZE
value: {{ .Values.orchard.database.poolSize | quote }}
{{- end }}
{{- if .Values.orchard.database.maxOverflow }}
- name: ORCHARD_DATABASE_MAX_OVERFLOW
value: {{ .Values.orchard.database.maxOverflow | quote }}
{{- end }}
{{- if .Values.orchard.database.poolTimeout }}
- name: ORCHARD_DATABASE_POOL_TIMEOUT
value: {{ .Values.orchard.database.poolTimeout | quote }}
{{- end }}
{{- if .Values.orchard.auth }}
{{- if or .Values.orchard.auth.secretsManager .Values.orchard.auth.existingSecret .Values.orchard.auth.adminPassword }}
- name: ORCHARD_ADMIN_PASSWORD
valueFrom:
secretKeyRef:
name: {{ include "orchard.auth.secretName" . }}
key: admin-password
{{- end }}
{{- end }}
{{- if or (and .Values.orchard.database.secretsManager .Values.orchard.database.secretsManager.enabled) (and .Values.orchard.auth .Values.orchard.auth.secretsManager .Values.orchard.auth.secretsManager.enabled) }}
volumeMounts:
{{- if and .Values.orchard.database.secretsManager .Values.orchard.database.secretsManager.enabled }}
- name: db-secrets
mountPath: /mnt/secrets-store/db
readOnly: true
{{- end }}
{{- if and .Values.orchard.auth .Values.orchard.auth.secretsManager .Values.orchard.auth.secretsManager.enabled }}
- name: auth-secrets
mountPath: /mnt/secrets-store/auth
readOnly: true
{{- end }}
{{- end }}
livenessProbe:
{{- toYaml .Values.livenessProbe | nindent 12 }}
readinessProbe:
{{- toYaml .Values.readinessProbe | nindent 12 }}
resources:
{{- toYaml .Values.resources | nindent 12 }}
{{- if or (and .Values.orchard.database.secretsManager .Values.orchard.database.secretsManager.enabled) (and .Values.orchard.auth .Values.orchard.auth.secretsManager .Values.orchard.auth.secretsManager.enabled) }}
volumes:
{{- if and .Values.orchard.database.secretsManager .Values.orchard.database.secretsManager.enabled }}
- name: db-secrets
csi:
driver: secrets-store.csi.k8s.io
readOnly: true
volumeAttributes:
secretProviderClass: {{ include "orchard.fullname" . }}-db-secret
{{- end }}
{{- if and .Values.orchard.auth .Values.orchard.auth.secretsManager .Values.orchard.auth.secretsManager.enabled }}
- name: auth-secrets
csi:
driver: secrets-store.csi.k8s.io
readOnly: true
volumeAttributes:
secretProviderClass: {{ include "orchard.fullname" . }}-auth-secret
{{- end }}
{{- end }}
{{- with .Values.nodeSelector }}
nodeSelector:
{{- toYaml . | nindent 8 }}

View File

@@ -0,0 +1,51 @@
{{- if and .Values.orchard.database.secretsManager .Values.orchard.database.secretsManager.enabled }}
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
name: {{ include "orchard.fullname" . }}-db-secret
labels:
{{- include "orchard.labels" . | nindent 4 }}
spec:
provider: aws
parameters:
objects: |
- objectName: "{{ .Values.orchard.database.secretsManager.secretArn }}"
objectType: "secretsmanager"
jmesPath:
- path: username
objectAlias: db-username
- path: password
objectAlias: db-password
secretObjects:
- secretName: {{ include "orchard.fullname" . }}-db-credentials
type: Opaque
data:
- objectName: db-username
key: username
- objectName: db-password
key: password
{{- end }}
---
{{- if and .Values.orchard.auth .Values.orchard.auth.secretsManager .Values.orchard.auth.secretsManager.enabled }}
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
name: {{ include "orchard.fullname" . }}-auth-secret
labels:
{{- include "orchard.labels" . | nindent 4 }}
spec:
provider: aws
parameters:
objects: |
- objectName: "{{ .Values.orchard.auth.secretsManager.secretArn }}"
objectType: "secretsmanager"
jmesPath:
- path: admin_password
objectAlias: admin-password
secretObjects:
- secretName: {{ include "orchard.fullname" . }}-auth-credentials
type: Opaque
data:
- objectName: admin-password
key: admin-password
{{- end }}

View File

@@ -22,3 +22,15 @@ data:
access-key-id: {{ .Values.orchard.s3.accessKeyId | b64enc | quote }}
secret-access-key: {{ .Values.orchard.s3.secretAccessKey | b64enc | quote }}
{{- end }}
---
{{- if and .Values.orchard.auth .Values.orchard.auth.adminPassword (not .Values.orchard.auth.existingSecret) (not (and .Values.orchard.auth.secretsManager .Values.orchard.auth.secretsManager.enabled)) }}
apiVersion: v1
kind: Secret
metadata:
name: {{ include "orchard.fullname" . }}-auth-secret
labels:
{{- include "orchard.labels" . | nindent 4 }}
type: Opaque
data:
admin-password: {{ .Values.orchard.auth.adminPassword | b64enc | quote }}
{{- end }}

View File

@@ -42,6 +42,7 @@ ingress:
className: "nginx"
annotations:
cert-manager.io/cluster-issuer: "letsencrypt"
nginx.ingress.kubernetes.io/proxy-body-size: "0" # Disable body size limit for uploads
hosts:
- host: orchard-dev.common.global.bsf.tools # Overridden by CI
paths:
@@ -52,15 +53,16 @@ ingress:
hosts:
- orchard-dev.common.global.bsf.tools # Overridden by CI
# Lighter resources for ephemeral environments
# Resources for dev/feature environments
# Bumped to handle concurrent integration tests
# Note: memory requests must equal limits per cluster policy
resources:
limits:
cpu: 250m
memory: 256Mi
cpu: 500m
memory: 512Mi
requests:
cpu: 100m
memory: 256Mi
cpu: 200m
memory: 512Mi
livenessProbe:
httpGet:
@@ -84,10 +86,15 @@ tolerations: []
affinity: {}
orchard:
env: "development" # Allows seed data for testing
server:
host: "0.0.0.0"
port: 8080
# Authentication settings
# Admin password is set via CI variable (DEV_ADMIN_PASSWORD) passed as --set flag
# This keeps the password out of version control
database:
host: ""
port: 5432
@@ -97,6 +104,10 @@ orchard:
sslmode: disable
existingSecret: ""
existingSecretPasswordKey: "password"
# Increased pool settings for concurrent integration tests
poolSize: 10
maxOverflow: 20
poolTimeout: 60
s3:
endpoint: ""
@@ -113,6 +124,10 @@ orchard:
mode: "presigned"
presignedUrlExpiry: 3600
# Relaxed rate limits for dev/feature environments (allows integration tests to run)
rateLimit:
login: "1000/minute" # Default is 5/minute, relaxed for CI integration tests
# PostgreSQL - ephemeral, no persistence
postgresql:
enabled: true
@@ -128,15 +143,16 @@ postgresql:
primary:
persistence:
enabled: false
# Resources with memory requests = limits per cluster policy
# Bumped resources for concurrent integration tests
# Note: memory requests must equal limits per cluster policy
resourcesPreset: "none"
resources:
limits:
cpu: 250m
memory: 256Mi
cpu: 500m
memory: 512Mi
requests:
cpu: 100m
memory: 256Mi
cpu: 200m
memory: 512Mi
# Volume permissions init container
volumePermissions:
resourcesPreset: "none"
@@ -162,15 +178,16 @@ minio:
defaultBuckets: "orchard-artifacts"
persistence:
enabled: false
# Resources with memory requests = limits per cluster policy
# Bumped resources for concurrent integration tests
# Note: memory requests must equal limits per cluster policy
resourcesPreset: "none" # Disable preset to use explicit resources
resources:
limits:
cpu: 250m
memory: 256Mi
cpu: 500m
memory: 512Mi
requests:
cpu: 100m
memory: 256Mi
cpu: 200m
memory: 512Mi
# Init container resources
defaultInitContainers:
volumePermissions:

View File

@@ -4,7 +4,7 @@ replicaCount: 1
image:
repository: registry.global.bsf.tools/esv/bsf/bsf-integration/orchard/orchard-mvp
pullPolicy: IfNotPresent # Don't always pull in prod
pullPolicy: Always
tag: "latest" # Overridden by CI
imagePullSecrets:
@@ -19,7 +19,8 @@ initContainer:
serviceAccount:
create: true
automount: true
annotations: {}
annotations:
eks.amazonaws.com/role-arn: arn:aws-us-gov:iam::052673043337:role/service-orchard-prd
name: "orchard"
podAnnotations: {}
@@ -41,6 +42,7 @@ ingress:
className: "nginx"
annotations:
cert-manager.io/cluster-issuer: "letsencrypt"
nginx.ingress.kubernetes.io/proxy-body-size: "0" # Disable body size limit for uploads
hosts:
- host: orchard.common.global.bsf.tools
paths:
@@ -86,126 +88,50 @@ tolerations: []
affinity: {}
orchard:
env: "production" # Disables seed data
server:
host: "0.0.0.0"
port: 8080
# Database configuration (used when postgresql.enabled is false)
# TODO: Configure for managed PostgreSQL when ready
database:
host: ""
port: 5432
user: orchard
password: ""
dbname: orchard
sslmode: disable
existingSecret: ""
existingSecretPasswordKey: "password"
# Authentication settings
auth:
# Admin password from AWS Secrets Manager
secretsManager:
enabled: true
secretArn: "arn:aws-us-gov:secretsmanager:us-gov-west-1:052673043337:secret:orch-prod-creds-0nhqkY"
# S3 configuration (used when minio.enabled is false)
# TODO: Configure for real S3 when ready
# Database configuration - uses AWS Secrets Manager via CSI driver
database:
host: "orchard-prd.cluster-cvw3jzjkozoc.us-gov-west-1.rds.amazonaws.com"
port: 5432
dbname: orchard_prod
sslmode: require
secretsManager:
enabled: true
secretArn: "arn:aws-us-gov:secretsmanager:us-gov-west-1:052673043337:secret:rds!cluster-0afc8af5-f644-4284-92fb-2ed545490f92-3v9uXV"
# S3 configuration - uses IRSA for credentials
s3:
endpoint: ""
region: us-east-1
bucket: orchard-artifacts
accessKeyId: ""
secretAccessKey: ""
usePathStyle: true
existingSecret: ""
existingSecretAccessKeyKey: "access-key-id"
existingSecretSecretKeyKey: "secret-access-key"
endpoint: "" # Empty = use AWS default
region: us-gov-west-1
bucket: orchard-artifacts-prod
usePathStyle: false # Real S3 uses virtual-hosted style
download:
mode: "presigned"
presignedUrlExpiry: 3600
# PostgreSQL subchart - MVP uses subchart, switch to managed later
# PostgreSQL subchart - disabled in prod, using RDS
postgresql:
enabled: true
image:
registry: containers.global.bsf.tools
repository: bitnami/postgresql
tag: "15"
pullPolicy: IfNotPresent
auth:
username: orchard
password: orchard-prod-password # TODO: Use existingSecret
database: orchard
primary:
persistence:
enabled: true # Enable persistence for prod
size: 20Gi
resourcesPreset: "none"
resources:
limits:
cpu: 500m
memory: 512Mi
requests:
cpu: 250m
memory: 512Mi
volumePermissions:
resourcesPreset: "none"
resources:
limits:
cpu: 100m
memory: 128Mi
requests:
cpu: 50m
memory: 128Mi
enabled: false
# MinIO subchart - MVP uses subchart, switch to real S3 later
# MinIO subchart - disabled in prod, using real S3
minio:
enabled: true
image:
registry: containers.global.bsf.tools
repository: bitnami/minio
tag: "latest"
pullPolicy: IfNotPresent
auth:
rootUser: minioadmin
rootPassword: minioadmin-prod # TODO: Use existingSecret
defaultBuckets: "orchard-artifacts"
persistence:
enabled: true # Enable persistence for prod
size: 100Gi
resourcesPreset: "none"
resources:
limits:
cpu: 500m
memory: 512Mi
requests:
cpu: 250m
memory: 512Mi
defaultInitContainers:
volumePermissions:
resourcesPreset: "none"
resources:
limits:
cpu: 100m
memory: 128Mi
requests:
cpu: 50m
memory: 128Mi
provisioning:
resources:
limits:
cpu: 200m
memory: 256Mi
requests:
cpu: 100m
memory: 256Mi
enabled: false
# MinIO external ingress for presigned URL access
# MinIO ingress - disabled in prod, using real S3
minioIngress:
enabled: true
className: "nginx"
annotations:
cert-manager.io/cluster-issuer: "letsencrypt"
nginx.ingress.kubernetes.io/proxy-body-size: "0"
host: "minio-orchard.common.global.bsf.tools"
tls:
enabled: true
secretName: minio-prod-tls
enabled: false
redis:
enabled: false

View File

@@ -19,7 +19,8 @@ initContainer:
serviceAccount:
create: true
automount: true
annotations: {}
annotations:
eks.amazonaws.com/role-arn: arn:aws-us-gov:iam::052673043337:role/service-orchard-stage
name: "orchard"
podAnnotations: {}
@@ -41,6 +42,7 @@ ingress:
className: "nginx"
annotations:
cert-manager.io/cluster-issuer: "letsencrypt"
nginx.ingress.kubernetes.io/proxy-body-size: "0" # Disable body size limit for uploads
hosts:
- host: orchard-stage.common.global.bsf.tools
paths:
@@ -88,130 +90,52 @@ affinity: {}
# Orchard server configuration
orchard:
env: "development" # Allows seed data for testing
server:
host: "0.0.0.0"
port: 8080
# Database configuration (used when postgresql.enabled is false)
database:
host: ""
port: 5432
user: orchard
password: ""
dbname: orchard
sslmode: disable
existingSecret: ""
existingSecretPasswordKey: "password"
# Authentication settings
# Admin password is set via CI variable (STAGE_ADMIN_PASSWORD) passed as --set flag
# This keeps the password out of version control
# S3 configuration (used when minio.enabled is false)
# Database configuration - uses AWS Secrets Manager via CSI driver
database:
host: "orchard-stage.cluster-cvw3jzjkozoc.us-gov-west-1.rds.amazonaws.com"
port: 5432
dbname: postgres
sslmode: require
secretsManager:
enabled: true
secretArn: "arn:aws-us-gov:secretsmanager:us-gov-west-1:052673043337:secret:rds!cluster-a573672b-1a38-4665-a654-1b7df37b5297-IaeFQL"
# S3 configuration - uses IRSA for credentials
s3:
endpoint: ""
region: us-east-1
bucket: orchard-artifacts
accessKeyId: ""
secretAccessKey: ""
usePathStyle: true
existingSecret: ""
existingSecretAccessKeyKey: "access-key-id"
existingSecretSecretKeyKey: "secret-access-key"
endpoint: "" # Empty = use AWS default
region: us-gov-west-1
bucket: orchard-artifacts-stage
usePathStyle: false # Real S3 uses virtual-hosted style
# Download configuration
download:
mode: "presigned" # presigned, redirect, or proxy
presignedUrlExpiry: 3600 # Presigned URL expiry in seconds
# PostgreSQL subchart configuration
# Relaxed rate limits for stage (allows CI integration tests to run)
rateLimit:
login: "1000/minute" # Default is 5/minute, relaxed for CI integration tests
# PostgreSQL subchart - disabled in stage, using RDS
postgresql:
enabled: true
image:
registry: containers.global.bsf.tools
repository: bitnami/postgresql
tag: "15"
pullPolicy: IfNotPresent
auth:
username: orchard
password: orchard-password
database: orchard
primary:
persistence:
enabled: false
size: 10Gi
# Resources with memory requests = limits per cluster policy
resourcesPreset: "none"
resources:
limits:
cpu: 500m
memory: 512Mi
requests:
cpu: 250m
memory: 512Mi
# Volume permissions init container
volumePermissions:
resourcesPreset: "none"
resources:
limits:
cpu: 100m
memory: 128Mi
requests:
cpu: 50m
memory: 128Mi
enabled: false
# MinIO subchart configuration
# MinIO subchart - disabled in stage, using real S3
minio:
enabled: true
image:
registry: containers.global.bsf.tools
repository: bitnami/minio
tag: "latest"
pullPolicy: IfNotPresent
auth:
rootUser: minioadmin
rootPassword: minioadmin
defaultBuckets: "orchard-artifacts"
persistence:
enabled: false
size: 50Gi
# Resources with memory requests = limits per cluster policy
resourcesPreset: "none" # Disable preset to use explicit resources
resources:
limits:
cpu: 500m
memory: 512Mi
requests:
cpu: 250m
memory: 512Mi
# Init container resources
defaultInitContainers:
volumePermissions:
resourcesPreset: "none"
resources:
limits:
cpu: 100m
memory: 128Mi
requests:
cpu: 50m
memory: 128Mi
# Provisioning job resources
provisioning:
resources:
limits:
cpu: 200m
memory: 256Mi
requests:
cpu: 100m
memory: 256Mi
enabled: false
# MinIO external ingress for presigned URL access (separate from subchart ingress)
# MinIO ingress - disabled in stage, using real S3
minioIngress:
enabled: true
className: "nginx"
annotations:
cert-manager.io/cluster-issuer: "letsencrypt"
nginx.ingress.kubernetes.io/proxy-body-size: "0" # Disable body size limit for uploads
host: "minio-orch-stage.common.global.bsf.tools"
tls:
enabled: true
secretName: minio-tls
enabled: false
# Redis subchart configuration (for future caching)
redis:

View File

@@ -120,6 +120,17 @@ orchard:
mode: "presigned" # presigned, redirect, or proxy
presignedUrlExpiry: 3600 # Presigned URL expiry in seconds
# Authentication settings
auth:
# Option 1: Plain admin password (creates K8s secret)
adminPassword: ""
# Option 2: Use existing K8s secret (must have 'admin-password' key)
existingSecret: ""
# Option 3: AWS Secrets Manager
# secretsManager:
# enabled: false
# secretArn: "" # Secret must have 'admin_password' field
# PostgreSQL subchart configuration
postgresql:
enabled: true

View File

@@ -0,0 +1,48 @@
-- Migration 008: Artifact Dependencies
-- Adds support for declaring dependencies between artifacts
-- Part of Package Dependency Management feature (#76)
-- Create artifact_dependencies table
CREATE TABLE IF NOT EXISTS artifact_dependencies (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
artifact_id VARCHAR(64) NOT NULL REFERENCES artifacts(id) ON DELETE CASCADE,
dependency_project VARCHAR(255) NOT NULL,
dependency_package VARCHAR(255) NOT NULL,
version_constraint VARCHAR(255),
tag_constraint VARCHAR(255),
created_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP,
-- Exactly one of version_constraint or tag_constraint must be set
CONSTRAINT check_constraint_type CHECK (
(version_constraint IS NOT NULL AND tag_constraint IS NULL) OR
(version_constraint IS NULL AND tag_constraint IS NOT NULL)
),
-- Each artifact can only have one dependency on a specific project/package
CONSTRAINT unique_artifact_dependency UNIQUE (artifact_id, dependency_project, dependency_package)
);
-- Index for fast lookups by artifact_id (get all deps for an artifact)
CREATE INDEX IF NOT EXISTS idx_artifact_dependencies_artifact_id
ON artifact_dependencies(artifact_id);
-- Index for reverse dependency lookups (find what depends on a package)
CREATE INDEX IF NOT EXISTS idx_artifact_dependencies_target
ON artifact_dependencies(dependency_project, dependency_package);
-- Index for finding dependencies with specific version constraints
CREATE INDEX IF NOT EXISTS idx_artifact_dependencies_version
ON artifact_dependencies(dependency_project, dependency_package, version_constraint)
WHERE version_constraint IS NOT NULL;
-- Index for finding dependencies with specific tag constraints
CREATE INDEX IF NOT EXISTS idx_artifact_dependencies_tag
ON artifact_dependencies(dependency_project, dependency_package, tag_constraint)
WHERE tag_constraint IS NOT NULL;
COMMENT ON TABLE artifact_dependencies IS 'Stores dependencies declared by artifacts on other packages';
COMMENT ON COLUMN artifact_dependencies.artifact_id IS 'The artifact that declares this dependency';
COMMENT ON COLUMN artifact_dependencies.dependency_project IS 'Project name of the dependency';
COMMENT ON COLUMN artifact_dependencies.dependency_package IS 'Package name of the dependency';
COMMENT ON COLUMN artifact_dependencies.version_constraint IS 'Exact version required (mutually exclusive with tag_constraint)';
COMMENT ON COLUMN artifact_dependencies.tag_constraint IS 'Tag name required (mutually exclusive with version_constraint)';