25 Commits

Author SHA1 Message Date
Mondo Diaz
b6e5a0fa4d Remove exists rule from frontend_tests for consistent behavior 2026-01-13 22:53:32 +00:00
Mondo Diaz
22c173ec37 Enhance test jobs with caching, coverage, and parallel execution
CI improvements:
- Add needs: [] to run tests parallel with build (faster pipeline)
- Add pip/npm caching (faster subsequent runs)
- Add 15m timeout to prevent hung jobs
- Add pytest coverage with cobertura report for GitLab
- Add pytest JUnit report for test tab in MR
- Add vitest coverage with cobertura report for GitLab
- Add coverage regex for badge display

Frontend:
- Add @vitest/coverage-v8 dependency
- Configure vitest coverage reporter (text, cobertura, html)
2026-01-13 22:50:50 +00:00
Mondo Diaz
acd31129e9 Use deps.global.bsf.tools registry for frontend_tests image 2026-01-13 22:45:12 +00:00
Mondo Diaz
b74276ba70 Fix frontend_tests to use correct container registry 2026-01-13 22:44:27 +00:00
Mondo Diaz
085ed898e2 Refactor CI pipeline with templates and add frontend tests
- Add frontend_tests job (npm run test with Vitest)
- Add verification checks to deploy_stage (health, API, frontend)
- Extract shared YAML anchors: deploy_template, helm_setup, verify_deployment
- Reduce code duplication across deploy jobs
2026-01-13 22:43:21 +00:00
Mondo Diaz
4ec91b46ed Fix security scan issues and harden docker-compose
Hadolint fixes:
- Use printf instead of echo for escape sequences
- Add hadolint ignore for apt pin version (DL3008)

KICS fixes (docker-compose):
- Add security_opt: no-new-privileges to all services
- Add mem_limit and cpus to prevent resource exhaustion
- Add healthcheck to orchard-server in docker-compose.yml

Gitleaks:
- Add .gitleaksignore for false positive (s3_key attribute name)
- Remove allow_failure from secrets job (now blocking)

Also:
- Remove || echo fallback from python_tests (tests should fail pipeline)
2026-01-13 22:40:51 +00:00
Mondo Diaz
d6644e9039 Add comprehensive deployment verification
- Health endpoint polling with retry loop
- API check (GET /api/v1/projects returns 200)
- Frontend check (HTML is served)
- Clear output with section headers
2026-01-13 22:28:48 +00:00
Mondo Diaz
f1b0c93f30 Add deployment verification with health check polling
- Add --wait --timeout 5m to helm upgrade
- Add kubectl rollout status check
- Poll health endpoint for up to 5 minutes (for cert provisioning)
2026-01-13 22:28:30 +00:00
Mondo Diaz
69180dd9eb Add PROSPER-NOTES.md to gitignore 2026-01-13 22:20:45 +00:00
Mondo Diaz
b07b6a8f8f Add build_image dependency to deploy jobs 2026-01-13 22:14:31 +00:00
Mondo Diaz
ace643bb2e Fix image tag format to match Prosper output (git.linux-amd64-SHA) 2026-01-13 22:06:02 +00:00
Mondo Diaz
f1aae057ed Clean up CI pipeline and remove unused values files
- Use branch name (CI_COMMIT_REF_SLUG) instead of commit SHA for feature IDs
- Remove commented-out code and unused deploy template
- Fix deploy_stage to use kubectl config use-context
- Remove values-production.yaml and values-external.yaml
2026-01-13 21:51:50 +00:00
Mondo Diaz
66515f24d4 Add kubectl context to cleanup_feature job 2026-01-13 21:49:28 +00:00
Mondo Diaz
892f63e685 Fix helm path by returning to project root before deploy 2026-01-13 21:37:37 +00:00
Mondo Diaz
cd7c930176 Use kubectl config use-context for agent authentication 2026-01-13 21:28:10 +00:00
Mondo Diaz
0bbfe38293 Test: hardcode agent path to rule out variable interpolation 2026-01-13 21:12:08 +00:00
Mondo Diaz
86567b4ae1 Add GitLab Agent configs with CI/CD access for deployments 2026-01-13 20:08:29 +00:00
Mondo Diaz
d3a47f6167 Fix GitLab Agent paths to use full project:agent format 2026-01-13 19:32:09 +00:00
Armando Diaz
089d4a50df update jobs to use correct image and agents. 2026-01-13 12:05:16 -06:00
Mondo Diaz
f6b79a7af0 Add feature branch deployment pipeline
- Add deploy_feature job for ephemeral dev environments
- Use unique identifier (feat-{short_sha}) for K8s resource isolation
- Dynamic hostnames for ingress (orchard-{sha}.common.global.bsf.tools)
- Add cleanup_feature job with on_stop for automatic cleanup on merge
- Add values-dev.yaml with lighter resources for ephemeral deployments
- Refactor deploy_stage to use dynamic image tag from CI
2026-01-13 16:45:48 +00:00
Dane Moss
deda96795b comment out rule block for now 2026-01-13 16:43:49 +00:00
Dane Moss
f555dd6bde try another rule 2026-01-13 16:43:49 +00:00
Dane Moss
36b79485ba update job name 2026-01-13 16:43:49 +00:00
Dane Moss
b58deb4a60 Update .gitlab-ci.yml file 2026-01-13 16:43:49 +00:00
Dane Moss
d3bacfe6b6 Update 2 files
- /helm/orchard/values-stage.yaml
- /.gitlab-ci.yml
2026-01-13 16:43:49 +00:00
56 changed files with 866 additions and 8245 deletions

View File

@@ -6,256 +6,34 @@ include:
variables:
# renovate: datasource=gitlab-tags depName=esv/bsf/pypi/prosper versioning=semver registryUrl=https://gitlab.global.bsf.tools
PROSPER_VERSION: v0.64.1
# Use internal PyPI proxy instead of public internet
PIP_INDEX_URL: https://deps.global.bsf.tools/artifactory/api/pypi/pypi.org/simple
# Environment URLs (used by deploy and test jobs)
STAGE_URL: https://orchard-stage.common.global.bsf.tools
PROD_URL: https://orchard.common.global.bsf.tools
# Stage environment AWS resources (used by reset job)
STAGE_RDS_HOST: orchard-stage.cluster-cvw3jzjkozoc.us-gov-west-1.rds.amazonaws.com
STAGE_RDS_DBNAME: postgres
STAGE_SECRET_ARN: "arn:aws-us-gov:secretsmanager:us-gov-west-1:052673043337:secret:rds!cluster-a573672b-1a38-4665-a654-1b7df37b5297-IaeFQL"
STAGE_S3_BUCKET: orchard-artifacts-stage
AWS_REGION: us-gov-west-1
# Shared pip cache directory
PIP_CACHE_DIR: "$CI_PROJECT_DIR/.pip-cache"
# Prevent duplicate pipelines for MRs
workflow:
rules:
- if: $CI_PIPELINE_SOURCE == "merge_request_event"
when: never
- when: always
# Define stages - extends Prosper's stages with our custom ones
stages:
- .pre
- lint
- build
- test
- analyze
- deploy
kics:
variables:
KICS_CONFIG: kics.config
allow_failure: true
# Full integration test suite template (for feature/stage deployments)
# Runs the complete pytest integration test suite against the deployed environment
.integration_test_template: &integration_test_template
stage: deploy # Runs in deploy stage, but after deployment due to 'needs'
image: deps.global.bsf.tools/docker/python:3.12-slim
timeout: 20m # Full suite takes longer than smoke tests
interruptible: true # Cancel if new pipeline starts
retry: 1 # Retry once on failure (network flakiness)
cache:
key: pip-$CI_COMMIT_REF_SLUG
paths:
- .pip-cache/
policy: pull-push
before_script:
- pip install --index-url "$PIP_INDEX_URL" -r backend/requirements.txt
- pip install --index-url "$PIP_INDEX_URL" pytest pytest-asyncio httpx
script:
- cd backend
# Run full integration test suite, excluding:
# - large/slow tests
# - requires_direct_s3 tests (can't access MinIO from outside K8s cluster)
# ORCHARD_TEST_URL tells the tests which server to connect to
# Note: Auth tests work because dev/stage deployments have relaxed rate limits
- |
python -m pytest tests/integration/ -v \
--junitxml=integration-report.xml \
-m "not large and not slow and not requires_direct_s3" \
--tb=short
artifacts:
when: always
expire_in: 1 week
paths:
- backend/integration-report.xml
reports:
junit: backend/integration-report.xml
hadolint:
allow_failure: true
# Lightweight smoke test template (for production - no test data creation)
.smoke_test_template: &smoke_test_template
stage: deploy
image: deps.global.bsf.tools/docker/python:3.12-slim
timeout: 5m
before_script:
- pip install --index-url "$PIP_INDEX_URL" httpx
script:
- |
python - <<'PYTEST_SCRIPT'
import httpx
import os
import sys
# secrets job is a blocking check - real credential leaks should fail the pipeline
BASE_URL = os.environ.get("ORCHARD_TEST_URL")
if not BASE_URL:
print("ERROR: ORCHARD_TEST_URL not set")
sys.exit(1)
print(f"Running smoke tests against {BASE_URL}")
client = httpx.Client(base_url=BASE_URL, timeout=30.0)
errors = []
# Test 1: Health endpoint
print("\n=== Test 1: Health endpoint ===")
r = client.get("/health")
if r.status_code == 200:
print("PASS: Health check passed")
else:
errors.append(f"Health check failed: {r.status_code}")
# Test 2: API responds (list projects)
print("\n=== Test 2: API responds ===")
r = client.get("/api/v1/projects")
if r.status_code == 200:
projects = r.json()
print(f"PASS: API responding, found {len(projects)} project(s)")
else:
errors.append(f"API check failed: {r.status_code}")
# Test 3: Frontend served
print("\n=== Test 3: Frontend served ===")
r = client.get("/")
if r.status_code == 200 and "</html>" in r.text:
print("PASS: Frontend is being served")
else:
errors.append(f"Frontend check failed: {r.status_code}")
# Report results
print("\n" + "=" * 50)
if errors:
print(f"FAILED: {len(errors)} error(s)")
for e in errors:
print(f" FAIL: {e}")
sys.exit(1)
else:
print("SUCCESS: All smoke tests passed!")
sys.exit(0)
PYTEST_SCRIPT
# Integration tests for stage deployment (full suite)
integration_test_stage:
<<: *integration_test_template
needs: [deploy_stage]
variables:
ORCHARD_TEST_URL: $STAGE_URL
rules:
- if: '$CI_COMMIT_BRANCH == "main"'
when: on_success
# Reset stage environment after integration tests (clean slate for next run)
# Calls the /api/v1/admin/factory-reset endpoint which handles DB and S3 cleanup
reset_stage:
stage: deploy
needs: [integration_test_stage]
image: deps.global.bsf.tools/docker/python:3.12-slim
timeout: 5m
retry: 1 # Retry once on transient failures
before_script:
- pip install --index-url "$PIP_INDEX_URL" httpx
script:
- |
python - <<'RESET_SCRIPT'
import httpx
import sys
import os
import time
BASE_URL = os.environ.get("STAGE_URL", "")
ADMIN_USER = "admin"
ADMIN_PASS = "changeme123" # Default admin password
MAX_RETRIES = 3
RETRY_DELAY = 5 # seconds
if not BASE_URL:
print("ERROR: STAGE_URL environment variable not set")
sys.exit(1)
print(f"=== Resetting stage environment at {BASE_URL} ===")
def do_reset():
with httpx.Client(base_url=BASE_URL, timeout=120.0) as client:
# Login as admin
print("Logging in as admin...")
login_response = client.post(
"/api/v1/auth/login",
json={"username": ADMIN_USER, "password": ADMIN_PASS},
)
if login_response.status_code != 200:
raise Exception(f"Login failed: {login_response.status_code} - {login_response.text}")
print("Login successful")
# Call factory reset endpoint
print("Calling factory reset endpoint...")
reset_response = client.post(
"/api/v1/admin/factory-reset",
headers={"X-Confirm-Reset": "yes-delete-all-data"},
)
if reset_response.status_code == 200:
result = reset_response.json()
print("Factory reset successful!")
print(f" Database tables dropped: {result['results']['database_tables_dropped']}")
print(f" S3 objects deleted: {result['results']['s3_objects_deleted']}")
print(f" Database reinitialized: {result['results']['database_reinitialized']}")
print(f" Seeded: {result['results']['seeded']}")
return True
else:
raise Exception(f"Factory reset failed: {reset_response.status_code} - {reset_response.text}")
# Retry loop
for attempt in range(1, MAX_RETRIES + 1):
try:
print(f"Attempt {attempt}/{MAX_RETRIES}")
if do_reset():
sys.exit(0)
except Exception as e:
print(f"Attempt {attempt} failed: {e}")
if attempt < MAX_RETRIES:
print(f"Retrying in {RETRY_DELAY} seconds...")
time.sleep(RETRY_DELAY)
else:
print("All retry attempts failed")
sys.exit(1)
RESET_SCRIPT
rules:
- if: '$CI_COMMIT_BRANCH == "main"'
when: on_success
allow_failure: true # Don't fail pipeline if reset has issues
# Integration tests for feature deployment (full suite)
integration_test_feature:
<<: *integration_test_template
needs: [deploy_feature]
variables:
ORCHARD_TEST_URL: https://orchard-$CI_COMMIT_REF_SLUG.common.global.bsf.tools
rules:
- if: '$CI_COMMIT_BRANCH && $CI_COMMIT_BRANCH != "main"'
when: on_success
# Run Python backend unit tests
python_unit_tests:
# Run Python backend tests
python_tests:
stage: test
needs: [] # Run in parallel with build
image: deps.global.bsf.tools/docker/python:3.12-slim
timeout: 15m
interruptible: true # Cancel if new pipeline starts
variables:
PIP_CACHE_DIR: "$CI_PROJECT_DIR/.pip-cache"
cache:
key: pip-$CI_COMMIT_REF_SLUG
paths:
- .pip-cache/
policy: pull-push
before_script:
- pip install --index-url "$PIP_INDEX_URL" -r backend/requirements.txt
- pip install --index-url "$PIP_INDEX_URL" pytest pytest-asyncio pytest-cov httpx
- pip install -r backend/requirements.txt
- pip install pytest pytest-asyncio pytest-cov httpx
script:
- cd backend
# Run unit tests (integration tests run post-deployment against live environment)
- python -m pytest tests/unit/ -v --cov=app --cov-report=term --cov-report=xml:coverage.xml --cov-report=html:coverage_html --junitxml=pytest-report.xml
- python -m pytest -v --cov=app --cov-report=term --cov-report=xml:coverage.xml --cov-report=html:coverage_html --junitxml=pytest-report.xml
artifacts:
when: always
expire_in: 1 week
@@ -276,7 +54,6 @@ frontend_tests:
needs: [] # Run in parallel with build
image: deps.global.bsf.tools/docker/node:20-alpine
timeout: 15m
interruptible: true # Cancel if new pipeline starts
cache:
key: npm-$CI_COMMIT_REF_SLUG
paths:
@@ -284,8 +61,7 @@ frontend_tests:
policy: pull-push
before_script:
- cd frontend
- npm config set registry https://deps.global.bsf.tools/artifactory/api/npm/registry.npmjs.org
- npm ci --verbose
- npm ci
script:
- npm run test -- --run --reporter=verbose --coverage
artifacts:
@@ -302,30 +78,58 @@ frontend_tests:
# Shared deploy configuration
.deploy_template: &deploy_template
stage: deploy
needs: [build_image, test_image, kics, hadolint, python_unit_tests, frontend_tests, secrets, app_deps_scan, cve_scan, cve_sbom_analysis, app_sbom_analysis]
needs: [build_image]
image: deps.global.bsf.tools/registry-1.docker.io/alpine/k8s:1.29.12
.helm_setup: &helm_setup
- helm version
- helm repo add stable https://charts.helm.sh/stable
- helm repo add bitnami https://charts.bitnami.com/bitnami
- cd helm/orchard
# OCI-based charts from internal registry - no repo add needed
- helm dependency update
- helm repo update
# Simplified deployment verification - just health check
# Full API/frontend checks are done by integration tests post-deployment
.verify_deployment: &verify_deployment |
echo "=== Waiting for health endpoint (certs may take a few minutes) ==="
for i in $(seq 1 30); do
if curl -sf --max-time 10 "$BASE_URL/health" > /dev/null 2>&1; then
echo "Health check passed!"
echo "Deployment URL: $BASE_URL"
exit 0
break
fi
echo "Attempt $i/30 - waiting 10s..."
sleep 10
done
echo "Health check failed after 30 attempts"
exit 1
# Verify health endpoint
echo ""
echo "=== Health Check ==="
curl -sf "$BASE_URL/health" || { echo "Health check failed"; exit 1; }
echo ""
# Verify API is responding
echo ""
echo "=== API Check (GET /api/v1/projects) ==="
HTTP_CODE=$(curl -sf -o /dev/null -w "%{http_code}" "$BASE_URL/api/v1/projects")
if [ "$HTTP_CODE" = "200" ]; then
echo "API responding: HTTP $HTTP_CODE"
else
echo "API check failed: HTTP $HTTP_CODE"
exit 1
fi
# Verify frontend is served
echo ""
echo "=== Frontend Check ==="
if curl -sf "$BASE_URL/" | grep -q "</html>"; then
echo "Frontend is being served"
else
echo "Frontend check failed"
exit 1
fi
echo ""
echo "=== All checks passed! ==="
echo "Deployment URL: $BASE_URL"
# Deploy to stage (main branch)
deploy_stage:
@@ -333,7 +137,7 @@ deploy_stage:
variables:
NAMESPACE: orch-stage-namespace
VALUES_FILE: helm/orchard/values-stage.yaml
BASE_URL: $STAGE_URL
BASE_URL: https://orchard-stage.common.global.bsf.tools
before_script:
- kubectl config use-context esv/bsf/bsf-integration/orchard/orchard-mvp:orchard-stage
- *helm_setup
@@ -346,18 +150,17 @@ deploy_stage:
-f $VALUES_FILE \
--set image.tag=git.linux-amd64-$CI_COMMIT_SHA \
--wait \
--atomic \
--timeout 10m
- kubectl rollout status deployment/orchard-stage-server -n $NAMESPACE --timeout=10m
--timeout 5m
- kubectl rollout status deployment/orchard-stage -n $NAMESPACE --timeout=5m
- *verify_deployment
environment:
name: stage
url: $STAGE_URL
url: https://orchard-stage.common.global.bsf.tools
kubernetes:
agent: esv/bsf/bsf-integration/orchard/orchard-mvp:orchard-stage
rules:
- if: '$CI_COMMIT_BRANCH == "main"'
when: on_success
when: always
# Deploy feature branch to dev namespace
deploy_feature:
@@ -382,31 +185,26 @@ deploy_feature:
--set minioIngress.host=minio-$CI_COMMIT_REF_SLUG.common.global.bsf.tools \
--set minioIngress.tls.secretName=minio-$CI_COMMIT_REF_SLUG-tls \
--wait \
--atomic \
--timeout 10m
- kubectl rollout status deployment/orchard-$CI_COMMIT_REF_SLUG-server -n $NAMESPACE --timeout=10m
--timeout 5m
- kubectl rollout status deployment/orchard-$CI_COMMIT_REF_SLUG -n $NAMESPACE --timeout=5m
- export BASE_URL="https://orchard-$CI_COMMIT_REF_SLUG.common.global.bsf.tools"
- *verify_deployment
environment:
name: review/$CI_COMMIT_REF_SLUG
url: https://orchard-$CI_COMMIT_REF_SLUG.common.global.bsf.tools
on_stop: cleanup_feature
auto_stop_in: 1 week
kubernetes:
agent: esv/bsf/bsf-integration/orchard/orchard-mvp:orchard
rules:
- if: '$CI_COMMIT_BRANCH && $CI_COMMIT_BRANCH != "main"'
when: on_success
when: always
# Cleanup feature branch deployment (standalone - doesn't need deploy dependencies)
# Cleanup feature branch deployment
cleanup_feature:
stage: deploy
<<: *deploy_template
needs: []
image: deps.global.bsf.tools/registry-1.docker.io/alpine/k8s:1.29.12
timeout: 5m
variables:
NAMESPACE: orch-dev-namespace
GIT_STRATEGY: none # No source needed, branch may be deleted
before_script:
- kubectl config use-context esv/bsf/bsf-integration/orchard/orchard-mvp:orchard
script:
@@ -421,51 +219,3 @@ cleanup_feature:
- if: '$CI_COMMIT_BRANCH && $CI_COMMIT_BRANCH != "main"'
when: manual
allow_failure: true
# Deploy to production (version tags only)
deploy_prod:
stage: deploy
# For tag pipelines, most jobs don't run (trusting main was tested)
# We only need build_image to have the image available
needs: [build_image]
image: deps.global.bsf.tools/registry-1.docker.io/alpine/k8s:1.29.12
variables:
NAMESPACE: orch-prod-namespace
VALUES_FILE: helm/orchard/values-prod.yaml
BASE_URL: $PROD_URL
before_script:
- kubectl config use-context esv/bsf/bsf-integration/orchard/orchard-mvp:orchard-prod
- *helm_setup
script:
- echo "Deploying to PRODUCTION - version $CI_COMMIT_TAG"
- cd $CI_PROJECT_DIR
- |
helm upgrade --install orchard-prod ./helm/orchard \
--namespace $NAMESPACE \
-f $VALUES_FILE \
--set image.tag=git.linux-amd64-$CI_COMMIT_SHA \
--wait \
--atomic \
--timeout 10m
- kubectl rollout status deployment/orchard-prod-server -n $NAMESPACE --timeout=10m
- *verify_deployment
environment:
name: production
url: $PROD_URL
kubernetes:
agent: esv/bsf/bsf-integration/orchard/orchard-mvp:orchard-prod
rules:
# Only run on semantic version tags (v1.0.0, v1.2.3, etc.)
- if: '$CI_COMMIT_TAG =~ /^v\d+\.\d+\.\d+$/'
when: on_success
allow_failure: false
# Smoke tests for production deployment (read-only, no test data creation)
smoke_test_prod:
<<: *smoke_test_template
needs: [deploy_prod]
variables:
ORCHARD_TEST_URL: $PROD_URL
rules:
- if: '$CI_COMMIT_TAG =~ /^v\d+\.\d+\.\d+$/'
when: on_success

View File

View File

@@ -1,3 +0,0 @@
ci_access:
projects:
- id: esv/bsf/bsf-integration/orchard/orchard-mvp

View File

@@ -1,18 +1,6 @@
# Gitleaks ignore file
# https://github.com/gitleaks/gitleaks#gitleaksignore
#
# False positive: s3_key is an attribute name in test assertions, not a secret
# These are historical commits - files have since been deleted or updated with inline comments
7e68baed0886a3c928644cd01aa3b39f92d4f976:backend/tests/test_duplicate_detection.py:generic-api-key:154
81458b3bcb5ace97109ba4c16f4afa6e55b1b8bd:backend/tests/test_duplicate_detection.py:generic-api-key:154
2f1891cf0126ec0e7d4c789d872a2cb2dd3a1745:backend/tests/unit/test_storage.py:generic-api-key:381
10d36947948de796f0bacea3827f4531529c405d:backend/tests/unit/test_storage.py:generic-api-key:381
bccbc71c13570d14b8b26a11335c45f102fe3072:backend/tests/unit/test_storage.py:generic-api-key:381
5c9da9003b844a2d655cce74a7c82c57e74f27c4:backend/tests/unit/test_storage.py:generic-api-key:381
90bb2a3a393d2361dc3136ee8d761debb0726d8a:backend/tests/unit/test_storage.py:generic-api-key:381
37666e41a72d2a4f34447c0d1a8728e1d7271d24:backend/tests/unit/test_storage.py:generic-api-key:381
0cc4f253621a9601c5193f6ae1e7ae33f0e7fc9b:backend/tests/unit/test_storage.py:generic-api-key:381
# False positive: s3_key is an attribute name, not a secret
35fda65d381acc5ab59bc592ee3013f75906c197:backend/tests/unit/test_storage.py:generic-api-key:381
08dce6cbb836b687002751fed4159bfc2da61f8b:backend/tests/unit/test_storage.py:generic-api-key:381
617bcbe89cff9a009d77e4f1f1864efed1820e63:backend/tests/unit/test_storage.py:generic-api-key:381
1cbd33544388e0fe6db752fa8886fab33cf9ce7c:backend/tests/unit/test_storage.py:generic-api-key:381

View File

@@ -7,132 +7,10 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
## [Unreleased]
### Added
- Added factory reset endpoint `POST /api/v1/admin/factory-reset` for test environment cleanup (#54)
- Requires admin authentication and `X-Confirm-Reset: yes-delete-all-data` header
- Drops all database tables, clears S3 bucket, reinitializes schema, re-seeds default data
- CI pipeline automatically calls this after integration tests on stage
- Added `delete_all()` method to storage backend for bulk S3 object deletion (#54)
- Added AWS Secrets Manager CSI driver support for database credentials (#54)
- Added SecretProviderClass template for Secrets Manager integration (#54)
- Added IRSA service account annotations for prod and stage environments (#54)
### Changed
- Configured stage and prod to use AWS RDS instead of PostgreSQL subchart (#54)
- Configured stage and prod to use AWS S3 instead of MinIO subchart (#54)
- Changed prod deployment from manual to automatic on version tags (#54)
- Updated S3 client to support IRSA credentials when no explicit keys provided (#54)
- Changed prod image pullPolicy to Always (#54)
- Added proxy-body-size annotation to prod ingress for large uploads (#54)
### Removed
- Disabled PostgreSQL subchart for stage and prod environments (#54)
- Disabled MinIO subchart for stage and prod environments (#54)
### Fixed
- Fixed factory reset not creating default admin user after reset (#60)
- Admin user was only created at server startup, not after factory reset
- CI reset job would fail to login because admin user didn't exist
- Improved reset_stage CI job reliability (#60)
- Added application-level retry logic (3 attempts with 5s delay)
- Added job-level retry for transient failures
- Fixed httpx client to use proper context manager
- Increased timeout to 120s for reset operations
### Added
- Added comprehensive upload/download tests for size boundaries (1B to 1GB) (#38)
- Added concurrent upload/download tests (2, 5, 10 parallel operations) (#38)
- Added data integrity tests (binary, text, unicode, compressed content) (#38)
- Added chunk boundary tests for edge cases (#38)
- Added `@pytest.mark.large` and `@pytest.mark.concurrent` test markers (#38)
- Added `generate_content()` and `generate_content_with_hash()` test helpers (#38)
- Added `sized_content` fixture for generating test content of specific sizes (#38)
- Added upload API tests: upload without tag, artifact creation verification, S3 object creation (#38)
- Added download API tests: tag: prefix resolution, 404 for nonexistent project/package/artifact (#38)
- Added download header tests: Content-Type, Content-Length, Content-Disposition, ETag, X-Checksum-SHA256 (#38)
- Added error handling tests: timeout behavior, checksum validation, resource cleanup, graceful error responses (#38)
- Added version API tests: version creation, auto-detection, listing, download by version prefix (#38)
- Added integrity verification tests: round-trip hash verification, client-side verification workflow, size variants (1KB-10MB) (#40)
- Added consistency check endpoint tests with response format validation (#40)
- Added corruption detection tests: bit flip, truncation, appended content, size mismatch, missing S3 objects (#40)
- Added Digest header tests (RFC 3230) and verification mode tests (#40)
- Added integrity verification documentation (`docs/integrity-verification.md`) (#40)
- Added conditional request support for downloads (If-None-Match, If-Modified-Since) returning 304 Not Modified (#42)
- Added caching headers to downloads: Cache-Control (immutable), Last-Modified (#42)
- Added 416 Range Not Satisfiable response for invalid range requests (#42)
- Added download completion logging with bytes transferred and throughput (#42)
- Added client disconnect handling during streaming downloads (#42)
- Added streaming download tests: range requests, conditional requests, caching headers, download resume (#42)
- Added upload duration and throughput metrics (`duration_ms`, `throughput_mbps`) to upload response (#43)
- Added upload progress logging for large files (hash computation and multipart upload phases) (#43)
- Added client disconnect handling during uploads with proper cleanup (#43)
- Added upload progress tracking endpoint `GET /upload/{upload_id}/progress` for resumable uploads (#43)
- Added large file upload tests (10MB, 100MB, 1GB) with multipart upload verification (#43)
- Added upload cancellation and timeout handling tests (#43)
- Added comprehensive API documentation for upload endpoints with curl, Python, and JavaScript examples (#43)
- Added `package_versions` table for immutable version tracking separate from mutable tags (#56)
- Versions are set at upload time via explicit `version` parameter or auto-detected from filename/metadata
- Version detection priority: explicit parameter > package metadata > filename pattern
- Versions are immutable once created (unlike tags which can be moved)
- Added version API endpoints (#56):
- `GET /api/v1/project/{project}/{package}/versions` - List all versions for a package
- `GET /api/v1/project/{project}/{package}/versions/{version}` - Get specific version details
- `DELETE /api/v1/project/{project}/{package}/versions/{version}` - Delete a version (admin only)
- Added version support to upload endpoint via `version` form parameter (#56)
- Added `version:X.Y.Z` prefix for explicit version resolution in download refs (#56)
- Added version field to tag responses (shows which version the artifact has, if any) (#56)
- Added migration `007_package_versions.sql` with ref_count triggers and data migration from semver tags (#56)
- Added production deployment job triggered by semantic version tags (v1.0.0) with manual approval gate (#63)
- Added production Helm values file with persistence enabled (20Gi PostgreSQL, 100Gi MinIO) (#63)
- Added integration tests for production deployment (#63)
- Added GitLab CI pipeline for feature branch deployments to dev namespace (#51)
- Added `deploy_feature` job with dynamic hostnames and unique release names (#51)
- Added `cleanup_feature` job with `on_stop` for automatic cleanup on merge (#51)
- Added `values-dev.yaml` Helm values for lightweight ephemeral environments (#51)
- Added main branch deployment to stage environment (#51)
- Added post-deployment integration tests (#51)
- Added internal proxy configuration for npm, pip, helm, and apt (#51)
### Changed
- CI integration tests now run full pytest suite (~350 tests) against deployed environment instead of 3 smoke tests
- CI production deployment uses lightweight smoke tests only (no test data creation in prod)
- CI pipeline improvements: shared pip cache, `interruptible` flag on test jobs, retry on integration tests
- Simplified deploy verification to health check only (full checks done by integration tests)
- Extracted environment URLs to global variables for maintainability
- Made `cleanup_feature` job standalone (no longer inherits deploy template dependencies)
- Renamed `integration_test_prod` to `smoke_test_prod` for clarity
- Updated download ref resolution to check versions before tags (version → tag → artifact ID) (#56)
- Deploy jobs now require all security scans to pass before deployment (added test_image, app_deps_scan, cve_scan, cve_sbom_analysis, app_sbom_analysis to dependencies) (#63)
- Increased deploy job timeout from 5m to 10m (#63)
- Added `--atomic` flag to Helm deployments for automatic rollback on failure
- Adjusted dark mode color palette to use lighter background tones for better readability and reduced eye strain (#52)
- Replaced project card grid with sortable data table on Home page for better handling of large project lists
- Replaced package card grid with sortable data table on Project page for consistency
- Replaced SortDropdown with table header sorting on Package page for consistency
- Enabled sorting on supported table columns (name, created, updated) via clickable headers
- Updated browser tab title to "Orchard" with custom favicon
- Improved pod naming: Orchard pods now named `orchard-{env}-server-*` for clarity (#51)
### Fixed
- Fixed CI integration test rate limiting: added configurable `ORCHARD_LOGIN_RATE_LIMIT` env var, relaxed to 1000/minute for dev/stage
- Fixed duplicate `TestSecurityEdgeCases` class definition in test_auth_api.py
- Fixed integration tests auth: session-scoped client, configurable credentials via env vars, fail-fast on auth errors
- Fixed 413 Request Entity Too Large errors on uploads by adding `proxy-body-size: "0"` nginx annotation to Orchard ingress
- Fixed CI tests that require direct S3 access: added `@pytest.mark.requires_direct_s3` marker and excluded from CI
- Fixed ref_count triggers not being created: added auto-migration for tags ref_count trigger functions
- Fixed Content-Disposition header encoding for non-ASCII filenames using RFC 5987 (#38)
- Fixed deploy jobs running even when tests or security scans fail (changed rules from `when: always` to `when: on_success`) (#63)
- Fixed python_tests job not using internal PyPI proxy (#63)
- Fixed `cleanup_feature` job failing when branch is deleted (`GIT_STRATEGY: none`) (#51)
- Fixed gitleaks false positives with fingerprints for historical commits (#51)
- Fixed integration tests running when deploy fails (`when: on_success`) (#51)
- Fixed static file serving for favicon and other files in frontend dist root
- Fixed deploy jobs running when secrets scan fails (added `secrets` to deploy dependencies)
- Fixed dev environment memory requests to equal limits per cluster Kyverno policy
- Fixed init containers missing resource limits (Kyverno policy compliance)
- Fixed Python SyntaxWarning for invalid escape sequence in database migration regex pattern
### Removed
- Removed unused `store_streaming()` method from storage.py (#51)
## [0.4.0] - 2026-01-12
### Added

View File

@@ -1,7 +1,7 @@
# Frontend build stage
FROM containers.global.bsf.tools/node:20-alpine AS frontend-builder
ARG NPM_REGISTRY=https://deps.global.bsf.tools/artifactory/api/npm/registry.npmjs.org
ARG NPM_REGISTRY=https://deps.global.bsf.tools/artifactory/api/npm/registry.npmjs.org/
WORKDIR /app/frontend
@@ -21,13 +21,6 @@ RUN npm run build
# Runtime stage
FROM containers.global.bsf.tools/python:3.12-slim
ARG PIP_INDEX_URL=https://deps.global.bsf.tools/artifactory/api/pypi/pypi.org/simple
# Configure apt to use internal Debian mirrors only (trixie = Debian testing)
RUN printf 'deb https://deps.global.bsf.tools/artifactory/deb.debian.org-debian trixie main\n\
deb https://deps.global.bsf.tools/artifactory/security.debian.org-debian-security trixie-security main\n' > /etc/apt/sources.list \
&& rm -rf /etc/apt/sources.list.d/* /var/lib/apt/lists/*
# Disable proxy cache
RUN printf 'Acquire::http::Pipeline-Depth 0;\nAcquire::http::No-Cache true;\nAcquire::BrokenProxy true;\n' > /etc/apt/apt.conf.d/99fixbadproxy
@@ -45,7 +38,7 @@ WORKDIR /app
# Copy requirements and install Python dependencies
COPY backend/requirements.txt .
RUN pip install --no-cache-dir --index-url "$PIP_INDEX_URL" -r requirements.txt
RUN pip install --no-cache-dir -r requirements.txt
# Copy backend source
COPY backend/ ./backend/

119
README.md
View File

@@ -22,7 +22,6 @@ Orchard is a centralized binary artifact storage system that provides content-ad
- **Package** - Named collection within a project
- **Artifact** - Specific content instance identified by SHA256
- **Tags** - Alias system for referencing artifacts by human-readable names (e.g., `v1.0.0`, `latest`, `stable`)
- **Versions** - Immutable version records set at upload time (explicit or auto-detected from filename/metadata), separate from mutable tags
- **Package Formats & Platforms** - Packages can be tagged with format (npm, pypi, docker, deb, rpm, etc.) and platform (linux, darwin, windows, etc.)
- **Rich Package Metadata** - Package listings include aggregated stats (tag count, artifact count, total size, latest tag)
- **S3-Compatible Backend** - Uses MinIO (or any S3-compatible storage) for artifact storage
@@ -47,12 +46,6 @@ Orchard is a centralized binary artifact storage system that provides content-ad
- `.whl` - Python wheels (name, version, author)
- `.jar` - Java JARs (manifest info, Maven coordinates)
- `.zip` - ZIP files (file count, uncompressed size)
- **Authentication** - Multiple authentication methods:
- Session-based login with username/password
- API keys for programmatic access (`orch_` prefixed tokens)
- OIDC integration for SSO
- Admin user management
- **Garbage Collection** - Clean up orphaned artifacts (ref_count=0) via admin API
### API Endpoints
@@ -74,9 +67,6 @@ Orchard is a centralized binary artifact storage system that provides content-ad
| `POST` | `/api/v1/project/:project/:package/tags` | Create a tag |
| `GET` | `/api/v1/project/:project/:package/tags/:tag_name` | Get single tag with artifact metadata |
| `GET` | `/api/v1/project/:project/:package/tags/:tag_name/history` | Get tag change history |
| `GET` | `/api/v1/project/:project/:package/versions` | List all versions for a package |
| `GET` | `/api/v1/project/:project/:package/versions/:version` | Get specific version details |
| `DELETE` | `/api/v1/project/:project/:package/versions/:version` | Delete a version (admin only) |
| `GET` | `/api/v1/project/:project/:package/artifacts` | List artifacts in package (with filtering) |
| `GET` | `/api/v1/project/:project/:package/consumers` | List consumers of a package |
| `GET` | `/api/v1/artifact/:id` | Get artifact metadata with referencing tags |
@@ -97,14 +87,12 @@ For large files, use the resumable upload API:
When downloading artifacts, the `:ref` parameter supports multiple formats:
- `latest` - Implicit lookup (checks version first, then tag, then artifact ID)
- `v1.0.0` - Implicit lookup (version takes precedence over tag with same name)
- `version:1.0.0` - Explicit version reference
- `latest` - Tag name directly
- `v1.0.0` - Version tag
- `tag:stable` - Explicit tag reference
- `version:2024.1` - Version reference
- `artifact:a3f5d8e12b4c6789...` - Direct SHA256 hash reference
**Resolution order for implicit refs:** version → tag → artifact ID
## Quick Start
### Prerequisites
@@ -236,16 +224,9 @@ curl "http://localhost:8080/api/v1/project/my-project/packages/releases?include_
### Upload an Artifact
```bash
# Upload with tag only (version auto-detected from filename)
curl -X POST http://localhost:8080/api/v1/project/my-project/releases/upload \
-F "file=@./build/app-v1.0.0.tar.gz" \
-F "tag=latest"
# Upload with explicit version and tag
curl -X POST http://localhost:8080/api/v1/project/my-project/releases/upload \
-F "file=@./build/app-v1.0.0.tar.gz" \
-F "tag=latest" \
-F "version=1.0.0"
-F "tag=v1.0.0"
```
Response:
@@ -255,9 +236,7 @@ Response:
"size": 1048576,
"project": "my-project",
"package": "releases",
"tag": "latest",
"version": "1.0.0",
"version_source": "explicit",
"tag": "v1.0.0",
"format_metadata": {
"format": "tarball",
"package_name": "app",
@@ -415,38 +394,6 @@ curl http://localhost:8080/api/v1/project/my-project/releases/tags/latest/histor
Returns list of artifact changes for the tag (most recent first).
### List Versions
```bash
# Basic listing
curl http://localhost:8080/api/v1/project/my-project/releases/versions
# With pagination and sorting
curl "http://localhost:8080/api/v1/project/my-project/releases/versions?sort=version&order=desc"
```
Response includes tags pointing to each version's artifact:
```json
{
"items": [
{
"id": "uuid",
"package_id": "uuid",
"version": "1.0.0",
"version_source": "explicit",
"artifact_id": "a3f5d8e...",
"size": 1048576,
"content_type": "application/gzip",
"original_name": "app-v1.0.0.tar.gz",
"created_at": "2025-01-01T00:00:00Z",
"created_by": "user",
"tags": ["latest", "stable"]
}
],
"pagination": {"page": 1, "limit": 20, "total": 1, "total_pages": 1}
}
```
### List Artifacts in Package
```bash
@@ -575,48 +522,15 @@ Configuration is provided via environment variables prefixed with `ORCHARD_`:
| `ORCHARD_DOWNLOAD_MODE` | Download mode: `presigned`, `redirect`, or `proxy` | `presigned` |
| `ORCHARD_PRESIGNED_URL_EXPIRY` | Presigned URL expiry in seconds | `3600` |
## CI/CD Pipeline
The GitLab CI/CD pipeline automates building, testing, and deploying Orchard.
### Pipeline Stages
| Stage | Jobs | Description |
|-------|------|-------------|
| lint | `kics`, `hadolint`, `secrets` | Security and code quality scanning |
| build | `build_image` | Build and push Docker image |
| test | `python_tests`, `frontend_tests` | Run unit tests with coverage |
| deploy | `deploy_stage`, `deploy_feature` | Deploy to Kubernetes |
| deploy | `integration_test_*` | Post-deployment integration tests |
### Environments
| Environment | Branch | Namespace | URL |
|-------------|--------|-----------|-----|
| Stage | `main` | `orch-stage-namespace` | `orchard-stage.common.global.bsf.tools` |
| Feature | `*` (non-main) | `orch-dev-namespace` | `orchard-{branch}.common.global.bsf.tools` |
### Feature Branch Workflow
1. Push a feature branch
2. Pipeline builds, tests, and deploys to isolated environment
3. Integration tests run against the deployed environment
4. GitLab UI shows environment link for manual testing
5. On merge to main, environment is automatically cleaned up
6. Environments also auto-expire after 1 week if branch is not deleted
### Manual Cleanup
Feature environments can be manually cleaned up via:
- GitLab UI: Environments → Stop environment
- CLI: `helm uninstall orchard-{branch} -n orch-dev-namespace`
## Kubernetes Deployment
### Using Helm
```bash
# Update dependencies (uses internal OCI registry)
# Add Bitnami repo for dependencies
helm repo add bitnami https://charts.bitnami.com/bitnami
# Update dependencies
cd helm/orchard
helm dependency update
@@ -661,8 +575,7 @@ See `helm/orchard/values.yaml` for all configuration options.
- **projects** - Top-level organizational containers
- **packages** - Collections within projects
- **artifacts** - Content-addressable artifacts (SHA256)
- **tags** - Mutable aliases pointing to artifacts
- **package_versions** - Immutable version records (set at upload time)
- **tags** - Aliases pointing to artifacts
- **tag_history** - Audit trail for tag changes
- **uploads** - Upload event records
- **consumers** - Dependency tracking
@@ -680,16 +593,10 @@ The following features are planned but not yet implemented:
- [ ] Export/Import for air-gapped systems
- [ ] Consumer notification
- [ ] Automated update propagation
- [ ] SAML authentication
- [ ] OIDC/SAML authentication
- [ ] API key management
- [ ] Redis caching layer
- [ ] Download integrity verification (see `docs/design/integrity-verification.md`)
### Recently Implemented
- [x] OIDC authentication
- [x] API key management
- [x] Garbage collection for orphaned artifacts
- [x] User authentication with sessions
- [ ] Garbage collection for orphaned artifacts
## License

View File

@@ -151,140 +151,6 @@ def _run_migrations():
END IF;
END $$;
""",
# Add package_versions indexes and triggers (007_package_versions.sql)
"""
DO $$
BEGIN
-- Create indexes for package_versions if table exists
IF EXISTS (SELECT 1 FROM information_schema.tables WHERE table_name = 'package_versions') THEN
-- Indexes for common queries
IF NOT EXISTS (SELECT 1 FROM pg_indexes WHERE indexname = 'idx_package_versions_package_id') THEN
CREATE INDEX idx_package_versions_package_id ON package_versions(package_id);
END IF;
IF NOT EXISTS (SELECT 1 FROM pg_indexes WHERE indexname = 'idx_package_versions_artifact_id') THEN
CREATE INDEX idx_package_versions_artifact_id ON package_versions(artifact_id);
END IF;
IF NOT EXISTS (SELECT 1 FROM pg_indexes WHERE indexname = 'idx_package_versions_package_version') THEN
CREATE INDEX idx_package_versions_package_version ON package_versions(package_id, version);
END IF;
END IF;
END $$;
""",
# Create ref_count trigger functions for tags (ensures triggers exist even if initial migration wasn't run)
"""
CREATE OR REPLACE FUNCTION increment_artifact_ref_count()
RETURNS TRIGGER AS $$
BEGIN
UPDATE artifacts SET ref_count = ref_count + 1 WHERE id = NEW.artifact_id;
RETURN NEW;
END;
$$ LANGUAGE plpgsql;
""",
"""
CREATE OR REPLACE FUNCTION decrement_artifact_ref_count()
RETURNS TRIGGER AS $$
BEGIN
UPDATE artifacts SET ref_count = ref_count - 1 WHERE id = OLD.artifact_id;
RETURN OLD;
END;
$$ LANGUAGE plpgsql;
""",
"""
CREATE OR REPLACE FUNCTION update_artifact_ref_count()
RETURNS TRIGGER AS $$
BEGIN
IF OLD.artifact_id != NEW.artifact_id THEN
UPDATE artifacts SET ref_count = ref_count - 1 WHERE id = OLD.artifact_id;
UPDATE artifacts SET ref_count = ref_count + 1 WHERE id = NEW.artifact_id;
END IF;
RETURN NEW;
END;
$$ LANGUAGE plpgsql;
""",
# Create triggers for tags ref_count management
"""
DO $$
BEGIN
-- Drop and recreate triggers to ensure they're current
DROP TRIGGER IF EXISTS tags_ref_count_insert_trigger ON tags;
CREATE TRIGGER tags_ref_count_insert_trigger
AFTER INSERT ON tags
FOR EACH ROW
EXECUTE FUNCTION increment_artifact_ref_count();
DROP TRIGGER IF EXISTS tags_ref_count_delete_trigger ON tags;
CREATE TRIGGER tags_ref_count_delete_trigger
AFTER DELETE ON tags
FOR EACH ROW
EXECUTE FUNCTION decrement_artifact_ref_count();
DROP TRIGGER IF EXISTS tags_ref_count_update_trigger ON tags;
CREATE TRIGGER tags_ref_count_update_trigger
AFTER UPDATE ON tags
FOR EACH ROW
WHEN (OLD.artifact_id IS DISTINCT FROM NEW.artifact_id)
EXECUTE FUNCTION update_artifact_ref_count();
END $$;
""",
# Create ref_count trigger functions for package_versions
"""
CREATE OR REPLACE FUNCTION increment_version_ref_count()
RETURNS TRIGGER AS $$
BEGIN
UPDATE artifacts SET ref_count = ref_count + 1 WHERE id = NEW.artifact_id;
RETURN NEW;
END;
$$ LANGUAGE plpgsql;
""",
"""
CREATE OR REPLACE FUNCTION decrement_version_ref_count()
RETURNS TRIGGER AS $$
BEGIN
UPDATE artifacts SET ref_count = ref_count - 1 WHERE id = OLD.artifact_id;
RETURN OLD;
END;
$$ LANGUAGE plpgsql;
""",
# Create triggers for package_versions ref_count
"""
DO $$
BEGIN
IF EXISTS (SELECT 1 FROM information_schema.tables WHERE table_name = 'package_versions') THEN
-- Drop and recreate triggers to ensure they're current
DROP TRIGGER IF EXISTS package_versions_ref_count_insert ON package_versions;
CREATE TRIGGER package_versions_ref_count_insert
AFTER INSERT ON package_versions
FOR EACH ROW
EXECUTE FUNCTION increment_version_ref_count();
DROP TRIGGER IF EXISTS package_versions_ref_count_delete ON package_versions;
CREATE TRIGGER package_versions_ref_count_delete
AFTER DELETE ON package_versions
FOR EACH ROW
EXECUTE FUNCTION decrement_version_ref_count();
END IF;
END $$;
""",
# Migrate existing semver tags to package_versions
r"""
DO $$
BEGIN
IF EXISTS (SELECT 1 FROM information_schema.tables WHERE table_name = 'package_versions') THEN
-- Migrate tags that look like versions (v1.0.0, 1.2.3, 2.0.0-beta, etc.)
INSERT INTO package_versions (package_id, artifact_id, version, version_source, created_by, created_at)
SELECT
t.package_id,
t.artifact_id,
CASE WHEN t.name LIKE 'v%' THEN substring(t.name from 2) ELSE t.name END,
'migrated_from_tag',
t.created_by,
t.created_at
FROM tags t
WHERE t.name ~ '^v?[0-9]+\.[0-9]+(\.[0-9]+)?([-.][a-zA-Z0-9]+)?$'
ON CONFLICT (package_id, version) DO NOTHING;
END IF;
END $$;
""",
]
with engine.connect() as conn:

View File

@@ -88,11 +88,6 @@ if os.path.exists(static_dir):
raise HTTPException(status_code=404, detail="Not found")
# Check if requesting a static file from dist root (favicon, etc.)
static_file_path = os.path.join(static_dir, full_path)
if os.path.isfile(static_file_path) and not full_path.startswith("."):
return FileResponse(static_file_path)
# Serve SPA for all other routes (including /project/*)
index_path = os.path.join(static_dir, "index.html")
if os.path.exists(index_path):

View File

@@ -245,10 +245,9 @@ def extract_tarball_metadata(file: BinaryIO, filename: str) -> Dict[str, Any]:
break
# Try to split name and version
# Handle optional 'v' prefix on version (e.g., package-v1.0.0)
patterns = [
r"^(.+)-v?(\d+\.\d+(?:\.\d+)?(?:[-_]\w+)?)$", # name-version or name-vversion
r"^(.+)_v?(\d+\.\d+(?:\.\d+)?(?:[-_]\w+)?)$", # name_version or name_vversion
r"^(.+)-(\d+\.\d+(?:\.\d+)?(?:[-._]\w+)?)$", # name-version
r"^(.+)_(\d+\.\d+(?:\.\d+)?(?:[-._]\w+)?)$", # name_version
]
for pattern in patterns:

View File

@@ -72,9 +72,6 @@ class Package(Base):
consumers = relationship(
"Consumer", back_populates="package", cascade="all, delete-orphan"
)
versions = relationship(
"PackageVersion", back_populates="package", cascade="all, delete-orphan"
)
__table_args__ = (
Index("idx_packages_project_id", "project_id"),
@@ -116,7 +113,6 @@ class Artifact(Base):
tags = relationship("Tag", back_populates="artifact")
uploads = relationship("Upload", back_populates="artifact")
versions = relationship("PackageVersion", back_populates="artifact")
@property
def sha256(self) -> str:
@@ -201,38 +197,6 @@ class TagHistory(Base):
)
class PackageVersion(Base):
"""Immutable version record for a package-artifact relationship.
Separates versions (immutable, set at upload) from tags (mutable labels).
Each artifact in a package can have at most one version.
"""
__tablename__ = "package_versions"
id = Column(UUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
package_id = Column(
UUID(as_uuid=True),
ForeignKey("packages.id", ondelete="CASCADE"),
nullable=False,
)
artifact_id = Column(String(64), ForeignKey("artifacts.id"), nullable=False)
version = Column(String(255), nullable=False)
version_source = Column(String(50)) # 'explicit', 'filename', 'metadata', 'migrated_from_tag'
created_at = Column(DateTime(timezone=True), default=datetime.utcnow)
created_by = Column(String(255), nullable=False)
package = relationship("Package", back_populates="versions")
artifact = relationship("Artifact", back_populates="versions")
__table_args__ = (
Index("idx_package_versions_package_id", "package_id"),
Index("idx_package_versions_artifact_id", "artifact_id"),
Index("idx_package_versions_package_version", "package_id", "version", unique=True),
Index("idx_package_versions_package_artifact", "package_id", "artifact_id", unique=True),
)
class Upload(Base):
__tablename__ = "uploads"

File diff suppressed because it is too large Load Diff

View File

@@ -173,7 +173,6 @@ class TagResponse(BaseModel):
artifact_id: str
created_at: datetime
created_by: str
version: Optional[str] = None # Version of the artifact this tag points to
class Config:
from_attributes = True
@@ -188,7 +187,6 @@ class TagDetailResponse(BaseModel):
artifact_id: str
created_at: datetime
created_by: str
version: Optional[str] = None # Version of the artifact this tag points to
# Artifact metadata
artifact_size: int
artifact_content_type: Optional[str]
@@ -385,7 +383,6 @@ class GlobalTagResponse(BaseModel):
package_name: str
artifact_size: Optional[int] = None
artifact_content_type: Optional[str] = None
version: Optional[str] = None # Version of the artifact this tag points to
class Config:
from_attributes = True
@@ -399,8 +396,6 @@ class UploadResponse(BaseModel):
project: str
package: str
tag: Optional[str]
version: Optional[str] = None # Version assigned to this artifact
version_source: Optional[str] = None # How version was determined: 'explicit', 'filename', 'metadata'
checksum_md5: Optional[str] = None
checksum_sha1: Optional[str] = None
s3_etag: Optional[str] = None
@@ -412,9 +407,6 @@ class UploadResponse(BaseModel):
content_type: Optional[str] = None
original_name: Optional[str] = None
created_at: Optional[datetime] = None
# Upload metrics (Issue #43)
duration_ms: Optional[int] = None # Upload duration in milliseconds
throughput_mbps: Optional[float] = None # Upload throughput in MB/s
# Resumable upload schemas
@@ -426,7 +418,6 @@ class ResumableUploadInitRequest(BaseModel):
content_type: Optional[str] = None
size: int
tag: Optional[str] = None
version: Optional[str] = None # Explicit version (auto-detected if not provided)
@field_validator("expected_hash")
@classmethod
@@ -481,21 +472,6 @@ class ResumableUploadStatusResponse(BaseModel):
total_uploaded_bytes: int
class UploadProgressResponse(BaseModel):
"""Progress information for an in-flight upload"""
upload_id: str
status: str # 'in_progress', 'completed', 'failed', 'not_found'
bytes_uploaded: int = 0
bytes_total: Optional[int] = None
percent_complete: Optional[float] = None
parts_uploaded: int = 0
parts_total: Optional[int] = None
started_at: Optional[datetime] = None
elapsed_seconds: Optional[float] = None
throughput_mbps: Optional[float] = None
# Consumer schemas
class ConsumerResponse(BaseModel):
id: UUID
@@ -508,35 +484,6 @@ class ConsumerResponse(BaseModel):
from_attributes = True
# Package version schemas
class PackageVersionResponse(BaseModel):
"""Immutable version record for an artifact in a package"""
id: UUID
package_id: UUID
artifact_id: str
version: str
version_source: Optional[str] = None # 'explicit', 'filename', 'metadata', 'migrated_from_tag'
created_at: datetime
created_by: str
# Enriched fields from joins
size: Optional[int] = None
content_type: Optional[str] = None
original_name: Optional[str] = None
tags: List[str] = [] # Tag names pointing to this artifact
class Config:
from_attributes = True
class PackageVersionDetailResponse(PackageVersionResponse):
"""Version with full artifact metadata"""
format_metadata: Optional[Dict[str, Any]] = None
checksum_md5: Optional[str] = None
checksum_sha1: Optional[str] = None
# Global search schemas
class SearchResultProject(BaseModel):
"""Project result for global search"""

View File

@@ -5,7 +5,7 @@ import hashlib
import logging
from sqlalchemy.orm import Session
from .models import Project, Package, Artifact, Tag, Upload, PackageVersion
from .models import Project, Package, Artifact, Tag, Upload
from .storage import get_storage
logger = logging.getLogger(__name__)
@@ -74,7 +74,7 @@ TEST_PROJECTS = [
},
]
# Sample artifacts to create (content, tags, version)
# Sample artifacts to create (content, tags)
TEST_ARTIFACTS = [
{
"project": "frontend-libs",
@@ -83,7 +83,6 @@ TEST_ARTIFACTS = [
"filename": "ui-components-1.0.0.js",
"content_type": "application/javascript",
"tags": ["v1.0.0", "latest"],
"version": "1.0.0",
},
{
"project": "frontend-libs",
@@ -92,7 +91,6 @@ TEST_ARTIFACTS = [
"filename": "ui-components-1.1.0.js",
"content_type": "application/javascript",
"tags": ["v1.1.0"],
"version": "1.1.0",
},
{
"project": "frontend-libs",
@@ -101,7 +99,6 @@ TEST_ARTIFACTS = [
"filename": "tokens.json",
"content_type": "application/json",
"tags": ["v1.0.0", "latest"],
"version": "1.0.0",
},
{
"project": "backend-services",
@@ -110,7 +107,6 @@ TEST_ARTIFACTS = [
"filename": "utils-2.0.0.py",
"content_type": "text/x-python",
"tags": ["v2.0.0", "stable", "latest"],
"version": "2.0.0",
},
{
"project": "backend-services",
@@ -119,7 +115,6 @@ TEST_ARTIFACTS = [
"filename": "auth-lib-1.0.0.go",
"content_type": "text/x-go",
"tags": ["v1.0.0", "latest"],
"version": "1.0.0",
},
]
@@ -165,10 +160,9 @@ def seed_database(db: Session) -> None:
logger.info(f"Created {len(project_map)} projects and {len(package_map)} packages")
# Create artifacts, tags, and versions
# Create artifacts and tags
artifact_count = 0
tag_count = 0
version_count = 0
for artifact_data in TEST_ARTIFACTS:
project = project_map[artifact_data["project"]]
@@ -190,11 +184,6 @@ def seed_database(db: Session) -> None:
logger.warning(f"Failed to store artifact in S3: {e}")
continue
# Calculate ref_count: tags + version (if present)
ref_count = len(artifact_data["tags"])
if artifact_data.get("version"):
ref_count += 1
# Create artifact record
artifact = Artifact(
id=sha256_hash,
@@ -203,7 +192,7 @@ def seed_database(db: Session) -> None:
original_name=artifact_data["filename"],
created_by="seed-user",
s3_key=s3_key,
ref_count=ref_count,
ref_count=len(artifact_data["tags"]),
)
db.add(artifact)
@@ -217,18 +206,6 @@ def seed_database(db: Session) -> None:
db.add(upload)
artifact_count += 1
# Create version record if specified
if artifact_data.get("version"):
version = PackageVersion(
package_id=package.id,
artifact_id=sha256_hash,
version=artifact_data["version"],
version_source="explicit",
created_by="seed-user",
)
db.add(version)
version_count += 1
# Create tags
for tag_name in artifact_data["tags"]:
tag = Tag(
@@ -241,5 +218,5 @@ def seed_database(db: Session) -> None:
tag_count += 1
db.commit()
logger.info(f"Created {artifact_count} artifacts, {tag_count} tags, and {version_count} versions")
logger.info(f"Created {artifact_count} artifacts and {tag_count} tags")
logger.info("Database seeding complete")

View File

@@ -6,6 +6,7 @@ from typing import (
Optional,
Dict,
Any,
Generator,
NamedTuple,
Protocol,
runtime_checkable,
@@ -242,19 +243,15 @@ class S3Storage:
},
)
# Build client kwargs - only include credentials if explicitly provided
# This allows IRSA/IAM role credentials to be used when no explicit creds are set
client_kwargs = {
"endpoint_url": settings.s3_endpoint if settings.s3_endpoint else None,
"region_name": settings.s3_region,
"config": config,
"verify": settings.s3_verify_ssl,
}
if settings.s3_access_key_id and settings.s3_secret_access_key:
client_kwargs["aws_access_key_id"] = settings.s3_access_key_id
client_kwargs["aws_secret_access_key"] = settings.s3_secret_access_key
self.client = boto3.client("s3", **client_kwargs)
self.client = boto3.client(
"s3",
endpoint_url=settings.s3_endpoint if settings.s3_endpoint else None,
region_name=settings.s3_region,
aws_access_key_id=settings.s3_access_key_id,
aws_secret_access_key=settings.s3_secret_access_key,
config=config,
verify=settings.s3_verify_ssl, # SSL/TLS verification
)
self.bucket = settings.s3_bucket
# Store active multipart uploads for resumable support
self._active_uploads: Dict[str, Dict[str, Any]] = {}
@@ -382,16 +379,10 @@ class S3Storage:
"""
# First pass: compute all hashes by streaming through file
try:
import time
sha256_hasher = hashlib.sha256()
md5_hasher = hashlib.md5()
sha1_hasher = hashlib.sha1()
size = 0
hash_start_time = time.time()
last_log_time = hash_start_time
log_interval_seconds = 5 # Log progress every 5 seconds
logger.info(f"Computing hashes for large file: expected_size={content_length}")
# Read file in chunks to compute hashes
while True:
@@ -403,18 +394,6 @@ class S3Storage:
sha1_hasher.update(chunk)
size += len(chunk)
# Log hash computation progress periodically
current_time = time.time()
if current_time - last_log_time >= log_interval_seconds:
elapsed = current_time - hash_start_time
percent = (size / content_length) * 100 if content_length > 0 else 0
throughput = (size / (1024 * 1024)) / elapsed if elapsed > 0 else 0
logger.info(
f"Hash computation progress: bytes={size}/{content_length} ({percent:.1f}%) "
f"throughput={throughput:.2f}MB/s"
)
last_log_time = current_time
# Enforce file size limit during streaming (protection against spoofing)
if size > settings.max_file_size:
raise FileSizeExceededError(
@@ -427,14 +406,6 @@ class S3Storage:
sha256_hash = sha256_hasher.hexdigest()
md5_hash = md5_hasher.hexdigest()
sha1_hash = sha1_hasher.hexdigest()
# Log hash computation completion
hash_elapsed = time.time() - hash_start_time
hash_throughput = (size / (1024 * 1024)) / hash_elapsed if hash_elapsed > 0 else 0
logger.info(
f"Hash computation completed: hash={sha256_hash[:16]}... "
f"size={size} duration={hash_elapsed:.2f}s throughput={hash_throughput:.2f}MB/s"
)
except (HashComputationError, FileSizeExceededError):
raise
except Exception as e:
@@ -488,19 +459,8 @@ class S3Storage:
upload_id = mpu["UploadId"]
try:
import time
parts = []
part_number = 1
bytes_uploaded = 0
upload_start_time = time.time()
last_log_time = upload_start_time
log_interval_seconds = 5 # Log progress every 5 seconds
total_parts = (content_length + MULTIPART_CHUNK_SIZE - 1) // MULTIPART_CHUNK_SIZE
logger.info(
f"Starting multipart upload: hash={sha256_hash[:16]}... "
f"size={content_length} parts={total_parts}"
)
while True:
chunk = file.read(MULTIPART_CHUNK_SIZE)
@@ -520,32 +480,8 @@ class S3Storage:
"ETag": response["ETag"],
}
)
bytes_uploaded += len(chunk)
# Log progress periodically
current_time = time.time()
if current_time - last_log_time >= log_interval_seconds:
elapsed = current_time - upload_start_time
percent = (bytes_uploaded / content_length) * 100
throughput = (bytes_uploaded / (1024 * 1024)) / elapsed if elapsed > 0 else 0
logger.info(
f"Upload progress: hash={sha256_hash[:16]}... "
f"part={part_number}/{total_parts} "
f"bytes={bytes_uploaded}/{content_length} ({percent:.1f}%) "
f"throughput={throughput:.2f}MB/s"
)
last_log_time = current_time
part_number += 1
# Log completion
total_elapsed = time.time() - upload_start_time
final_throughput = (content_length / (1024 * 1024)) / total_elapsed if total_elapsed > 0 else 0
logger.info(
f"Multipart upload completed: hash={sha256_hash[:16]}... "
f"size={content_length} duration={total_elapsed:.2f}s throughput={final_throughput:.2f}MB/s"
)
# Complete multipart upload
complete_response = self.client.complete_multipart_upload(
Bucket=self.bucket,
@@ -567,29 +503,134 @@ class S3Storage:
except Exception as e:
# Abort multipart upload on failure
error_str = str(e).lower()
is_client_disconnect = (
isinstance(e, (ConnectionResetError, BrokenPipeError)) or
"connection" in error_str or "broken pipe" in error_str or "reset" in error_str
logger.error(f"Multipart upload failed: {e}")
self.client.abort_multipart_upload(
Bucket=self.bucket,
Key=s3_key,
UploadId=upload_id,
)
if is_client_disconnect:
logger.warning(
f"Multipart upload aborted (client disconnect): hash={sha256_hash[:16]}... "
f"parts_uploaded={len(parts)} bytes_uploaded={bytes_uploaded}"
)
else:
logger.error(f"Multipart upload failed: hash={sha256_hash[:16]}... error={e}")
raise
def store_streaming(self, chunks: Generator[bytes, None, None]) -> StorageResult:
"""
Store a file from a stream of chunks.
First accumulates to compute hash, then uploads.
For truly large files, consider using initiate_resumable_upload instead.
"""
# Accumulate chunks and compute all hashes
sha256_hasher = hashlib.sha256()
md5_hasher = hashlib.md5()
sha1_hasher = hashlib.sha1()
all_chunks = []
size = 0
for chunk in chunks:
sha256_hasher.update(chunk)
md5_hasher.update(chunk)
sha1_hasher.update(chunk)
all_chunks.append(chunk)
size += len(chunk)
sha256_hash = sha256_hasher.hexdigest()
md5_hash = md5_hasher.hexdigest()
sha1_hash = sha1_hasher.hexdigest()
s3_key = f"fruits/{sha256_hash[:2]}/{sha256_hash[2:4]}/{sha256_hash}"
s3_etag = None
# Check if already exists
if self._exists(s3_key):
obj_info = self.get_object_info(s3_key)
s3_etag = obj_info.get("etag", "").strip('"') if obj_info else None
return StorageResult(
sha256=sha256_hash,
size=size,
s3_key=s3_key,
md5=md5_hash,
sha1=sha1_hash,
s3_etag=s3_etag,
already_existed=True,
)
# Upload based on size
if size < MULTIPART_THRESHOLD:
content = b"".join(all_chunks)
response = self.client.put_object(
Bucket=self.bucket, Key=s3_key, Body=content
)
s3_etag = response.get("ETag", "").strip('"')
else:
# Use multipart for large files
mpu = self.client.create_multipart_upload(Bucket=self.bucket, Key=s3_key)
upload_id = mpu["UploadId"]
try:
parts = []
part_number = 1
buffer = b""
for chunk in all_chunks:
buffer += chunk
while len(buffer) >= MULTIPART_CHUNK_SIZE:
part_data = buffer[:MULTIPART_CHUNK_SIZE]
buffer = buffer[MULTIPART_CHUNK_SIZE:]
response = self.client.upload_part(
Bucket=self.bucket,
Key=s3_key,
UploadId=upload_id,
PartNumber=part_number,
Body=part_data,
)
parts.append(
{
"PartNumber": part_number,
"ETag": response["ETag"],
}
)
part_number += 1
# Upload remaining buffer
if buffer:
response = self.client.upload_part(
Bucket=self.bucket,
Key=s3_key,
UploadId=upload_id,
PartNumber=part_number,
Body=buffer,
)
parts.append(
{
"PartNumber": part_number,
"ETag": response["ETag"],
}
)
complete_response = self.client.complete_multipart_upload(
Bucket=self.bucket,
Key=s3_key,
UploadId=upload_id,
MultipartUpload={"Parts": parts},
)
s3_etag = complete_response.get("ETag", "").strip('"')
except Exception as e:
logger.error(f"Streaming multipart upload failed: {e}")
self.client.abort_multipart_upload(
Bucket=self.bucket,
Key=s3_key,
UploadId=upload_id,
)
logger.info(f"Multipart upload aborted and cleaned up: upload_id={upload_id[:16]}...")
except Exception as abort_error:
logger.error(f"Failed to abort multipart upload: {abort_error}")
raise
raise
return StorageResult(
sha256=sha256_hash,
size=size,
s3_key=s3_key,
md5=md5_hash,
sha1=sha1_hash,
s3_etag=s3_etag,
already_existed=False,
)
def initiate_resumable_upload(self, expected_hash: str) -> Dict[str, Any]:
"""
@@ -610,17 +651,12 @@ class S3Storage:
mpu = self.client.create_multipart_upload(Bucket=self.bucket, Key=s3_key)
upload_id = mpu["UploadId"]
import time
session = {
"upload_id": upload_id,
"s3_key": s3_key,
"already_exists": False,
"parts": [],
"expected_hash": expected_hash,
"started_at": time.time(),
"bytes_uploaded": 0,
"expected_size": None, # Set when init provides size
"status": "in_progress",
}
self._active_uploads[upload_id] = session
return session
@@ -647,57 +683,10 @@ class S3Storage:
part_info = {
"PartNumber": part_number,
"ETag": response["ETag"],
"size": len(data),
}
session["parts"].append(part_info)
session["bytes_uploaded"] = session.get("bytes_uploaded", 0) + len(data)
return part_info
def get_upload_progress(self, upload_id: str) -> Optional[Dict[str, Any]]:
"""
Get progress information for a resumable upload.
Returns None if upload not found.
"""
import time
session = self._active_uploads.get(upload_id)
if not session:
return None
bytes_uploaded = session.get("bytes_uploaded", 0)
expected_size = session.get("expected_size")
started_at = session.get("started_at")
progress = {
"upload_id": upload_id,
"status": session.get("status", "in_progress"),
"bytes_uploaded": bytes_uploaded,
"bytes_total": expected_size,
"parts_uploaded": len(session.get("parts", [])),
"parts_total": None,
"started_at": started_at,
"elapsed_seconds": None,
"percent_complete": None,
"throughput_mbps": None,
}
if expected_size and expected_size > 0:
progress["percent_complete"] = round((bytes_uploaded / expected_size) * 100, 2)
progress["parts_total"] = (expected_size + MULTIPART_CHUNK_SIZE - 1) // MULTIPART_CHUNK_SIZE
if started_at:
elapsed = time.time() - started_at
progress["elapsed_seconds"] = round(elapsed, 2)
if elapsed > 0 and bytes_uploaded > 0:
progress["throughput_mbps"] = round((bytes_uploaded / (1024 * 1024)) / elapsed, 2)
return progress
def set_upload_expected_size(self, upload_id: str, size: int):
"""Set the expected size for an upload (for progress tracking)."""
session = self._active_uploads.get(upload_id)
if session:
session["expected_size"] = size
def complete_resumable_upload(self, upload_id: str) -> Tuple[str, str]:
"""
Complete a resumable upload.
@@ -835,36 +824,6 @@ class S3Storage:
except ClientError:
return False
def delete_all(self) -> int:
"""
Delete all objects in the bucket.
Returns:
Number of objects deleted
"""
deleted_count = 0
try:
paginator = self.client.get_paginator("list_objects_v2")
for page in paginator.paginate(Bucket=self.bucket):
objects = page.get("Contents", [])
if not objects:
continue
# Delete objects in batches of 1000 (S3 limit)
delete_keys = [{"Key": obj["Key"]} for obj in objects]
if delete_keys:
self.client.delete_objects(
Bucket=self.bucket, Delete={"Objects": delete_keys}
)
deleted_count += len(delete_keys)
logger.info(f"Deleted {len(delete_keys)} objects from S3")
logger.info(f"Total objects deleted from S3: {deleted_count}")
return deleted_count
except ClientError as e:
logger.error(f"Failed to delete all S3 objects: {e}")
raise
def generate_presigned_url(
self,
s3_key: str,

View File

@@ -12,8 +12,6 @@ markers =
unit: Unit tests (no external dependencies)
integration: Integration tests (require database/storage)
slow: Slow tests (skip with -m "not slow")
large: Large file tests (100MB+, skip with -m "not large")
concurrent: Concurrent operation tests
# Coverage configuration
[coverage:run]

View File

@@ -9,37 +9,6 @@ This module provides:
import os
import pytest
# =============================================================================
# Pytest Markers
# =============================================================================
def pytest_configure(config):
"""Register custom pytest markers."""
config.addinivalue_line(
"markers",
"auth_intensive: marks tests that make many login requests (excluded from CI integration tests due to rate limiting)",
)
config.addinivalue_line(
"markers",
"integration: marks tests as integration tests",
)
config.addinivalue_line(
"markers",
"large: marks tests that handle large files (slow)",
)
config.addinivalue_line(
"markers",
"slow: marks tests as slow running",
)
config.addinivalue_line(
"markers",
"requires_direct_s3: marks tests that require direct S3/MinIO access (skipped in CI where S3 is not directly accessible)",
)
import io
from typing import Generator
from unittest.mock import MagicMock
@@ -63,8 +32,6 @@ from tests.factories import (
compute_md5,
compute_sha1,
upload_test_file,
generate_content,
generate_content_with_hash,
TEST_CONTENT_HELLO,
TEST_HASH_HELLO,
TEST_MD5_HELLO,
@@ -212,64 +179,29 @@ def test_app():
# =============================================================================
@pytest.fixture(scope="session")
@pytest.fixture
def integration_client():
"""
Create an authenticated test client for integration tests.
Uses the real database and MinIO from docker-compose.local.yml or deployed environment.
Authenticates as admin for write operations. Session-scoped to reuse login across tests.
Environment variables:
ORCHARD_TEST_URL: Base URL of the Orchard server (default: http://localhost:8080)
ORCHARD_TEST_USERNAME: Admin username for authentication (default: admin)
ORCHARD_TEST_PASSWORD: Admin password for authentication (default: changeme123)
Uses the real database and MinIO from docker-compose.local.yml.
Authenticates as admin for write operations.
"""
import httpx
from httpx import Client
# Connect to the running orchard-server container or deployed environment
# Connect to the running orchard-server container
base_url = os.environ.get("ORCHARD_TEST_URL", "http://localhost:8080")
username = os.environ.get("ORCHARD_TEST_USERNAME", "admin")
password = os.environ.get("ORCHARD_TEST_PASSWORD", "changeme123")
with httpx.Client(base_url=base_url, timeout=30.0) as client:
with Client(base_url=base_url, timeout=30.0) as client:
# Login as admin to enable write operations
login_response = client.post(
"/api/v1/auth/login",
json={"username": username, "password": password},
json={"username": "admin", "password": "changeme123"},
)
# If login fails, tests will fail - that's expected if auth is broken
if login_response.status_code != 200:
pytest.fail(
f"Authentication failed against {base_url}: {login_response.status_code} - {login_response.text}. "
f"Set ORCHARD_TEST_USERNAME and ORCHARD_TEST_PASSWORD environment variables if using non-default credentials."
)
# Verify cookie was set
if not client.cookies:
pytest.fail(
f"Login succeeded but no session cookie was set. Response headers: {login_response.headers}"
)
yield client
@pytest.fixture
def auth_client():
"""
Create a function-scoped test client for authentication tests.
Unlike integration_client (session-scoped), this creates a fresh client
for each test. Use this for tests that manipulate authentication state
(login, logout, cookie clearing) to avoid polluting other tests.
Environment variables:
ORCHARD_TEST_URL: Base URL of the Orchard server (default: http://localhost:8080)
"""
import httpx
base_url = os.environ.get("ORCHARD_TEST_URL", "http://localhost:8080")
with httpx.Client(base_url=base_url, timeout=30.0) as client:
# Try to continue without auth for backward compatibility
pass
yield client
@@ -339,41 +271,3 @@ def test_content():
content = f"test-content-{uuid.uuid4().hex}".encode()
sha256 = compute_sha256(content)
return (content, sha256)
@pytest.fixture
def sized_content():
"""
Factory fixture for generating content of specific sizes.
Usage:
def test_example(sized_content):
content, hash = sized_content(1024) # 1KB
content, hash = sized_content(1024 * 1024) # 1MB
"""
def _generate(size: int, seed: int = None):
return generate_content_with_hash(size, seed)
return _generate
# =============================================================================
# Size Constants for Tests
# =============================================================================
# Common file sizes for boundary testing
SIZE_1B = 1
SIZE_1KB = 1024
SIZE_10KB = 10 * 1024
SIZE_100KB = 100 * 1024
SIZE_1MB = 1024 * 1024
SIZE_5MB = 5 * 1024 * 1024
SIZE_10MB = 10 * 1024 * 1024
SIZE_50MB = 50 * 1024 * 1024
SIZE_100MB = 100 * 1024 * 1024
SIZE_250MB = 250 * 1024 * 1024
SIZE_500MB = 500 * 1024 * 1024
SIZE_1GB = 1024 * 1024 * 1024
# Chunk size boundaries (based on typical S3 multipart chunk sizes)
CHUNK_SIZE = 64 * 1024 # 64KB typical chunk
MULTIPART_THRESHOLD = 100 * 1024 * 1024 # 100MB multipart threshold

View File

@@ -97,7 +97,6 @@ def upload_test_file(
content: bytes,
filename: str = "test.bin",
tag: Optional[str] = None,
version: Optional[str] = None,
) -> dict:
"""
Helper function to upload a test file via the API.
@@ -109,7 +108,6 @@ def upload_test_file(
content: File content as bytes
filename: Original filename
tag: Optional tag to assign
version: Optional version to assign
Returns:
The upload response as a dict
@@ -118,8 +116,6 @@ def upload_test_file(
data = {}
if tag:
data["tag"] = tag
if version:
data["version"] = version
response = client.post(
f"/api/v1/project/{project}/{package}/upload",
@@ -130,41 +126,6 @@ def upload_test_file(
return response.json()
def generate_content(size: int, seed: Optional[int] = None) -> bytes:
"""
Generate deterministic or random content of a specified size.
Args:
size: Size of content in bytes
seed: Optional seed for reproducible content (None for random)
Returns:
Bytes of the specified size
"""
if size == 0:
return b""
if seed is not None:
import random
rng = random.Random(seed)
return bytes(rng.randint(0, 255) for _ in range(size))
return os.urandom(size)
def generate_content_with_hash(size: int, seed: Optional[int] = None) -> tuple[bytes, str]:
"""
Generate content of specified size and compute its SHA256 hash.
Args:
size: Size of content in bytes
seed: Optional seed for reproducible content
Returns:
Tuple of (content_bytes, sha256_hash)
"""
content = generate_content(size, seed)
return content, compute_sha256(content)
# =============================================================================
# Project/Package Factories
# =============================================================================

View File

@@ -1,25 +1,16 @@
"""Integration tests for authentication API endpoints.
Note: These tests are marked as auth_intensive because they make many login
requests. Dev/stage deployments have relaxed rate limits (1000/minute) to
allow these tests to run. Production uses strict rate limits (5/minute).
"""
"""Integration tests for authentication API endpoints."""
import pytest
from uuid import uuid4
# Mark all tests in this module as auth_intensive (informational, not excluded from CI)
pytestmark = pytest.mark.auth_intensive
class TestAuthLogin:
"""Tests for login endpoint."""
@pytest.mark.integration
def test_login_success(self, auth_client):
def test_login_success(self, integration_client):
"""Test successful login with default admin credentials."""
response = auth_client.post(
response = integration_client.post(
"/api/v1/auth/login",
json={"username": "admin", "password": "changeme123"},
)
@@ -30,9 +21,9 @@ class TestAuthLogin:
assert "orchard_session" in response.cookies
@pytest.mark.integration
def test_login_invalid_password(self, auth_client):
def test_login_invalid_password(self, integration_client):
"""Test login with wrong password."""
response = auth_client.post(
response = integration_client.post(
"/api/v1/auth/login",
json={"username": "admin", "password": "wrongpassword"},
)
@@ -40,9 +31,9 @@ class TestAuthLogin:
assert "Invalid username or password" in response.json()["detail"]
@pytest.mark.integration
def test_login_nonexistent_user(self, auth_client):
def test_login_nonexistent_user(self, integration_client):
"""Test login with non-existent user."""
response = auth_client.post(
response = integration_client.post(
"/api/v1/auth/login",
json={"username": "nonexistent", "password": "password"},
)
@@ -53,24 +44,24 @@ class TestAuthLogout:
"""Tests for logout endpoint."""
@pytest.mark.integration
def test_logout_success(self, auth_client):
def test_logout_success(self, integration_client):
"""Test successful logout."""
# First login
login_response = auth_client.post(
login_response = integration_client.post(
"/api/v1/auth/login",
json={"username": "admin", "password": "changeme123"},
)
assert login_response.status_code == 200
# Then logout
logout_response = auth_client.post("/api/v1/auth/logout")
logout_response = integration_client.post("/api/v1/auth/logout")
assert logout_response.status_code == 200
assert "Logged out successfully" in logout_response.json()["message"]
@pytest.mark.integration
def test_logout_without_session(self, auth_client):
def test_logout_without_session(self, integration_client):
"""Test logout without being logged in."""
response = auth_client.post("/api/v1/auth/logout")
response = integration_client.post("/api/v1/auth/logout")
# Should succeed even without session
assert response.status_code == 200
@@ -79,15 +70,15 @@ class TestAuthMe:
"""Tests for get current user endpoint."""
@pytest.mark.integration
def test_get_me_authenticated(self, auth_client):
def test_get_me_authenticated(self, integration_client):
"""Test getting current user when authenticated."""
# Login first
auth_client.post(
integration_client.post(
"/api/v1/auth/login",
json={"username": "admin", "password": "changeme123"},
)
response = auth_client.get("/api/v1/auth/me")
response = integration_client.get("/api/v1/auth/me")
assert response.status_code == 200
data = response.json()
assert data["username"] == "admin"
@@ -96,88 +87,67 @@ class TestAuthMe:
assert "created_at" in data
@pytest.mark.integration
def test_get_me_unauthenticated(self, auth_client):
def test_get_me_unauthenticated(self, integration_client):
"""Test getting current user without authentication."""
# Clear any existing cookies
auth_client.cookies.clear()
integration_client.cookies.clear()
response = auth_client.get("/api/v1/auth/me")
response = integration_client.get("/api/v1/auth/me")
assert response.status_code == 401
assert "Not authenticated" in response.json()["detail"]
class TestAuthChangePassword:
"""Tests for change password endpoint.
Note: These tests use dedicated test users instead of admin to avoid
invalidating the integration_client session (which uses admin).
"""
"""Tests for change password endpoint."""
@pytest.mark.integration
def test_change_password_success(self, auth_client):
def test_change_password_success(self, integration_client):
"""Test successful password change."""
# Login as admin to create a test user
auth_client.post(
# Login first
integration_client.post(
"/api/v1/auth/login",
json={"username": "admin", "password": "changeme123"},
)
test_username = f"pwchange_{uuid4().hex[:8]}"
auth_client.post(
"/api/v1/admin/users",
json={"username": test_username, "password": "oldpassword123"},
)
# Login as test user
auth_client.cookies.clear()
auth_client.post(
"/api/v1/auth/login",
json={"username": test_username, "password": "oldpassword123"},
)
# Change password
response = auth_client.post(
response = integration_client.post(
"/api/v1/auth/change-password",
json={"current_password": "oldpassword123", "new_password": "newpassword123"},
json={"current_password": "changeme123", "new_password": "newpassword123"},
)
assert response.status_code == 200
# Verify old password no longer works
auth_client.cookies.clear()
response = auth_client.post(
integration_client.cookies.clear()
response = integration_client.post(
"/api/v1/auth/login",
json={"username": test_username, "password": "oldpassword123"},
json={"username": "admin", "password": "changeme123"},
)
assert response.status_code == 401
# Verify new password works
response = auth_client.post(
response = integration_client.post(
"/api/v1/auth/login",
json={"username": test_username, "password": "newpassword123"},
json={"username": "admin", "password": "newpassword123"},
)
assert response.status_code == 200
# Reset password back to original for other tests
reset_response = integration_client.post(
"/api/v1/auth/change-password",
json={"current_password": "newpassword123", "new_password": "changeme123"},
)
assert reset_response.status_code == 200, "Failed to reset admin password back to default"
@pytest.mark.integration
def test_change_password_wrong_current(self, auth_client):
def test_change_password_wrong_current(self, integration_client):
"""Test password change with wrong current password."""
# Login as admin to create a test user
auth_client.post(
# Login first
integration_client.post(
"/api/v1/auth/login",
json={"username": "admin", "password": "changeme123"},
)
test_username = f"pwwrong_{uuid4().hex[:8]}"
auth_client.post(
"/api/v1/admin/users",
json={"username": test_username, "password": "password123"},
)
# Login as test user
auth_client.cookies.clear()
auth_client.post(
"/api/v1/auth/login",
json={"username": test_username, "password": "password123"},
)
response = auth_client.post(
response = integration_client.post(
"/api/v1/auth/change-password",
json={"current_password": "wrongpassword", "new_password": "newpassword"},
)
@@ -189,16 +159,16 @@ class TestAPIKeys:
"""Tests for API key management endpoints."""
@pytest.mark.integration
def test_create_and_list_api_key(self, auth_client):
def test_create_and_list_api_key(self, integration_client):
"""Test creating and listing API keys."""
# Login first
auth_client.post(
integration_client.post(
"/api/v1/auth/login",
json={"username": "admin", "password": "changeme123"},
)
# Create API key
create_response = auth_client.post(
create_response = integration_client.post(
"/api/v1/auth/keys",
json={"name": "test-key", "description": "Test API key"},
)
@@ -212,23 +182,23 @@ class TestAPIKeys:
api_key = data["key"]
# List API keys
list_response = auth_client.get("/api/v1/auth/keys")
list_response = integration_client.get("/api/v1/auth/keys")
assert list_response.status_code == 200
keys = list_response.json()
assert any(k["id"] == key_id for k in keys)
# Clean up - delete the key
auth_client.delete(f"/api/v1/auth/keys/{key_id}")
integration_client.delete(f"/api/v1/auth/keys/{key_id}")
@pytest.mark.integration
def test_use_api_key_for_auth(self, auth_client):
def test_use_api_key_for_auth(self, integration_client):
"""Test using API key for authentication."""
# Login and create API key
auth_client.post(
integration_client.post(
"/api/v1/auth/login",
json={"username": "admin", "password": "changeme123"},
)
create_response = auth_client.post(
create_response = integration_client.post(
"/api/v1/auth/keys",
json={"name": "auth-test-key"},
)
@@ -236,8 +206,8 @@ class TestAPIKeys:
key_id = create_response.json()["id"]
# Clear cookies and use API key
auth_client.cookies.clear()
response = auth_client.get(
integration_client.cookies.clear()
response = integration_client.get(
"/api/v1/auth/me",
headers={"Authorization": f"Bearer {api_key}"},
)
@@ -245,21 +215,21 @@ class TestAPIKeys:
assert response.json()["username"] == "admin"
# Clean up
auth_client.post(
integration_client.post(
"/api/v1/auth/login",
json={"username": "admin", "password": "changeme123"},
)
auth_client.delete(f"/api/v1/auth/keys/{key_id}")
integration_client.delete(f"/api/v1/auth/keys/{key_id}")
@pytest.mark.integration
def test_delete_api_key(self, auth_client):
def test_delete_api_key(self, integration_client):
"""Test revoking an API key."""
# Login and create API key
auth_client.post(
integration_client.post(
"/api/v1/auth/login",
json={"username": "admin", "password": "changeme123"},
)
create_response = auth_client.post(
create_response = integration_client.post(
"/api/v1/auth/keys",
json={"name": "delete-test-key"},
)
@@ -267,12 +237,12 @@ class TestAPIKeys:
api_key = create_response.json()["key"]
# Delete the key
delete_response = auth_client.delete(f"/api/v1/auth/keys/{key_id}")
delete_response = integration_client.delete(f"/api/v1/auth/keys/{key_id}")
assert delete_response.status_code == 200
# Verify key no longer works
auth_client.cookies.clear()
response = auth_client.get(
integration_client.cookies.clear()
response = integration_client.get(
"/api/v1/auth/me",
headers={"Authorization": f"Bearer {api_key}"},
)
@@ -283,32 +253,32 @@ class TestAdminUserManagement:
"""Tests for admin user management endpoints."""
@pytest.mark.integration
def test_list_users(self, auth_client):
def test_list_users(self, integration_client):
"""Test listing users as admin."""
# Login as admin
auth_client.post(
integration_client.post(
"/api/v1/auth/login",
json={"username": "admin", "password": "changeme123"},
)
response = auth_client.get("/api/v1/admin/users")
response = integration_client.get("/api/v1/admin/users")
assert response.status_code == 200
users = response.json()
assert len(users) >= 1
assert any(u["username"] == "admin" for u in users)
@pytest.mark.integration
def test_create_user(self, auth_client):
def test_create_user(self, integration_client):
"""Test creating a new user as admin."""
# Login as admin
auth_client.post(
integration_client.post(
"/api/v1/auth/login",
json={"username": "admin", "password": "changeme123"},
)
# Create new user
test_username = f"testuser_{uuid4().hex[:8]}"
response = auth_client.post(
response = integration_client.post(
"/api/v1/admin/users",
json={
"username": test_username,
@@ -323,31 +293,31 @@ class TestAdminUserManagement:
assert data["is_admin"] is False
# Verify new user can login
auth_client.cookies.clear()
login_response = auth_client.post(
integration_client.cookies.clear()
login_response = integration_client.post(
"/api/v1/auth/login",
json={"username": test_username, "password": "testpassword"},
)
assert login_response.status_code == 200
@pytest.mark.integration
def test_update_user(self, auth_client):
def test_update_user(self, integration_client):
"""Test updating a user as admin."""
# Login as admin
auth_client.post(
integration_client.post(
"/api/v1/auth/login",
json={"username": "admin", "password": "changeme123"},
)
# Create a test user
test_username = f"updateuser_{uuid4().hex[:8]}"
auth_client.post(
integration_client.post(
"/api/v1/admin/users",
json={"username": test_username, "password": "password"},
)
# Update the user
response = auth_client.put(
response = integration_client.put(
f"/api/v1/admin/users/{test_username}",
json={"email": "updated@example.com", "is_admin": True},
)
@@ -357,59 +327,59 @@ class TestAdminUserManagement:
assert data["is_admin"] is True
@pytest.mark.integration
def test_reset_user_password(self, auth_client):
def test_reset_user_password(self, integration_client):
"""Test resetting a user's password as admin."""
# Login as admin
auth_client.post(
integration_client.post(
"/api/v1/auth/login",
json={"username": "admin", "password": "changeme123"},
)
# Create a test user
test_username = f"resetuser_{uuid4().hex[:8]}"
auth_client.post(
integration_client.post(
"/api/v1/admin/users",
json={"username": test_username, "password": "oldpassword"},
)
# Reset password
response = auth_client.post(
response = integration_client.post(
f"/api/v1/admin/users/{test_username}/reset-password",
json={"new_password": "newpassword"},
)
assert response.status_code == 200
# Verify new password works
auth_client.cookies.clear()
login_response = auth_client.post(
integration_client.cookies.clear()
login_response = integration_client.post(
"/api/v1/auth/login",
json={"username": test_username, "password": "newpassword"},
)
assert login_response.status_code == 200
@pytest.mark.integration
def test_non_admin_cannot_access_admin_endpoints(self, auth_client):
def test_non_admin_cannot_access_admin_endpoints(self, integration_client):
"""Test that non-admin users cannot access admin endpoints."""
# Login as admin and create non-admin user
auth_client.post(
integration_client.post(
"/api/v1/auth/login",
json={"username": "admin", "password": "changeme123"},
)
test_username = f"nonadmin_{uuid4().hex[:8]}"
auth_client.post(
integration_client.post(
"/api/v1/admin/users",
json={"username": test_username, "password": "password", "is_admin": False},
)
# Login as non-admin
auth_client.cookies.clear()
auth_client.post(
integration_client.cookies.clear()
integration_client.post(
"/api/v1/auth/login",
json={"username": test_username, "password": "password"},
)
# Try to access admin endpoints
response = auth_client.get("/api/v1/admin/users")
response = integration_client.get("/api/v1/admin/users")
assert response.status_code == 403
assert "Admin privileges required" in response.json()["detail"]
@@ -418,28 +388,28 @@ class TestSecurityEdgeCases:
"""Tests for security edge cases and validation."""
@pytest.mark.integration
def test_login_inactive_user(self, auth_client):
def test_login_inactive_user(self, integration_client):
"""Test that inactive users cannot login."""
# Login as admin and create a user
auth_client.post(
integration_client.post(
"/api/v1/auth/login",
json={"username": "admin", "password": "changeme123"},
)
test_username = f"inactive_{uuid4().hex[:8]}"
auth_client.post(
integration_client.post(
"/api/v1/admin/users",
json={"username": test_username, "password": "password123"},
)
# Deactivate the user
auth_client.put(
integration_client.put(
f"/api/v1/admin/users/{test_username}",
json={"is_active": False},
)
# Try to login as inactive user
auth_client.cookies.clear()
response = auth_client.post(
integration_client.cookies.clear()
response = integration_client.post(
"/api/v1/auth/login",
json={"username": test_username, "password": "password123"},
)
@@ -447,14 +417,14 @@ class TestSecurityEdgeCases:
assert "Invalid username or password" in response.json()["detail"]
@pytest.mark.integration
def test_password_too_short_on_create(self, auth_client):
def test_password_too_short_on_create(self, integration_client):
"""Test that short passwords are rejected when creating users."""
auth_client.post(
integration_client.post(
"/api/v1/auth/login",
json={"username": "admin", "password": "changeme123"},
)
response = auth_client.post(
response = integration_client.post(
"/api/v1/admin/users",
json={"username": f"shortpw_{uuid4().hex[:8]}", "password": "short"},
)
@@ -462,49 +432,36 @@ class TestSecurityEdgeCases:
assert "at least 8 characters" in response.json()["detail"]
@pytest.mark.integration
def test_password_too_short_on_change(self, auth_client):
def test_password_too_short_on_change(self, integration_client):
"""Test that short passwords are rejected when changing password."""
# Create test user
auth_client.post(
integration_client.post(
"/api/v1/auth/login",
json={"username": "admin", "password": "changeme123"},
)
test_username = f"shortchange_{uuid4().hex[:8]}"
auth_client.post(
"/api/v1/admin/users",
json={"username": test_username, "password": "password123"},
)
# Login as test user
auth_client.cookies.clear()
auth_client.post(
"/api/v1/auth/login",
json={"username": test_username, "password": "password123"},
)
response = auth_client.post(
response = integration_client.post(
"/api/v1/auth/change-password",
json={"current_password": "password123", "new_password": "short"},
json={"current_password": "changeme123", "new_password": "short"},
)
assert response.status_code == 400
assert "at least 8 characters" in response.json()["detail"]
@pytest.mark.integration
def test_password_too_short_on_reset(self, auth_client):
def test_password_too_short_on_reset(self, integration_client):
"""Test that short passwords are rejected when resetting password."""
auth_client.post(
integration_client.post(
"/api/v1/auth/login",
json={"username": "admin", "password": "changeme123"},
)
# Create a test user first
test_username = f"resetshort_{uuid4().hex[:8]}"
auth_client.post(
integration_client.post(
"/api/v1/admin/users",
json={"username": test_username, "password": "password123"},
)
response = auth_client.post(
response = integration_client.post(
f"/api/v1/admin/users/{test_username}/reset-password",
json={"new_password": "short"},
)
@@ -512,23 +469,23 @@ class TestSecurityEdgeCases:
assert "at least 8 characters" in response.json()["detail"]
@pytest.mark.integration
def test_duplicate_username_rejected(self, auth_client):
def test_duplicate_username_rejected(self, integration_client):
"""Test that duplicate usernames are rejected."""
auth_client.post(
integration_client.post(
"/api/v1/auth/login",
json={"username": "admin", "password": "changeme123"},
)
test_username = f"duplicate_{uuid4().hex[:8]}"
# Create user first time
response1 = auth_client.post(
response1 = integration_client.post(
"/api/v1/admin/users",
json={"username": test_username, "password": "password123"},
)
assert response1.status_code == 200
# Try to create same username again
response2 = auth_client.post(
response2 = integration_client.post(
"/api/v1/admin/users",
json={"username": test_username, "password": "password456"},
)
@@ -536,14 +493,14 @@ class TestSecurityEdgeCases:
assert "already exists" in response2.json()["detail"]
@pytest.mark.integration
def test_cannot_delete_other_users_api_key(self, auth_client):
def test_cannot_delete_other_users_api_key(self, integration_client):
"""Test that users cannot delete API keys owned by other users."""
# Login as admin and create an API key
auth_client.post(
integration_client.post(
"/api/v1/auth/login",
json={"username": "admin", "password": "changeme123"},
)
create_response = auth_client.post(
create_response = integration_client.post(
"/api/v1/auth/keys",
json={"name": "admin-key"},
)
@@ -551,65 +508,253 @@ class TestSecurityEdgeCases:
# Create a non-admin user
test_username = f"nonadmin_{uuid4().hex[:8]}"
auth_client.post(
integration_client.post(
"/api/v1/admin/users",
json={"username": test_username, "password": "password123"},
)
# Login as non-admin
auth_client.cookies.clear()
auth_client.post(
integration_client.cookies.clear()
integration_client.post(
"/api/v1/auth/login",
json={"username": test_username, "password": "password123"},
)
# Try to delete admin's API key
response = auth_client.delete(f"/api/v1/auth/keys/{admin_key_id}")
response = integration_client.delete(f"/api/v1/auth/keys/{admin_key_id}")
assert response.status_code == 403
assert "Cannot delete another user's API key" in response.json()["detail"]
# Cleanup: login as admin and delete the key
auth_client.cookies.clear()
auth_client.post(
integration_client.cookies.clear()
integration_client.post(
"/api/v1/auth/login",
json={"username": "admin", "password": "changeme123"},
)
auth_client.delete(f"/api/v1/auth/keys/{admin_key_id}")
integration_client.delete(f"/api/v1/auth/keys/{admin_key_id}")
@pytest.mark.integration
def test_sessions_invalidated_on_password_change(self, auth_client):
def test_sessions_invalidated_on_password_change(self, integration_client):
"""Test that all sessions are invalidated when password is changed."""
# Create a test user
auth_client.post(
integration_client.post(
"/api/v1/auth/login",
json={"username": "admin", "password": "changeme123"},
)
test_username = f"sessiontest_{uuid4().hex[:8]}"
auth_client.post(
integration_client.post(
"/api/v1/admin/users",
json={"username": test_username, "password": "password123"},
)
# Login as test user
auth_client.cookies.clear()
login_response = auth_client.post(
integration_client.cookies.clear()
login_response = integration_client.post(
"/api/v1/auth/login",
json={"username": test_username, "password": "password123"},
)
assert login_response.status_code == 200
# Verify session works
me_response = auth_client.get("/api/v1/auth/me")
me_response = integration_client.get("/api/v1/auth/me")
assert me_response.status_code == 200
# Change password
auth_client.post(
integration_client.post(
"/api/v1/auth/change-password",
json={"current_password": "password123", "new_password": "newpassword123"},
)
# Old session should be invalidated - try to access /me
# (note: the change-password call itself may have cleared the session cookie)
me_response2 = auth_client.get("/api/v1/auth/me")
me_response2 = integration_client.get("/api/v1/auth/me")
# This should fail because all sessions were invalidated
assert me_response2.status_code == 401
class TestSecurityEdgeCases:
"""Tests for security edge cases and validation."""
@pytest.mark.integration
def test_login_inactive_user(self, integration_client):
"""Test that inactive users cannot login."""
# Login as admin and create a user
integration_client.post(
"/api/v1/auth/login",
json={"username": "admin", "password": "changeme123"},
)
test_username = f"inactive_{uuid4().hex[:8]}"
integration_client.post(
"/api/v1/admin/users",
json={"username": test_username, "password": "password123"},
)
# Deactivate the user
integration_client.put(
f"/api/v1/admin/users/{test_username}",
json={"is_active": False},
)
# Try to login as inactive user
integration_client.cookies.clear()
response = integration_client.post(
"/api/v1/auth/login",
json={"username": test_username, "password": "password123"},
)
assert response.status_code == 401
assert "Invalid username or password" in response.json()["detail"]
@pytest.mark.integration
def test_password_too_short_on_create(self, integration_client):
"""Test that short passwords are rejected when creating users."""
integration_client.post(
"/api/v1/auth/login",
json={"username": "admin", "password": "changeme123"},
)
response = integration_client.post(
"/api/v1/admin/users",
json={"username": f"shortpw_{uuid4().hex[:8]}", "password": "short"},
)
assert response.status_code == 400
assert "at least 8 characters" in response.json()["detail"]
@pytest.mark.integration
def test_password_too_short_on_change(self, integration_client):
"""Test that short passwords are rejected when changing password."""
integration_client.post(
"/api/v1/auth/login",
json={"username": "admin", "password": "changeme123"},
)
response = integration_client.post(
"/api/v1/auth/change-password",
json={"current_password": "changeme123", "new_password": "short"},
)
assert response.status_code == 400
assert "at least 8 characters" in response.json()["detail"]
@pytest.mark.integration
def test_password_too_short_on_reset(self, integration_client):
"""Test that short passwords are rejected when resetting password."""
integration_client.post(
"/api/v1/auth/login",
json={"username": "admin", "password": "changeme123"},
)
# Create a test user first
test_username = f"resetshort_{uuid4().hex[:8]}"
integration_client.post(
"/api/v1/admin/users",
json={"username": test_username, "password": "password123"},
)
response = integration_client.post(
f"/api/v1/admin/users/{test_username}/reset-password",
json={"new_password": "short"},
)
assert response.status_code == 400
assert "at least 8 characters" in response.json()["detail"]
@pytest.mark.integration
def test_duplicate_username_rejected(self, integration_client):
"""Test that duplicate usernames are rejected."""
integration_client.post(
"/api/v1/auth/login",
json={"username": "admin", "password": "changeme123"},
)
test_username = f"duplicate_{uuid4().hex[:8]}"
# Create user first time
response1 = integration_client.post(
"/api/v1/admin/users",
json={"username": test_username, "password": "password123"},
)
assert response1.status_code == 200
# Try to create same username again
response2 = integration_client.post(
"/api/v1/admin/users",
json={"username": test_username, "password": "password456"},
)
assert response2.status_code == 409
assert "already exists" in response2.json()["detail"]
@pytest.mark.integration
def test_cannot_delete_other_users_api_key(self, integration_client):
"""Test that users cannot delete API keys owned by other users."""
# Login as admin and create an API key
integration_client.post(
"/api/v1/auth/login",
json={"username": "admin", "password": "changeme123"},
)
create_response = integration_client.post(
"/api/v1/auth/keys",
json={"name": "admin-key"},
)
admin_key_id = create_response.json()["id"]
# Create a non-admin user
test_username = f"nonadmin_{uuid4().hex[:8]}"
integration_client.post(
"/api/v1/admin/users",
json={"username": test_username, "password": "password123"},
)
# Login as non-admin
integration_client.cookies.clear()
integration_client.post(
"/api/v1/auth/login",
json={"username": test_username, "password": "password123"},
)
# Try to delete admin's API key
response = integration_client.delete(f"/api/v1/auth/keys/{admin_key_id}")
assert response.status_code == 403
assert "Cannot delete another user's API key" in response.json()["detail"]
# Cleanup: login as admin and delete the key
integration_client.cookies.clear()
integration_client.post(
"/api/v1/auth/login",
json={"username": "admin", "password": "changeme123"},
)
integration_client.delete(f"/api/v1/auth/keys/{admin_key_id}")
@pytest.mark.integration
def test_sessions_invalidated_on_password_change(self, integration_client):
"""Test that all sessions are invalidated when password is changed."""
# Create a test user
integration_client.post(
"/api/v1/auth/login",
json={"username": "admin", "password": "changeme123"},
)
test_username = f"sessiontest_{uuid4().hex[:8]}"
integration_client.post(
"/api/v1/admin/users",
json={"username": test_username, "password": "password123"},
)
# Login as test user
integration_client.cookies.clear()
login_response = integration_client.post(
"/api/v1/auth/login",
json={"username": test_username, "password": "password123"},
)
assert login_response.status_code == 200
# Verify session works
me_response = integration_client.get("/api/v1/auth/me")
assert me_response.status_code == 200
# Change password
integration_client.post(
"/api/v1/auth/change-password",
json={"current_password": "password123", "new_password": "newpassword123"},
)
# Old session should be invalidated - try to access /me
# (note: the change-password call itself may have cleared the session cookie)
me_response2 = integration_client.get("/api/v1/auth/me")
# This should fail because all sessions were invalidated
assert me_response2.status_code == 401

View File

@@ -1,737 +0,0 @@
"""
Integration tests for concurrent upload and download operations.
Tests cover:
- Concurrent uploads of different files
- Concurrent uploads of same file (deduplication race)
- Concurrent downloads of same artifact
- Concurrent downloads of different artifacts
- Mixed concurrent uploads and downloads
- Data corruption prevention under concurrency
"""
import pytest
import io
import os
from concurrent.futures import ThreadPoolExecutor, as_completed
from tests.factories import (
compute_sha256,
upload_test_file,
generate_content_with_hash,
)
def get_api_key(integration_client):
"""Create an API key for concurrent test workers."""
import uuid
response = integration_client.post(
"/api/v1/auth/keys",
json={"name": f"concurrent-test-{uuid.uuid4().hex[:8]}"},
)
if response.status_code == 200:
return response.json()["key"]
return None
class TestConcurrentUploads:
"""Tests for concurrent upload operations."""
@pytest.mark.integration
@pytest.mark.concurrent
def test_2_concurrent_uploads_different_files(self, integration_client, test_package):
"""Test 2 concurrent uploads of different files."""
project, package = test_package
api_key = get_api_key(integration_client)
assert api_key, "Failed to create API key"
files_data = [
generate_content_with_hash(1024, seed=i) for i in range(2)
]
results = []
errors = []
def upload_worker(idx, content, expected_hash):
try:
from httpx import Client
base_url = os.environ.get("ORCHARD_TEST_URL", "http://localhost:8080")
with Client(base_url=base_url, timeout=60.0) as client:
files = {
"file": (f"file-{idx}.bin", io.BytesIO(content), "application/octet-stream")
}
response = client.post(
f"/api/v1/project/{project}/{package}/upload",
files=files,
data={"tag": f"concurrent-{idx}"},
headers={"Authorization": f"Bearer {api_key}"},
)
if response.status_code == 200:
result = response.json()
results.append((idx, result, expected_hash))
else:
errors.append(f"Worker {idx}: Status {response.status_code}: {response.text}")
except Exception as e:
errors.append(f"Worker {idx}: {str(e)}")
with ThreadPoolExecutor(max_workers=2) as executor:
futures = [
executor.submit(upload_worker, i, content, hash)
for i, (content, hash) in enumerate(files_data)
]
for future in as_completed(futures):
pass
assert len(errors) == 0, f"Errors: {errors}"
assert len(results) == 2
# Verify each upload returned correct artifact_id
for idx, result, expected_hash in results:
assert result["artifact_id"] == expected_hash
@pytest.mark.integration
@pytest.mark.concurrent
def test_5_concurrent_uploads_different_files(self, integration_client, test_package):
"""Test 5 concurrent uploads of different files."""
project, package = test_package
api_key = get_api_key(integration_client)
assert api_key, "Failed to create API key"
num_files = 5
files_data = [
generate_content_with_hash(2048, seed=100 + i) for i in range(num_files)
]
results = []
errors = []
def upload_worker(idx, content, expected_hash):
try:
from httpx import Client
base_url = os.environ.get("ORCHARD_TEST_URL", "http://localhost:8080")
with Client(base_url=base_url, timeout=60.0) as client:
files = {
"file": (f"file-{idx}.bin", io.BytesIO(content), "application/octet-stream")
}
response = client.post(
f"/api/v1/project/{project}/{package}/upload",
files=files,
data={"tag": f"concurrent5-{idx}"},
headers={"Authorization": f"Bearer {api_key}"},
)
if response.status_code == 200:
result = response.json()
results.append((idx, result, expected_hash))
else:
errors.append(f"Worker {idx}: Status {response.status_code}")
except Exception as e:
errors.append(f"Worker {idx}: {str(e)}")
with ThreadPoolExecutor(max_workers=num_files) as executor:
futures = [
executor.submit(upload_worker, i, content, hash)
for i, (content, hash) in enumerate(files_data)
]
for future in as_completed(futures):
pass
assert len(errors) == 0, f"Errors: {errors}"
assert len(results) == num_files
# Verify all uploads have unique artifact_ids
artifact_ids = set(r[1]["artifact_id"] for r in results)
assert len(artifact_ids) == num_files
@pytest.mark.integration
@pytest.mark.concurrent
def test_10_concurrent_uploads_different_files(self, integration_client, test_package):
"""Test 10 concurrent uploads of different files."""
project, package = test_package
api_key = get_api_key(integration_client)
assert api_key, "Failed to create API key"
num_files = 10
files_data = [
generate_content_with_hash(1024, seed=200 + i) for i in range(num_files)
]
results = []
errors = []
def upload_worker(idx, content, expected_hash):
try:
from httpx import Client
base_url = os.environ.get("ORCHARD_TEST_URL", "http://localhost:8080")
with Client(base_url=base_url, timeout=60.0) as client:
files = {
"file": (f"file-{idx}.bin", io.BytesIO(content), "application/octet-stream")
}
response = client.post(
f"/api/v1/project/{project}/{package}/upload",
files=files,
data={"tag": f"concurrent10-{idx}"},
headers={"Authorization": f"Bearer {api_key}"},
)
if response.status_code == 200:
result = response.json()
results.append((idx, result, expected_hash))
else:
errors.append(f"Worker {idx}: Status {response.status_code}")
except Exception as e:
errors.append(f"Worker {idx}: {str(e)}")
with ThreadPoolExecutor(max_workers=num_files) as executor:
futures = [
executor.submit(upload_worker, i, content, hash)
for i, (content, hash) in enumerate(files_data)
]
for future in as_completed(futures):
pass
assert len(errors) == 0, f"Errors: {errors}"
assert len(results) == num_files
@pytest.mark.integration
@pytest.mark.concurrent
def test_concurrent_uploads_same_file_deduplication(self, integration_client, test_package):
"""Test concurrent uploads of same file handle deduplication correctly."""
project, package = test_package
api_key = get_api_key(integration_client)
assert api_key, "Failed to create API key"
content, expected_hash = generate_content_with_hash(4096, seed=999)
num_concurrent = 5
results = []
errors = []
def upload_worker(idx):
try:
from httpx import Client
base_url = os.environ.get("ORCHARD_TEST_URL", "http://localhost:8080")
with Client(base_url=base_url, timeout=60.0) as client:
files = {
"file": (f"same-{idx}.bin", io.BytesIO(content), "application/octet-stream")
}
response = client.post(
f"/api/v1/project/{project}/{package}/upload",
files=files,
data={"tag": f"dedup-{idx}"},
headers={"Authorization": f"Bearer {api_key}"},
)
if response.status_code == 200:
results.append(response.json())
else:
errors.append(f"Worker {idx}: Status {response.status_code}")
except Exception as e:
errors.append(f"Worker {idx}: {str(e)}")
with ThreadPoolExecutor(max_workers=num_concurrent) as executor:
futures = [executor.submit(upload_worker, i) for i in range(num_concurrent)]
for future in as_completed(futures):
pass
assert len(errors) == 0, f"Errors: {errors}"
assert len(results) == num_concurrent
# All should have same artifact_id
artifact_ids = set(r["artifact_id"] for r in results)
assert len(artifact_ids) == 1
assert expected_hash in artifact_ids
# Verify final ref_count equals number of uploads
response = integration_client.get(f"/api/v1/artifact/{expected_hash}")
assert response.status_code == 200
assert response.json()["ref_count"] == num_concurrent
@pytest.mark.integration
@pytest.mark.concurrent
def test_concurrent_uploads_to_different_packages(self, integration_client, test_project, unique_test_id):
"""Test concurrent uploads to different packages."""
project = test_project
api_key = get_api_key(integration_client)
assert api_key, "Failed to create API key"
num_packages = 3
package_names = []
# Create multiple packages
for i in range(num_packages):
pkg_name = f"pkg-{unique_test_id}-{i}"
response = integration_client.post(
f"/api/v1/project/{project}/packages",
json={"name": pkg_name, "description": f"Package {i}"},
)
assert response.status_code == 200
package_names.append(pkg_name)
files_data = [
generate_content_with_hash(1024, seed=300 + i) for i in range(num_packages)
]
results = []
errors = []
def upload_worker(idx, package, content, expected_hash):
try:
from httpx import Client
base_url = os.environ.get("ORCHARD_TEST_URL", "http://localhost:8080")
with Client(base_url=base_url, timeout=60.0) as client:
files = {
"file": (f"file-{idx}.bin", io.BytesIO(content), "application/octet-stream")
}
response = client.post(
f"/api/v1/project/{project}/{package}/upload",
files=files,
data={"tag": "latest"},
headers={"Authorization": f"Bearer {api_key}"},
)
if response.status_code == 200:
result = response.json()
results.append((package, result, expected_hash))
else:
errors.append(f"Worker {idx}: Status {response.status_code}")
except Exception as e:
errors.append(f"Worker {idx}: {str(e)}")
with ThreadPoolExecutor(max_workers=num_packages) as executor:
futures = [
executor.submit(upload_worker, i, package_names[i], content, hash)
for i, (content, hash) in enumerate(files_data)
]
for future in as_completed(futures):
pass
assert len(errors) == 0, f"Errors: {errors}"
assert len(results) == num_packages
class TestConcurrentDownloads:
"""Tests for concurrent download operations."""
@pytest.mark.integration
@pytest.mark.concurrent
def test_2_concurrent_downloads_same_artifact(self, integration_client, test_package):
"""Test 2 concurrent downloads of same artifact."""
project, package = test_package
content, expected_hash = generate_content_with_hash(2048, seed=400)
# Upload first
upload_test_file(integration_client, project, package, content, tag="download-test")
results = []
errors = []
def download_worker(idx):
try:
from httpx import Client
base_url = os.environ.get("ORCHARD_TEST_URL", "http://localhost:8080")
with Client(base_url=base_url, timeout=60.0) as client:
response = client.get(
f"/api/v1/project/{project}/{package}/+/download-test",
params={"mode": "proxy"},
)
if response.status_code == 200:
results.append((idx, response.content))
else:
errors.append(f"Worker {idx}: Status {response.status_code}")
except Exception as e:
errors.append(f"Worker {idx}: {str(e)}")
with ThreadPoolExecutor(max_workers=2) as executor:
futures = [executor.submit(download_worker, i) for i in range(2)]
for future in as_completed(futures):
pass
assert len(errors) == 0, f"Errors: {errors}"
assert len(results) == 2
# All downloads should match original
for idx, downloaded in results:
assert downloaded == content
@pytest.mark.integration
@pytest.mark.concurrent
def test_5_concurrent_downloads_same_artifact(self, integration_client, test_package):
"""Test 5 concurrent downloads of same artifact."""
project, package = test_package
content, expected_hash = generate_content_with_hash(4096, seed=500)
upload_test_file(integration_client, project, package, content, tag="download5-test")
num_downloads = 5
results = []
errors = []
def download_worker(idx):
try:
from httpx import Client
base_url = os.environ.get("ORCHARD_TEST_URL", "http://localhost:8080")
with Client(base_url=base_url, timeout=60.0) as client:
response = client.get(
f"/api/v1/project/{project}/{package}/+/download5-test",
params={"mode": "proxy"},
)
if response.status_code == 200:
results.append((idx, response.content))
else:
errors.append(f"Worker {idx}: Status {response.status_code}")
except Exception as e:
errors.append(f"Worker {idx}: {str(e)}")
with ThreadPoolExecutor(max_workers=num_downloads) as executor:
futures = [executor.submit(download_worker, i) for i in range(num_downloads)]
for future in as_completed(futures):
pass
assert len(errors) == 0, f"Errors: {errors}"
assert len(results) == num_downloads
for idx, downloaded in results:
assert downloaded == content
@pytest.mark.integration
@pytest.mark.concurrent
def test_10_concurrent_downloads_same_artifact(self, integration_client, test_package):
"""Test 10 concurrent downloads of same artifact."""
project, package = test_package
content, expected_hash = generate_content_with_hash(8192, seed=600)
upload_test_file(integration_client, project, package, content, tag="download10-test")
num_downloads = 10
results = []
errors = []
def download_worker(idx):
try:
from httpx import Client
base_url = os.environ.get("ORCHARD_TEST_URL", "http://localhost:8080")
with Client(base_url=base_url, timeout=60.0) as client:
response = client.get(
f"/api/v1/project/{project}/{package}/+/download10-test",
params={"mode": "proxy"},
)
if response.status_code == 200:
results.append((idx, response.content))
else:
errors.append(f"Worker {idx}: Status {response.status_code}")
except Exception as e:
errors.append(f"Worker {idx}: {str(e)}")
with ThreadPoolExecutor(max_workers=num_downloads) as executor:
futures = [executor.submit(download_worker, i) for i in range(num_downloads)]
for future in as_completed(futures):
pass
assert len(errors) == 0, f"Errors: {errors}"
assert len(results) == num_downloads
for idx, downloaded in results:
assert downloaded == content
@pytest.mark.integration
@pytest.mark.concurrent
def test_concurrent_downloads_different_artifacts(self, integration_client, test_package):
"""Test concurrent downloads of different artifacts."""
project, package = test_package
# Upload multiple files
num_files = 5
uploads = []
for i in range(num_files):
content, expected_hash = generate_content_with_hash(1024, seed=700 + i)
upload_test_file(
integration_client, project, package, content,
tag=f"multi-download-{i}"
)
uploads.append((f"multi-download-{i}", content))
results = []
errors = []
def download_worker(tag, expected_content):
try:
from httpx import Client
base_url = os.environ.get("ORCHARD_TEST_URL", "http://localhost:8080")
with Client(base_url=base_url, timeout=60.0) as client:
response = client.get(
f"/api/v1/project/{project}/{package}/+/{tag}",
params={"mode": "proxy"},
)
if response.status_code == 200:
results.append((tag, response.content, expected_content))
else:
errors.append(f"Tag {tag}: Status {response.status_code}")
except Exception as e:
errors.append(f"Tag {tag}: {str(e)}")
with ThreadPoolExecutor(max_workers=num_files) as executor:
futures = [
executor.submit(download_worker, tag, content)
for tag, content in uploads
]
for future in as_completed(futures):
pass
assert len(errors) == 0, f"Errors: {errors}"
assert len(results) == num_files
for tag, downloaded, expected in results:
assert downloaded == expected, f"Content mismatch for {tag}"
class TestMixedConcurrentOperations:
"""Tests for mixed concurrent upload and download operations."""
@pytest.mark.integration
@pytest.mark.concurrent
def test_upload_while_download_in_progress(self, integration_client, test_package):
"""Test uploading while a download is in progress."""
project, package = test_package
api_key = get_api_key(integration_client)
assert api_key, "Failed to create API key"
# Upload initial content
content1, hash1 = generate_content_with_hash(10240, seed=800) # 10KB
upload_test_file(integration_client, project, package, content1, tag="initial")
# New content for upload during download
content2, hash2 = generate_content_with_hash(10240, seed=801)
results = {"downloads": [], "uploads": []}
errors = []
def download_worker():
try:
from httpx import Client
base_url = os.environ.get("ORCHARD_TEST_URL", "http://localhost:8080")
with Client(base_url=base_url, timeout=60.0) as client:
response = client.get(
f"/api/v1/project/{project}/{package}/+/initial",
params={"mode": "proxy"},
)
if response.status_code == 200:
results["downloads"].append(response.content)
else:
errors.append(f"Download: Status {response.status_code}")
except Exception as e:
errors.append(f"Download: {str(e)}")
def upload_worker():
try:
from httpx import Client
base_url = os.environ.get("ORCHARD_TEST_URL", "http://localhost:8080")
with Client(base_url=base_url, timeout=60.0) as client:
files = {
"file": ("new.bin", io.BytesIO(content2), "application/octet-stream")
}
response = client.post(
f"/api/v1/project/{project}/{package}/upload",
files=files,
data={"tag": "during-download"},
headers={"Authorization": f"Bearer {api_key}"},
)
if response.status_code == 200:
results["uploads"].append(response.json())
else:
errors.append(f"Upload: Status {response.status_code}")
except Exception as e:
errors.append(f"Upload: {str(e)}")
with ThreadPoolExecutor(max_workers=2) as executor:
futures = [
executor.submit(download_worker),
executor.submit(upload_worker),
]
for future in as_completed(futures):
pass
assert len(errors) == 0, f"Errors: {errors}"
assert len(results["downloads"]) == 1
assert len(results["uploads"]) == 1
# Verify download got correct content
assert results["downloads"][0] == content1
# Verify upload succeeded
assert results["uploads"][0]["artifact_id"] == hash2
@pytest.mark.integration
@pytest.mark.concurrent
def test_multiple_uploads_and_downloads_simultaneously(self, integration_client, test_package):
"""Test multiple uploads and downloads running simultaneously."""
project, package = test_package
api_key = get_api_key(integration_client)
assert api_key, "Failed to create API key"
# Pre-upload some files for downloading
existing_files = []
for i in range(3):
content, hash = generate_content_with_hash(2048, seed=900 + i)
upload_test_file(integration_client, project, package, content, tag=f"existing-{i}")
existing_files.append((f"existing-{i}", content))
# New files for uploading
new_files = [
generate_content_with_hash(2048, seed=910 + i) for i in range(3)
]
results = {"downloads": [], "uploads": []}
errors = []
def download_worker(tag, expected):
try:
from httpx import Client
base_url = os.environ.get("ORCHARD_TEST_URL", "http://localhost:8080")
with Client(base_url=base_url, timeout=60.0) as client:
response = client.get(
f"/api/v1/project/{project}/{package}/+/{tag}",
params={"mode": "proxy"},
)
if response.status_code == 200:
results["downloads"].append((tag, response.content, expected))
else:
errors.append(f"Download {tag}: Status {response.status_code}")
except Exception as e:
errors.append(f"Download {tag}: {str(e)}")
def upload_worker(idx, content, expected_hash):
try:
from httpx import Client
base_url = os.environ.get("ORCHARD_TEST_URL", "http://localhost:8080")
with Client(base_url=base_url, timeout=60.0) as client:
files = {
"file": (f"new-{idx}.bin", io.BytesIO(content), "application/octet-stream")
}
response = client.post(
f"/api/v1/project/{project}/{package}/upload",
files=files,
data={"tag": f"new-{idx}"},
headers={"Authorization": f"Bearer {api_key}"},
)
if response.status_code == 200:
results["uploads"].append((idx, response.json(), expected_hash))
else:
errors.append(f"Upload {idx}: Status {response.status_code}")
except Exception as e:
errors.append(f"Upload {idx}: {str(e)}")
with ThreadPoolExecutor(max_workers=6) as executor:
futures = []
# Submit downloads
for tag, content in existing_files:
futures.append(executor.submit(download_worker, tag, content))
# Submit uploads
for i, (content, hash) in enumerate(new_files):
futures.append(executor.submit(upload_worker, i, content, hash))
for future in as_completed(futures):
pass
assert len(errors) == 0, f"Errors: {errors}"
assert len(results["downloads"]) == 3
assert len(results["uploads"]) == 3
# Verify downloads
for tag, downloaded, expected in results["downloads"]:
assert downloaded == expected, f"Download mismatch for {tag}"
# Verify uploads
for idx, result, expected_hash in results["uploads"]:
assert result["artifact_id"] == expected_hash
@pytest.mark.integration
@pytest.mark.concurrent
def test_no_data_corruption_under_concurrency(self, integration_client, test_package):
"""Test that no data corruption occurs under concurrent operations."""
project, package = test_package
api_key = get_api_key(integration_client)
assert api_key, "Failed to create API key"
# Create content with recognizable patterns
num_files = 5
files_data = []
for i in range(num_files):
# Each file has unique repeating pattern for easy corruption detection
pattern = bytes([i] * 256)
content = pattern * 40 # 10KB each
hash = compute_sha256(content)
files_data.append((content, hash))
results = []
errors = []
def upload_and_verify(idx, content, expected_hash):
try:
from httpx import Client
base_url = os.environ.get("ORCHARD_TEST_URL", "http://localhost:8080")
with Client(base_url=base_url, timeout=60.0) as client:
# Upload
files = {
"file": (f"pattern-{idx}.bin", io.BytesIO(content), "application/octet-stream")
}
upload_resp = client.post(
f"/api/v1/project/{project}/{package}/upload",
files=files,
data={"tag": f"pattern-{idx}"},
headers={"Authorization": f"Bearer {api_key}"},
)
if upload_resp.status_code != 200:
errors.append(f"Upload {idx}: Status {upload_resp.status_code}")
return
upload_result = upload_resp.json()
if upload_result["artifact_id"] != expected_hash:
errors.append(f"Upload {idx}: Hash mismatch")
return
# Immediately download and verify
download_resp = client.get(
f"/api/v1/project/{project}/{package}/+/pattern-{idx}",
params={"mode": "proxy"},
)
if download_resp.status_code != 200:
errors.append(f"Download {idx}: Status {download_resp.status_code}")
return
if download_resp.content != content:
errors.append(f"Worker {idx}: DATA CORRUPTION DETECTED")
return
# Verify the downloaded content hash
downloaded_hash = compute_sha256(download_resp.content)
if downloaded_hash != expected_hash:
errors.append(f"Worker {idx}: Hash verification failed")
return
results.append(idx)
except Exception as e:
errors.append(f"Worker {idx}: {str(e)}")
with ThreadPoolExecutor(max_workers=num_files) as executor:
futures = [
executor.submit(upload_and_verify, i, content, hash)
for i, (content, hash) in enumerate(files_data)
]
for future in as_completed(futures):
pass
assert len(errors) == 0, f"Errors: {errors}"
assert len(results) == num_files

View File

@@ -1,322 +0,0 @@
"""
Integration tests for error handling in upload and download operations.
Tests cover:
- Timeout handling
- Invalid request handling
- Resource cleanup on failures
- Graceful error responses
"""
import pytest
import io
import time
from tests.factories import (
compute_sha256,
upload_test_file,
generate_content_with_hash,
)
class TestUploadErrorHandling:
"""Tests for upload error handling."""
@pytest.mark.integration
def test_upload_to_nonexistent_project_returns_404(
self, integration_client, unique_test_id
):
"""Test upload to nonexistent project returns 404."""
content = b"test content for nonexistent project"
files = {"file": ("test.bin", io.BytesIO(content), "application/octet-stream")}
response = integration_client.post(
f"/api/v1/project/nonexistent-project-{unique_test_id}/nonexistent-pkg/upload",
files=files,
)
assert response.status_code == 404
@pytest.mark.integration
def test_upload_to_nonexistent_package_returns_404(
self, integration_client, test_project, unique_test_id
):
"""Test upload to nonexistent package returns 404."""
content = b"test content for nonexistent package"
files = {"file": ("test.bin", io.BytesIO(content), "application/octet-stream")}
response = integration_client.post(
f"/api/v1/project/{test_project}/nonexistent-package-{unique_test_id}/upload",
files=files,
)
assert response.status_code == 404
@pytest.mark.integration
def test_upload_empty_file_rejected(self, integration_client, test_package):
"""Test empty file upload is rejected."""
project, package = test_package
files = {"file": ("empty.bin", io.BytesIO(b""), "application/octet-stream")}
response = integration_client.post(
f"/api/v1/project/{project}/{package}/upload",
files=files,
)
assert response.status_code in [400, 422]
@pytest.mark.integration
def test_upload_missing_file_returns_422(self, integration_client, test_package):
"""Test upload without file field returns 422."""
project, package = test_package
response = integration_client.post(
f"/api/v1/project/{project}/{package}/upload",
data={"tag": "no-file-provided"},
)
assert response.status_code == 422
@pytest.mark.integration
def test_upload_invalid_checksum_format_returns_400(
self, integration_client, test_package
):
"""Test upload with invalid checksum format returns 400."""
project, package = test_package
content = b"checksum format test"
files = {"file": ("test.bin", io.BytesIO(content), "application/octet-stream")}
response = integration_client.post(
f"/api/v1/project/{project}/{package}/upload",
files=files,
headers={"X-Checksum-SHA256": "invalid-hash-format"},
)
assert response.status_code == 400
@pytest.mark.integration
def test_upload_checksum_mismatch_returns_422(
self, integration_client, test_package
):
"""Test upload with mismatched checksum returns 422."""
project, package = test_package
content = b"checksum mismatch test"
wrong_hash = "0" * 64 # Valid format but wrong hash
files = {"file": ("test.bin", io.BytesIO(content), "application/octet-stream")}
response = integration_client.post(
f"/api/v1/project/{project}/{package}/upload",
files=files,
headers={"X-Checksum-SHA256": wrong_hash},
)
assert response.status_code == 422
@pytest.mark.integration
def test_upload_with_correct_checksum_succeeds(
self, integration_client, test_package
):
"""Test upload with correct checksum succeeds."""
project, package = test_package
content = b"correct checksum test"
correct_hash = compute_sha256(content)
files = {"file": ("test.bin", io.BytesIO(content), "application/octet-stream")}
response = integration_client.post(
f"/api/v1/project/{project}/{package}/upload",
files=files,
headers={"X-Checksum-SHA256": correct_hash},
)
assert response.status_code == 200
assert response.json()["artifact_id"] == correct_hash
class TestDownloadErrorHandling:
"""Tests for download error handling."""
@pytest.mark.integration
def test_download_nonexistent_tag_returns_404(
self, integration_client, test_package
):
"""Test download of nonexistent tag returns 404."""
project, package = test_package
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/nonexistent-tag-xyz"
)
assert response.status_code == 404
@pytest.mark.integration
def test_download_nonexistent_artifact_returns_404(
self, integration_client, test_package
):
"""Test download of nonexistent artifact ID returns 404."""
project, package = test_package
fake_hash = "a" * 64
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/artifact:{fake_hash}"
)
assert response.status_code == 404
@pytest.mark.integration
def test_download_invalid_artifact_id_format(
self, integration_client, test_package
):
"""Test download with invalid artifact ID format."""
project, package = test_package
# Too short
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/artifact:abc123"
)
assert response.status_code == 404
@pytest.mark.integration
def test_download_from_nonexistent_project_returns_404(
self, integration_client, unique_test_id
):
"""Test download from nonexistent project returns 404."""
response = integration_client.get(
f"/api/v1/project/nonexistent-{unique_test_id}/pkg/+/tag"
)
assert response.status_code == 404
@pytest.mark.integration
def test_download_from_nonexistent_package_returns_404(
self, integration_client, test_project, unique_test_id
):
"""Test download from nonexistent package returns 404."""
response = integration_client.get(
f"/api/v1/project/{test_project}/nonexistent-{unique_test_id}/+/tag"
)
assert response.status_code == 404
class TestTimeoutBehavior:
"""Tests for timeout behavior (integration level)."""
@pytest.mark.integration
@pytest.mark.slow
def test_large_upload_completes_within_reasonable_time(
self, integration_client, test_package, sized_content
):
"""Test that a 10MB upload completes within reasonable time."""
project, package = test_package
content, expected_hash = sized_content(10 * 1024 * 1024, seed=999) # 10MB
start_time = time.time()
result = upload_test_file(
integration_client, project, package, content, tag="timeout-test"
)
elapsed = time.time() - start_time
assert result["artifact_id"] == expected_hash
# Should complete within 60 seconds for 10MB on local docker
assert elapsed < 60, f"Upload took too long: {elapsed:.2f}s"
@pytest.mark.integration
@pytest.mark.slow
def test_large_download_completes_within_reasonable_time(
self, integration_client, test_package, sized_content
):
"""Test that a 10MB download completes within reasonable time."""
project, package = test_package
content, expected_hash = sized_content(10 * 1024 * 1024, seed=998) # 10MB
# First upload
upload_test_file(
integration_client, project, package, content, tag="download-timeout-test"
)
# Then download and time it
start_time = time.time()
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/download-timeout-test",
params={"mode": "proxy"},
)
elapsed = time.time() - start_time
assert response.status_code == 200
assert len(response.content) == len(content)
# Should complete within 60 seconds for 10MB on local docker
assert elapsed < 60, f"Download took too long: {elapsed:.2f}s"
class TestResourceCleanup:
"""Tests for proper resource cleanup on failures.
Note: More comprehensive cleanup tests are in test_upload_download_api.py
(TestUploadFailureCleanup class) including S3 object cleanup verification.
"""
@pytest.mark.integration
def test_checksum_mismatch_no_orphaned_artifact(
self, integration_client, test_package, unique_test_id
):
"""Test checksum mismatch doesn't leave orphaned artifact."""
project, package = test_package
# Use unique content to ensure artifact doesn't exist from prior tests
content = f"checksum mismatch orphan test {unique_test_id}".encode()
wrong_hash = "0" * 64
actual_hash = compute_sha256(content)
# Verify artifact doesn't exist before test
pre_check = integration_client.get(f"/api/v1/artifact/{actual_hash}")
assert pre_check.status_code == 404, "Artifact should not exist before test"
files = {"file": ("test.bin", io.BytesIO(content), "application/octet-stream")}
response = integration_client.post(
f"/api/v1/project/{project}/{package}/upload",
files=files,
headers={"X-Checksum-SHA256": wrong_hash},
)
assert response.status_code == 422
# Verify no artifact was created with either hash
response1 = integration_client.get(f"/api/v1/artifact/{wrong_hash}")
response2 = integration_client.get(f"/api/v1/artifact/{actual_hash}")
assert response1.status_code == 404
assert response2.status_code == 404
class TestGracefulErrorResponses:
"""Tests for graceful and informative error responses."""
@pytest.mark.integration
def test_404_response_has_detail_message(
self, integration_client, test_package
):
"""Test 404 responses include a detail message."""
project, package = test_package
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/nonexistent-tag"
)
assert response.status_code == 404
data = response.json()
assert "detail" in data
assert len(data["detail"]) > 0
@pytest.mark.integration
def test_422_response_has_detail_message(self, integration_client, test_package):
"""Test 422 responses include a detail message."""
project, package = test_package
# Upload with mismatched checksum
content = b"detail message test"
wrong_hash = "0" * 64
files = {"file": ("test.bin", io.BytesIO(content), "application/octet-stream")}
response = integration_client.post(
f"/api/v1/project/{project}/{package}/upload",
files=files,
headers={"X-Checksum-SHA256": wrong_hash},
)
assert response.status_code == 422
data = response.json()
assert "detail" in data
@pytest.mark.integration
def test_error_response_is_json(self, integration_client, unique_test_id):
"""Test error responses are valid JSON."""
response = integration_client.get(
f"/api/v1/project/nonexistent-{unique_test_id}/pkg/+/tag"
)
assert response.status_code == 404
# Should not raise exception - valid JSON
data = response.json()
assert isinstance(data, dict)

View File

@@ -1,768 +0,0 @@
"""
Integration tests for artifact integrity verification.
Tests cover:
- Round-trip verification (upload -> download -> verify hash)
- Consistency check endpoint
- Header-based verification
- Integrity verification across file sizes
- Client-side verification workflow
"""
import pytest
import io
import hashlib
from tests.factories import (
compute_sha256,
upload_test_file,
generate_content_with_hash,
s3_object_exists,
get_s3_client,
get_s3_bucket,
)
from tests.conftest import (
SIZE_1KB,
SIZE_10KB,
SIZE_100KB,
SIZE_1MB,
SIZE_10MB,
)
class TestRoundTripVerification:
"""Tests for complete round-trip integrity verification."""
@pytest.mark.integration
def test_upload_download_hash_matches(self, integration_client, test_package):
"""Test that upload -> download round trip preserves content integrity."""
project, package = test_package
content = b"Round trip integrity test content"
expected_hash = compute_sha256(content)
# Upload and capture returned hash
result = upload_test_file(
integration_client, project, package, content, tag="roundtrip"
)
uploaded_hash = result["artifact_id"]
# Verify upload returned correct hash
assert uploaded_hash == expected_hash
# Download artifact
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/roundtrip",
params={"mode": "proxy"},
)
assert response.status_code == 200
# Compute hash of downloaded content
downloaded_hash = compute_sha256(response.content)
# All three hashes should match
assert downloaded_hash == expected_hash
assert downloaded_hash == uploaded_hash
@pytest.mark.integration
def test_upload_response_contains_hash(self, integration_client, test_package):
"""Test upload response contains artifact_id which is the SHA256 hash."""
project, package = test_package
content = b"Upload response hash test"
expected_hash = compute_sha256(content)
result = upload_test_file(integration_client, project, package, content)
assert "artifact_id" in result
assert result["artifact_id"] == expected_hash
assert len(result["artifact_id"]) == 64
assert all(c in "0123456789abcdef" for c in result["artifact_id"])
@pytest.mark.integration
def test_download_header_matches_artifact_id(self, integration_client, test_package):
"""Test X-Checksum-SHA256 header matches artifact ID."""
project, package = test_package
content = b"Header verification test"
expected_hash = compute_sha256(content)
upload_test_file(
integration_client, project, package, content, tag="header-check"
)
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/header-check",
params={"mode": "proxy"},
)
assert response.status_code == 200
assert response.headers.get("X-Checksum-SHA256") == expected_hash
@pytest.mark.integration
def test_etag_matches_artifact_id(self, integration_client, test_package):
"""Test ETag header matches artifact ID."""
project, package = test_package
content = b"ETag verification test"
expected_hash = compute_sha256(content)
upload_test_file(
integration_client, project, package, content, tag="etag-check"
)
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/etag-check",
params={"mode": "proxy"},
)
assert response.status_code == 200
etag = response.headers.get("ETag", "").strip('"')
assert etag == expected_hash
@pytest.mark.integration
def test_artifact_endpoint_returns_correct_hash(self, integration_client, test_package):
"""Test artifact endpoint returns correct hash/ID."""
project, package = test_package
content = b"Artifact endpoint hash test"
expected_hash = compute_sha256(content)
upload_test_file(integration_client, project, package, content)
# Query artifact directly
response = integration_client.get(f"/api/v1/artifact/{expected_hash}")
assert response.status_code == 200
data = response.json()
assert data["id"] == expected_hash
assert data.get("sha256") == expected_hash
class TestClientSideVerificationWorkflow:
"""Tests for client-side verification workflow."""
@pytest.mark.integration
def test_client_can_verify_before_upload(self, integration_client, test_package):
"""Test client can compute hash before upload and verify response matches."""
project, package = test_package
content = b"Client pre-upload verification test"
# Client computes hash locally before upload
client_hash = compute_sha256(content)
# Upload
result = upload_test_file(integration_client, project, package, content)
# Client verifies server returned the same hash
assert result["artifact_id"] == client_hash
@pytest.mark.integration
def test_client_can_provide_checksum_header(self, integration_client, test_package):
"""Test client can provide X-Checksum-SHA256 header for verification."""
project, package = test_package
content = b"Client checksum header test"
client_hash = compute_sha256(content)
files = {"file": ("test.bin", io.BytesIO(content), "application/octet-stream")}
response = integration_client.post(
f"/api/v1/project/{project}/{package}/upload",
files=files,
headers={"X-Checksum-SHA256": client_hash},
)
assert response.status_code == 200
assert response.json()["artifact_id"] == client_hash
@pytest.mark.integration
def test_checksum_mismatch_rejected(self, integration_client, test_package):
"""Test upload with wrong client checksum is rejected."""
project, package = test_package
content = b"Checksum mismatch test"
wrong_hash = "0" * 64
files = {"file": ("test.bin", io.BytesIO(content), "application/octet-stream")}
response = integration_client.post(
f"/api/v1/project/{project}/{package}/upload",
files=files,
headers={"X-Checksum-SHA256": wrong_hash},
)
assert response.status_code == 422
@pytest.mark.integration
def test_client_can_verify_after_download(self, integration_client, test_package):
"""Test client can verify downloaded content matches header hash."""
project, package = test_package
content = b"Client post-download verification"
upload_test_file(
integration_client, project, package, content, tag="verify-after"
)
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/verify-after",
params={"mode": "proxy"},
)
assert response.status_code == 200
# Client gets hash from header
header_hash = response.headers.get("X-Checksum-SHA256")
# Client computes hash of downloaded content
downloaded_hash = compute_sha256(response.content)
# Client verifies they match
assert downloaded_hash == header_hash
class TestIntegritySizeVariants:
"""Tests for integrity verification across different file sizes."""
@pytest.mark.integration
def test_integrity_1kb(self, integration_client, test_package, sized_content):
"""Test integrity verification for 1KB file."""
project, package = test_package
content, expected_hash = sized_content(SIZE_1KB, seed=100)
result = upload_test_file(
integration_client, project, package, content, tag="int-1kb"
)
assert result["artifact_id"] == expected_hash
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/int-1kb",
params={"mode": "proxy"},
)
assert response.status_code == 200
assert compute_sha256(response.content) == expected_hash
assert response.headers.get("X-Checksum-SHA256") == expected_hash
@pytest.mark.integration
def test_integrity_100kb(self, integration_client, test_package, sized_content):
"""Test integrity verification for 100KB file."""
project, package = test_package
content, expected_hash = sized_content(SIZE_100KB, seed=101)
result = upload_test_file(
integration_client, project, package, content, tag="int-100kb"
)
assert result["artifact_id"] == expected_hash
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/int-100kb",
params={"mode": "proxy"},
)
assert response.status_code == 200
assert compute_sha256(response.content) == expected_hash
assert response.headers.get("X-Checksum-SHA256") == expected_hash
@pytest.mark.integration
def test_integrity_1mb(self, integration_client, test_package, sized_content):
"""Test integrity verification for 1MB file."""
project, package = test_package
content, expected_hash = sized_content(SIZE_1MB, seed=102)
result = upload_test_file(
integration_client, project, package, content, tag="int-1mb"
)
assert result["artifact_id"] == expected_hash
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/int-1mb",
params={"mode": "proxy"},
)
assert response.status_code == 200
assert compute_sha256(response.content) == expected_hash
assert response.headers.get("X-Checksum-SHA256") == expected_hash
@pytest.mark.integration
@pytest.mark.slow
def test_integrity_10mb(self, integration_client, test_package, sized_content):
"""Test integrity verification for 10MB file."""
project, package = test_package
content, expected_hash = sized_content(SIZE_10MB, seed=103)
result = upload_test_file(
integration_client, project, package, content, tag="int-10mb"
)
assert result["artifact_id"] == expected_hash
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/int-10mb",
params={"mode": "proxy"},
)
assert response.status_code == 200
assert compute_sha256(response.content) == expected_hash
assert response.headers.get("X-Checksum-SHA256") == expected_hash
class TestConsistencyCheck:
"""Tests for the admin consistency check endpoint."""
@pytest.mark.integration
def test_consistency_check_returns_200(self, integration_client):
"""Test consistency check endpoint returns 200."""
response = integration_client.get("/api/v1/admin/consistency-check")
assert response.status_code == 200
@pytest.mark.integration
def test_consistency_check_response_format(self, integration_client):
"""Test consistency check returns expected response format."""
response = integration_client.get("/api/v1/admin/consistency-check")
assert response.status_code == 200
data = response.json()
# Check expected fields
assert "total_artifacts_checked" in data
assert "orphaned_s3_objects" in data
assert "missing_s3_objects" in data
assert "size_mismatches" in data
assert "healthy" in data
assert "orphaned_s3_keys" in data
assert "missing_s3_keys" in data
assert "size_mismatch_artifacts" in data
# Verify types
assert isinstance(data["total_artifacts_checked"], int)
assert isinstance(data["orphaned_s3_objects"], int)
assert isinstance(data["missing_s3_objects"], int)
assert isinstance(data["size_mismatches"], int)
assert isinstance(data["healthy"], bool)
assert isinstance(data["orphaned_s3_keys"], list)
assert isinstance(data["missing_s3_keys"], list)
assert isinstance(data["size_mismatch_artifacts"], list)
@pytest.mark.integration
def test_consistency_check_after_upload(self, integration_client, test_package):
"""Test consistency check passes after valid upload."""
project, package = test_package
content = b"Consistency check test content"
# Upload artifact
upload_test_file(integration_client, project, package, content)
# Run consistency check
response = integration_client.get("/api/v1/admin/consistency-check")
assert response.status_code == 200
data = response.json()
# Verify check ran and no issues
assert data["total_artifacts_checked"] >= 1
assert data["healthy"] is True
@pytest.mark.integration
def test_consistency_check_limit_parameter(self, integration_client):
"""Test consistency check respects limit parameter."""
response = integration_client.get(
"/api/v1/admin/consistency-check",
params={"limit": 10}
)
assert response.status_code == 200
data = response.json()
# Lists should not exceed limit
assert len(data["orphaned_s3_keys"]) <= 10
assert len(data["missing_s3_keys"]) <= 10
assert len(data["size_mismatch_artifacts"]) <= 10
class TestDigestHeader:
"""Tests for RFC 3230 Digest header."""
@pytest.mark.integration
def test_download_includes_digest_header(self, integration_client, test_package):
"""Test download includes Digest header in RFC 3230 format."""
project, package = test_package
content = b"Digest header test"
expected_hash = compute_sha256(content)
upload_test_file(
integration_client, project, package, content, tag="digest-test"
)
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/digest-test",
params={"mode": "proxy"},
)
assert response.status_code == 200
assert "Digest" in response.headers
# Verify Digest format (sha-256=base64hash)
digest = response.headers["Digest"]
assert digest.startswith("sha-256=")
@pytest.mark.integration
def test_digest_header_base64_valid(self, integration_client, test_package):
"""Test Digest header contains valid base64 encoding."""
import base64
project, package = test_package
content = b"Digest base64 test"
expected_hash = compute_sha256(content)
upload_test_file(
integration_client, project, package, content, tag="digest-b64"
)
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/digest-b64",
params={"mode": "proxy"},
)
assert response.status_code == 200
digest = response.headers["Digest"]
base64_part = digest.split("=", 1)[1]
# Should be valid base64
try:
decoded = base64.b64decode(base64_part)
assert len(decoded) == 32 # SHA256 is 32 bytes
except Exception as e:
pytest.fail(f"Invalid base64 in Digest header: {e}")
class TestVerificationModes:
"""Tests for download verification modes."""
@pytest.mark.integration
def test_pre_verification_mode(self, integration_client, test_package):
"""Test pre-verification mode verifies before streaming."""
project, package = test_package
content = b"Pre-verification mode test"
upload_test_file(
integration_client, project, package, content, tag="pre-verify"
)
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/pre-verify",
params={"mode": "proxy", "verify": "true", "verify_mode": "pre"},
)
assert response.status_code == 200
assert response.content == content
# X-Verified header should be true
assert response.headers.get("X-Verified") == "true"
@pytest.mark.integration
def test_stream_verification_mode(self, integration_client, test_package):
"""Test streaming verification mode."""
project, package = test_package
content = b"Stream verification mode test"
upload_test_file(
integration_client, project, package, content, tag="stream-verify"
)
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/stream-verify",
params={"mode": "proxy", "verify": "true", "verify_mode": "stream"},
)
assert response.status_code == 200
assert response.content == content
class TestArtifactIntegrityEndpoint:
"""Tests for artifact-specific integrity operations."""
@pytest.mark.integration
def test_artifact_size_matches(self, integration_client, test_package):
"""Test artifact endpoint returns correct size."""
project, package = test_package
content = b"Artifact size test content"
expected_size = len(content)
result = upload_test_file(integration_client, project, package, content)
artifact_id = result["artifact_id"]
response = integration_client.get(f"/api/v1/artifact/{artifact_id}")
assert response.status_code == 200
data = response.json()
assert data["size"] == expected_size
@pytest.mark.integration
def test_content_length_header_matches_size(self, integration_client, test_package):
"""Test Content-Length header matches artifact size."""
project, package = test_package
content = b"Content-Length header test"
expected_size = len(content)
upload_test_file(
integration_client, project, package, content, tag="content-len"
)
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/content-len",
params={"mode": "proxy"},
)
assert response.status_code == 200
assert int(response.headers.get("Content-Length", 0)) == expected_size
assert len(response.content) == expected_size
@pytest.mark.requires_direct_s3
class TestCorruptionDetection:
"""Tests for detecting corrupted S3 objects.
These tests directly manipulate S3 objects to simulate corruption
and verify that the system can detect hash mismatches.
Note: These tests require direct S3/MinIO access and are skipped in CI
where S3 is not directly accessible from the test runner.
"""
@pytest.mark.integration
def test_detection_of_corrupted_content(self, integration_client, test_package):
"""Test that corrupted S3 content is detected via hash mismatch.
Uploads content, then directly modifies the S3 object, then
verifies that the downloaded content hash doesn't match.
"""
project, package = test_package
content = b"Original content for corruption test"
expected_hash = compute_sha256(content)
# Upload original content
result = upload_test_file(
integration_client, project, package, content, tag="corrupt-test"
)
assert result["artifact_id"] == expected_hash
# Get the S3 object and corrupt it
s3_client = get_s3_client()
bucket = get_s3_bucket()
s3_key = f"fruits/{expected_hash[:2]}/{expected_hash[2:4]}/{expected_hash}"
# Replace with corrupted content
corrupted_content = b"Corrupted content - different from original!"
s3_client.put_object(Bucket=bucket, Key=s3_key, Body=corrupted_content)
# Download via proxy (bypasses hash verification)
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/corrupt-test",
params={"mode": "proxy"},
)
assert response.status_code == 200
# Verify the downloaded content doesn't match original hash
downloaded_hash = compute_sha256(response.content)
assert downloaded_hash != expected_hash, "Corruption was not detected - hashes match"
assert response.content == corrupted_content
# The X-Checksum-SHA256 header should still show the original hash (from DB)
# but the actual content hash is different
header_hash = response.headers.get("X-Checksum-SHA256")
assert header_hash == expected_hash # Header shows expected hash
assert downloaded_hash != header_hash # But content is corrupted
# Restore original content for cleanup
s3_client.put_object(Bucket=bucket, Key=s3_key, Body=content)
@pytest.mark.integration
def test_detection_of_single_bit_flip(self, integration_client, test_package):
"""Test detection of a single bit flip in S3 object content."""
project, package = test_package
content = b"Content for single bit flip detection test"
expected_hash = compute_sha256(content)
result = upload_test_file(
integration_client, project, package, content, tag="bitflip-test"
)
assert result["artifact_id"] == expected_hash
# Get S3 object and flip a single bit
s3_client = get_s3_client()
bucket = get_s3_bucket()
s3_key = f"fruits/{expected_hash[:2]}/{expected_hash[2:4]}/{expected_hash}"
# Flip the first bit of the first byte
corrupted_content = bytearray(content)
corrupted_content[0] ^= 0x01
corrupted_content = bytes(corrupted_content)
s3_client.put_object(Bucket=bucket, Key=s3_key, Body=corrupted_content)
# Download and verify hash mismatch
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/bitflip-test",
params={"mode": "proxy"},
)
assert response.status_code == 200
downloaded_hash = compute_sha256(response.content)
assert downloaded_hash != expected_hash, "Single bit flip not detected"
# Restore original
s3_client.put_object(Bucket=bucket, Key=s3_key, Body=content)
@pytest.mark.integration
def test_detection_of_truncated_content(self, integration_client, test_package):
"""Test detection of truncated S3 object."""
project, package = test_package
content = b"This is content that will be truncated for testing purposes"
expected_hash = compute_sha256(content)
result = upload_test_file(
integration_client, project, package, content, tag="truncate-test"
)
assert result["artifact_id"] == expected_hash
# Get S3 object and truncate it
s3_client = get_s3_client()
bucket = get_s3_bucket()
s3_key = f"fruits/{expected_hash[:2]}/{expected_hash[2:4]}/{expected_hash}"
# Truncate to half the original size
truncated_content = content[: len(content) // 2]
s3_client.put_object(Bucket=bucket, Key=s3_key, Body=truncated_content)
# Download and verify hash mismatch
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/truncate-test",
params={"mode": "proxy"},
)
assert response.status_code == 200
downloaded_hash = compute_sha256(response.content)
assert downloaded_hash != expected_hash, "Truncation not detected"
assert len(response.content) < len(content), "Content was not truncated"
# Restore original
s3_client.put_object(Bucket=bucket, Key=s3_key, Body=content)
@pytest.mark.integration
def test_detection_of_appended_content(self, integration_client, test_package):
"""Test detection of content with extra bytes appended."""
project, package = test_package
content = b"Original content"
expected_hash = compute_sha256(content)
result = upload_test_file(
integration_client, project, package, content, tag="append-test"
)
assert result["artifact_id"] == expected_hash
# Get S3 object and append extra bytes
s3_client = get_s3_client()
bucket = get_s3_bucket()
s3_key = f"fruits/{expected_hash[:2]}/{expected_hash[2:4]}/{expected_hash}"
appended_content = content + b" - extra bytes appended"
s3_client.put_object(Bucket=bucket, Key=s3_key, Body=appended_content)
# Download and verify hash mismatch
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/append-test",
params={"mode": "proxy"},
)
assert response.status_code == 200
downloaded_hash = compute_sha256(response.content)
assert downloaded_hash != expected_hash, "Appended content not detected"
assert len(response.content) > len(content), "Content was not extended"
# Restore original
s3_client.put_object(Bucket=bucket, Key=s3_key, Body=content)
@pytest.mark.integration
def test_client_detects_hash_mismatch_post_download(
self, integration_client, test_package
):
"""Test that a client can detect hash mismatch after downloading corrupted content.
This simulates the full client verification workflow:
1. Download content
2. Get expected hash from header
3. Compute actual hash of content
4. Verify they match (or detect corruption)
"""
project, package = test_package
content = b"Content for client-side corruption detection"
expected_hash = compute_sha256(content)
result = upload_test_file(
integration_client, project, package, content, tag="client-detect"
)
# Corrupt the S3 object
s3_client = get_s3_client()
bucket = get_s3_bucket()
s3_key = f"fruits/{expected_hash[:2]}/{expected_hash[2:4]}/{expected_hash}"
corrupted = b"This is completely different content"
s3_client.put_object(Bucket=bucket, Key=s3_key, Body=corrupted)
# Simulate client download and verification
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/client-detect",
params={"mode": "proxy"},
)
assert response.status_code == 200
# Client gets expected hash from header
header_hash = response.headers.get("X-Checksum-SHA256")
# Client computes hash of downloaded content
actual_hash = compute_sha256(response.content)
# Client detects the mismatch
corruption_detected = actual_hash != header_hash
assert corruption_detected, "Client should detect hash mismatch"
# Restore original
s3_client.put_object(Bucket=bucket, Key=s3_key, Body=content)
@pytest.mark.integration
def test_consistency_check_detects_size_mismatch(
self, integration_client, test_package, unique_test_id
):
"""Test that consistency check detects size mismatches.
Uploads content, modifies S3 object size, then runs consistency check.
"""
project, package = test_package
content = b"Content for size mismatch consistency check test " + unique_test_id.encode()
expected_hash = compute_sha256(content)
result = upload_test_file(
integration_client, project, package, content, tag="size-mismatch"
)
# Modify S3 object to have different size
s3_client = get_s3_client()
bucket = get_s3_bucket()
s3_key = f"fruits/{expected_hash[:2]}/{expected_hash[2:4]}/{expected_hash}"
different_size_content = content + b"extra extra extra"
s3_client.put_object(Bucket=bucket, Key=s3_key, Body=different_size_content)
# Run consistency check
response = integration_client.get("/api/v1/admin/consistency-check")
assert response.status_code == 200
data = response.json()
# Should detect the size mismatch
assert data["size_mismatches"] >= 1 or len(data["size_mismatch_artifacts"]) >= 1
# Restore original
s3_client.put_object(Bucket=bucket, Key=s3_key, Body=content)
@pytest.mark.integration
def test_consistency_check_detects_missing_s3_object(
self, integration_client, test_package, unique_test_id
):
"""Test that consistency check detects missing S3 objects.
Uploads content, deletes S3 object, then runs consistency check.
"""
project, package = test_package
content = b"Content for missing S3 object test " + unique_test_id.encode()
expected_hash = compute_sha256(content)
result = upload_test_file(
integration_client, project, package, content, tag="missing-s3"
)
# Delete the S3 object
s3_client = get_s3_client()
bucket = get_s3_bucket()
s3_key = f"fruits/{expected_hash[:2]}/{expected_hash[2:4]}/{expected_hash}"
s3_client.delete_object(Bucket=bucket, Key=s3_key)
# Run consistency check
response = integration_client.get("/api/v1/admin/consistency-check")
assert response.status_code == 200
data = response.json()
# Should detect the missing S3 object
assert data["missing_s3_objects"] >= 1 or len(data["missing_s3_keys"]) >= 1
# Restore the object for cleanup
s3_client.put_object(Bucket=bucket, Key=s3_key, Body=content)

View File

@@ -1,552 +0,0 @@
"""
Integration tests for large file upload functionality.
Tests cover:
- Large file uploads (100MB, 1GB)
- Multipart upload behavior
- Upload metrics (duration, throughput)
- Memory efficiency during uploads
- Upload progress tracking
Note: Large tests are marked with @pytest.mark.slow and will be skipped
by default. Run with `pytest --run-slow` to include them.
"""
import os
import pytest
import io
import time
from tests.factories import (
compute_sha256,
upload_test_file,
s3_object_exists,
)
from tests.conftest import (
SIZE_1KB,
SIZE_100KB,
SIZE_1MB,
SIZE_10MB,
SIZE_100MB,
SIZE_1GB,
)
class TestUploadMetrics:
"""Tests for upload duration and throughput metrics."""
@pytest.mark.integration
def test_upload_response_includes_duration_ms(self, integration_client, test_package):
"""Test upload response includes duration_ms field."""
project, package = test_package
content = b"duration test content"
result = upload_test_file(
integration_client, project, package, content, tag="duration-test"
)
assert "duration_ms" in result
assert result["duration_ms"] is not None
assert result["duration_ms"] >= 0
@pytest.mark.integration
def test_upload_response_includes_throughput(self, integration_client, test_package):
"""Test upload response includes throughput_mbps field."""
project, package = test_package
content = b"throughput test content"
result = upload_test_file(
integration_client, project, package, content, tag="throughput-test"
)
assert "throughput_mbps" in result
# For small files throughput may be very high or None
# Just verify the field exists
@pytest.mark.integration
def test_upload_duration_reasonable(
self, integration_client, test_package, sized_content
):
"""Test upload duration is reasonable for file size."""
project, package = test_package
content, _ = sized_content(SIZE_1MB, seed=100)
start = time.time()
result = upload_test_file(
integration_client, project, package, content, tag="duration-check"
)
actual_duration = (time.time() - start) * 1000 # ms
# Reported duration should be close to actual
assert result["duration_ms"] is not None
# Allow some variance (network overhead)
assert result["duration_ms"] <= actual_duration + 1000 # Within 1s
class TestLargeFileUploads:
"""Tests for large file uploads using multipart."""
@pytest.mark.integration
def test_upload_10mb_file(self, integration_client, test_package, sized_content):
"""Test uploading a 10MB file."""
project, package = test_package
content, expected_hash = sized_content(SIZE_10MB, seed=200)
result = upload_test_file(
integration_client, project, package, content, tag="large-10mb"
)
assert result["artifact_id"] == expected_hash
assert result["size"] == SIZE_10MB
assert result["duration_ms"] is not None
assert result["throughput_mbps"] is not None
@pytest.mark.integration
@pytest.mark.slow
@pytest.mark.requires_direct_s3
def test_upload_100mb_file(self, integration_client, test_package, sized_content):
"""Test uploading a 100MB file (triggers multipart upload)."""
project, package = test_package
content, expected_hash = sized_content(SIZE_100MB, seed=300)
result = upload_test_file(
integration_client, project, package, content, tag="large-100mb"
)
assert result["artifact_id"] == expected_hash
assert result["size"] == SIZE_100MB
# Verify S3 object exists
assert s3_object_exists(expected_hash)
@pytest.mark.integration
@pytest.mark.slow
@pytest.mark.large
def test_upload_1gb_file(self, integration_client, test_package, sized_content):
"""Test uploading a 1GB file."""
project, package = test_package
content, expected_hash = sized_content(SIZE_1GB, seed=400)
result = upload_test_file(
integration_client, project, package, content, tag="large-1gb"
)
assert result["artifact_id"] == expected_hash
assert result["size"] == SIZE_1GB
# Should have measurable throughput
assert result["throughput_mbps"] is not None
assert result["throughput_mbps"] > 0
@pytest.mark.integration
def test_large_file_deduplication(
self, integration_client, test_package, sized_content, unique_test_id
):
"""Test deduplication works for large files."""
project, package = test_package
# Use unique_test_id to ensure unique content per test run
seed = hash(unique_test_id) % 10000
content, expected_hash = sized_content(SIZE_10MB, seed=seed)
# First upload
result1 = upload_test_file(
integration_client, project, package, content, tag=f"dedup-{unique_test_id}-1"
)
# Note: may be True if previous test uploaded same content
first_dedupe = result1["deduplicated"]
# Second upload of same content
result2 = upload_test_file(
integration_client, project, package, content, tag=f"dedup-{unique_test_id}-2"
)
assert result2["artifact_id"] == expected_hash
# Second upload MUST be deduplicated
assert result2["deduplicated"] is True
class TestUploadProgress:
"""Tests for upload progress tracking endpoint."""
@pytest.mark.integration
def test_progress_endpoint_returns_not_found_for_invalid_id(
self, integration_client, test_package
):
"""Test progress endpoint returns not_found status for invalid upload ID."""
project, package = test_package
response = integration_client.get(
f"/api/v1/project/{project}/{package}/upload/invalid-upload-id/progress"
)
assert response.status_code == 200
data = response.json()
assert data["status"] == "not_found"
assert data["upload_id"] == "invalid-upload-id"
@pytest.mark.integration
def test_progress_endpoint_requires_valid_project(
self, integration_client, unique_test_id
):
"""Test progress endpoint validates project exists."""
response = integration_client.get(
f"/api/v1/project/nonexistent-{unique_test_id}/pkg/upload/upload-id/progress"
)
assert response.status_code == 404
@pytest.mark.integration
def test_progress_endpoint_requires_valid_package(
self, integration_client, test_project, unique_test_id
):
"""Test progress endpoint validates package exists."""
response = integration_client.get(
f"/api/v1/project/{test_project}/nonexistent-{unique_test_id}/upload/upload-id/progress"
)
assert response.status_code == 404
class TestResumableUploadProgress:
"""Tests for progress tracking during resumable uploads."""
@pytest.mark.integration
def test_resumable_upload_init_and_progress(
self, integration_client, test_package, sized_content
):
"""Test initializing resumable upload and checking progress."""
project, package = test_package
content, expected_hash = sized_content(SIZE_100KB, seed=600)
# Get API key for auth
api_key_response = integration_client.post(
"/api/v1/auth/keys",
json={"name": "progress-test-key"},
)
assert api_key_response.status_code == 200
api_key = api_key_response.json()["key"]
# Initialize resumable upload
init_response = integration_client.post(
f"/api/v1/project/{project}/{package}/upload/init",
json={
"expected_hash": expected_hash,
"filename": "progress-test.bin",
"size": SIZE_100KB,
},
headers={"Authorization": f"Bearer {api_key}"},
)
assert init_response.status_code == 200
upload_id = init_response.json().get("upload_id")
if upload_id:
# Check initial progress
progress_response = integration_client.get(
f"/api/v1/project/{project}/{package}/upload/{upload_id}/progress",
headers={"Authorization": f"Bearer {api_key}"},
)
assert progress_response.status_code == 200
progress = progress_response.json()
assert progress["status"] == "in_progress"
assert progress["bytes_uploaded"] == 0
assert progress["bytes_total"] == SIZE_100KB
# Abort to clean up
integration_client.delete(
f"/api/v1/project/{project}/{package}/upload/{upload_id}",
headers={"Authorization": f"Bearer {api_key}"},
)
class TestUploadSizeLimits:
"""Tests for upload size limit enforcement."""
@pytest.mark.integration
def test_empty_file_rejected(self, integration_client, test_package):
"""Test empty files are rejected."""
project, package = test_package
files = {"file": ("empty.txt", io.BytesIO(b""), "application/octet-stream")}
response = integration_client.post(
f"/api/v1/project/{project}/{package}/upload",
files=files,
)
assert response.status_code in [400, 422]
@pytest.mark.integration
def test_minimum_size_accepted(self, integration_client, test_package):
"""Test 1-byte file is accepted."""
project, package = test_package
content = b"X"
result = upload_test_file(
integration_client, project, package, content, tag="min-size"
)
assert result["size"] == 1
@pytest.mark.integration
def test_content_length_header_used_in_response(self, integration_client, test_package):
"""Test that upload response size matches Content-Length."""
project, package = test_package
content = b"content length verification test"
result = upload_test_file(
integration_client, project, package, content, tag="content-length-test"
)
# Size in response should match actual content length
assert result["size"] == len(content)
class TestUploadErrorHandling:
"""Tests for upload error handling."""
@pytest.mark.integration
def test_upload_to_nonexistent_project_returns_404(
self, integration_client, unique_test_id
):
"""Test upload to nonexistent project returns 404."""
content = b"test content"
files = {"file": ("test.bin", io.BytesIO(content), "application/octet-stream")}
response = integration_client.post(
f"/api/v1/project/nonexistent-{unique_test_id}/pkg/upload",
files=files,
)
assert response.status_code == 404
@pytest.mark.integration
def test_upload_to_nonexistent_package_returns_404(
self, integration_client, test_project, unique_test_id
):
"""Test upload to nonexistent package returns 404."""
content = b"test content"
files = {"file": ("test.bin", io.BytesIO(content), "application/octet-stream")}
response = integration_client.post(
f"/api/v1/project/{test_project}/nonexistent-{unique_test_id}/upload",
files=files,
)
assert response.status_code == 404
@pytest.mark.integration
def test_upload_without_file_returns_422(self, integration_client, test_package):
"""Test upload without file field returns 422."""
project, package = test_package
response = integration_client.post(
f"/api/v1/project/{project}/{package}/upload",
data={"tag": "no-file"},
)
assert response.status_code == 422
@pytest.mark.integration
def test_upload_with_invalid_checksum_rejected(
self, integration_client, test_package
):
"""Test upload with invalid checksum header format is rejected."""
project, package = test_package
content = b"checksum test"
files = {"file": ("test.bin", io.BytesIO(content), "application/octet-stream")}
response = integration_client.post(
f"/api/v1/project/{project}/{package}/upload",
files=files,
headers={"X-Checksum-SHA256": "invalid-checksum"},
)
assert response.status_code == 400
@pytest.mark.integration
def test_upload_with_mismatched_checksum_rejected(
self, integration_client, test_package
):
"""Test upload with wrong checksum is rejected."""
project, package = test_package
content = b"mismatch test"
wrong_hash = "0" * 64
files = {"file": ("test.bin", io.BytesIO(content), "application/octet-stream")}
response = integration_client.post(
f"/api/v1/project/{project}/{package}/upload",
files=files,
headers={"X-Checksum-SHA256": wrong_hash},
)
assert response.status_code == 422
assert "verification failed" in response.json().get("detail", "").lower()
class TestResumableUploadCancellation:
"""Tests for resumable upload cancellation."""
@pytest.mark.integration
def test_abort_resumable_upload(self, integration_client, test_package, sized_content):
"""Test aborting a resumable upload cleans up properly."""
project, package = test_package
content, expected_hash = sized_content(SIZE_100KB, seed=700)
# Get API key for auth
api_key_response = integration_client.post(
"/api/v1/auth/keys",
json={"name": "abort-test-key"},
)
assert api_key_response.status_code == 200
api_key = api_key_response.json()["key"]
# Initialize resumable upload
init_response = integration_client.post(
f"/api/v1/project/{project}/{package}/upload/init",
json={
"expected_hash": expected_hash,
"filename": "abort-test.bin",
"size": SIZE_100KB,
},
headers={"Authorization": f"Bearer {api_key}"},
)
assert init_response.status_code == 200
upload_id = init_response.json().get("upload_id")
if upload_id:
# Abort the upload (without uploading any parts)
abort_response = integration_client.delete(
f"/api/v1/project/{project}/{package}/upload/{upload_id}",
headers={"Authorization": f"Bearer {api_key}"},
)
assert abort_response.status_code in [200, 204]
# Verify progress shows not_found after abort
progress_response = integration_client.get(
f"/api/v1/project/{project}/{package}/upload/{upload_id}/progress",
headers={"Authorization": f"Bearer {api_key}"},
)
assert progress_response.status_code == 200
assert progress_response.json()["status"] == "not_found"
@pytest.mark.integration
def test_abort_nonexistent_upload(self, integration_client, test_package):
"""Test aborting nonexistent upload returns appropriate error."""
project, package = test_package
# Get API key for auth
api_key_response = integration_client.post(
"/api/v1/auth/keys",
json={"name": "abort-nonexistent-key"},
)
assert api_key_response.status_code == 200
api_key = api_key_response.json()["key"]
response = integration_client.delete(
f"/api/v1/project/{project}/{package}/upload/nonexistent-upload-id",
headers={"Authorization": f"Bearer {api_key}"},
)
# Should return 404 or 200 (idempotent delete)
assert response.status_code in [200, 204, 404]
class TestUploadTimeout:
"""Tests for upload timeout handling."""
@pytest.mark.integration
def test_upload_with_short_timeout_succeeds_for_small_file(
self, integration_client, test_package
):
"""Test small file upload succeeds with reasonable timeout."""
project, package = test_package
content = b"small timeout test"
# httpx client should handle this quickly
result = upload_test_file(
integration_client, project, package, content, tag="timeout-small"
)
assert result["artifact_id"] is not None
@pytest.mark.integration
def test_upload_response_duration_under_timeout(
self, integration_client, test_package, sized_content
):
"""Test upload completes within reasonable time."""
project, package = test_package
content, _ = sized_content(SIZE_1MB, seed=800)
start = time.time()
result = upload_test_file(
integration_client, project, package, content, tag="timeout-check"
)
duration = time.time() - start
# 1MB should upload in well under 60 seconds on local
assert duration < 60
assert result["artifact_id"] is not None
class TestConcurrentUploads:
"""Tests for concurrent upload handling."""
@pytest.mark.integration
def test_concurrent_different_files(
self, integration_client, test_package, sized_content
):
"""Test concurrent uploads of different files succeed."""
from concurrent.futures import ThreadPoolExecutor, as_completed
project, package = test_package
# Get API key for auth
api_key_response = integration_client.post(
"/api/v1/auth/keys",
json={"name": "concurrent-diff-key"},
)
assert api_key_response.status_code == 200
api_key = api_key_response.json()["key"]
num_uploads = 3
results = []
errors = []
def upload_unique_file(idx):
try:
from httpx import Client
content, expected_hash = sized_content(SIZE_100KB, seed=900 + idx)
base_url = os.environ.get("ORCHARD_TEST_URL", "http://localhost:8080")
with Client(base_url=base_url, timeout=30.0) as client:
files = {
"file": (
f"concurrent-{idx}.bin",
io.BytesIO(content),
"application/octet-stream",
)
}
response = client.post(
f"/api/v1/project/{project}/{package}/upload",
files=files,
data={"tag": f"concurrent-diff-{idx}"},
headers={"Authorization": f"Bearer {api_key}"},
)
if response.status_code == 200:
results.append((idx, response.json(), expected_hash))
else:
errors.append(f"Upload {idx}: {response.status_code} - {response.text}")
except Exception as e:
errors.append(f"Upload {idx}: {str(e)}")
with ThreadPoolExecutor(max_workers=num_uploads) as executor:
futures = [executor.submit(upload_unique_file, i) for i in range(num_uploads)]
for future in as_completed(futures):
pass
assert len(errors) == 0, f"Concurrent upload errors: {errors}"
assert len(results) == num_uploads
# Each upload should have unique artifact ID
artifact_ids = set(r[1]["artifact_id"] for r in results)
assert len(artifact_ids) == num_uploads
# Each should match expected hash
for idx, result, expected_hash in results:
assert result["artifact_id"] == expected_hash

View File

@@ -1,583 +0,0 @@
"""
Integration tests for upload/download with various file sizes.
Tests cover:
- Small files (0B - 100KB)
- Medium files (1MB - 50MB)
- Large files (100MB - 1GB) - marked as slow/large
- Exact chunk boundaries
- Data integrity verification across all sizes
"""
import pytest
import io
import time
from concurrent.futures import ThreadPoolExecutor, as_completed
from tests.factories import (
compute_sha256,
upload_test_file,
generate_content,
generate_content_with_hash,
)
from tests.conftest import (
SIZE_1B,
SIZE_1KB,
SIZE_10KB,
SIZE_100KB,
SIZE_1MB,
SIZE_5MB,
SIZE_10MB,
SIZE_50MB,
SIZE_100MB,
SIZE_250MB,
SIZE_500MB,
SIZE_1GB,
CHUNK_SIZE,
MULTIPART_THRESHOLD,
)
class TestSmallFileSizes:
"""Tests for small file uploads/downloads (0B - 100KB)."""
@pytest.mark.integration
def test_upload_download_1_byte(self, integration_client, test_package, sized_content):
"""Test upload/download of 1 byte file."""
project, package = test_package
content, expected_hash = sized_content(SIZE_1B, seed=1)
result = upload_test_file(
integration_client, project, package, content,
filename="1byte.bin", tag="1byte"
)
assert result["artifact_id"] == expected_hash
assert result["size"] == SIZE_1B
# Download and verify
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/1byte",
params={"mode": "proxy"},
)
assert response.status_code == 200
assert response.content == content
assert len(response.content) == SIZE_1B
@pytest.mark.integration
def test_upload_download_1kb(self, integration_client, test_package, sized_content):
"""Test upload/download of 1KB file."""
project, package = test_package
content, expected_hash = sized_content(SIZE_1KB, seed=2)
result = upload_test_file(
integration_client, project, package, content,
filename="1kb.bin", tag="1kb"
)
assert result["artifact_id"] == expected_hash
assert result["size"] == SIZE_1KB
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/1kb",
params={"mode": "proxy"},
)
assert response.status_code == 200
assert response.content == content
@pytest.mark.integration
def test_upload_download_10kb(self, integration_client, test_package, sized_content):
"""Test upload/download of 10KB file."""
project, package = test_package
content, expected_hash = sized_content(SIZE_10KB, seed=3)
result = upload_test_file(
integration_client, project, package, content,
filename="10kb.bin", tag="10kb"
)
assert result["artifact_id"] == expected_hash
assert result["size"] == SIZE_10KB
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/10kb",
params={"mode": "proxy"},
)
assert response.status_code == 200
assert response.content == content
@pytest.mark.integration
def test_upload_download_100kb(self, integration_client, test_package, sized_content):
"""Test upload/download of 100KB file."""
project, package = test_package
content, expected_hash = sized_content(SIZE_100KB, seed=4)
result = upload_test_file(
integration_client, project, package, content,
filename="100kb.bin", tag="100kb"
)
assert result["artifact_id"] == expected_hash
assert result["size"] == SIZE_100KB
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/100kb",
params={"mode": "proxy"},
)
assert response.status_code == 200
assert response.content == content
class TestMediumFileSizes:
"""Tests for medium file uploads/downloads (1MB - 50MB)."""
@pytest.mark.integration
def test_upload_download_1mb(self, integration_client, test_package, sized_content):
"""Test upload/download of 1MB file."""
project, package = test_package
content, expected_hash = sized_content(SIZE_1MB, seed=10)
result = upload_test_file(
integration_client, project, package, content,
filename="1mb.bin", tag="1mb"
)
assert result["artifact_id"] == expected_hash
assert result["size"] == SIZE_1MB
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/1mb",
params={"mode": "proxy"},
)
assert response.status_code == 200
assert len(response.content) == SIZE_1MB
assert compute_sha256(response.content) == expected_hash
@pytest.mark.integration
def test_upload_download_5mb(self, integration_client, test_package, sized_content):
"""Test upload/download of 5MB file (multipart threshold boundary area)."""
project, package = test_package
content, expected_hash = sized_content(SIZE_5MB, seed=11)
result = upload_test_file(
integration_client, project, package, content,
filename="5mb.bin", tag="5mb"
)
assert result["artifact_id"] == expected_hash
assert result["size"] == SIZE_5MB
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/5mb",
params={"mode": "proxy"},
)
assert response.status_code == 200
assert len(response.content) == SIZE_5MB
assert compute_sha256(response.content) == expected_hash
@pytest.mark.integration
@pytest.mark.slow
def test_upload_download_10mb(self, integration_client, test_package, sized_content):
"""Test upload/download of 10MB file."""
project, package = test_package
content, expected_hash = sized_content(SIZE_10MB, seed=12)
result = upload_test_file(
integration_client, project, package, content,
filename="10mb.bin", tag="10mb"
)
assert result["artifact_id"] == expected_hash
assert result["size"] == SIZE_10MB
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/10mb",
params={"mode": "proxy"},
)
assert response.status_code == 200
assert len(response.content) == SIZE_10MB
assert compute_sha256(response.content) == expected_hash
@pytest.mark.integration
@pytest.mark.slow
def test_upload_download_50mb(self, integration_client, test_package, sized_content):
"""Test upload/download of 50MB file."""
project, package = test_package
content, expected_hash = sized_content(SIZE_50MB, seed=13)
start_time = time.time()
result = upload_test_file(
integration_client, project, package, content,
filename="50mb.bin", tag="50mb"
)
upload_time = time.time() - start_time
assert result["artifact_id"] == expected_hash
assert result["size"] == SIZE_50MB
start_time = time.time()
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/50mb",
params={"mode": "proxy"},
)
download_time = time.time() - start_time
assert response.status_code == 200
assert len(response.content) == SIZE_50MB
assert compute_sha256(response.content) == expected_hash
# Log timing for performance tracking
print(f"\n50MB upload: {upload_time:.2f}s, download: {download_time:.2f}s")
class TestLargeFileSizes:
"""Tests for large file uploads/downloads (100MB - 1GB).
These tests are marked as slow and large, skipped by default.
Run with: pytest -m "large" to include these tests.
"""
@pytest.mark.integration
@pytest.mark.slow
@pytest.mark.large
def test_upload_download_100mb(self, integration_client, test_package, sized_content):
"""Test upload/download of 100MB file (multipart threshold)."""
project, package = test_package
content, expected_hash = sized_content(SIZE_100MB, seed=100)
start_time = time.time()
result = upload_test_file(
integration_client, project, package, content,
filename="100mb.bin", tag="100mb"
)
upload_time = time.time() - start_time
assert result["artifact_id"] == expected_hash
assert result["size"] == SIZE_100MB
start_time = time.time()
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/100mb",
params={"mode": "proxy"},
)
download_time = time.time() - start_time
assert response.status_code == 200
assert len(response.content) == SIZE_100MB
assert compute_sha256(response.content) == expected_hash
print(f"\n100MB upload: {upload_time:.2f}s, download: {download_time:.2f}s")
@pytest.mark.integration
@pytest.mark.slow
@pytest.mark.large
def test_upload_download_250mb(self, integration_client, test_package, sized_content):
"""Test upload/download of 250MB file."""
project, package = test_package
content, expected_hash = sized_content(SIZE_250MB, seed=250)
start_time = time.time()
result = upload_test_file(
integration_client, project, package, content,
filename="250mb.bin", tag="250mb"
)
upload_time = time.time() - start_time
assert result["artifact_id"] == expected_hash
assert result["size"] == SIZE_250MB
start_time = time.time()
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/250mb",
params={"mode": "proxy"},
)
download_time = time.time() - start_time
assert response.status_code == 200
assert len(response.content) == SIZE_250MB
assert compute_sha256(response.content) == expected_hash
print(f"\n250MB upload: {upload_time:.2f}s, download: {download_time:.2f}s")
@pytest.mark.integration
@pytest.mark.slow
@pytest.mark.large
def test_upload_download_500mb(self, integration_client, test_package, sized_content):
"""Test upload/download of 500MB file."""
project, package = test_package
content, expected_hash = sized_content(SIZE_500MB, seed=500)
start_time = time.time()
result = upload_test_file(
integration_client, project, package, content,
filename="500mb.bin", tag="500mb"
)
upload_time = time.time() - start_time
assert result["artifact_id"] == expected_hash
assert result["size"] == SIZE_500MB
start_time = time.time()
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/500mb",
params={"mode": "proxy"},
)
download_time = time.time() - start_time
assert response.status_code == 200
assert len(response.content) == SIZE_500MB
assert compute_sha256(response.content) == expected_hash
print(f"\n500MB upload: {upload_time:.2f}s, download: {download_time:.2f}s")
@pytest.mark.integration
@pytest.mark.slow
@pytest.mark.large
def test_upload_download_1gb(self, integration_client, test_package, sized_content):
"""Test upload/download of 1GB file.
This test may take several minutes depending on network/disk speed.
"""
project, package = test_package
content, expected_hash = sized_content(SIZE_1GB, seed=1024)
start_time = time.time()
result = upload_test_file(
integration_client, project, package, content,
filename="1gb.bin", tag="1gb"
)
upload_time = time.time() - start_time
assert result["artifact_id"] == expected_hash
assert result["size"] == SIZE_1GB
start_time = time.time()
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/1gb",
params={"mode": "proxy"},
)
download_time = time.time() - start_time
assert response.status_code == 200
assert len(response.content) == SIZE_1GB
assert compute_sha256(response.content) == expected_hash
print(f"\n1GB upload: {upload_time:.2f}s, download: {download_time:.2f}s")
class TestChunkBoundaries:
"""Tests for exact chunk size boundaries."""
@pytest.mark.integration
def test_upload_download_at_chunk_size(self, integration_client, test_package, sized_content):
"""Test upload/download at exact chunk size (64KB)."""
project, package = test_package
content, expected_hash = sized_content(CHUNK_SIZE, seed=64)
result = upload_test_file(
integration_client, project, package, content,
filename="chunk.bin", tag="chunk-exact"
)
assert result["artifact_id"] == expected_hash
assert result["size"] == CHUNK_SIZE
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/chunk-exact",
params={"mode": "proxy"},
)
assert response.status_code == 200
assert response.content == content
@pytest.mark.integration
def test_upload_download_chunk_size_plus_1(self, integration_client, test_package, sized_content):
"""Test upload/download at chunk size + 1 byte."""
project, package = test_package
size = CHUNK_SIZE + 1
content, expected_hash = sized_content(size, seed=65)
result = upload_test_file(
integration_client, project, package, content,
filename="chunk_plus.bin", tag="chunk-plus"
)
assert result["artifact_id"] == expected_hash
assert result["size"] == size
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/chunk-plus",
params={"mode": "proxy"},
)
assert response.status_code == 200
assert response.content == content
@pytest.mark.integration
def test_upload_download_chunk_size_minus_1(self, integration_client, test_package, sized_content):
"""Test upload/download at chunk size - 1 byte."""
project, package = test_package
size = CHUNK_SIZE - 1
content, expected_hash = sized_content(size, seed=63)
result = upload_test_file(
integration_client, project, package, content,
filename="chunk_minus.bin", tag="chunk-minus"
)
assert result["artifact_id"] == expected_hash
assert result["size"] == size
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/chunk-minus",
params={"mode": "proxy"},
)
assert response.status_code == 200
assert response.content == content
@pytest.mark.integration
def test_upload_download_multiple_chunks(self, integration_client, test_package, sized_content):
"""Test upload/download spanning multiple chunks."""
project, package = test_package
size = CHUNK_SIZE * 3 + 1000 # 3 full chunks + partial
content, expected_hash = sized_content(size, seed=300)
result = upload_test_file(
integration_client, project, package, content,
filename="multi_chunk.bin", tag="multi-chunk"
)
assert result["artifact_id"] == expected_hash
assert result["size"] == size
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/multi-chunk",
params={"mode": "proxy"},
)
assert response.status_code == 200
assert response.content == content
class TestDataIntegrity:
"""Tests for data integrity with various content types."""
@pytest.mark.integration
def test_binary_content_integrity(self, integration_client, test_package):
"""Test binary content (all byte values 0-255) integrity."""
project, package = test_package
# Content with all 256 possible byte values
content = bytes(range(256)) * 100 # 25.6KB
expected_hash = compute_sha256(content)
result = upload_test_file(
integration_client, project, package, content,
filename="binary.bin", tag="binary"
)
assert result["artifact_id"] == expected_hash
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/binary",
params={"mode": "proxy"},
)
assert response.status_code == 200
assert response.content == content
@pytest.mark.integration
def test_text_content_integrity(self, integration_client, test_package):
"""Test UTF-8 text content integrity."""
project, package = test_package
content = "Hello, World! 你好世界 🌍 مرحبا العالم".encode("utf-8")
expected_hash = compute_sha256(content)
result = upload_test_file(
integration_client, project, package, content,
filename="text.txt", tag="text"
)
assert result["artifact_id"] == expected_hash
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/text",
params={"mode": "proxy"},
)
assert response.status_code == 200
assert response.content == content
assert response.content.decode("utf-8") == "Hello, World! 你好世界 🌍 مرحبا العالم"
@pytest.mark.integration
def test_null_bytes_content_integrity(self, integration_client, test_package):
"""Test content with null bytes."""
project, package = test_package
content = b"before\x00null\x00bytes\x00after"
expected_hash = compute_sha256(content)
result = upload_test_file(
integration_client, project, package, content,
filename="nulls.bin", tag="nulls"
)
assert result["artifact_id"] == expected_hash
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/nulls",
params={"mode": "proxy"},
)
assert response.status_code == 200
assert response.content == content
assert b"\x00" in response.content
@pytest.mark.integration
def test_unicode_filename_integrity(self, integration_client, test_package):
"""Test file with unicode filename."""
project, package = test_package
content = b"unicode filename test"
expected_hash = compute_sha256(content)
result = upload_test_file(
integration_client, project, package, content,
filename="文件名.txt", tag="unicode-name"
)
assert result["artifact_id"] == expected_hash
assert result["original_name"] == "文件名.txt"
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/unicode-name",
params={"mode": "proxy"},
)
assert response.status_code == 200
assert response.content == content
@pytest.mark.integration
def test_compressed_content_integrity(self, integration_client, test_package):
"""Test gzip-compressed content integrity."""
import gzip
project, package = test_package
original = b"This is some text that will be compressed " * 100
content = gzip.compress(original)
expected_hash = compute_sha256(content)
result = upload_test_file(
integration_client, project, package, content,
filename="data.gz", tag="compressed"
)
assert result["artifact_id"] == expected_hash
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/compressed",
params={"mode": "proxy"},
)
assert response.status_code == 200
assert response.content == content
# Verify we can decompress
assert gzip.decompress(response.content) == original
@pytest.mark.integration
def test_hash_verification_matches(self, integration_client, test_package, sized_content):
"""Test that computed hash matches artifact_id for various sizes."""
project, package = test_package
sizes = [SIZE_1B, SIZE_1KB, SIZE_10KB, SIZE_100KB, SIZE_1MB]
for i, size in enumerate(sizes):
content, expected_hash = sized_content(size, seed=1000 + i)
result = upload_test_file(
integration_client, project, package, content,
filename=f"hash_test_{size}.bin", tag=f"hash-{size}"
)
# Verify artifact_id matches expected hash
assert result["artifact_id"] == expected_hash
# Download and verify hash of downloaded content
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/hash-{size}",
params={"mode": "proxy"},
)
downloaded_hash = compute_sha256(response.content)
assert downloaded_hash == expected_hash

View File

@@ -1,535 +0,0 @@
"""
Integration tests for streaming download functionality.
Tests cover:
- HTTP Range requests (partial downloads, resume)
- Conditional requests (If-None-Match, If-Modified-Since)
- Caching headers (Cache-Control, Last-Modified, Accept-Ranges)
- Large file streaming
- Download modes (proxy, redirect, presigned)
"""
import pytest
import io
import time
from email.utils import formatdate
from tests.factories import (
compute_sha256,
upload_test_file,
)
from tests.conftest import (
SIZE_1KB,
SIZE_100KB,
SIZE_1MB,
)
class TestRangeRequests:
"""Tests for HTTP Range request support (partial downloads)."""
@pytest.mark.integration
def test_range_request_first_bytes(self, integration_client, test_package):
"""Test range request for first N bytes."""
project, package = test_package
content = b"0123456789" * 100 # 1000 bytes
upload_test_file(integration_client, project, package, content, tag="range-test")
# Request first 10 bytes
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/range-test",
params={"mode": "proxy"},
headers={"Range": "bytes=0-9"},
)
assert response.status_code == 206 # Partial Content
assert response.content == b"0123456789"
assert "Content-Range" in response.headers
assert response.headers["Content-Range"].startswith("bytes 0-9/")
@pytest.mark.integration
def test_range_request_middle_bytes(self, integration_client, test_package):
"""Test range request for bytes in the middle."""
project, package = test_package
content = b"ABCDEFGHIJKLMNOPQRSTUVWXYZ"
upload_test_file(integration_client, project, package, content, tag="range-mid")
# Request bytes 10-19 (KLMNOPQRST)
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/range-mid",
params={"mode": "proxy"},
headers={"Range": "bytes=10-19"},
)
assert response.status_code == 206
assert response.content == b"KLMNOPQRST"
@pytest.mark.integration
def test_range_request_suffix_bytes(self, integration_client, test_package):
"""Test range request for last N bytes (suffix range)."""
project, package = test_package
content = b"0123456789ABCDEF" # 16 bytes
upload_test_file(integration_client, project, package, content, tag="range-suffix")
# Request last 4 bytes
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/range-suffix",
params={"mode": "proxy"},
headers={"Range": "bytes=-4"},
)
assert response.status_code == 206
assert response.content == b"CDEF"
@pytest.mark.integration
def test_range_request_open_ended(self, integration_client, test_package):
"""Test range request from offset to end."""
project, package = test_package
content = b"0123456789"
upload_test_file(integration_client, project, package, content, tag="range-open")
# Request from byte 5 to end
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/range-open",
params={"mode": "proxy"},
headers={"Range": "bytes=5-"},
)
assert response.status_code == 206
assert response.content == b"56789"
@pytest.mark.integration
def test_range_request_includes_accept_ranges_header(
self, integration_client, test_package
):
"""Test that range requests include Accept-Ranges header."""
project, package = test_package
content = b"test content"
upload_test_file(integration_client, project, package, content, tag="accept-ranges")
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/accept-ranges",
params={"mode": "proxy"},
headers={"Range": "bytes=0-4"},
)
assert response.status_code == 206
assert response.headers.get("Accept-Ranges") == "bytes"
@pytest.mark.integration
def test_full_download_advertises_accept_ranges(
self, integration_client, test_package
):
"""Test that full downloads advertise range support."""
project, package = test_package
content = b"test content"
upload_test_file(integration_client, project, package, content, tag="full-accept")
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/full-accept",
params={"mode": "proxy"},
)
assert response.status_code == 200
assert response.headers.get("Accept-Ranges") == "bytes"
class TestConditionalRequests:
"""Tests for conditional request handling (304 Not Modified)."""
@pytest.mark.integration
def test_if_none_match_returns_304(self, integration_client, test_package):
"""Test If-None-Match with matching ETag returns 304."""
project, package = test_package
content = b"conditional request test content"
expected_hash = compute_sha256(content)
upload_test_file(integration_client, project, package, content, tag="cond-etag")
# Request with matching ETag
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/cond-etag",
params={"mode": "proxy"},
headers={"If-None-Match": f'"{expected_hash}"'},
)
assert response.status_code == 304
assert response.content == b"" # No body for 304
@pytest.mark.integration
def test_if_none_match_without_quotes(self, integration_client, test_package):
"""Test If-None-Match works with or without quotes."""
project, package = test_package
content = b"etag no quotes test"
expected_hash = compute_sha256(content)
upload_test_file(integration_client, project, package, content, tag="cond-noquote")
# Request with ETag without quotes
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/cond-noquote",
params={"mode": "proxy"},
headers={"If-None-Match": expected_hash},
)
assert response.status_code == 304
@pytest.mark.integration
def test_if_none_match_mismatch_returns_200(self, integration_client, test_package):
"""Test If-None-Match with non-matching ETag returns 200."""
project, package = test_package
content = b"etag mismatch test"
upload_test_file(integration_client, project, package, content, tag="cond-mismatch")
# Request with different ETag
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/cond-mismatch",
params={"mode": "proxy"},
headers={"If-None-Match": '"different-etag-value"'},
)
assert response.status_code == 200
assert response.content == content
@pytest.mark.integration
def test_if_modified_since_returns_304(self, integration_client, test_package):
"""Test If-Modified-Since with future date returns 304."""
project, package = test_package
content = b"modified since test"
upload_test_file(integration_client, project, package, content, tag="cond-modified")
# Request with future date (artifact was definitely created before this)
future_date = formatdate(time.time() + 86400, usegmt=True) # Tomorrow
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/cond-modified",
params={"mode": "proxy"},
headers={"If-Modified-Since": future_date},
)
assert response.status_code == 304
@pytest.mark.integration
def test_if_modified_since_old_date_returns_200(
self, integration_client, test_package
):
"""Test If-Modified-Since with old date returns 200."""
project, package = test_package
content = b"old date test"
upload_test_file(integration_client, project, package, content, tag="cond-old")
# Request with old date (2020-01-01)
old_date = "Wed, 01 Jan 2020 00:00:00 GMT"
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/cond-old",
params={"mode": "proxy"},
headers={"If-Modified-Since": old_date},
)
assert response.status_code == 200
assert response.content == content
@pytest.mark.integration
def test_304_includes_etag(self, integration_client, test_package):
"""Test 304 response includes ETag header."""
project, package = test_package
content = b"304 etag test"
expected_hash = compute_sha256(content)
upload_test_file(integration_client, project, package, content, tag="304-etag")
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/304-etag",
params={"mode": "proxy"},
headers={"If-None-Match": f'"{expected_hash}"'},
)
assert response.status_code == 304
assert response.headers.get("ETag") == f'"{expected_hash}"'
@pytest.mark.integration
def test_304_includes_cache_control(self, integration_client, test_package):
"""Test 304 response includes Cache-Control header."""
project, package = test_package
content = b"304 cache test"
expected_hash = compute_sha256(content)
upload_test_file(integration_client, project, package, content, tag="304-cache")
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/304-cache",
params={"mode": "proxy"},
headers={"If-None-Match": f'"{expected_hash}"'},
)
assert response.status_code == 304
assert "immutable" in response.headers.get("Cache-Control", "")
class TestCachingHeaders:
"""Tests for caching headers on download responses."""
@pytest.mark.integration
def test_download_includes_cache_control(self, integration_client, test_package):
"""Test download response includes Cache-Control header."""
project, package = test_package
content = b"cache control test"
upload_test_file(integration_client, project, package, content, tag="cache-ctl")
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/cache-ctl",
params={"mode": "proxy"},
)
assert response.status_code == 200
cache_control = response.headers.get("Cache-Control", "")
assert "public" in cache_control
assert "immutable" in cache_control
assert "max-age" in cache_control
@pytest.mark.integration
def test_download_includes_last_modified(self, integration_client, test_package):
"""Test download response includes Last-Modified header."""
project, package = test_package
content = b"last modified test"
upload_test_file(integration_client, project, package, content, tag="last-mod")
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/last-mod",
params={"mode": "proxy"},
)
assert response.status_code == 200
assert "Last-Modified" in response.headers
# Should be in RFC 7231 format
last_modified = response.headers["Last-Modified"]
assert "GMT" in last_modified
@pytest.mark.integration
def test_download_includes_etag(self, integration_client, test_package):
"""Test download response includes ETag header."""
project, package = test_package
content = b"etag header test"
expected_hash = compute_sha256(content)
upload_test_file(integration_client, project, package, content, tag="etag-hdr")
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/etag-hdr",
params={"mode": "proxy"},
)
assert response.status_code == 200
assert response.headers.get("ETag") == f'"{expected_hash}"'
class TestDownloadResume:
"""Tests for download resume functionality using range requests."""
@pytest.mark.integration
def test_resume_download_after_partial(self, integration_client, test_package):
"""Test resuming download from where it left off."""
project, package = test_package
content = b"ABCDEFGHIJ" * 100 # 1000 bytes
upload_test_file(integration_client, project, package, content, tag="resume-test")
# Simulate partial download (first 500 bytes)
response1 = integration_client.get(
f"/api/v1/project/{project}/{package}/+/resume-test",
params={"mode": "proxy"},
headers={"Range": "bytes=0-499"},
)
assert response1.status_code == 206
first_half = response1.content
assert len(first_half) == 500
# Resume from byte 500
response2 = integration_client.get(
f"/api/v1/project/{project}/{package}/+/resume-test",
params={"mode": "proxy"},
headers={"Range": "bytes=500-"},
)
assert response2.status_code == 206
second_half = response2.content
assert len(second_half) == 500
# Combine and verify
combined = first_half + second_half
assert combined == content
@pytest.mark.integration
def test_resume_with_etag_verification(self, integration_client, test_package):
"""Test that resumed download can verify content hasn't changed."""
project, package = test_package
content = b"resume etag verification test content"
expected_hash = compute_sha256(content)
upload_test_file(integration_client, project, package, content, tag="resume-etag")
# Get ETag from first request
response1 = integration_client.get(
f"/api/v1/project/{project}/{package}/+/resume-etag",
params={"mode": "proxy"},
headers={"Range": "bytes=0-9"},
)
assert response1.status_code == 206
etag = response1.headers.get("ETag")
assert etag == f'"{expected_hash}"'
# Resume with If-Match to ensure content hasn't changed
# (Note: If-Match would fail and return 412 if content changed)
response2 = integration_client.get(
f"/api/v1/project/{project}/{package}/+/resume-etag",
params={"mode": "proxy"},
headers={"Range": "bytes=10-"},
)
assert response2.status_code == 206
# ETag should be the same
assert response2.headers.get("ETag") == etag
class TestLargeFileStreaming:
"""Tests for streaming large files."""
@pytest.mark.integration
def test_stream_1mb_file(self, integration_client, test_package, sized_content):
"""Test streaming a 1MB file."""
project, package = test_package
content, expected_hash = sized_content(SIZE_1MB, seed=500)
upload_test_file(integration_client, project, package, content, tag="stream-1mb")
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/stream-1mb",
params={"mode": "proxy"},
)
assert response.status_code == 200
assert len(response.content) == SIZE_1MB
assert compute_sha256(response.content) == expected_hash
@pytest.mark.integration
def test_stream_large_file_has_correct_headers(
self, integration_client, test_package, sized_content
):
"""Test that large file streaming has correct headers."""
project, package = test_package
content, expected_hash = sized_content(SIZE_100KB, seed=501)
upload_test_file(integration_client, project, package, content, tag="stream-hdr")
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/stream-hdr",
params={"mode": "proxy"},
)
assert response.status_code == 200
assert int(response.headers.get("Content-Length", 0)) == SIZE_100KB
assert response.headers.get("X-Checksum-SHA256") == expected_hash
assert response.headers.get("Accept-Ranges") == "bytes"
@pytest.mark.integration
def test_range_request_on_large_file(
self, integration_client, test_package, sized_content
):
"""Test range request on a larger file."""
project, package = test_package
content, _ = sized_content(SIZE_100KB, seed=502)
upload_test_file(integration_client, project, package, content, tag="range-large")
# Request a slice from the middle
start = 50000
end = 50999
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/range-large",
params={"mode": "proxy"},
headers={"Range": f"bytes={start}-{end}"},
)
assert response.status_code == 206
assert len(response.content) == 1000
assert response.content == content[start : end + 1]
class TestDownloadModes:
"""Tests for different download modes."""
@pytest.mark.integration
def test_proxy_mode_streams_content(self, integration_client, test_package):
"""Test proxy mode streams content through backend."""
project, package = test_package
content = b"proxy mode test content"
upload_test_file(integration_client, project, package, content, tag="mode-proxy")
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/mode-proxy",
params={"mode": "proxy"},
)
assert response.status_code == 200
assert response.content == content
@pytest.mark.integration
def test_presigned_mode_returns_url(self, integration_client, test_package):
"""Test presigned mode returns JSON with URL."""
project, package = test_package
content = b"presigned mode test"
upload_test_file(integration_client, project, package, content, tag="mode-presign")
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/mode-presign",
params={"mode": "presigned"},
)
assert response.status_code == 200
data = response.json()
assert "url" in data
assert "expires_at" in data
assert data["url"].startswith("http")
@pytest.mark.integration
def test_redirect_mode_returns_302(self, integration_client, test_package):
"""Test redirect mode returns 302 to presigned URL."""
project, package = test_package
content = b"redirect mode test"
upload_test_file(integration_client, project, package, content, tag="mode-redir")
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/mode-redir",
params={"mode": "redirect"},
follow_redirects=False,
)
assert response.status_code == 302
assert "Location" in response.headers
class TestIntegrityDuringStreaming:
"""Tests for data integrity during streaming downloads."""
@pytest.mark.integration
def test_checksum_header_matches_content(self, integration_client, test_package):
"""Test X-Checksum-SHA256 header matches actual downloaded content."""
project, package = test_package
content = b"integrity check content"
expected_hash = compute_sha256(content)
upload_test_file(integration_client, project, package, content, tag="integrity")
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/integrity",
params={"mode": "proxy"},
)
assert response.status_code == 200
header_hash = response.headers.get("X-Checksum-SHA256")
actual_hash = compute_sha256(response.content)
assert header_hash == expected_hash
assert actual_hash == expected_hash
assert header_hash == actual_hash
@pytest.mark.integration
def test_etag_matches_content_hash(self, integration_client, test_package):
"""Test ETag header matches content hash."""
project, package = test_package
content = b"etag integrity test"
expected_hash = compute_sha256(content)
upload_test_file(integration_client, project, package, content, tag="etag-int")
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/etag-int",
params={"mode": "proxy"},
)
assert response.status_code == 200
etag = response.headers.get("ETag", "").strip('"')
actual_hash = compute_sha256(response.content)
assert etag == expected_hash
assert actual_hash == expected_hash
@pytest.mark.integration
def test_digest_header_present(self, integration_client, test_package):
"""Test Digest header is present in RFC 3230 format."""
project, package = test_package
content = b"digest header test"
upload_test_file(integration_client, project, package, content, tag="digest")
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/digest",
params={"mode": "proxy"},
)
assert response.status_code == 200
assert "Digest" in response.headers
assert response.headers["Digest"].startswith("sha-256=")

View File

@@ -10,7 +10,6 @@ Tests cover:
- S3 storage verification
"""
import os
import pytest
import io
import threading
@@ -26,19 +25,6 @@ from tests.factories import (
class TestUploadBasics:
"""Tests for basic upload functionality."""
@pytest.mark.integration
def test_upload_returns_200(self, integration_client, test_package):
"""Test upload with valid file returns 200."""
project, package = test_package
content = b"valid file upload test"
files = {"file": ("test.bin", io.BytesIO(content), "application/octet-stream")}
response = integration_client.post(
f"/api/v1/project/{project}/{package}/upload",
files=files,
)
assert response.status_code == 200
@pytest.mark.integration
def test_upload_returns_artifact_id(self, integration_client, test_package):
"""Test upload returns the artifact ID (SHA256 hash)."""
@@ -115,83 +101,6 @@ class TestUploadBasics:
assert "created_at" in result
assert result["created_at"] is not None
@pytest.mark.integration
def test_upload_without_tag_succeeds(self, integration_client, test_package):
"""Test upload without tag succeeds (no tag created)."""
project, package = test_package
content = b"upload without tag test"
expected_hash = compute_sha256(content)
files = {"file": ("no_tag.bin", io.BytesIO(content), "application/octet-stream")}
response = integration_client.post(
f"/api/v1/project/{project}/{package}/upload",
files=files,
# No tag parameter
)
assert response.status_code == 200
result = response.json()
assert result["artifact_id"] == expected_hash
# Verify no tag was created - list tags and check
tags_response = integration_client.get(
f"/api/v1/project/{project}/{package}/tags"
)
assert tags_response.status_code == 200
tags = tags_response.json()
# Filter for tags pointing to this artifact
artifact_tags = [t for t in tags.get("items", tags) if t.get("artifact_id") == expected_hash]
assert len(artifact_tags) == 0, "Tag should not be created when not specified"
@pytest.mark.integration
def test_upload_creates_artifact_in_database(self, integration_client, test_package):
"""Test upload creates artifact record in database."""
project, package = test_package
content = b"database artifact test"
expected_hash = compute_sha256(content)
upload_test_file(integration_client, project, package, content)
# Verify artifact exists via API
response = integration_client.get(f"/api/v1/artifact/{expected_hash}")
assert response.status_code == 200
artifact = response.json()
assert artifact["id"] == expected_hash
assert artifact["size"] == len(content)
@pytest.mark.integration
@pytest.mark.requires_direct_s3
def test_upload_creates_object_in_s3(self, integration_client, test_package):
"""Test upload creates object in S3 storage."""
project, package = test_package
content = b"s3 object creation test"
expected_hash = compute_sha256(content)
upload_test_file(integration_client, project, package, content)
# Verify S3 object exists
assert s3_object_exists(expected_hash), "S3 object should exist after upload"
@pytest.mark.integration
def test_upload_with_tag_creates_tag_record(self, integration_client, test_package):
"""Test upload with tag creates tag record."""
project, package = test_package
content = b"tag creation test"
expected_hash = compute_sha256(content)
tag_name = "my-tag-v1"
upload_test_file(
integration_client, project, package, content, tag=tag_name
)
# Verify tag exists
tags_response = integration_client.get(
f"/api/v1/project/{project}/{package}/tags"
)
assert tags_response.status_code == 200
tags = tags_response.json()
tag_names = [t["name"] for t in tags.get("items", tags)]
assert tag_name in tag_names
class TestDuplicateUploads:
"""Tests for duplicate upload deduplication behavior."""
@@ -339,23 +248,6 @@ class TestDownload:
assert response.status_code == 200
assert response.content == original_content
@pytest.mark.integration
def test_download_by_tag_prefix(self, integration_client, test_package):
"""Test downloading artifact using tag: prefix."""
project, package = test_package
original_content = b"download by tag prefix test"
upload_test_file(
integration_client, project, package, original_content, tag="prefix-tag"
)
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/tag:prefix-tag",
params={"mode": "proxy"},
)
assert response.status_code == 200
assert response.content == original_content
@pytest.mark.integration
def test_download_nonexistent_tag(self, integration_client, test_package):
"""Test downloading nonexistent tag returns 404."""
@@ -366,33 +258,6 @@ class TestDownload:
)
assert response.status_code == 404
@pytest.mark.integration
def test_download_nonexistent_artifact(self, integration_client, test_package):
"""Test downloading nonexistent artifact ID returns 404."""
project, package = test_package
fake_hash = "0" * 64
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/artifact:{fake_hash}"
)
assert response.status_code == 404
@pytest.mark.integration
def test_download_from_nonexistent_project(self, integration_client, unique_test_id):
"""Test downloading from nonexistent project returns 404."""
response = integration_client.get(
f"/api/v1/project/nonexistent-project-{unique_test_id}/somepackage/+/sometag"
)
assert response.status_code == 404
@pytest.mark.integration
def test_download_from_nonexistent_package(self, integration_client, test_project, unique_test_id):
"""Test downloading from nonexistent package returns 404."""
response = integration_client.get(
f"/api/v1/project/{test_project}/nonexistent-package-{unique_test_id}/+/sometag"
)
assert response.status_code == 404
@pytest.mark.integration
def test_content_matches_original(self, integration_client, test_package):
"""Test downloaded content matches original exactly."""
@@ -410,111 +275,6 @@ class TestDownload:
assert response.content == original_content
class TestDownloadHeaders:
"""Tests for download response headers."""
@pytest.mark.integration
def test_download_content_type_header(self, integration_client, test_package):
"""Test download returns correct Content-Type header."""
project, package = test_package
content = b"content type header test"
upload_test_file(
integration_client, project, package, content,
filename="test.txt", tag="content-type-test"
)
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/content-type-test",
params={"mode": "proxy"},
)
assert response.status_code == 200
# Content-Type should be set (either text/plain or application/octet-stream)
assert "content-type" in response.headers
@pytest.mark.integration
def test_download_content_length_header(self, integration_client, test_package):
"""Test download returns correct Content-Length header."""
project, package = test_package
content = b"content length header test - exactly 41 bytes!"
expected_length = len(content)
upload_test_file(
integration_client, project, package, content, tag="content-length-test"
)
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/content-length-test",
params={"mode": "proxy"},
)
assert response.status_code == 200
assert "content-length" in response.headers
assert int(response.headers["content-length"]) == expected_length
@pytest.mark.integration
def test_download_content_disposition_header(self, integration_client, test_package):
"""Test download returns correct Content-Disposition header."""
project, package = test_package
content = b"content disposition test"
filename = "my-test-file.bin"
upload_test_file(
integration_client, project, package, content,
filename=filename, tag="disposition-test"
)
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/disposition-test",
params={"mode": "proxy"},
)
assert response.status_code == 200
assert "content-disposition" in response.headers
disposition = response.headers["content-disposition"]
assert "attachment" in disposition
assert filename in disposition
@pytest.mark.integration
def test_download_checksum_headers(self, integration_client, test_package):
"""Test download returns checksum headers."""
project, package = test_package
content = b"checksum header test content"
expected_hash = compute_sha256(content)
upload_test_file(
integration_client, project, package, content, tag="checksum-headers"
)
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/checksum-headers",
params={"mode": "proxy"},
)
assert response.status_code == 200
# Check for checksum headers
assert "x-checksum-sha256" in response.headers
assert response.headers["x-checksum-sha256"] == expected_hash
@pytest.mark.integration
def test_download_etag_header(self, integration_client, test_package):
"""Test download returns ETag header (artifact ID)."""
project, package = test_package
content = b"etag header test"
expected_hash = compute_sha256(content)
upload_test_file(
integration_client, project, package, content, tag="etag-test"
)
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/etag-test",
params={"mode": "proxy"},
)
assert response.status_code == 200
assert "etag" in response.headers
# ETag should contain the artifact ID (hash)
etag = response.headers["etag"].strip('"')
assert etag == expected_hash
class TestConcurrentUploads:
"""Tests for concurrent upload handling."""
@@ -541,7 +301,7 @@ class TestConcurrentUploads:
try:
from httpx import Client
base_url = os.environ.get("ORCHARD_TEST_URL", "http://localhost:8080")
base_url = "http://localhost:8080"
with Client(base_url=base_url, timeout=30.0) as client:
files = {
"file": (
@@ -637,7 +397,6 @@ class TestUploadFailureCleanup:
"""Tests for cleanup when uploads fail."""
@pytest.mark.integration
@pytest.mark.requires_direct_s3
def test_upload_failure_invalid_project_no_orphaned_s3(
self, integration_client, unique_test_id
):
@@ -660,7 +419,6 @@ class TestUploadFailureCleanup:
)
@pytest.mark.integration
@pytest.mark.requires_direct_s3
def test_upload_failure_invalid_package_no_orphaned_s3(
self, integration_client, test_project, unique_test_id
):
@@ -708,7 +466,6 @@ class TestS3StorageVerification:
"""Tests to verify S3 storage behavior."""
@pytest.mark.integration
@pytest.mark.requires_direct_s3
def test_s3_single_object_after_duplicates(
self, integration_client, test_package, unique_test_id
):
@@ -764,7 +521,6 @@ class TestSecurityPathTraversal:
"""
@pytest.mark.integration
@pytest.mark.requires_direct_s3
def test_path_traversal_in_filename_stored_safely(
self, integration_client, test_package
):

View File

@@ -1,347 +0,0 @@
"""
Integration tests for package version API endpoints.
Tests cover:
- Version creation via upload
- Version auto-detection from filename
- Version listing and retrieval
- Download by version prefix
- Version deletion
"""
import pytest
import io
from tests.factories import (
compute_sha256,
upload_test_file,
)
class TestVersionCreation:
"""Tests for creating versions via upload."""
@pytest.mark.integration
def test_upload_with_explicit_version(self, integration_client, test_package):
"""Test upload with explicit version parameter creates version record."""
project, package = test_package
content = b"version creation test"
expected_hash = compute_sha256(content)
files = {"file": ("app.tar.gz", io.BytesIO(content), "application/octet-stream")}
response = integration_client.post(
f"/api/v1/project/{project}/{package}/upload",
files=files,
data={"version": "1.0.0"},
)
assert response.status_code == 200
result = response.json()
assert result["artifact_id"] == expected_hash
assert result.get("version") == "1.0.0"
assert result.get("version_source") == "explicit"
@pytest.mark.integration
def test_upload_with_version_and_tag(self, integration_client, test_package):
"""Test upload with both version and tag creates both records."""
project, package = test_package
content = b"version and tag test"
files = {"file": ("app.tar.gz", io.BytesIO(content), "application/octet-stream")}
response = integration_client.post(
f"/api/v1/project/{project}/{package}/upload",
files=files,
data={"version": "2.0.0", "tag": "latest"},
)
assert response.status_code == 200
result = response.json()
assert result.get("version") == "2.0.0"
# Verify tag was also created
tags_response = integration_client.get(
f"/api/v1/project/{project}/{package}/tags"
)
assert tags_response.status_code == 200
tags = tags_response.json()
tag_names = [t["name"] for t in tags.get("items", tags)]
assert "latest" in tag_names
@pytest.mark.integration
def test_duplicate_version_same_content_succeeds(self, integration_client, test_package):
"""Test uploading same version with same content succeeds (deduplication)."""
project, package = test_package
content = b"version dedup test"
# First upload with version
files1 = {"file": ("app1.tar.gz", io.BytesIO(content), "application/octet-stream")}
response1 = integration_client.post(
f"/api/v1/project/{project}/{package}/upload",
files=files1,
data={"version": "3.0.0"},
)
assert response1.status_code == 200
# Second upload with same version and same content succeeds
files2 = {"file": ("app2.tar.gz", io.BytesIO(content), "application/octet-stream")}
response2 = integration_client.post(
f"/api/v1/project/{project}/{package}/upload",
files=files2,
data={"version": "3.0.0"},
)
# This succeeds because it's the same artifact (deduplication)
assert response2.status_code == 200
class TestVersionAutoDetection:
"""Tests for automatic version detection from filename."""
@pytest.mark.integration
def test_version_detected_from_filename_tarball(self, integration_client, test_package):
"""Test version is auto-detected from tarball filename or metadata."""
project, package = test_package
content = b"auto detect version tarball"
files = {"file": ("myapp-1.2.3.tar.gz", io.BytesIO(content), "application/octet-stream")}
response = integration_client.post(
f"/api/v1/project/{project}/{package}/upload",
files=files,
)
assert response.status_code == 200
result = response.json()
assert result.get("version") == "1.2.3"
# Version source can be 'filename' or 'metadata' depending on detection order
assert result.get("version_source") in ["filename", "metadata"]
@pytest.mark.integration
def test_version_detected_from_filename_zip(self, integration_client, test_package):
"""Test version is auto-detected from zip filename."""
project, package = test_package
content = b"auto detect version zip"
files = {"file": ("package-2.0.0.zip", io.BytesIO(content), "application/octet-stream")}
response = integration_client.post(
f"/api/v1/project/{project}/{package}/upload",
files=files,
)
assert response.status_code == 200
result = response.json()
assert result.get("version") == "2.0.0"
assert result.get("version_source") == "filename"
@pytest.mark.integration
def test_explicit_version_overrides_filename(self, integration_client, test_package):
"""Test explicit version parameter overrides filename detection."""
project, package = test_package
content = b"explicit override test"
files = {"file": ("myapp-1.0.0.tar.gz", io.BytesIO(content), "application/octet-stream")}
response = integration_client.post(
f"/api/v1/project/{project}/{package}/upload",
files=files,
data={"version": "9.9.9"},
)
assert response.status_code == 200
result = response.json()
assert result.get("version") == "9.9.9"
assert result.get("version_source") == "explicit"
@pytest.mark.integration
def test_no_version_detected_from_plain_filename(self, integration_client, test_package):
"""Test no version is created for filenames without version pattern."""
project, package = test_package
content = b"no version in filename"
files = {"file": ("plain-file.bin", io.BytesIO(content), "application/octet-stream")}
response = integration_client.post(
f"/api/v1/project/{project}/{package}/upload",
files=files,
)
assert response.status_code == 200
result = response.json()
# Version should be None or not present
assert result.get("version") is None
class TestVersionListing:
"""Tests for listing and retrieving versions."""
@pytest.mark.integration
def test_list_versions(self, integration_client, test_package):
"""Test listing all versions for a package."""
project, package = test_package
# Create multiple versions
for ver in ["1.0.0", "1.1.0", "2.0.0"]:
content = f"version {ver} content".encode()
files = {"file": (f"app-{ver}.tar.gz", io.BytesIO(content), "application/octet-stream")}
response = integration_client.post(
f"/api/v1/project/{project}/{package}/upload",
files=files,
data={"version": ver},
)
assert response.status_code == 200
# List versions
response = integration_client.get(
f"/api/v1/project/{project}/{package}/versions"
)
assert response.status_code == 200
data = response.json()
versions = [v["version"] for v in data.get("items", data)]
assert "1.0.0" in versions
assert "1.1.0" in versions
assert "2.0.0" in versions
@pytest.mark.integration
def test_get_specific_version(self, integration_client, test_package):
"""Test getting details for a specific version."""
project, package = test_package
content = b"specific version test"
expected_hash = compute_sha256(content)
# Create version
files = {"file": ("app-4.0.0.tar.gz", io.BytesIO(content), "application/octet-stream")}
integration_client.post(
f"/api/v1/project/{project}/{package}/upload",
files=files,
data={"version": "4.0.0"},
)
# Get version details
response = integration_client.get(
f"/api/v1/project/{project}/{package}/versions/4.0.0"
)
assert response.status_code == 200
data = response.json()
assert data["version"] == "4.0.0"
assert data["artifact_id"] == expected_hash
@pytest.mark.integration
def test_get_nonexistent_version_returns_404(self, integration_client, test_package):
"""Test getting nonexistent version returns 404."""
project, package = test_package
response = integration_client.get(
f"/api/v1/project/{project}/{package}/versions/99.99.99"
)
assert response.status_code == 404
class TestDownloadByVersion:
"""Tests for downloading artifacts by version."""
@pytest.mark.integration
def test_download_by_version_prefix(self, integration_client, test_package):
"""Test downloading artifact using version: prefix."""
project, package = test_package
content = b"download by version test"
expected_hash = compute_sha256(content)
# Upload with version
files = {"file": ("app.tar.gz", io.BytesIO(content), "application/octet-stream")}
integration_client.post(
f"/api/v1/project/{project}/{package}/upload",
files=files,
data={"version": "5.0.0"},
)
# Download by version prefix
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/version:5.0.0",
params={"mode": "proxy"},
)
assert response.status_code == 200
assert response.content == content
@pytest.mark.integration
def test_download_nonexistent_version_returns_404(self, integration_client, test_package):
"""Test downloading nonexistent version returns 404."""
project, package = test_package
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/version:99.0.0"
)
assert response.status_code == 404
@pytest.mark.integration
def test_version_resolution_priority(self, integration_client, test_package):
"""Test that version: prefix explicitly resolves to version, not tag."""
project, package = test_package
version_content = b"this is the version content"
tag_content = b"this is the tag content"
# Create a version 6.0.0
files1 = {"file": ("app-v.tar.gz", io.BytesIO(version_content), "application/octet-stream")}
integration_client.post(
f"/api/v1/project/{project}/{package}/upload",
files=files1,
data={"version": "6.0.0"},
)
# Create a tag named "6.0.0" pointing to different content
files2 = {"file": ("app-t.tar.gz", io.BytesIO(tag_content), "application/octet-stream")}
integration_client.post(
f"/api/v1/project/{project}/{package}/upload",
files=files2,
data={"tag": "6.0.0"},
)
# Download with version: prefix should get version content
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/version:6.0.0",
params={"mode": "proxy"},
)
assert response.status_code == 200
assert response.content == version_content
# Download with tag: prefix should get tag content
response2 = integration_client.get(
f"/api/v1/project/{project}/{package}/+/tag:6.0.0",
params={"mode": "proxy"},
)
assert response2.status_code == 200
assert response2.content == tag_content
class TestVersionDeletion:
"""Tests for deleting versions."""
@pytest.mark.integration
def test_delete_version(self, integration_client, test_package):
"""Test deleting a version."""
project, package = test_package
content = b"delete version test"
# Create version
files = {"file": ("app.tar.gz", io.BytesIO(content), "application/octet-stream")}
integration_client.post(
f"/api/v1/project/{project}/{package}/upload",
files=files,
data={"version": "7.0.0"},
)
# Verify version exists
response = integration_client.get(
f"/api/v1/project/{project}/{package}/versions/7.0.0"
)
assert response.status_code == 200
# Delete version - returns 204 No Content on success
delete_response = integration_client.delete(
f"/api/v1/project/{project}/{package}/versions/7.0.0"
)
assert delete_response.status_code == 204
# Verify version no longer exists
response2 = integration_client.get(
f"/api/v1/project/{project}/{package}/versions/7.0.0"
)
assert response2.status_code == 404
@pytest.mark.integration
def test_delete_nonexistent_version_returns_404(self, integration_client, test_package):
"""Test deleting nonexistent version returns 404."""
project, package = test_package
response = integration_client.delete(
f"/api/v1/project/{project}/{package}/versions/99.0.0"
)
assert response.status_code == 404

View File

@@ -1,412 +0,0 @@
"""
Integration tests for version API endpoints.
Tests cover:
- Version creation via upload
- Version auto-detection from filename
- Version listing with pagination
- Version deletion
- Download by version ref
- ref_count behavior with version operations
"""
import pytest
from tests.factories import upload_test_file
class TestVersionCreation:
"""Tests for version creation during upload."""
@pytest.mark.integration
def test_upload_with_explicit_version(self, integration_client, test_package):
"""Test creating a version via explicit version parameter."""
project_name, package_name = test_package
result = upload_test_file(
integration_client,
project_name,
package_name,
b"version create test",
tag="latest",
version="1.0.0",
)
assert result["tag"] == "latest"
assert result["version"] == "1.0.0"
assert result["version_source"] == "explicit"
assert result["artifact_id"]
@pytest.mark.integration
def test_upload_with_version_auto_detect_from_tarball(
self, integration_client, test_package
):
"""Test version auto-detection from tarball filename pattern."""
project_name, package_name = test_package
result = upload_test_file(
integration_client,
project_name,
package_name,
b"auto version test",
filename="myapp-2.1.0.tar.gz",
)
assert result["version"] == "2.1.0"
# Tarball metadata extractor parses version from filename
assert result["version_source"] == "metadata"
@pytest.mark.integration
def test_upload_with_version_auto_detect_v_prefix(
self, integration_client, test_package
):
"""Test version auto-detection strips 'v' prefix from tarball filename."""
project_name, package_name = test_package
result = upload_test_file(
integration_client,
project_name,
package_name,
b"v prefix test",
filename="package-v3.0.0.tar.gz",
)
assert result["version"] == "3.0.0"
# Tarball metadata extractor parses version from filename
assert result["version_source"] == "metadata"
@pytest.mark.integration
def test_upload_duplicate_version_warning(self, integration_client, test_package):
"""Test that duplicate version during upload returns response without error."""
project_name, package_name = test_package
# Upload with version 1.0.0
upload_test_file(
integration_client,
project_name,
package_name,
b"first upload",
version="1.0.0",
)
# Upload different content with same version - should succeed but no new version
result = upload_test_file(
integration_client,
project_name,
package_name,
b"second upload different content",
version="1.0.0",
)
# Upload succeeds but version may not be set (duplicate)
assert result["artifact_id"]
class TestVersionCRUD:
"""Tests for version list, get, delete operations."""
@pytest.mark.integration
def test_list_versions(self, integration_client, test_package):
"""Test listing versions for a package."""
project_name, package_name = test_package
# Create some versions
upload_test_file(
integration_client,
project_name,
package_name,
b"v1 content",
version="1.0.0",
)
upload_test_file(
integration_client,
project_name,
package_name,
b"v2 content",
version="2.0.0",
)
response = integration_client.get(
f"/api/v1/project/{project_name}/{package_name}/versions"
)
assert response.status_code == 200
data = response.json()
assert "items" in data
assert "pagination" in data
versions = [v["version"] for v in data["items"]]
assert "1.0.0" in versions
assert "2.0.0" in versions
@pytest.mark.integration
def test_list_versions_with_artifact_info(self, integration_client, test_package):
"""Test that version list includes artifact metadata."""
project_name, package_name = test_package
upload_test_file(
integration_client,
project_name,
package_name,
b"version with info",
version="1.0.0",
tag="release",
)
response = integration_client.get(
f"/api/v1/project/{project_name}/{package_name}/versions"
)
assert response.status_code == 200
data = response.json()
assert len(data["items"]) >= 1
version_item = next(
(v for v in data["items"] if v["version"] == "1.0.0"), None
)
assert version_item is not None
assert "size" in version_item
assert "artifact_id" in version_item
assert "tags" in version_item
assert "release" in version_item["tags"]
@pytest.mark.integration
def test_get_version(self, integration_client, test_package):
"""Test getting a specific version."""
project_name, package_name = test_package
upload_result = upload_test_file(
integration_client,
project_name,
package_name,
b"get version test",
version="3.0.0",
)
response = integration_client.get(
f"/api/v1/project/{project_name}/{package_name}/versions/3.0.0"
)
assert response.status_code == 200
data = response.json()
assert data["version"] == "3.0.0"
assert data["artifact_id"] == upload_result["artifact_id"]
assert data["version_source"] == "explicit"
@pytest.mark.integration
def test_get_version_not_found(self, integration_client, test_package):
"""Test getting a non-existent version returns 404."""
project_name, package_name = test_package
response = integration_client.get(
f"/api/v1/project/{project_name}/{package_name}/versions/99.99.99"
)
assert response.status_code == 404
@pytest.mark.integration
def test_delete_version(self, integration_client, test_package):
"""Test deleting a version."""
project_name, package_name = test_package
upload_test_file(
integration_client,
project_name,
package_name,
b"delete version test",
version="4.0.0",
)
# Delete version
response = integration_client.delete(
f"/api/v1/project/{project_name}/{package_name}/versions/4.0.0"
)
assert response.status_code == 204
# Verify deleted
response = integration_client.get(
f"/api/v1/project/{project_name}/{package_name}/versions/4.0.0"
)
assert response.status_code == 404
class TestVersionDownload:
"""Tests for downloading artifacts by version reference."""
@pytest.mark.integration
def test_download_by_version_prefix(self, integration_client, test_package):
"""Test downloading an artifact using version: prefix."""
project_name, package_name = test_package
content = b"download by version test"
upload_test_file(
integration_client,
project_name,
package_name,
content,
version="5.0.0",
)
response = integration_client.get(
f"/api/v1/project/{project_name}/{package_name}/+/version:5.0.0",
follow_redirects=False,
)
# Should either redirect or return content
assert response.status_code in [200, 302, 307]
@pytest.mark.integration
def test_download_by_implicit_version(self, integration_client, test_package):
"""Test downloading an artifact using version number directly (no prefix)."""
project_name, package_name = test_package
content = b"implicit version download test"
upload_test_file(
integration_client,
project_name,
package_name,
content,
version="6.0.0",
)
response = integration_client.get(
f"/api/v1/project/{project_name}/{package_name}/+/6.0.0",
follow_redirects=False,
)
# Should resolve version first (before tag)
assert response.status_code in [200, 302, 307]
@pytest.mark.integration
def test_version_takes_precedence_over_tag(self, integration_client, test_package):
"""Test that version is checked before tag when resolving refs."""
project_name, package_name = test_package
# Upload with version "1.0"
version_result = upload_test_file(
integration_client,
project_name,
package_name,
b"version content",
version="1.0",
)
# Create a tag with the same name "1.0" pointing to different artifact
tag_result = upload_test_file(
integration_client,
project_name,
package_name,
b"tag content different",
tag="1.0",
)
# Download by "1.0" should resolve to version, not tag
# Since version:1.0 artifact was uploaded first
response = integration_client.get(
f"/api/v1/project/{project_name}/{package_name}/+/1.0",
follow_redirects=False,
)
assert response.status_code in [200, 302, 307]
class TestTagVersionEnrichment:
"""Tests for tag responses including version information."""
@pytest.mark.integration
def test_tag_response_includes_version(self, integration_client, test_package):
"""Test that tag responses include version of the artifact."""
project_name, package_name = test_package
# Upload with both version and tag
upload_test_file(
integration_client,
project_name,
package_name,
b"enriched tag test",
version="7.0.0",
tag="stable",
)
# Get tag and check version field
response = integration_client.get(
f"/api/v1/project/{project_name}/{package_name}/tags/stable"
)
assert response.status_code == 200
data = response.json()
assert data["name"] == "stable"
assert data["version"] == "7.0.0"
@pytest.mark.integration
def test_tag_list_includes_versions(self, integration_client, test_package):
"""Test that tag list responses include version for each tag."""
project_name, package_name = test_package
upload_test_file(
integration_client,
project_name,
package_name,
b"list version test",
version="8.0.0",
tag="latest",
)
response = integration_client.get(
f"/api/v1/project/{project_name}/{package_name}/tags"
)
assert response.status_code == 200
data = response.json()
tag_item = next((t for t in data["items"] if t["name"] == "latest"), None)
assert tag_item is not None
assert tag_item.get("version") == "8.0.0"
class TestVersionPagination:
"""Tests for version listing pagination and sorting."""
@pytest.mark.integration
def test_versions_pagination(self, integration_client, test_package):
"""Test version listing respects pagination."""
project_name, package_name = test_package
response = integration_client.get(
f"/api/v1/project/{project_name}/{package_name}/versions?limit=5"
)
assert response.status_code == 200
data = response.json()
assert "pagination" in data
assert data["pagination"]["limit"] == 5
@pytest.mark.integration
def test_versions_sorting(self, integration_client, test_package):
"""Test version listing can be sorted."""
project_name, package_name = test_package
# Create versions with different timestamps
upload_test_file(
integration_client,
project_name,
package_name,
b"sort test 1",
version="1.0.0",
)
upload_test_file(
integration_client,
project_name,
package_name,
b"sort test 2",
version="2.0.0",
)
# Test ascending sort
response = integration_client.get(
f"/api/v1/project/{project_name}/{package_name}/versions?sort=version&order=asc"
)
assert response.status_code == 200
data = response.json()
versions = [v["version"] for v in data["items"]]
# First version should be 1.0.0 when sorted ascending
if len(versions) >= 2:
assert versions.index("1.0.0") < versions.index("2.0.0")

View File

@@ -378,7 +378,7 @@ class TestDeduplicationBehavior:
result2 = mock_storage._store_simple(file2)
assert result1.sha256 == result2.sha256
assert result1.s3_key == result2.s3_key # gitleaks:allow
assert result1.s3_key == result2.s3_key
@pytest.mark.unit
def test_different_content_different_keys(self, mock_storage):
@@ -393,7 +393,7 @@ class TestDeduplicationBehavior:
result2 = mock_storage._store_simple(file2)
assert result1.sha256 != result2.sha256
assert result1.s3_key != result2.s3_key # gitleaks:allow
assert result1.s3_key != result2.s3_key
# =============================================================================

View File

@@ -6,7 +6,7 @@ services:
context: .
dockerfile: Dockerfile.local
ports:
- "0.0.0.0:8080:8080"
- "8080:8080"
environment:
- ORCHARD_SERVER_HOST=0.0.0.0
- ORCHARD_SERVER_PORT=8080
@@ -44,13 +44,8 @@ services:
retries: 3
security_opt:
- no-new-privileges:true
cap_drop:
- ALL
deploy:
resources:
limits:
cpus: '1.0'
memory: 1G
mem_limit: 1g
cpus: 1.0
postgres:
image: postgres:16-alpine
@@ -62,7 +57,7 @@ services:
- postgres-data-local:/var/lib/postgresql/data
- ./migrations:/docker-entrypoint-initdb.d:ro
ports:
- "127.0.0.1:5432:5432"
- "5432:5432"
healthcheck:
test: ["CMD-SHELL", "pg_isready -U orchard -d orchard"]
interval: 10s
@@ -71,11 +66,10 @@ services:
networks:
- orchard-network
restart: unless-stopped
deploy:
resources:
limits:
cpus: '0.5'
memory: 512M
security_opt:
- no-new-privileges:true
mem_limit: 512m
cpus: 0.5
minio:
image: minio/minio:latest
@@ -86,8 +80,8 @@ services:
volumes:
- minio-data-local:/data
ports:
- "127.0.0.1:9000:9000"
- "127.0.0.1:9001:9001"
- "9000:9000"
- "9001:9001"
healthcheck:
test: ["CMD", "mc", "ready", "local"]
interval: 10s
@@ -96,11 +90,10 @@ services:
networks:
- orchard-network
restart: unless-stopped
deploy:
resources:
limits:
cpus: '0.5'
memory: 512M
security_opt:
- no-new-privileges:true
mem_limit: 512m
cpus: 0.5
minio-init:
image: minio/mc:latest
@@ -116,11 +109,10 @@ services:
"
networks:
- orchard-network
deploy:
resources:
limits:
cpus: '0.25'
memory: 128M
security_opt:
- no-new-privileges:true
mem_limit: 128m
cpus: 0.25
redis:
image: redis:7-alpine
@@ -128,7 +120,7 @@ services:
volumes:
- redis-data-local:/data
ports:
- "127.0.0.1:6379:6379"
- "6379:6379"
healthcheck:
test: ["CMD", "redis-cli", "ping"]
interval: 10s
@@ -137,11 +129,10 @@ services:
networks:
- orchard-network
restart: unless-stopped
deploy:
resources:
limits:
cpus: '0.25'
memory: 256M
security_opt:
- no-new-privileges:true
mem_limit: 256m
cpus: 0.25
volumes:
postgres-data-local:

View File

@@ -6,7 +6,7 @@ services:
context: .
dockerfile: Dockerfile
ports:
- "127.0.0.1:8080:8080"
- "8080:8080"
environment:
- ORCHARD_SERVER_HOST=0.0.0.0
- ORCHARD_SERVER_PORT=8080
@@ -42,13 +42,8 @@ services:
retries: 3
security_opt:
- no-new-privileges:true
cap_drop:
- ALL
deploy:
resources:
limits:
cpus: '1.0'
memory: 1G
mem_limit: 1g
cpus: 1.0
postgres:
image: containers.global.bsf.tools/postgres:16-alpine
@@ -60,7 +55,7 @@ services:
- postgres-data:/var/lib/postgresql/data
- ./migrations:/docker-entrypoint-initdb.d:ro
ports:
- "127.0.0.1:5432:5432"
- "5432:5432"
healthcheck:
test: ["CMD-SHELL", "pg_isready -U orchard -d orchard"]
interval: 10s
@@ -71,13 +66,8 @@ services:
restart: unless-stopped
security_opt:
- no-new-privileges:true
cap_drop:
- ALL
deploy:
resources:
limits:
cpus: '0.5'
memory: 512M
mem_limit: 512m
cpus: 0.5
minio:
image: containers.global.bsf.tools/minio/minio:latest
@@ -88,8 +78,8 @@ services:
volumes:
- minio-data:/data
ports:
- "127.0.0.1:9000:9000"
- "127.0.0.1:9001:9001"
- "9000:9000"
- "9001:9001"
healthcheck:
test: ["CMD", "mc", "ready", "local"]
interval: 10s
@@ -100,13 +90,8 @@ services:
restart: unless-stopped
security_opt:
- no-new-privileges:true
cap_drop:
- ALL
deploy:
resources:
limits:
cpus: '0.5'
memory: 512M
mem_limit: 512m
cpus: 0.5
minio-init:
image: containers.global.bsf.tools/minio/mc:latest
@@ -124,13 +109,8 @@ services:
- orchard-network
security_opt:
- no-new-privileges:true
cap_drop:
- ALL
deploy:
resources:
limits:
cpus: '0.25'
memory: 128M
mem_limit: 128m
cpus: 0.25
redis:
image: containers.global.bsf.tools/redis:7-alpine
@@ -138,7 +118,7 @@ services:
volumes:
- redis-data:/data
ports:
- "127.0.0.1:6379:6379"
- "6379:6379"
healthcheck:
test: ["CMD", "redis-cli", "ping"]
interval: 10s
@@ -149,13 +129,8 @@ services:
restart: unless-stopped
security_opt:
- no-new-privileges:true
cap_drop:
- ALL
deploy:
resources:
limits:
cpus: '0.25'
memory: 256M
mem_limit: 256m
cpus: 0.25
volumes:
postgres-data:

View File

@@ -1,294 +0,0 @@
# Integrity Verification
Orchard uses content-addressable storage with SHA256 hashing to ensure artifact integrity. This document describes how integrity verification works and how to use it.
## How It Works
### Content-Addressable Storage
Orchard stores artifacts using their SHA256 hash as the unique identifier. This provides several benefits:
1. **Automatic deduplication**: Identical content is stored only once
2. **Built-in integrity**: The artifact ID *is* the content hash
3. **Tamper detection**: Any modification changes the hash, making corruption detectable
When you upload a file:
1. Orchard computes the SHA256 hash of the content
2. The hash becomes the artifact ID (64-character hex string)
3. The file is stored in S3 at `fruits/{hash[0:2]}/{hash[2:4]}/{hash}`
4. The hash and metadata are recorded in the database
### Hash Format
- Algorithm: SHA256
- Format: 64-character lowercase hexadecimal string
- Example: `dffd6021bb2bd5b0af676290809ec3a53191dd81c7f70a4b28688a362182986f`
## Client-Side Verification
### Before Upload
Compute the hash locally before uploading to verify the server received your content correctly:
```python
import hashlib
def compute_sha256(content: bytes) -> str:
return hashlib.sha256(content).hexdigest()
# Compute hash before upload
content = open("myfile.tar.gz", "rb").read()
local_hash = compute_sha256(content)
# Upload the file
response = requests.post(
f"{base_url}/api/v1/project/{project}/{package}/upload",
files={"file": ("myfile.tar.gz", content)},
)
result = response.json()
# Verify server computed the same hash
assert result["artifact_id"] == local_hash, "Hash mismatch!"
```
### Providing Expected Hash on Upload
You can provide the expected hash in the upload request. The server will reject the upload if the computed hash doesn't match:
```python
response = requests.post(
f"{base_url}/api/v1/project/{project}/{package}/upload",
files={"file": ("myfile.tar.gz", content)},
headers={"X-Checksum-SHA256": local_hash},
)
# Returns 422 if hash doesn't match
if response.status_code == 422:
print("Checksum mismatch - upload rejected")
```
### After Download
Verify downloaded content matches the expected hash using response headers:
```python
response = requests.get(
f"{base_url}/api/v1/project/{project}/{package}/+/{tag}",
params={"mode": "proxy"},
)
# Get expected hash from header
expected_hash = response.headers.get("X-Checksum-SHA256")
# Compute hash of downloaded content
actual_hash = compute_sha256(response.content)
# Verify
if actual_hash != expected_hash:
raise Exception(f"Integrity check failed! Expected {expected_hash}, got {actual_hash}")
```
### Response Headers for Verification
Download responses include multiple headers for verification:
| Header | Format | Description |
|--------|--------|-------------|
| `X-Checksum-SHA256` | Hex string | SHA256 hash (64 chars) |
| `ETag` | `"<hash>"` | SHA256 hash in quotes |
| `Digest` | `sha-256=<base64>` | RFC 3230 format (base64-encoded) |
| `Content-Length` | Integer | File size in bytes |
### Server-Side Verification on Download
Request server-side verification during download:
```bash
# Pre-verification: Server verifies before streaming (returns 500 if corrupt)
curl "${base_url}/api/v1/project/${project}/${package}/+/${tag}?mode=proxy&verify=true&verify_mode=pre"
# Stream verification: Server verifies while streaming (logs error if corrupt)
curl "${base_url}/api/v1/project/${project}/${package}/+/${tag}?mode=proxy&verify=true&verify_mode=stream"
```
The `X-Verified` header indicates whether server-side verification was performed:
- `X-Verified: true` - Content was verified by the server
## Server-Side Consistency Check
### Consistency Check Endpoint
Administrators can run a consistency check to verify all stored artifacts:
```bash
curl "${base_url}/api/v1/admin/consistency-check"
```
Response:
```json
{
"total_artifacts_checked": 1234,
"healthy": true,
"orphaned_s3_objects": 0,
"missing_s3_objects": 0,
"size_mismatches": 0,
"orphaned_s3_keys": [],
"missing_s3_keys": [],
"size_mismatch_artifacts": []
}
```
### What the Check Verifies
1. **Missing S3 objects**: Database records with no corresponding S3 object
2. **Orphaned S3 objects**: S3 objects with no database record
3. **Size mismatches**: S3 object size doesn't match database record
### Running Consistency Checks
**Manual check:**
```bash
# Check all artifacts
curl "${base_url}/api/v1/admin/consistency-check"
# Limit results (for large deployments)
curl "${base_url}/api/v1/admin/consistency-check?limit=100"
```
**Scheduled checks (recommended):**
Set up a cron job or Kubernetes CronJob to run periodic checks:
```yaml
# Kubernetes CronJob example
apiVersion: batch/v1
kind: CronJob
metadata:
name: orchard-consistency-check
spec:
schedule: "0 2 * * *" # Daily at 2 AM
jobTemplate:
spec:
template:
spec:
containers:
- name: check
image: curlimages/curl
command:
- /bin/sh
- -c
- |
response=$(curl -s "${ORCHARD_URL}/api/v1/admin/consistency-check")
healthy=$(echo "$response" | jq -r '.healthy')
if [ "$healthy" != "true" ]; then
echo "ALERT: Consistency check failed!"
echo "$response"
exit 1
fi
echo "Consistency check passed"
restartPolicy: OnFailure
```
## Recovery Procedures
### Corrupted Artifact (Size Mismatch)
If the consistency check reports size mismatches:
1. **Identify affected artifacts:**
```bash
curl "${base_url}/api/v1/admin/consistency-check" | jq '.size_mismatch_artifacts'
```
2. **Check if artifact can be re-uploaded:**
- If the original content is available, delete the corrupted artifact and re-upload
- The same content will produce the same artifact ID
3. **If original content is lost:**
- The artifact data is corrupted and cannot be recovered
- Delete the artifact record and notify affected users
- Consider restoring from backup if available
### Missing S3 Object
If database records exist but S3 objects are missing:
1. **Identify affected artifacts:**
```bash
curl "${base_url}/api/v1/admin/consistency-check" | jq '.missing_s3_keys'
```
2. **Check S3 bucket:**
- Verify the S3 bucket exists and is accessible
- Check S3 access logs for deletion events
- Check if objects were moved or lifecycle-deleted
3. **Recovery options:**
- Restore from S3 versioning (if enabled)
- Restore from backup
- Re-upload original content (if available)
- Delete orphaned database records
### Orphaned S3 Objects
If S3 objects exist without database records:
1. **Identify orphaned objects:**
```bash
curl "${base_url}/api/v1/admin/consistency-check" | jq '.orphaned_s3_keys'
```
2. **Investigate cause:**
- Upload interrupted before database commit?
- Database record deleted but S3 cleanup failed?
3. **Resolution:**
- If content is needed, create database record manually
- If content is not needed, delete the S3 object to reclaim storage
### Preventive Measures
1. **Enable S3 versioning** to recover from accidental deletions
2. **Regular backups** of both database and S3 bucket
3. **Scheduled consistency checks** to detect issues early
4. **Monitoring and alerting** on consistency check failures
5. **Audit logging** to track all artifact operations
## Verification in CI/CD
### Verifying Artifacts in Pipelines
```bash
#!/bin/bash
# Download and verify artifact in CI pipeline
ARTIFACT_URL="${ORCHARD_URL}/api/v1/project/${PROJECT}/${PACKAGE}/+/${TAG}"
# Download with verification headers
response=$(curl -s -D - "${ARTIFACT_URL}?mode=proxy" -o artifact.tar.gz)
expected_hash=$(echo "$response" | grep -i "X-Checksum-SHA256" | cut -d: -f2 | tr -d ' \r')
# Compute actual hash
actual_hash=$(sha256sum artifact.tar.gz | cut -d' ' -f1)
# Verify
if [ "$actual_hash" != "$expected_hash" ]; then
echo "ERROR: Integrity check failed!"
echo "Expected: $expected_hash"
echo "Actual: $actual_hash"
exit 1
fi
echo "Integrity verified: $actual_hash"
```
### Using Server-Side Verification
For critical deployments, use server-side pre-verification:
```bash
# Server verifies before streaming - returns 500 if corrupt
curl -f "${ARTIFACT_URL}?mode=proxy&verify=true&verify_mode=pre" -o artifact.tar.gz
```
This ensures the artifact is verified before any bytes are streamed to your pipeline.

View File

@@ -2,9 +2,9 @@
<html lang="en">
<head>
<meta charset="UTF-8" />
<link rel="icon" type="image/svg+xml" href="/orchard.svg" />
<link rel="icon" type="image/svg+xml" href="/vite.svg" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>Orchard</title>
<title>Orchard - Content-Addressable Storage</title>
</head>
<body>
<div id="root"></div>

View File

@@ -19,7 +19,6 @@
"@types/react": "^18.2.48",
"@types/react-dom": "^18.2.18",
"@vitejs/plugin-react": "^4.2.1",
"@vitest/coverage-v8": "^1.3.1",
"jsdom": "^24.0.0",
"typescript": "^5.3.3",
"vite": "^5.0.12",
@@ -33,19 +32,6 @@
"dev": true,
"license": "MIT"
},
"node_modules/@ampproject/remapping": {
"version": "2.3.0",
"resolved": "https://registry.npmjs.org/@ampproject/remapping/-/remapping-2.3.0.tgz",
"integrity": "sha512-30iZtAPgz+LTIYoeivqYo853f02jBYSd5uGnGpkFV0M3xOt9aN73erkgYAmZU43x4VfqcnLxW9Kpg3R5LC4YYw==",
"dev": true,
"dependencies": {
"@jridgewell/gen-mapping": "^0.3.5",
"@jridgewell/trace-mapping": "^0.3.24"
},
"engines": {
"node": ">=6.0.0"
}
},
"node_modules/@asamuzakjp/css-color": {
"version": "3.2.0",
"resolved": "https://registry.npmjs.org/@asamuzakjp/css-color/-/css-color-3.2.0.tgz",
@@ -359,12 +345,6 @@
"node": ">=6.9.0"
}
},
"node_modules/@bcoe/v8-coverage": {
"version": "0.2.3",
"resolved": "https://registry.npmjs.org/@bcoe/v8-coverage/-/v8-coverage-0.2.3.tgz",
"integrity": "sha512-0hYQ8SB4Db5zvZB4axdMHGwEaQjkZzFjQiN9LVYvIFB2nSUHW9tYpxWriPrWDASIxiaXax83REcLxuSdnGPZtw==",
"dev": true
},
"node_modules/@csstools/color-helpers": {
"version": "5.1.0",
"resolved": "https://registry.npmjs.org/@csstools/color-helpers/-/color-helpers-5.1.0.tgz",
@@ -871,15 +851,6 @@
"node": ">=12"
}
},
"node_modules/@istanbuljs/schema": {
"version": "0.1.3",
"resolved": "https://registry.npmjs.org/@istanbuljs/schema/-/schema-0.1.3.tgz",
"integrity": "sha512-ZXRY4jNvVgSVQ8DL3LTcakaAtXwTVUxE81hslsyD2AtoXW/wVob10HkOJ1X/pAlcI7D+2YoZKg5do8G/w6RYgA==",
"dev": true,
"engines": {
"node": ">=8"
}
},
"node_modules/@jest/schemas": {
"version": "29.6.3",
"resolved": "https://registry.npmjs.org/@jest/schemas/-/schemas-29.6.3.tgz",
@@ -1493,33 +1464,6 @@
"vite": "^4.2.0 || ^5.0.0 || ^6.0.0 || ^7.0.0"
}
},
"node_modules/@vitest/coverage-v8": {
"version": "1.6.1",
"resolved": "https://registry.npmjs.org/@vitest/coverage-v8/-/coverage-v8-1.6.1.tgz",
"integrity": "sha512-6YeRZwuO4oTGKxD3bijok756oktHSIm3eczVVzNe3scqzuhLwltIF3S9ZL/vwOVIpURmU6SnZhziXXAfw8/Qlw==",
"dev": true,
"dependencies": {
"@ampproject/remapping": "^2.2.1",
"@bcoe/v8-coverage": "^0.2.3",
"debug": "^4.3.4",
"istanbul-lib-coverage": "^3.2.2",
"istanbul-lib-report": "^3.0.1",
"istanbul-lib-source-maps": "^5.0.4",
"istanbul-reports": "^3.1.6",
"magic-string": "^0.30.5",
"magicast": "^0.3.3",
"picocolors": "^1.0.0",
"std-env": "^3.5.0",
"strip-literal": "^2.0.0",
"test-exclude": "^6.0.0"
},
"funding": {
"url": "https://opencollective.com/vitest"
},
"peerDependencies": {
"vitest": "1.6.1"
}
},
"node_modules/@vitest/expect": {
"version": "1.6.1",
"resolved": "https://registry.npmjs.org/@vitest/expect/-/expect-1.6.1.tgz",
@@ -1757,12 +1701,6 @@
"url": "https://github.com/sponsors/ljharb"
}
},
"node_modules/balanced-match": {
"version": "1.0.2",
"resolved": "https://registry.npmjs.org/balanced-match/-/balanced-match-1.0.2.tgz",
"integrity": "sha512-3oSeUO0TMV67hN1AmbXsK4yaqU7tjiHlbxRDZOpH0KW9+CeX4bRAaX0Anxt0tx2MrpRpWwQaPwIlISEJhYU5Pw==",
"dev": true
},
"node_modules/baseline-browser-mapping": {
"version": "2.9.5",
"resolved": "https://registry.npmjs.org/baseline-browser-mapping/-/baseline-browser-mapping-2.9.5.tgz",
@@ -1773,16 +1711,6 @@
"baseline-browser-mapping": "dist/cli.js"
}
},
"node_modules/brace-expansion": {
"version": "1.1.12",
"resolved": "https://registry.npmjs.org/brace-expansion/-/brace-expansion-1.1.12.tgz",
"integrity": "sha512-9T9UjW3r0UW5c1Q7GTwllptXwhvYmEzFhzMfZ9H7FQWt+uZePjZPjBP/W1ZEyZ1twGWom5/56TF4lPcqjnDHcg==",
"dev": true,
"dependencies": {
"balanced-match": "^1.0.0",
"concat-map": "0.0.1"
}
},
"node_modules/browserslist": {
"version": "4.28.1",
"resolved": "https://registry.npmjs.org/browserslist/-/browserslist-4.28.1.tgz",
@@ -1996,12 +1924,6 @@
"node": ">= 0.8"
}
},
"node_modules/concat-map": {
"version": "0.0.1",
"resolved": "https://registry.npmjs.org/concat-map/-/concat-map-0.0.1.tgz",
"integrity": "sha512-/Srv4dswyQNBfohGpz9o6Yb3Gz3SrUDqBH5rTuhGR7ahtlbYKnVxw2bCFMRljaA7EXHaXZ8wsHdodFvbkhKmqg==",
"dev": true
},
"node_modules/confbox": {
"version": "0.1.8",
"resolved": "https://registry.npmjs.org/confbox/-/confbox-0.1.8.tgz",
@@ -2445,12 +2367,6 @@
"node": ">= 6"
}
},
"node_modules/fs.realpath": {
"version": "1.0.0",
"resolved": "https://registry.npmjs.org/fs.realpath/-/fs.realpath-1.0.0.tgz",
"integrity": "sha512-OO0pH2lK6a0hZnAdau5ItzHPI6pUlvI7jMVnxUQRtw4owF2wk8lOSabtGDCTP4Ggrg2MbGnWO9X8K1t4+fGMDw==",
"dev": true
},
"node_modules/fsevents": {
"version": "2.3.3",
"resolved": "https://registry.npmjs.org/fsevents/-/fsevents-2.3.3.tgz",
@@ -2558,27 +2474,6 @@
"url": "https://github.com/sponsors/sindresorhus"
}
},
"node_modules/glob": {
"version": "7.2.3",
"resolved": "https://registry.npmjs.org/glob/-/glob-7.2.3.tgz",
"integrity": "sha512-nFR0zLpU2YCaRxwoCJvL6UvCH2JFyFVIvwTLsIf21AuHlMskA1hhTdk+LlYJtOlYt9v6dvszD2BGRqBL+iQK9Q==",
"deprecated": "Glob versions prior to v9 are no longer supported",
"dev": true,
"dependencies": {
"fs.realpath": "^1.0.0",
"inflight": "^1.0.4",
"inherits": "2",
"minimatch": "^3.1.1",
"once": "^1.3.0",
"path-is-absolute": "^1.0.0"
},
"engines": {
"node": "*"
},
"funding": {
"url": "https://github.com/sponsors/isaacs"
}
},
"node_modules/gopd": {
"version": "1.2.0",
"resolved": "https://registry.npmjs.org/gopd/-/gopd-1.2.0.tgz",
@@ -2683,12 +2578,6 @@
"node": ">=18"
}
},
"node_modules/html-escaper": {
"version": "2.0.2",
"resolved": "https://registry.npmjs.org/html-escaper/-/html-escaper-2.0.2.tgz",
"integrity": "sha512-H2iMtd0I4Mt5eYiapRdIDjp+XzelXQ0tFE4JS7YFwFevXXMmOp9myNrUvCg0D6ws8iqkRPBfKHgbwig1SmlLfg==",
"dev": true
},
"node_modules/http-proxy-agent": {
"version": "7.0.2",
"resolved": "https://registry.npmjs.org/http-proxy-agent/-/http-proxy-agent-7.0.2.tgz",
@@ -2750,23 +2639,6 @@
"node": ">=8"
}
},
"node_modules/inflight": {
"version": "1.0.6",
"resolved": "https://registry.npmjs.org/inflight/-/inflight-1.0.6.tgz",
"integrity": "sha512-k92I/b08q4wvFscXCLvqfsHCrjrF7yiXsQuIVvVE7N82W3+aqpzuUdBbfhWcy/FZR3/4IgflMgKLOsvPDrGCJA==",
"deprecated": "This module is not supported, and leaks memory. Do not use it. Check out lru-cache if you want a good and tested way to coalesce async requests by a key value, which is much more comprehensive and powerful.",
"dev": true,
"dependencies": {
"once": "^1.3.0",
"wrappy": "1"
}
},
"node_modules/inherits": {
"version": "2.0.4",
"resolved": "https://registry.npmjs.org/inherits/-/inherits-2.0.4.tgz",
"integrity": "sha512-k/vGaX4/Yla3WzyMCvTQOXYeIHvqOKtnqBduzTHpzpQZzAskKMhZ2K+EnBiSM9zGSoIFeMpXKxa4dYeZIQqewQ==",
"dev": true
},
"node_modules/internal-slot": {
"version": "1.1.0",
"resolved": "https://registry.npmjs.org/internal-slot/-/internal-slot-1.1.0.tgz",
@@ -3057,56 +2929,6 @@
"dev": true,
"license": "ISC"
},
"node_modules/istanbul-lib-coverage": {
"version": "3.2.2",
"resolved": "https://registry.npmjs.org/istanbul-lib-coverage/-/istanbul-lib-coverage-3.2.2.tgz",
"integrity": "sha512-O8dpsF+r0WV/8MNRKfnmrtCWhuKjxrq2w+jpzBL5UZKTi2LeVWnWOmWRxFlesJONmc+wLAGvKQZEOanko0LFTg==",
"dev": true,
"engines": {
"node": ">=8"
}
},
"node_modules/istanbul-lib-report": {
"version": "3.0.1",
"resolved": "https://registry.npmjs.org/istanbul-lib-report/-/istanbul-lib-report-3.0.1.tgz",
"integrity": "sha512-GCfE1mtsHGOELCU8e/Z7YWzpmybrx/+dSTfLrvY8qRmaY6zXTKWn6WQIjaAFw069icm6GVMNkgu0NzI4iPZUNw==",
"dev": true,
"dependencies": {
"istanbul-lib-coverage": "^3.0.0",
"make-dir": "^4.0.0",
"supports-color": "^7.1.0"
},
"engines": {
"node": ">=10"
}
},
"node_modules/istanbul-lib-source-maps": {
"version": "5.0.6",
"resolved": "https://registry.npmjs.org/istanbul-lib-source-maps/-/istanbul-lib-source-maps-5.0.6.tgz",
"integrity": "sha512-yg2d+Em4KizZC5niWhQaIomgf5WlL4vOOjZ5xGCmF8SnPE/mDWWXgvRExdcpCgh9lLRRa1/fSYp2ymmbJ1pI+A==",
"dev": true,
"dependencies": {
"@jridgewell/trace-mapping": "^0.3.23",
"debug": "^4.1.1",
"istanbul-lib-coverage": "^3.0.0"
},
"engines": {
"node": ">=10"
}
},
"node_modules/istanbul-reports": {
"version": "3.2.0",
"resolved": "https://registry.npmjs.org/istanbul-reports/-/istanbul-reports-3.2.0.tgz",
"integrity": "sha512-HGYWWS/ehqTV3xN10i23tkPkpH46MLCIMFNCaaKNavAXTF1RkqxawEPtnjnGZ6XKSInBKkiOA5BKS+aZiY3AvA==",
"dev": true,
"dependencies": {
"html-escaper": "^2.0.0",
"istanbul-lib-report": "^3.0.0"
},
"engines": {
"node": ">=8"
}
},
"node_modules/js-tokens": {
"version": "4.0.0",
"resolved": "https://registry.npmjs.org/js-tokens/-/js-tokens-4.0.0.tgz",
@@ -3249,44 +3071,6 @@
"@jridgewell/sourcemap-codec": "^1.5.5"
}
},
"node_modules/magicast": {
"version": "0.3.5",
"resolved": "https://registry.npmjs.org/magicast/-/magicast-0.3.5.tgz",
"integrity": "sha512-L0WhttDl+2BOsybvEOLK7fW3UA0OQ0IQ2d6Zl2x/a6vVRs3bAY0ECOSHHeL5jD+SbOpOCUEi0y1DgHEn9Qn1AQ==",
"dev": true,
"dependencies": {
"@babel/parser": "^7.25.4",
"@babel/types": "^7.25.4",
"source-map-js": "^1.2.0"
}
},
"node_modules/make-dir": {
"version": "4.0.0",
"resolved": "https://registry.npmjs.org/make-dir/-/make-dir-4.0.0.tgz",
"integrity": "sha512-hXdUTZYIVOt1Ex//jAQi+wTZZpUpwBj/0QsOzqegb3rGMMeJiSEu5xLHnYfBrRV4RH2+OCSOO95Is/7x1WJ4bw==",
"dev": true,
"dependencies": {
"semver": "^7.5.3"
},
"engines": {
"node": ">=10"
},
"funding": {
"url": "https://github.com/sponsors/sindresorhus"
}
},
"node_modules/make-dir/node_modules/semver": {
"version": "7.7.3",
"resolved": "https://registry.npmjs.org/semver/-/semver-7.7.3.tgz",
"integrity": "sha512-SdsKMrI9TdgjdweUSR9MweHA4EJ8YxHn8DFaDisvhVlUOe4BF1tLD7GAj0lIqWVl+dPb/rExr0Btby5loQm20Q==",
"dev": true,
"bin": {
"semver": "bin/semver.js"
},
"engines": {
"node": ">=10"
}
},
"node_modules/math-intrinsics": {
"version": "1.1.0",
"resolved": "https://registry.npmjs.org/math-intrinsics/-/math-intrinsics-1.1.0.tgz",
@@ -3350,18 +3134,6 @@
"node": ">=4"
}
},
"node_modules/minimatch": {
"version": "3.1.2",
"resolved": "https://registry.npmjs.org/minimatch/-/minimatch-3.1.2.tgz",
"integrity": "sha512-J7p63hRiAjw1NDEww1W7i37+ByIrOWO5XQQAzZ3VOcL0PNybwpfmV/N05zFAzwQ9USyEcX6t3UO+K5aqBQOIHw==",
"dev": true,
"dependencies": {
"brace-expansion": "^1.1.7"
},
"engines": {
"node": "*"
}
},
"node_modules/mlly": {
"version": "1.8.0",
"resolved": "https://registry.npmjs.org/mlly/-/mlly-1.8.0.tgz",
@@ -3512,15 +3284,6 @@
"url": "https://github.com/sponsors/ljharb"
}
},
"node_modules/once": {
"version": "1.4.0",
"resolved": "https://registry.npmjs.org/once/-/once-1.4.0.tgz",
"integrity": "sha512-lNaJgI+2Q5URQBkccEKHTQOPaXdUxnZZElQTZY0MFUAuaEqe1E+Nyvgdz/aIyNi6Z9MzO5dv1H8n58/GELp3+w==",
"dev": true,
"dependencies": {
"wrappy": "1"
}
},
"node_modules/onetime": {
"version": "6.0.0",
"resolved": "https://registry.npmjs.org/onetime/-/onetime-6.0.0.tgz",
@@ -3566,15 +3329,6 @@
"url": "https://github.com/inikulin/parse5?sponsor=1"
}
},
"node_modules/path-is-absolute": {
"version": "1.0.1",
"resolved": "https://registry.npmjs.org/path-is-absolute/-/path-is-absolute-1.0.1.tgz",
"integrity": "sha512-AVbw3UJ2e9bq64vSaS9Am0fje1Pa8pbGqTTsmXfaIiMpnr5DlDhfJOuLj9Sf95ZPVDAUerDfEk88MPmPe7UCQg==",
"dev": true,
"engines": {
"node": ">=0.10.0"
}
},
"node_modules/path-key": {
"version": "3.1.1",
"resolved": "https://registry.npmjs.org/path-key/-/path-key-3.1.1.tgz",
@@ -4191,20 +3945,6 @@
"dev": true,
"license": "MIT"
},
"node_modules/test-exclude": {
"version": "6.0.0",
"resolved": "https://registry.npmjs.org/test-exclude/-/test-exclude-6.0.0.tgz",
"integrity": "sha512-cAGWPIyOHU6zlmg88jwm7VRyXnMN7iV68OGAbYDk/Mh/xC/pzVPlQtY6ngoIH/5/tciuhGfvESU8GrHrcxD56w==",
"dev": true,
"dependencies": {
"@istanbuljs/schema": "^0.1.2",
"glob": "^7.1.4",
"minimatch": "^3.0.4"
},
"engines": {
"node": ">=8"
}
},
"node_modules/tinybench": {
"version": "2.9.0",
"resolved": "https://registry.npmjs.org/tinybench/-/tinybench-2.9.0.tgz",
@@ -4648,12 +4388,6 @@
"node": ">=8"
}
},
"node_modules/wrappy": {
"version": "1.0.2",
"resolved": "https://registry.npmjs.org/wrappy/-/wrappy-1.0.2.tgz",
"integrity": "sha512-l4Sp/DRseor9wL6EvV2+TuQn63dMkPjZ/sp9XkghTEbV9KlPS1xUsZ3u7/IQO4wxtcFB4bgpQPRcR3QCvezPcQ==",
"dev": true
},
"node_modules/ws": {
"version": "8.18.0",
"resolved": "https://registry.npmjs.org/ws/-/ws-8.18.0.tgz",

View File

@@ -1,18 +0,0 @@
<svg width="32" height="32" viewBox="0 0 24 24" fill="none" xmlns="http://www.w3.org/2000/svg">
<!-- Green background -->
<rect width="24" height="24" rx="4" fill="#4CAF50"/>
<!-- Three fruit trees representing an orchard - shifted down to center -->
<g transform="translate(0, 2)">
<!-- Left tree - rounded canopy -->
<path d="M6 14 Q6 8 3 8 Q6 4 6 4 Q6 4 9 8 Q6 8 6 14" fill="white" opacity="0.7"/>
<rect x="5.25" y="13" width="1.5" height="4" fill="white" opacity="0.7"/>
<!-- Center tree - larger rounded canopy -->
<path d="M12 12 Q12 5 8 5 Q12 1 12 1 Q12 1 16 5 Q12 5 12 12" fill="white"/>
<rect x="11.25" y="11" width="1.5" height="5" fill="white"/>
<!-- Right tree - rounded canopy -->
<path d="M18 14 Q18 8 15 8 Q18 4 18 4 Q18 4 21 8 Q18 8 18 14" fill="white" opacity="0.7"/>
<rect x="17.25" y="13" width="1.5" height="4" fill="white" opacity="0.7"/>
<!-- Ground -->
<ellipse cx="12" cy="18" rx="8" ry="1.5" fill="white" opacity="0.4"/>
</g>
</svg>

Before

Width:  |  Height:  |  Size: 1012 B

View File

@@ -32,7 +32,6 @@ import {
OIDCConfig,
OIDCConfigUpdate,
OIDCStatus,
PackageVersion,
} from './types';
const API_BASE = '/api/v1';
@@ -240,21 +239,12 @@ export async function listPackageArtifacts(
}
// Upload
export async function uploadArtifact(
projectName: string,
packageName: string,
file: File,
tag?: string,
version?: string
): Promise<UploadResponse> {
export async function uploadArtifact(projectName: string, packageName: string, file: File, tag?: string): Promise<UploadResponse> {
const formData = new FormData();
formData.append('file', file);
if (tag) {
formData.append('tag', tag);
}
if (version) {
formData.append('version', version);
}
const response = await fetch(`${API_BASE}/project/${projectName}/${packageName}/upload`, {
method: 'POST',
@@ -453,38 +443,3 @@ export function getOIDCLoginUrl(returnTo?: string): string {
const query = params.toString();
return `${API_BASE}/auth/oidc/login${query ? `?${query}` : ''}`;
}
// Version API
export async function listVersions(
projectName: string,
packageName: string,
params: ListParams = {}
): Promise<PaginatedResponse<PackageVersion>> {
const query = buildQueryString(params as Record<string, unknown>);
const response = await fetch(`${API_BASE}/project/${projectName}/${packageName}/versions${query}`);
return handleResponse<PaginatedResponse<PackageVersion>>(response);
}
export async function getVersion(
projectName: string,
packageName: string,
version: string
): Promise<PackageVersion> {
const response = await fetch(`${API_BASE}/project/${projectName}/${packageName}/versions/${version}`);
return handleResponse<PackageVersion>(response);
}
export async function deleteVersion(
projectName: string,
packageName: string,
version: string
): Promise<void> {
const response = await fetch(`${API_BASE}/project/${projectName}/${packageName}/versions/${version}`, {
method: 'DELETE',
credentials: 'include',
});
if (!response.ok) {
const error = await response.json().catch(() => ({ detail: 'Unknown error' }));
throw new Error(error.detail || `HTTP ${response.status}`);
}
}

View File

@@ -98,58 +98,3 @@
text-overflow: ellipsis;
white-space: nowrap;
}
/* Clickable rows */
.data-table__row--clickable {
cursor: pointer;
}
.data-table__row--clickable:hover {
background: var(--bg-hover);
}
/* Responsive table wrapper */
.data-table--responsive {
overflow-x: auto;
-webkit-overflow-scrolling: touch;
}
.data-table--responsive table {
min-width: 800px;
}
/* Cell with name and icon */
.data-table .cell-name {
display: flex;
align-items: center;
gap: 8px;
font-weight: 500;
color: var(--text-primary);
}
.data-table .cell-name:hover {
color: var(--accent-primary);
}
/* Date cells */
.data-table .cell-date {
color: var(--text-tertiary);
font-size: 0.8125rem;
white-space: nowrap;
}
/* Description cell */
.data-table .cell-description {
max-width: 300px;
overflow: hidden;
text-overflow: ellipsis;
white-space: nowrap;
color: var(--text-secondary);
font-size: 0.875rem;
}
/* Owner cell */
.data-table .cell-owner {
color: var(--text-secondary);
font-size: 0.875rem;
}

View File

@@ -18,7 +18,6 @@ interface DataTableProps<T> {
onSort?: (key: string) => void;
sortKey?: string;
sortOrder?: 'asc' | 'desc';
onRowClick?: (item: T) => void;
}
export function DataTable<T>({
@@ -30,7 +29,6 @@ export function DataTable<T>({
onSort,
sortKey,
sortOrder,
onRowClick,
}: DataTableProps<T>) {
if (data.length === 0) {
return (
@@ -73,11 +71,7 @@ export function DataTable<T>({
</thead>
<tbody>
{data.map((item) => (
<tr
key={keyExtractor(item)}
onClick={() => onRowClick?.(item)}
className={onRowClick ? 'data-table__row--clickable' : ''}
>
<tr key={keyExtractor(item)}>
{columns.map((column) => (
<td key={column.key} className={column.className}>
{column.render(item)}

View File

@@ -14,7 +14,7 @@
top: 0;
z-index: 100;
backdrop-filter: blur(12px);
background: rgba(37, 37, 41, 0.85);
background: rgba(17, 17, 19, 0.85);
}
.header-content {

View File

@@ -5,12 +5,12 @@
}
:root {
/* Dark mode color palette - lighter tones for better readability */
--bg-primary: #1e1e22;
--bg-secondary: #252529;
--bg-tertiary: #2d2d32;
--bg-elevated: #35353a;
--bg-hover: #3d3d42;
/* Dark mode color palette */
--bg-primary: #0a0a0b;
--bg-secondary: #111113;
--bg-tertiary: #1a1a1d;
--bg-elevated: #222225;
--bg-hover: #2a2a2e;
/* Accent colors - Green/Emerald theme */
--accent-primary: #10b981;
@@ -24,9 +24,9 @@
--text-tertiary: #9ca3af;
--text-muted: #6b7280;
/* Border colors - slightly more visible */
--border-primary: #37373d;
--border-secondary: #48484e;
/* Border colors */
--border-primary: #27272a;
--border-secondary: #3f3f46;
--border-accent: #10b981;
/* Status colors */

View File

@@ -1,6 +1,6 @@
/* Page Layout */
.home {
max-width: 1200px;
max-width: 1000px;
margin: 0 auto;
}

View File

@@ -1,9 +1,9 @@
import { useState, useEffect, useCallback } from 'react';
import { Link, useSearchParams, useNavigate } from 'react-router-dom';
import { Link, useSearchParams } from 'react-router-dom';
import { Project, PaginatedResponse } from '../types';
import { listProjects, createProject } from '../api';
import { Badge } from '../components/Badge';
import { DataTable } from '../components/DataTable';
import { SortDropdown, SortOption } from '../components/SortDropdown';
import { FilterDropdown, FilterOption } from '../components/FilterDropdown';
import { FilterChip, FilterChipGroup } from '../components/FilterChip';
import { Pagination } from '../components/Pagination';
@@ -20,6 +20,12 @@ function LockIcon() {
);
}
const SORT_OPTIONS: SortOption[] = [
{ value: 'name', label: 'Name' },
{ value: 'created_at', label: 'Created' },
{ value: 'updated_at', label: 'Updated' },
];
const VISIBILITY_OPTIONS: FilterOption[] = [
{ value: '', label: 'All Projects' },
{ value: 'public', label: 'Public Only' },
@@ -28,7 +34,6 @@ const VISIBILITY_OPTIONS: FilterOption[] = [
function Home() {
const [searchParams, setSearchParams] = useSearchParams();
const navigate = useNavigate();
const { user } = useAuth();
const [projectsData, setProjectsData] = useState<PaginatedResponse<Project> | null>(null);
@@ -96,10 +101,8 @@ function Home() {
}
}
const handleSortChange = (columnKey: string) => {
// Toggle order if clicking the same column, otherwise default to asc
const newOrder = columnKey === sort ? (order === 'asc' ? 'desc' : 'asc') : 'asc';
updateParams({ sort: columnKey, order: newOrder, page: '1' });
const handleSortChange = (newSort: string, newOrder: 'asc' | 'desc') => {
updateParams({ sort: newSort, order: newOrder, page: '1' });
};
const handleVisibilityChange = (value: string) => {
@@ -186,6 +189,7 @@ function Home() {
value={visibility}
onChange={handleVisibilityChange}
/>
<SortDropdown options={SORT_OPTIONS} value={sort} order={order} onChange={handleSortChange} />
</div>
{hasActiveFilters && (
@@ -200,106 +204,69 @@ function Home() {
</FilterChipGroup>
)}
<div className="data-table--responsive">
<DataTable
data={projects}
keyExtractor={(project) => project.id}
onRowClick={(project) => navigate(`/project/${project.name}`)}
onSort={handleSortChange}
sortKey={sort}
sortOrder={order}
emptyMessage={
hasActiveFilters
? 'No projects match your filters. Try adjusting your search.'
: 'No projects yet. Create your first project to get started!'
}
columns={[
{
key: 'name',
header: 'Name',
sortable: true,
render: (project) => (
<span className="cell-name">
{projects.length === 0 ? (
<div className="empty-state">
{hasActiveFilters ? (
<p>No projects match your filters. Try adjusting your search.</p>
) : (
<p>No projects yet. Create your first project to get started!</p>
)}
</div>
) : (
<>
<div className="project-grid">
{projects.map((project) => (
<Link to={`/project/${project.name}`} key={project.id} className="project-card card">
<h3>
{!project.is_public && <LockIcon />}
{project.name}
</span>
),
},
{
key: 'description',
header: 'Description',
className: 'cell-description',
render: (project) => project.description || '—',
},
{
key: 'visibility',
header: 'Visibility',
render: (project) => (
<Badge variant={project.is_public ? 'public' : 'private'}>
{project.is_public ? 'Public' : 'Private'}
</Badge>
),
},
{
key: 'created_by',
header: 'Owner',
className: 'cell-owner',
render: (project) => project.created_by,
},
...(user
? [
{
key: 'access_level',
header: 'Access',
render: (project: Project) =>
project.access_level ? (
<Badge
variant={
project.is_owner
</h3>
{project.description && <p>{project.description}</p>}
<div className="project-meta">
<div className="project-badges">
<Badge variant={project.is_public ? 'public' : 'private'}>
{project.is_public ? 'Public' : 'Private'}
</Badge>
{user && project.access_level && (
<Badge
variant={
project.is_owner
? 'success'
: project.access_level === 'admin'
? 'success'
: project.access_level === 'admin'
? 'success'
: project.access_level === 'write'
? 'info'
: 'default'
}
>
{project.is_owner
? 'Owner'
: project.access_level.charAt(0).toUpperCase() + project.access_level.slice(1)}
</Badge>
) : (
'—'
),
},
]
: []),
{
key: 'created_at',
header: 'Created',
sortable: true,
className: 'cell-date',
render: (project) => new Date(project.created_at).toLocaleDateString(),
},
{
key: 'updated_at',
header: 'Updated',
sortable: true,
className: 'cell-date',
render: (project) => new Date(project.updated_at).toLocaleDateString(),
},
]}
/>
</div>
: project.access_level === 'write'
? 'info'
: 'default'
}
>
{project.is_owner ? 'Owner' : project.access_level.charAt(0).toUpperCase() + project.access_level.slice(1)}
</Badge>
)}
</div>
<div className="project-meta__dates">
<span className="date">Created {new Date(project.created_at).toLocaleDateString()}</span>
{project.updated_at !== project.created_at && (
<span className="date">Updated {new Date(project.updated_at).toLocaleDateString()}</span>
)}
</div>
</div>
<div className="project-meta__owner">
<span className="owner">by {project.created_by}</span>
</div>
</Link>
))}
</div>
{pagination && pagination.total_pages > 1 && (
<Pagination
page={pagination.page}
totalPages={pagination.total_pages}
total={pagination.total}
limit={pagination.limit}
onPageChange={handlePageChange}
/>
{pagination && pagination.total_pages > 1 && (
<Pagination
page={pagination.page}
totalPages={pagination.total_pages}
total={pagination.total}
limit={pagination.limit}
onPageChange={handlePageChange}
/>
)}
</>
)}
</div>
);

View File

@@ -324,86 +324,6 @@ tr:hover .copy-btn {
color: var(--text-muted);
}
/* Version badge */
.version-badge {
font-family: 'JetBrains Mono', 'Fira Code', 'Consolas', monospace;
font-size: 0.8125rem;
color: var(--accent-primary);
background: rgba(16, 185, 129, 0.1);
padding: 2px 8px;
border-radius: var(--radius-sm);
}
/* Create Tag Section */
.create-tag-section {
margin-top: 32px;
background: var(--bg-secondary);
}
.create-tag-section h3 {
margin-bottom: 4px;
color: var(--text-primary);
font-size: 1rem;
font-weight: 600;
}
.section-description {
color: var(--text-muted);
font-size: 0.875rem;
margin-bottom: 16px;
}
.create-tag-form .form-row {
display: flex;
gap: 12px;
align-items: flex-end;
flex-wrap: wrap;
}
.create-tag-form .form-group {
flex: 1;
min-width: 150px;
}
.create-tag-form .form-group--wide {
flex: 2;
min-width: 300px;
}
.create-tag-form .form-group label {
display: block;
margin-bottom: 6px;
font-size: 0.75rem;
font-weight: 500;
color: var(--text-secondary);
text-transform: uppercase;
letter-spacing: 0.05em;
}
.create-tag-form .form-group input {
width: 100%;
padding: 10px 14px;
background: var(--bg-tertiary);
border: 1px solid var(--border-primary);
border-radius: var(--radius-md);
color: var(--text-primary);
font-size: 0.875rem;
}
.create-tag-form .form-group input:focus {
outline: none;
border-color: var(--accent-primary);
}
.create-tag-form .form-group input:disabled {
opacity: 0.6;
cursor: not-allowed;
}
.create-tag-form button {
flex-shrink: 0;
}
/* Created cell */
.created-cell {
display: flex;

View File

@@ -1,10 +1,11 @@
import { useState, useEffect, useCallback } from 'react';
import { useParams, useSearchParams, useNavigate, useLocation } from 'react-router-dom';
import { TagDetail, Package, PaginatedResponse, AccessLevel } from '../types';
import { listTags, getDownloadUrl, getPackage, getMyProjectAccess, createTag, UnauthorizedError, ForbiddenError } from '../api';
import { listTags, getDownloadUrl, getPackage, getMyProjectAccess, UnauthorizedError, ForbiddenError } from '../api';
import { Breadcrumb } from '../components/Breadcrumb';
import { Badge } from '../components/Badge';
import { SearchInput } from '../components/SearchInput';
import { SortDropdown, SortOption } from '../components/SortDropdown';
import { FilterChip, FilterChipGroup } from '../components/FilterChip';
import { DataTable } from '../components/DataTable';
import { Pagination } from '../components/Pagination';
@@ -13,6 +14,11 @@ import { useAuth } from '../contexts/AuthContext';
import './Home.css';
import './PackagePage.css';
const SORT_OPTIONS: SortOption[] = [
{ value: 'name', label: 'Name' },
{ value: 'created_at', label: 'Created' },
];
function formatBytes(bytes: number): string {
if (bytes === 0) return '0 B';
const k = 1024;
@@ -64,9 +70,6 @@ function PackagePage() {
const [uploadSuccess, setUploadSuccess] = useState<string | null>(null);
const [artifactIdInput, setArtifactIdInput] = useState('');
const [accessLevel, setAccessLevel] = useState<AccessLevel | null>(null);
const [createTagName, setCreateTagName] = useState('');
const [createTagArtifactId, setCreateTagArtifactId] = useState('');
const [createTagLoading, setCreateTagLoading] = useState(false);
// Derived permissions
const canWrite = accessLevel === 'write' || accessLevel === 'admin';
@@ -157,37 +160,12 @@ function PackagePage() {
setError(errorMsg);
}, []);
const handleCreateTag = async (e: React.FormEvent) => {
e.preventDefault();
if (!createTagName.trim() || createTagArtifactId.length !== 64) return;
setCreateTagLoading(true);
setError(null);
try {
await createTag(projectName!, packageName!, {
name: createTagName.trim(),
artifact_id: createTagArtifactId,
});
setUploadSuccess(`Tag "${createTagName}" created successfully!`);
setCreateTagName('');
setCreateTagArtifactId('');
loadData();
setTimeout(() => setUploadSuccess(null), 5000);
} catch (err) {
setError(err instanceof Error ? err.message : 'Failed to create tag');
} finally {
setCreateTagLoading(false);
}
};
const handleSearchChange = (value: string) => {
updateParams({ search: value, page: '1' });
};
const handleSortChange = (columnKey: string) => {
const newOrder = columnKey === sort ? (order === 'asc' ? 'desc' : 'asc') : 'asc';
updateParams({ sort: columnKey, order: newOrder, page: '1' });
const handleSortChange = (newSort: string, newOrder: 'asc' | 'desc') => {
updateParams({ sort: newSort, order: newOrder, page: '1' });
};
const handlePageChange = (newPage: number) => {
@@ -209,13 +187,6 @@ function PackagePage() {
sortable: true,
render: (t: TagDetail) => <strong>{t.name}</strong>,
},
{
key: 'version',
header: 'Version',
render: (t: TagDetail) => (
<span className="version-badge">{t.version || '-'}</span>
),
},
{
key: 'artifact_id',
header: 'Artifact ID',
@@ -227,19 +198,19 @@ function PackagePage() {
),
},
{
key: 'artifact_size',
key: 'size',
header: 'Size',
render: (t: TagDetail) => <span>{formatBytes(t.artifact_size)}</span>,
},
{
key: 'artifact_content_type',
key: 'content_type',
header: 'Type',
render: (t: TagDetail) => (
<span className="content-type">{t.artifact_content_type || '-'}</span>
),
},
{
key: 'artifact_original_name',
key: 'original_name',
header: 'Filename',
className: 'cell-truncate',
render: (t: TagDetail) => (
@@ -405,6 +376,7 @@ function PackagePage() {
placeholder="Filter tags..."
className="list-controls__search"
/>
<SortDropdown options={SORT_OPTIONS} value={sort} order={order} onChange={handleSortChange} />
</div>
{hasActiveFilters && (
@@ -413,21 +385,25 @@ function PackagePage() {
</FilterChipGroup>
)}
<div className="data-table--responsive">
<DataTable
data={tags}
columns={columns}
keyExtractor={(t) => t.id}
emptyMessage={
hasActiveFilters
? 'No tags match your filters. Try adjusting your search.'
: 'No tags yet. Upload an artifact with a tag to create one!'
<DataTable
data={tags}
columns={columns}
keyExtractor={(t) => t.id}
emptyMessage={
hasActiveFilters
? 'No tags match your filters. Try adjusting your search.'
: 'No tags yet. Upload an artifact with a tag to create one!'
}
onSort={(key) => {
if (key === sort) {
handleSortChange(key, order === 'asc' ? 'desc' : 'asc');
} else {
handleSortChange(key, 'asc');
}
onSort={handleSortChange}
sortKey={sort}
sortOrder={order}
/>
</div>
}}
sortKey={sort}
sortOrder={order}
/>
{pagination && pagination.total_pages > 1 && (
<Pagination
@@ -467,50 +443,6 @@ function PackagePage() {
)}
</div>
{user && canWrite && (
<div className="create-tag-section card">
<h3>Create / Update Tag</h3>
<p className="section-description">Point a tag at any existing artifact by its ID</p>
<form onSubmit={handleCreateTag} className="create-tag-form">
<div className="form-row">
<div className="form-group">
<label htmlFor="create-tag-name">Tag Name</label>
<input
id="create-tag-name"
type="text"
value={createTagName}
onChange={(e) => setCreateTagName(e.target.value)}
placeholder="latest, stable, v1.0.0..."
disabled={createTagLoading}
/>
</div>
<div className="form-group form-group--wide">
<label htmlFor="create-tag-artifact">Artifact ID</label>
<input
id="create-tag-artifact"
type="text"
value={createTagArtifactId}
onChange={(e) => setCreateTagArtifactId(e.target.value.toLowerCase().replace(/[^a-f0-9]/g, '').slice(0, 64))}
placeholder="SHA256 hash (64 hex characters)"
className="artifact-id-input"
disabled={createTagLoading}
/>
</div>
<button
type="submit"
className="btn btn-primary"
disabled={createTagLoading || !createTagName.trim() || createTagArtifactId.length !== 64}
>
{createTagLoading ? 'Creating...' : 'Create Tag'}
</button>
</div>
{createTagArtifactId.length > 0 && createTagArtifactId.length !== 64 && (
<p className="validation-hint">Artifact ID must be exactly 64 hex characters ({createTagArtifactId.length}/64)</p>
)}
</form>
</div>
)}
<div className="usage-section card">
<h3>Usage</h3>
<p>Download artifacts using:</p>

View File

@@ -1,17 +1,23 @@
import { useState, useEffect, useCallback } from 'react';
import { useParams, useSearchParams, useNavigate, useLocation } from 'react-router-dom';
import { useParams, Link, useSearchParams, useNavigate, useLocation } from 'react-router-dom';
import { Project, Package, PaginatedResponse, AccessLevel } from '../types';
import { getProject, listPackages, createPackage, getMyProjectAccess, UnauthorizedError, ForbiddenError } from '../api';
import { Breadcrumb } from '../components/Breadcrumb';
import { Badge } from '../components/Badge';
import { DataTable } from '../components/DataTable';
import { SearchInput } from '../components/SearchInput';
import { SortDropdown, SortOption } from '../components/SortDropdown';
import { FilterChip, FilterChipGroup } from '../components/FilterChip';
import { Pagination } from '../components/Pagination';
import { AccessManagement } from '../components/AccessManagement';
import { useAuth } from '../contexts/AuthContext';
import './Home.css';
const SORT_OPTIONS: SortOption[] = [
{ value: 'name', label: 'Name' },
{ value: 'created_at', label: 'Created' },
{ value: 'updated_at', label: 'Updated' },
];
const FORMAT_OPTIONS = ['generic', 'npm', 'pypi', 'docker', 'deb', 'rpm', 'maven', 'nuget', 'helm'];
function formatBytes(bytes: number): string {
@@ -134,9 +140,8 @@ function ProjectPage() {
updateParams({ search: value, page: '1' });
};
const handleSortChange = (columnKey: string) => {
const newOrder = columnKey === sort ? (order === 'asc' ? 'desc' : 'asc') : 'asc';
updateParams({ sort: columnKey, order: newOrder, page: '1' });
const handleSortChange = (newSort: string, newOrder: 'asc' | 'desc') => {
updateParams({ sort: newSort, order: newOrder, page: '1' });
};
const handleFormatChange = (value: string) => {
@@ -289,6 +294,7 @@ function ProjectPage() {
</option>
))}
</select>
<SortDropdown options={SORT_OPTIONS} value={sort} order={order} onChange={handleSortChange} />
</div>
{hasActiveFilters && (
@@ -298,78 +304,70 @@ function ProjectPage() {
</FilterChipGroup>
)}
<div className="data-table--responsive">
<DataTable
data={packages}
keyExtractor={(pkg) => pkg.id}
onRowClick={(pkg) => navigate(`/project/${projectName}/${pkg.name}`)}
onSort={handleSortChange}
sortKey={sort}
sortOrder={order}
emptyMessage={
hasActiveFilters
? 'No packages match your filters. Try adjusting your search.'
: 'No packages yet. Create your first package to start uploading artifacts!'
}
columns={[
{
key: 'name',
header: 'Name',
sortable: true,
render: (pkg) => <span className="cell-name">{pkg.name}</span>,
},
{
key: 'description',
header: 'Description',
className: 'cell-description',
render: (pkg) => pkg.description || '—',
},
{
key: 'format',
header: 'Format',
render: (pkg) => <Badge variant="default">{pkg.format}</Badge>,
},
{
key: 'tag_count',
header: 'Tags',
render: (pkg) => pkg.tag_count ?? '—',
},
{
key: 'artifact_count',
header: 'Artifacts',
render: (pkg) => pkg.artifact_count ?? '—',
},
{
key: 'total_size',
header: 'Size',
render: (pkg) =>
pkg.total_size !== undefined && pkg.total_size > 0 ? formatBytes(pkg.total_size) : '—',
},
{
key: 'latest_tag',
header: 'Latest',
render: (pkg) =>
pkg.latest_tag ? <strong style={{ color: 'var(--accent-primary)' }}>{pkg.latest_tag}</strong> : '—',
},
{
key: 'created_at',
header: 'Created',
sortable: true,
className: 'cell-date',
render: (pkg) => new Date(pkg.created_at).toLocaleDateString(),
},
]}
/>
</div>
{packages.length === 0 ? (
<div className="empty-state">
{hasActiveFilters ? (
<p>No packages match your filters. Try adjusting your search.</p>
) : (
<p>No packages yet. Create your first package to start uploading artifacts!</p>
)}
</div>
) : (
<>
<div className="project-grid">
{packages.map((pkg) => (
<Link to={`/project/${projectName}/${pkg.name}`} key={pkg.id} className="project-card card">
<div className="package-card__header">
<h3>{pkg.name}</h3>
<Badge variant="default">{pkg.format}</Badge>
</div>
{pkg.description && <p>{pkg.description}</p>}
{pagination && pagination.total_pages > 1 && (
<Pagination
page={pagination.page}
totalPages={pagination.total_pages}
total={pagination.total}
limit={pagination.limit}
onPageChange={handlePageChange}
/>
{(pkg.tag_count !== undefined || pkg.artifact_count !== undefined) && (
<div className="package-stats">
{pkg.tag_count !== undefined && (
<div className="package-stats__item">
<span className="package-stats__value">{pkg.tag_count}</span>
<span className="package-stats__label">Tags</span>
</div>
)}
{pkg.artifact_count !== undefined && (
<div className="package-stats__item">
<span className="package-stats__value">{pkg.artifact_count}</span>
<span className="package-stats__label">Artifacts</span>
</div>
)}
{pkg.total_size !== undefined && pkg.total_size > 0 && (
<div className="package-stats__item">
<span className="package-stats__value">{formatBytes(pkg.total_size)}</span>
<span className="package-stats__label">Size</span>
</div>
)}
</div>
)}
<div className="project-meta">
{pkg.latest_tag && (
<span className="latest-tag">
Latest: <strong>{pkg.latest_tag}</strong>
</span>
)}
<span className="date">Created {new Date(pkg.created_at).toLocaleDateString()}</span>
</div>
</Link>
))}
</div>
{pagination && pagination.total_pages > 1 && (
<Pagination
page={pagination.page}
totalPages={pagination.total_pages}
total={pagination.total}
limit={pagination.limit}
onPageChange={handlePageChange}
/>
)}
</>
)}
{canAdmin && projectName && (

View File

@@ -63,22 +63,6 @@ export interface TagDetail extends Tag {
artifact_original_name: string | null;
artifact_created_at: string;
artifact_format_metadata: Record<string, unknown> | null;
version: string | null;
}
export interface PackageVersion {
id: string;
package_id: string;
artifact_id: string;
version: string;
version_source: string | null;
created_at: string;
created_by: string;
// Enriched fields from joins
size?: number;
content_type?: string | null;
original_name?: string | null;
tags?: string[];
}
export interface ArtifactTagInfo {
@@ -138,8 +122,6 @@ export interface UploadResponse {
project: string;
package: string;
tag: string | null;
version: string | null;
version_source: string | null;
}
// Global search types

View File

@@ -17,13 +17,13 @@ maintainers:
dependencies:
- name: postgresql
version: "15.5.x"
repository: oci://deps.global.bsf.tools/registry-1.docker.io-helmoci/bitnamicharts
repository: https://charts.bitnami.com/bitnami
condition: postgresql.enabled
- name: minio
version: "14.x.x"
repository: oci://deps.global.bsf.tools/registry-1.docker.io-helmoci/bitnamicharts
repository: https://charts.bitnami.com/bitnami
condition: minio.enabled
- name: redis
version: "19.x.x"
repository: oci://deps.global.bsf.tools/registry-1.docker.io-helmoci/bitnamicharts
repository: https://charts.bitnami.com/bitnami
condition: redis.enabled

View File

@@ -7,7 +7,6 @@ Expand the name of the chart.
{{/*
Create a default fully qualified app name.
Appends "-server" to distinguish from subcharts (minio, postgresql, redis).
*/}}
{{- define "orchard.fullname" -}}
{{- if .Values.fullnameOverride }}
@@ -15,9 +14,9 @@ Appends "-server" to distinguish from subcharts (minio, postgresql, redis).
{{- else }}
{{- $name := default .Chart.Name .Values.nameOverride }}
{{- if contains $name .Release.Name }}
{{- printf "%s-server" .Release.Name | trunc 63 | trimSuffix "-" }}
{{- .Release.Name | trunc 63 | trimSuffix "-" }}
{{- else }}
{{- printf "%s-%s-server" .Release.Name $name | trunc 63 | trimSuffix "-" }}
{{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" }}
{{- end }}
{{- end }}
{{- end }}
@@ -77,8 +76,6 @@ PostgreSQL secret name
{{- define "orchard.postgresql.secretName" -}}
{{- if .Values.orchard.database.existingSecret }}
{{- .Values.orchard.database.existingSecret }}
{{- else if and .Values.orchard.database.secretsManager .Values.orchard.database.secretsManager.enabled }}
{{- printf "%s-db-credentials" (include "orchard.fullname" .) }}
{{- else if .Values.postgresql.enabled }}
{{- printf "%s-postgresql" .Release.Name }}
{{- else }}
@@ -92,8 +89,6 @@ PostgreSQL password key in secret
{{- define "orchard.postgresql.passwordKey" -}}
{{- if .Values.orchard.database.existingSecret -}}
{{- .Values.orchard.database.existingSecretPasswordKey -}}
{{- else if and .Values.orchard.database.secretsManager .Values.orchard.database.secretsManager.enabled -}}
password
{{- else if .Values.postgresql.enabled -}}
password
{{- else -}}

View File

@@ -37,26 +37,12 @@ spec:
image: "{{ .Values.initContainer.image.repository }}:{{ .Values.initContainer.image.tag }}"
imagePullPolicy: {{ .Values.initContainer.image.pullPolicy }}
command: ['sh', '-c', 'until nc -z {{ include "orchard.postgresql.host" . }} 5432; do echo waiting for database; sleep 2; done;']
resources:
limits:
cpu: 50m
memory: 32Mi
requests:
cpu: 10m
memory: 32Mi
{{- end }}
{{- if .Values.minio.enabled }}
- name: wait-for-minio
image: "{{ .Values.initContainer.image.repository }}:{{ .Values.initContainer.image.tag }}"
imagePullPolicy: {{ .Values.initContainer.image.pullPolicy }}
command: ['sh', '-c', 'until nc -z {{ .Release.Name }}-minio 9000; do echo waiting for minio; sleep 2; done;']
resources:
limits:
cpu: 50m
memory: 32Mi
requests:
cpu: 10m
memory: 32Mi
{{- end }}
containers:
- name: {{ .Chart.Name }}
@@ -77,16 +63,8 @@ spec:
value: {{ include "orchard.postgresql.host" . | quote }}
- name: ORCHARD_DATABASE_PORT
value: {{ .Values.orchard.database.port | quote }}
{{- if and .Values.orchard.database.secretsManager .Values.orchard.database.secretsManager.enabled }}
- name: ORCHARD_DATABASE_USER
valueFrom:
secretKeyRef:
name: {{ include "orchard.postgresql.secretName" . }}
key: username
{{- else }}
- name: ORCHARD_DATABASE_USER
value: {{ .Values.orchard.database.user | default .Values.postgresql.auth.username | quote }}
{{- end }}
- name: ORCHARD_DATABASE_DBNAME
value: {{ .Values.orchard.database.dbname | default .Values.postgresql.auth.database | quote }}
- name: ORCHARD_DATABASE_SSLMODE
@@ -104,7 +82,6 @@ spec:
value: {{ .Values.orchard.s3.bucket | quote }}
- name: ORCHARD_S3_USE_PATH_STYLE
value: {{ .Values.orchard.s3.usePathStyle | quote }}
{{- if or .Values.minio.enabled .Values.orchard.s3.existingSecret .Values.orchard.s3.accessKeyId }}
- name: ORCHARD_S3_ACCESS_KEY_ID
valueFrom:
secretKeyRef:
@@ -115,38 +92,16 @@ spec:
secretKeyRef:
name: {{ include "orchard.minio.secretName" . }}
key: {{ if .Values.minio.enabled }}root-password{{ else }}{{ .Values.orchard.s3.existingSecretSecretKeyKey }}{{ end }}
{{- end }}
- name: ORCHARD_DOWNLOAD_MODE
value: {{ .Values.orchard.download.mode | quote }}
- name: ORCHARD_PRESIGNED_URL_EXPIRY
value: {{ .Values.orchard.download.presignedUrlExpiry | quote }}
{{- if .Values.orchard.rateLimit }}
{{- if .Values.orchard.rateLimit.login }}
- name: ORCHARD_LOGIN_RATE_LIMIT
value: {{ .Values.orchard.rateLimit.login | quote }}
{{- end }}
{{- end }}
{{- if and .Values.orchard.database.secretsManager .Values.orchard.database.secretsManager.enabled }}
volumeMounts:
- name: db-secrets
mountPath: /mnt/secrets-store
readOnly: true
{{- end }}
livenessProbe:
{{- toYaml .Values.livenessProbe | nindent 12 }}
readinessProbe:
{{- toYaml .Values.readinessProbe | nindent 12 }}
resources:
{{- toYaml .Values.resources | nindent 12 }}
{{- if and .Values.orchard.database.secretsManager .Values.orchard.database.secretsManager.enabled }}
volumes:
- name: db-secrets
csi:
driver: secrets-store.csi.k8s.io
readOnly: true
volumeAttributes:
secretProviderClass: {{ include "orchard.fullname" . }}-db-secret
{{- end }}
{{- with .Values.nodeSelector }}
nodeSelector:
{{- toYaml . | nindent 8 }}

View File

@@ -1,27 +0,0 @@
{{- if and .Values.orchard.database.secretsManager .Values.orchard.database.secretsManager.enabled }}
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
name: {{ include "orchard.fullname" . }}-db-secret
labels:
{{- include "orchard.labels" . | nindent 4 }}
spec:
provider: aws
parameters:
objects: |
- objectName: "{{ .Values.orchard.database.secretsManager.secretArn }}"
objectType: "secretsmanager"
jmesPath:
- path: username
objectAlias: db-username
- path: password
objectAlias: db-password
secretObjects:
- secretName: {{ include "orchard.fullname" . }}-db-credentials
type: Opaque
data:
- objectName: db-username
key: username
- objectName: db-password
key: password
{{- end }}

View File

@@ -42,7 +42,6 @@ ingress:
className: "nginx"
annotations:
cert-manager.io/cluster-issuer: "letsencrypt"
nginx.ingress.kubernetes.io/proxy-body-size: "0" # Disable body size limit for uploads
hosts:
- host: orchard-dev.common.global.bsf.tools # Overridden by CI
paths:
@@ -54,14 +53,13 @@ ingress:
- orchard-dev.common.global.bsf.tools # Overridden by CI
# Lighter resources for ephemeral environments
# Note: memory requests must equal limits per cluster policy
resources:
limits:
cpu: 250m
memory: 256Mi
requests:
cpu: 100m
memory: 256Mi
memory: 128Mi
livenessProbe:
httpGet:
@@ -114,10 +112,6 @@ orchard:
mode: "presigned"
presignedUrlExpiry: 3600
# Relaxed rate limits for dev/feature environments (allows integration tests to run)
rateLimit:
login: "1000/minute" # Default is 5/minute, relaxed for CI integration tests
# PostgreSQL - ephemeral, no persistence
postgresql:
enabled: true
@@ -133,25 +127,6 @@ postgresql:
primary:
persistence:
enabled: false
# Resources with memory requests = limits per cluster policy
resourcesPreset: "none"
resources:
limits:
cpu: 250m
memory: 256Mi
requests:
cpu: 100m
memory: 256Mi
# Volume permissions init container
volumePermissions:
resourcesPreset: "none"
resources:
limits:
cpu: 50m
memory: 64Mi
requests:
cpu: 10m
memory: 64Mi
# MinIO - ephemeral, no persistence
minio:
@@ -167,35 +142,6 @@ minio:
defaultBuckets: "orchard-artifacts"
persistence:
enabled: false
# Resources with memory requests = limits per cluster policy
resourcesPreset: "none" # Disable preset to use explicit resources
resources:
limits:
cpu: 250m
memory: 256Mi
requests:
cpu: 100m
memory: 256Mi
# Init container resources
defaultInitContainers:
volumePermissions:
resourcesPreset: "none"
resources:
limits:
cpu: 50m
memory: 64Mi
requests:
cpu: 10m
memory: 64Mi
# Provisioning job resources
provisioning:
resources:
limits:
cpu: 100m
memory: 128Mi
requests:
cpu: 50m
memory: 128Mi
# MinIO ingress - hostname overridden by CI
minioIngress:

View File

@@ -1,135 +0,0 @@
# Production values for orchard
# TODO: Replace subcharts with managed services (RDS, S3) when ready
replicaCount: 1
image:
repository: registry.global.bsf.tools/esv/bsf/bsf-integration/orchard/orchard-mvp
pullPolicy: Always
tag: "latest" # Overridden by CI
imagePullSecrets:
- name: orchard-pull-secret
initContainer:
image:
repository: containers.global.bsf.tools/busybox
tag: "1.36"
pullPolicy: IfNotPresent
serviceAccount:
create: true
automount: true
annotations:
eks.amazonaws.com/role-arn: arn:aws-us-gov:iam::052673043337:role/service-orchard-prd
name: "orchard"
podAnnotations: {}
podLabels: {}
podSecurityContext: {}
securityContext:
readOnlyRootFilesystem: false
runAsNonRoot: true
runAsUser: 1000
service:
type: ClusterIP
port: 8080
ingress:
enabled: true
className: "nginx"
annotations:
cert-manager.io/cluster-issuer: "letsencrypt"
nginx.ingress.kubernetes.io/proxy-body-size: "0" # Disable body size limit for uploads
hosts:
- host: orchard.common.global.bsf.tools
paths:
- path: /
pathType: Prefix
tls:
- secretName: orchard-prod-tls
hosts:
- orchard.common.global.bsf.tools
# Production resources - same as stage for MVP, increase as needed
resources:
limits:
cpu: 500m
memory: 512Mi
requests:
cpu: 500m
memory: 512Mi
livenessProbe:
httpGet:
path: /health
port: http
initialDelaySeconds: 10
periodSeconds: 10
readinessProbe:
httpGet:
path: /health
port: http
initialDelaySeconds: 5
periodSeconds: 5
autoscaling:
enabled: false
minReplicas: 1
maxReplicas: 10
targetCPUUtilizationPercentage: 80
targetMemoryUtilizationPercentage: 80
nodeSelector: {}
tolerations: []
affinity: {}
orchard:
server:
host: "0.0.0.0"
port: 8080
# Database configuration - uses AWS Secrets Manager via CSI driver
database:
host: "orchard-prd.cluster-cvw3jzjkozoc.us-gov-west-1.rds.amazonaws.com"
port: 5432
dbname: orchard_prod
sslmode: require
secretsManager:
enabled: true
secretArn: "arn:aws-us-gov:secretsmanager:us-gov-west-1:052673043337:secret:rds!cluster-0afc8af5-f644-4284-92fb-2ed545490f92-3v9uXV"
# S3 configuration - uses IRSA for credentials
s3:
endpoint: "" # Empty = use AWS default
region: us-gov-west-1
bucket: orchard-artifacts-prod
usePathStyle: false # Real S3 uses virtual-hosted style
download:
mode: "presigned"
presignedUrlExpiry: 3600
# PostgreSQL subchart - disabled in prod, using RDS
postgresql:
enabled: false
# MinIO subchart - disabled in prod, using real S3
minio:
enabled: false
# MinIO ingress - disabled in prod, using real S3
minioIngress:
enabled: false
redis:
enabled: false
waitForDatabase: true
global:
security:
allowInsecureImages: true

View File

@@ -19,8 +19,7 @@ initContainer:
serviceAccount:
create: true
automount: true
annotations:
eks.amazonaws.com/role-arn: arn:aws-us-gov:iam::052673043337:role/service-orchard-stage
annotations: {}
name: "orchard"
podAnnotations: {}
@@ -42,7 +41,6 @@ ingress:
className: "nginx"
annotations:
cert-manager.io/cluster-issuer: "letsencrypt"
nginx.ingress.kubernetes.io/proxy-body-size: "0" # Disable body size limit for uploads
hosts:
- host: orchard-stage.common.global.bsf.tools
paths:
@@ -94,43 +92,78 @@ orchard:
host: "0.0.0.0"
port: 8080
# Database configuration - uses AWS Secrets Manager via CSI driver
# Database configuration (used when postgresql.enabled is false)
database:
host: "orchard-stage.cluster-cvw3jzjkozoc.us-gov-west-1.rds.amazonaws.com"
host: ""
port: 5432
dbname: postgres
sslmode: require
secretsManager:
enabled: true
secretArn: "arn:aws-us-gov:secretsmanager:us-gov-west-1:052673043337:secret:rds!cluster-a573672b-1a38-4665-a654-1b7df37b5297-IaeFQL"
user: orchard
password: ""
dbname: orchard
sslmode: disable
existingSecret: ""
existingSecretPasswordKey: "password"
# S3 configuration - uses IRSA for credentials
# S3 configuration (used when minio.enabled is false)
s3:
endpoint: "" # Empty = use AWS default
region: us-gov-west-1
bucket: orchard-artifacts-stage
usePathStyle: false # Real S3 uses virtual-hosted style
endpoint: ""
region: us-east-1
bucket: orchard-artifacts
accessKeyId: ""
secretAccessKey: ""
usePathStyle: true
existingSecret: ""
existingSecretAccessKeyKey: "access-key-id"
existingSecretSecretKeyKey: "secret-access-key"
# Download configuration
download:
mode: "presigned" # presigned, redirect, or proxy
presignedUrlExpiry: 3600 # Presigned URL expiry in seconds
# Relaxed rate limits for stage (allows CI integration tests to run)
rateLimit:
login: "1000/minute" # Default is 5/minute, relaxed for CI integration tests
# PostgreSQL subchart - disabled in stage, using RDS
# PostgreSQL subchart configuration
postgresql:
enabled: false
enabled: true
image:
registry: containers.global.bsf.tools
repository: bitnami/postgresql
tag: "15"
pullPolicy: IfNotPresent
auth:
username: orchard
password: orchard-password
database: orchard
primary:
persistence:
enabled: false
size: 10Gi
# MinIO subchart - disabled in stage, using real S3
# MinIO subchart configuration
minio:
enabled: false
enabled: true
image:
registry: containers.global.bsf.tools
repository: bitnami/minio
tag: "latest"
pullPolicy: IfNotPresent
auth:
rootUser: minioadmin
rootPassword: minioadmin
defaultBuckets: "orchard-artifacts"
persistence:
enabled: false
size: 50Gi
# MinIO ingress - disabled in stage, using real S3
# MinIO external ingress for presigned URL access (separate from subchart ingress)
minioIngress:
enabled: false
enabled: true
className: "nginx"
annotations:
cert-manager.io/cluster-issuer: "letsencrypt"
nginx.ingress.kubernetes.io/proxy-body-size: "0" # Disable body size limit for uploads
host: "minio-orch-stage.common.global.bsf.tools"
tls:
enabled: true
secretName: minio-tls
# Redis subchart configuration (for future caching)
redis:

View File

@@ -1,51 +0,0 @@
# KICS Configuration File
# https://docs.kics.io/latest/configuration-file/
# Exclude specific queries that are acceptable for this project
exclude-queries:
# Shared Volumes Between Containers (INFO)
# Reason: Database services (postgres, minio, redis) require persistent volumes
# for data storage. This is expected and necessary behavior.
- 8c978947-0ff6-485c-b0c2-0bfca6026466
# Passwords And Secrets - Generic Password (HIGH)
# Reason: These are LOCAL DEVELOPMENT configs only. Production deployments
# use Kubernetes secrets injected at runtime. The passwords in docker-compose
# and helm values files are placeholder/dev values, not real secrets.
- a88baa34-e2ad-44ea-ad6f-8cac87bc7c71
# Healthcheck Not Set (MEDIUM)
# Reason: minio-init is an init container that runs once and exits.
# Healthchecks are not applicable to containers that are designed to exit.
- 698ed579-b239-4f8f-a388-baa4bcb13ef8
# Apt Get Install Pin Version Not Defined (MEDIUM)
# Reason: We intentionally don't pin curl version to get security updates.
# This is documented with hadolint ignore comment in Dockerfile.
- 965a08d7-ef86-4f14-8792-4a3b2098937e
# Container Capabilities Unrestricted (MEDIUM)
# Reason: LOCAL DEVELOPMENT ONLY. Stock postgres, redis, minio images require
# certain capabilities (SETUID, SETGID, CHOWN) to switch users at startup.
# cap_drop: ALL breaks these containers. Production Kubernetes deployments
# use securityContext with appropriate settings.
- ce76b7d0-9e77-464d-b86f-c5c48e03e22d
# No New Privileges Not Set (HIGH)
# Reason: LOCAL DEVELOPMENT ONLY. Stock postgres, redis, minio images need
# to escalate privileges during initialization (e.g., postgres switches from
# root to postgres user). no-new-privileges:true prevents this and causes
# containers to crash. Production Kubernetes deployments handle this via
# securityContext.
- 27fcc7d6-c49b-46e0-98f1-6c082a6a2750
# Security Opt Not Set (MEDIUM)
# Reason: LOCAL DEVELOPMENT ONLY. Related to above - security_opt is not set
# on database services because no-new-privileges breaks them.
- 610e266e-6c12-4bca-9925-1ed0cd29742b
# Container Traffic Not Bound To Host Interface (MEDIUM)
# Reason: LOCAL DEVELOPMENT ONLY. The orchard-server port is bound to 0.0.0.0
# to allow testing from other machines on the local network. This is only in
# docker-compose.local.yml, not production deployments.
- 451d79dc-0588-476a-ad03-3c7f0320abb3

View File

@@ -1,67 +0,0 @@
-- Migration: Add package_versions table for separate version tracking
-- This separates immutable versions from mutable tags
-- Create package_versions table
CREATE TABLE IF NOT EXISTS package_versions (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
package_id UUID NOT NULL REFERENCES packages(id) ON DELETE CASCADE,
artifact_id VARCHAR(64) NOT NULL REFERENCES artifacts(id),
version VARCHAR(255) NOT NULL,
version_source VARCHAR(50), -- 'explicit', 'filename', 'metadata', 'migrated_from_tag'
created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
created_by VARCHAR(255) NOT NULL,
UNIQUE(package_id, version),
UNIQUE(package_id, artifact_id)
);
-- Indexes for common queries
CREATE INDEX IF NOT EXISTS idx_package_versions_package_id ON package_versions(package_id);
CREATE INDEX IF NOT EXISTS idx_package_versions_artifact_id ON package_versions(artifact_id);
CREATE INDEX IF NOT EXISTS idx_package_versions_package_version ON package_versions(package_id, version);
-- Trigger functions for ref_count management (same pattern as tags)
CREATE OR REPLACE FUNCTION increment_version_ref_count()
RETURNS TRIGGER AS $$
BEGIN
UPDATE artifacts SET ref_count = ref_count + 1 WHERE id = NEW.artifact_id;
RETURN NEW;
END;
$$ LANGUAGE plpgsql;
CREATE OR REPLACE FUNCTION decrement_version_ref_count()
RETURNS TRIGGER AS $$
BEGIN
UPDATE artifacts SET ref_count = ref_count - 1 WHERE id = OLD.artifact_id;
RETURN OLD;
END;
$$ LANGUAGE plpgsql;
-- Create triggers
DROP TRIGGER IF EXISTS package_versions_ref_count_insert ON package_versions;
CREATE TRIGGER package_versions_ref_count_insert
AFTER INSERT ON package_versions
FOR EACH ROW
EXECUTE FUNCTION increment_version_ref_count();
DROP TRIGGER IF EXISTS package_versions_ref_count_delete ON package_versions;
CREATE TRIGGER package_versions_ref_count_delete
AFTER DELETE ON package_versions
FOR EACH ROW
EXECUTE FUNCTION decrement_version_ref_count();
-- Data migration: populate from existing semver-pattern tags
-- This extracts versions from tags that look like version numbers
-- Tags like "v1.0.0", "1.2.3", "2.0.0-beta" will be migrated
-- Tags like "latest", "stable", "dev" will NOT be migrated
INSERT INTO package_versions (package_id, artifact_id, version, version_source, created_by, created_at)
SELECT
t.package_id,
t.artifact_id,
-- Strip leading 'v' if present
CASE WHEN t.name LIKE 'v%' THEN substring(t.name from 2) ELSE t.name END,
'migrated_from_tag',
t.created_by,
t.created_at
FROM tags t
WHERE t.name ~ '^v?[0-9]+\.[0-9]+(\.[0-9]+)?([-.][a-zA-Z0-9]+)?$'
ON CONFLICT (package_id, version) DO NOTHING;