58 Commits

Author SHA1 Message Date
Dane Moss
d6b5531885 Merge branch 'cut_release' into 'main'
cut 0.6.0 release

See merge request esv/bsf/bsf-integration/orchard/orchard-mvp!58
2026-02-05 10:00:12 -07:00
Dane Moss
21cc4c6149 cut 0.6.0 release 2026-02-05 09:50:00 -07:00
Mondo Diaz
4b0140d020 Merge branch 'fix/pypi-proxy-timeout' into 'main'
Fix httpx.Timeout configuration in PyPI proxy

See merge request esv/bsf/bsf-integration/orchard/orchard-mvp!57
2026-02-05 10:31:04 -06:00
Mondo Diaz
dfa089376a Fix httpx.Timeout configuration in PyPI proxy 2026-02-05 10:31:04 -06:00
Evan Cohen-Doty
11c5aee0f1 Merge branch 's3_bucket_provisioner' into 'main'
59 Add S3 Bucket Provisioner

Closes #59

See merge request esv/bsf/bsf-integration/orchard/orchard-mvp!49
2026-02-04 11:32:12 -08:00
Evan Cohen-Doty
1b706fe858 59 Add S3 Bucket Provisioner 2026-02-04 11:32:12 -08:00
Mondo Diaz
dcd405679a Merge branch 'feature/transparent-proxy' into 'main'
Add transparent PyPI proxy and improve upstream sources UI

Closes #108

See merge request esv/bsf/bsf-integration/orchard/orchard-mvp!56
2026-01-29 16:12:57 -06:00
Mondo Diaz
97498b2f86 Add transparent PyPI proxy and improve upstream sources UI 2026-01-29 16:12:57 -06:00
Mondo Diaz
e8cf2462b7 Merge branch 'fix/upstream-caching-bugs-2' into 'main'
Simplify cache management UI and improve test status display (#107)

See merge request esv/bsf/bsf-integration/orchard/orchard-mvp!55
2026-01-29 14:25:19 -06:00
Mondo Diaz
038ad4ed1b Simplify cache management UI and improve test status display (#107) 2026-01-29 14:25:19 -06:00
Mondo Diaz
858b45d434 Merge branch 'fix/purge-seed-data-user-id' into 'main'
Fix purge_seed_data type mismatch for access_permissions.user_id (#107)

See merge request esv/bsf/bsf-integration/orchard/orchard-mvp!54
2026-01-29 13:48:21 -06:00
Mondo Diaz
95470b2bf6 Fix purge_seed_data type mismatch for access_permissions.user_id (#107) 2026-01-29 13:48:21 -06:00
Mondo Diaz
c512d85f9e Merge branch 'fix/upstream-caching-bugs' into 'main'
Remove public internet features and fix upstream source UI (#107)

See merge request esv/bsf/bsf-integration/orchard/orchard-mvp!53
2026-01-29 13:26:29 -06:00
Mondo Diaz
82f67539bd Remove public internet features and fix upstream source UI (#107) 2026-01-29 13:26:28 -06:00
Mondo Diaz
e93e7e7021 Merge branch 'feature/proxy-schema' into 'main'
Add upstream caching infrastructure and refactor CI pipeline

See merge request esv/bsf/bsf-integration/orchard/orchard-mvp!52
2026-01-29 11:55:15 -06:00
Mondo Diaz
1d51c856b0 Add upstream caching infrastructure and refactor CI pipeline 2026-01-29 11:55:15 -06:00
Mondo Diaz
c92895ffe9 Merge branch 'fix/migration-rollback' into 'main'
Add rollback after failed migration to allow subsequent migrations to run

See merge request esv/bsf/bsf-integration/orchard/orchard-mvp!51
2026-01-28 15:23:51 -06:00
Mondo Diaz
b147af43d2 Add rollback after failed migration to allow subsequent migrations to run 2026-01-28 15:23:51 -06:00
Mondo Diaz
aed48bb4a2 Merge branch 'fix/teams-migration-runtime-v2' into 'main'
Add teams migration to runtime migrations

See merge request esv/bsf/bsf-integration/orchard/orchard-mvp!50
2026-01-28 14:19:35 -06:00
Mondo Diaz
0e67ebf94f Add teams migration to runtime migrations 2026-01-28 14:19:35 -06:00
Mondo Diaz
0a69910e8b Merge branch 'feature/multi-tenancy-teams' into 'main'
Add multi-tenancy with Teams feature

Closes #88, #89, #90, #91, #92, #93, #94, #95, #96, #97, #98, #99, #100, #101, #102, #103, and #104

See merge request esv/bsf/bsf-integration/orchard/orchard-mvp!48
2026-01-28 12:50:58 -06:00
Mondo Diaz
576791d19e Add multi-tenancy with Teams feature 2026-01-28 12:50:58 -06:00
Mondo Diaz
a5796f5437 Merge branch 'feature/stage-password-ci-var' into 'main'
Use CI variable for stage admin password

See merge request esv/bsf/bsf-integration/orchard/orchard-mvp!47
2026-01-27 15:44:35 -06:00
Mondo Diaz
284945ba33 Use CI variable for stage admin password 2026-01-27 15:44:34 -06:00
Mondo Diaz
fe07638485 Merge branch 'feature/admin-password-env-var' into 'main'
Add configurable admin password via environment variable

Closes #87

See merge request esv/bsf/bsf-integration/orchard/orchard-mvp!46
2026-01-27 14:23:41 -06:00
Mondo Diaz
7120cf64f1 Add configurable admin password via environment variable 2026-01-27 14:23:40 -06:00
Mondo Diaz
718e6e7193 Merge branch 'feature/package-dependencies' into 'main'
Add package dependencies system and project settings page

Closes #76, #77, #78, #79, #80, and #81

See merge request esv/bsf/bsf-integration/orchard/orchard-mvp!45
2026-01-27 10:11:04 -06:00
Mondo Diaz
abba90ebac Add package dependencies system and project settings page 2026-01-27 10:11:04 -06:00
Mondo Diaz
6c8b922818 Merge branch 'fix/reset-stage-before-tests' into 'main'
Add pre-test stage reset to ensure known environment state

See merge request esv/bsf/bsf-integration/orchard/orchard-mvp!44
2026-01-26 09:13:04 -06:00
Mondo Diaz
99d28cf9c6 Add pre-test stage reset to ensure known environment state 2026-01-26 09:13:03 -06:00
Dane Moss
b5579f1643 Merge branch 'release_0.5.1' into 'main'
add CL entry to bump version

See merge request esv/bsf/bsf-integration/orchard/orchard-mvp!43
2026-01-23 15:37:09 -07:00
Dane Moss
fafa03e4ce add CL entry to bump version 2026-01-23 15:37:09 -07:00
Mondo Diaz
d4b2da3232 Merge branch 'fix/release-wait-for-stage-tests' into 'main'
Add gitleaks fingerprint for test file false positive

See merge request esv/bsf/bsf-integration/orchard/orchard-mvp!42
2026-01-23 16:16:03 -06:00
Mondo Diaz
7b04bbdf05 Add gitleaks fingerprint for test file false positive 2026-01-23 16:16:02 -06:00
Mondo Diaz
3a807870a3 Merge branch 'fix/ci-prod-namespace' into 'main'
Fix production CI deployment and simplify tag pipeline

See merge request esv/bsf/bsf-integration/orchard/orchard-mvp!41
2026-01-23 15:50:24 -06:00
Mondo Diaz
f966fde7df Fix production CI deployment and simplify tag pipeline 2026-01-23 15:50:24 -06:00
Mondo Diaz
133d9cbfd6 Merge branch 'bump_version' into 'main'
add changelog entry to cut a new release

See merge request esv/bsf/bsf-integration/orchard/orchard-mvp!40
2026-01-23 13:00:51 -06:00
Dane Moss
276b4f2743 add changelog entry to cut a new release 2026-01-23 10:46:20 -07:00
Mondo Diaz
67ac6bb3f8 Merge branch 'fix/factory-reset-admin-user' into 'main'
Update CHANGELOG with factory reset fixes (#60)

See merge request esv/bsf/bsf-integration/orchard/orchard-mvp!39
2026-01-23 09:33:01 -06:00
Mondo Diaz
b0bb3ed569 Update CHANGELOG with factory reset fixes (#60) 2026-01-21 23:44:45 +00:00
Mondo Diaz
1ac75e1017 Fix factory reset and improve reset_stage CI job
- Add create_default_admin() call to factory reset (admin user wasn't being
  created after reset, only on server restart)
- Add retry logic to reset_stage CI job (3 attempts with 5s delay)
- Use proper context manager for httpx client
- Increase timeout to 120s for reset operation
- Add retry: 1 at job level for transient failures
2026-01-21 23:20:48 +00:00
Mondo Diaz
693613f111 Fix factory reset - capture username before dropping tables 2026-01-21 23:18:29 +00:00
Mondo Diaz
9da4ae8c0d Add gitleaks fingerprint for test file false positive 2026-01-21 22:59:08 +00:00
Mondo Diaz
7ffdc64364 Fix seed_database call in factory reset - pass fresh db session 2026-01-21 22:51:03 +00:00
Mondo Diaz
6abc0c88b0 Merge branch 'feature/stage-reset-job' into 'main'
Fix reset_stage job to read STAGE_URL from environment

See merge request esv/bsf/bsf-integration/orchard/orchard-mvp!38
2026-01-21 16:39:39 -06:00
Mondo Diaz
e96dc5cde8 Fix reset_stage job to read STAGE_URL from environment 2026-01-21 22:25:04 +00:00
Mondo Diaz
cba5bac383 Merge branch 'feature/stage-reset-job' into 'main'
Add factory reset endpoint for stage environment cleanup (#54)

See merge request esv/bsf/bsf-integration/orchard/orchard-mvp!37
2026-01-21 16:00:02 -06:00
Mondo Diaz
535280a783 Add factory reset endpoint for stage environment cleanup (#54) 2026-01-21 16:00:02 -06:00
Dane Moss
c9026e1950 Merge branch 'fix/s3-irsa-credentials' into 'main'
Fix S3 client to support IRSA credentials (#54)

See merge request esv/bsf/bsf-integration/orchard/orchard-mvp!36
2026-01-21 13:42:53 -07:00
Mondo Diaz
fedbd95cf4 Fix S3 client to support IRSA credentials (#54) 2026-01-21 13:42:53 -07:00
Dane Moss
255e25d66d Merge branch 'fix/prod-values-config' into 'main'
Configure prod and stage for AWS services (RDS, S3, Secrets Manager)

See merge request esv/bsf/bsf-integration/orchard/orchard-mvp!35
2026-01-21 12:32:44 -07:00
Mondo Diaz
427d2fec70 Configure prod and stage for AWS services (RDS, S3, Secrets Manager) 2026-01-21 12:32:44 -07:00
Mondo Diaz
199821b34d Merge branch 'feature/upload-download-tests' into 'main'
Add comprehensive upload/download tests and streaming enhancements (#38, #40, #42, #43)

Closes #38, #40, #42, and #43

See merge request esv/bsf/bsf-integration/orchard/orchard-mvp!34
2026-01-21 09:35:12 -06:00
Mondo Diaz
584acd1e90 Add comprehensive upload/download tests and streaming enhancements (#38, #40, #42, #43) 2026-01-21 09:35:12 -06:00
Mondo Diaz
f7ffc1c877 Merge branch 'feature/separate-version-tag' into 'main'
Add separate version tracking for artifacts

Closes orchard-python-poetry#56

See merge request esv/bsf/bsf-integration/orchard/orchard-mvp!33
2026-01-16 11:36:08 -06:00
Mondo Diaz
b93d5a9c68 Add separate version tracking for artifacts 2026-01-16 11:36:08 -06:00
Mondo Diaz
a98ac154d5 Merge branch 'fix/deploy-job-dependencies' into 'main'
Fix deploy job dependencies and add production deployment (#63)

Closes #63

See merge request esv/bsf/bsf-integration/orchard/orchard-mvp!32
2026-01-15 15:16:07 -06:00
Mondo Diaz
823dfcb400 Fix deploy job dependencies and add production deployment (#63) 2026-01-15 15:16:07 -06:00
128 changed files with 34100 additions and 3495 deletions

7
.env.example Normal file
View File

@@ -0,0 +1,7 @@
# Orchard Local Development Environment
# Copy this file to .env and customize as needed
# Note: .env is gitignored and will not be committed
# Admin account password (required for local development)
# This sets the initial admin password when the database is first created
ORCHARD_ADMIN_PASSWORD=changeme123

View File

@@ -8,6 +8,11 @@ variables:
PROSPER_VERSION: v0.64.1
# Use internal PyPI proxy instead of public internet
PIP_INDEX_URL: https://deps.global.bsf.tools/artifactory/api/pypi/pypi.org/simple
# Environment URLs (used by deploy and test jobs)
STAGE_URL: https://orchard-stage.common.global.bsf.tools
PROD_URL: https://orchard.common.global.bsf.tools
# Shared pip cache directory
PIP_CACHE_DIR: "$CI_PROJECT_DIR/.pip-cache"
# Prevent duplicate pipelines for MRs
workflow:
@@ -25,15 +30,121 @@ stages:
- analyze
- deploy
# Override Prosper template jobs to exclude tag pipelines
# Tags only run deploy_prod and smoke_test_prod (image already built on main)
build_image:
rules:
- if: '$CI_COMMIT_TAG'
when: never
- when: on_success
test_image:
rules:
- if: '$CI_COMMIT_TAG'
when: never
- when: on_success
hadolint:
rules:
- if: '$CI_COMMIT_TAG'
when: never
- when: on_success
kics:
variables:
KICS_CONFIG: kics.config
rules:
- if: '$CI_COMMIT_TAG'
when: never
- when: on_success
# Post-deployment integration tests template
secrets:
rules:
- if: '$CI_COMMIT_TAG'
when: never
- when: on_success
app_deps_scan:
rules:
- if: '$CI_COMMIT_TAG'
when: never
- when: on_success
cve_scan:
rules:
- if: '$CI_COMMIT_TAG'
when: never
- when: on_success
app_sbom_analysis:
rules:
- if: '$CI_COMMIT_TAG'
when: never
- when: on_success
cve_sbom_analysis:
rules:
- if: '$CI_COMMIT_TAG'
when: never
- when: on_success
# Disable prosper_setup for tag pipelines since no build/analysis jobs run
# (image is already built when commit was on main, and deploy uses helm directly)
prosper_setup:
rules:
- if: '$CI_COMMIT_TAG'
when: never
- when: on_success
# Override release job to wait for stage deployment and smoke tests before creating tag
# This ensures the tag (which triggers prod deploy) is only created after stage passes
release:
needs: [smoke_test_stage, changelog]
# Full integration test suite template (for feature/stage deployments)
# Runs the complete pytest integration test suite against the deployed environment
.integration_test_template: &integration_test_template
stage: deploy # Runs in deploy stage, but after deployment due to 'needs'
image: deps.global.bsf.tools/docker/python:3.12-slim
timeout: 10m
timeout: 20m # Full suite takes longer than smoke tests
interruptible: true # Cancel if new pipeline starts
retry: 1 # Retry once on failure (network flakiness)
cache:
key: pip-$CI_COMMIT_REF_SLUG
paths:
- .pip-cache/
policy: pull-push
before_script:
- pip install --index-url "$PIP_INDEX_URL" -r backend/requirements.txt
- pip install --index-url "$PIP_INDEX_URL" pytest pytest-asyncio httpx
script:
- cd backend
# Debug: Print environment variables for test configuration
- echo "ORCHARD_TEST_URL=$ORCHARD_TEST_URL"
- echo "ORCHARD_TEST_PASSWORD is set to '${ORCHARD_TEST_PASSWORD:-NOT SET}'"
# Run full integration test suite, excluding:
# - large/slow tests
# - requires_direct_s3 tests (can't access MinIO from outside K8s cluster)
# ORCHARD_TEST_URL tells the tests which server to connect to
# Note: Auth tests work because dev/stage deployments have relaxed rate limits
- |
python -m pytest tests/integration/ -v \
--junitxml=integration-report.xml \
-m "not large and not slow and not requires_direct_s3" \
--tb=short
artifacts:
when: always
expire_in: 1 week
paths:
- backend/integration-report.xml
reports:
junit: backend/integration-report.xml
# Lightweight smoke test template (for production - no test data creation)
.smoke_test_template: &smoke_test_template
stage: deploy
image: deps.global.bsf.tools/docker/python:3.12-slim
timeout: 5m
before_script:
- pip install --index-url "$PIP_INDEX_URL" httpx
script:
@@ -43,12 +154,12 @@ kics:
import os
import sys
BASE_URL = os.environ.get("BASE_URL")
BASE_URL = os.environ.get("ORCHARD_TEST_URL")
if not BASE_URL:
print("ERROR: BASE_URL not set")
print("ERROR: ORCHARD_TEST_URL not set")
sys.exit(1)
print(f"Running integration tests against {BASE_URL}")
print(f"Running smoke tests against {BASE_URL}")
client = httpx.Client(base_url=BASE_URL, timeout=30.0)
errors = []
@@ -86,49 +197,108 @@ kics:
print(f" FAIL: {e}")
sys.exit(1)
else:
print("SUCCESS: All integration tests passed!")
print("SUCCESS: All smoke tests passed!")
sys.exit(0)
PYTEST_SCRIPT
# Integration tests for stage deployment
integration_test_stage:
<<: *integration_test_template
needs: [deploy_stage]
variables:
BASE_URL: https://orchard-stage.common.global.bsf.tools
rules:
- if: '$CI_COMMIT_BRANCH == "main"'
when: on_success
# Integration tests for feature deployment
# Integration tests for feature deployment (full suite)
# Uses DEV_ADMIN_PASSWORD CI variable (same as deploy_feature)
integration_test_feature:
<<: *integration_test_template
needs: [deploy_feature]
variables:
BASE_URL: https://orchard-$CI_COMMIT_REF_SLUG.common.global.bsf.tools
ORCHARD_TEST_URL: https://orchard-$CI_COMMIT_REF_SLUG.common.global.bsf.tools
ORCHARD_TEST_PASSWORD: $DEV_ADMIN_PASSWORD
rules:
- if: '$CI_COMMIT_BRANCH && $CI_COMMIT_BRANCH != "main"'
when: on_success
# Run Python backend tests
python_tests:
# Reset feature environment after integration tests
# Calls factory-reset to clean up test data created during integration tests
reset_feature:
stage: deploy
needs: [integration_test_feature]
image: deps.global.bsf.tools/docker/python:3.12-slim
timeout: 5m
before_script:
- pip install --index-url "$PIP_INDEX_URL" httpx
script:
# Debug: Check if variable is set at shell level
- echo "RESET_ADMIN_PASSWORD length at shell level:${#RESET_ADMIN_PASSWORD}"
- |
python - <<'RESET_SCRIPT'
import httpx
import os
import sys
BASE_URL = f"https://orchard-{os.environ['CI_COMMIT_REF_SLUG']}.common.global.bsf.tools"
PASSWORD_RAW = os.environ.get("RESET_ADMIN_PASSWORD")
if not PASSWORD_RAW:
print("ERROR: RESET_ADMIN_PASSWORD not set")
sys.exit(1)
# Debug: check for hidden characters
print(f"Raw password repr (first 3 chars): {repr(PASSWORD_RAW[:3])}")
print(f"Raw password repr (last 3 chars): {repr(PASSWORD_RAW[-3:])}")
print(f"Raw length: {len(PASSWORD_RAW)}")
# Strip any whitespace
PASSWORD = PASSWORD_RAW.strip()
print(f"Stripped length: {len(PASSWORD)}")
print(f"Resetting environment at {BASE_URL}")
client = httpx.Client(base_url=BASE_URL, timeout=60.0)
# Login as admin
login_resp = client.post("/api/v1/auth/login", json={
"username": "admin",
"password": PASSWORD
})
if login_resp.status_code != 200:
print(f"ERROR: Login failed: {login_resp.status_code}")
print(f"Response: {login_resp.text}")
sys.exit(1)
# Call factory reset
reset_resp = client.post(
"/api/v1/admin/factory-reset",
headers={"X-Confirm-Reset": "yes-delete-all-data"}
)
if reset_resp.status_code == 200:
print("SUCCESS: Factory reset completed")
print(reset_resp.json())
else:
print(f"ERROR: Factory reset failed: {reset_resp.status_code}")
print(reset_resp.text)
sys.exit(1)
RESET_SCRIPT
variables:
# Use same pattern as integration_test_feature - create new variable from CI variable
RESET_ADMIN_PASSWORD: $DEV_ADMIN_PASSWORD
rules:
- if: '$CI_COMMIT_BRANCH && $CI_COMMIT_BRANCH != "main"'
when: on_success
allow_failure: true # Don't fail the pipeline if reset fails
# Run Python backend unit tests
python_unit_tests:
stage: test
needs: [] # Run in parallel with build
image: deps.global.bsf.tools/docker/python:3.12-slim
timeout: 15m
variables:
PIP_CACHE_DIR: "$CI_PROJECT_DIR/.pip-cache"
interruptible: true # Cancel if new pipeline starts
cache:
key: pip-$CI_COMMIT_REF_SLUG
paths:
- .pip-cache/
policy: pull-push
before_script:
- pip install -r backend/requirements.txt
- pip install pytest pytest-asyncio pytest-cov httpx
- pip install --index-url "$PIP_INDEX_URL" -r backend/requirements.txt
- pip install --index-url "$PIP_INDEX_URL" pytest pytest-asyncio pytest-cov httpx
script:
- cd backend
# Only run unit tests - integration tests require Docker Compose services
# Run unit tests (integration tests run post-deployment against live environment)
- python -m pytest tests/unit/ -v --cov=app --cov-report=term --cov-report=xml:coverage.xml --cov-report=html:coverage_html --junitxml=pytest-report.xml
artifacts:
when: always
@@ -143,6 +313,10 @@ python_tests:
coverage_format: cobertura
path: backend/coverage.xml
coverage: '/TOTAL.*\s+(\d+%)/'
rules:
- if: '$CI_COMMIT_TAG'
when: never
- when: on_success
# Run frontend tests
frontend_tests:
@@ -150,6 +324,7 @@ frontend_tests:
needs: [] # Run in parallel with build
image: deps.global.bsf.tools/docker/node:20-alpine
timeout: 15m
interruptible: true # Cancel if new pipeline starts
cache:
key: npm-$CI_COMMIT_REF_SLUG
paths:
@@ -171,11 +346,15 @@ frontend_tests:
coverage_format: cobertura
path: frontend/coverage/cobertura-coverage.xml
coverage: '/All files[^|]*\|[^|]*\s+([\d\.]+)/'
rules:
- if: '$CI_COMMIT_TAG'
when: never
- when: on_success
# Shared deploy configuration
.deploy_template: &deploy_template
stage: deploy
needs: [build_image, kics, hadolint, python_tests, frontend_tests, secrets]
needs: [build_image, test_image, kics, hadolint, python_unit_tests, frontend_tests, secrets, app_deps_scan, cve_scan, cve_sbom_analysis, app_sbom_analysis]
image: deps.global.bsf.tools/registry-1.docker.io/alpine/k8s:1.29.12
.helm_setup: &helm_setup
@@ -184,79 +363,143 @@ frontend_tests:
# OCI-based charts from internal registry - no repo add needed
- helm dependency update
# Simplified deployment verification - just health check
# Full API/frontend checks are done by integration tests post-deployment
.verify_deployment: &verify_deployment |
echo "=== Waiting for health endpoint (certs may take a few minutes) ==="
for i in $(seq 1 30); do
if curl -sf --max-time 10 "$BASE_URL/health" > /dev/null 2>&1; then
echo "Health check passed!"
break
echo "Deployment URL: $BASE_URL"
exit 0
fi
echo "Attempt $i/30 - waiting 10s..."
sleep 10
done
echo "Health check failed after 30 attempts"
exit 1
# Verify health endpoint
echo ""
echo "=== Health Check ==="
curl -sf "$BASE_URL/health" || { echo "Health check failed"; exit 1; }
echo ""
# Verify API is responding
echo ""
echo "=== API Check (GET /api/v1/projects) ==="
HTTP_CODE=$(curl -sf -o /dev/null -w "%{http_code}" "$BASE_URL/api/v1/projects")
if [ "$HTTP_CODE" = "200" ]; then
echo "API responding: HTTP $HTTP_CODE"
else
echo "API check failed: HTTP $HTTP_CODE"
exit 1
fi
# Verify frontend is served
echo ""
echo "=== Frontend Check ==="
if curl -sf "$BASE_URL/" | grep -q "</html>"; then
echo "Frontend is being served"
else
echo "Frontend check failed"
exit 1
fi
echo ""
echo "=== All checks passed! ==="
echo "Deployment URL: $BASE_URL"
# Deploy to stage (main branch)
deploy_stage:
# Ephemeral test deployment in stage namespace (main branch only)
# Runs integration tests before promoting to long-running stage
deploy_test:
<<: *deploy_template
variables:
NAMESPACE: orch-stage-namespace
VALUES_FILE: helm/orchard/values-stage.yaml
BASE_URL: https://orchard-stage.common.global.bsf.tools
VALUES_FILE: helm/orchard/values-dev.yaml
BASE_URL: https://orchard-test.common.global.bsf.tools
before_script:
- kubectl config use-context esv/bsf/bsf-integration/orchard/orchard-mvp:orchard-stage
- *helm_setup
script:
- echo "Deploying to stage environment"
- echo "Deploying ephemeral test environment"
- cd $CI_PROJECT_DIR
- |
helm upgrade --install orchard-test ./helm/orchard \
--namespace $NAMESPACE \
-f $VALUES_FILE \
--set image.tag=git.linux-amd64-$CI_COMMIT_SHA \
--set orchard.auth.adminPassword=$STAGE_ADMIN_PASSWORD \
--set ingress.hosts[0].host=orchard-test.common.global.bsf.tools \
--set ingress.tls[0].hosts[0]=orchard-test.common.global.bsf.tools \
--set ingress.tls[0].secretName=orchard-test-tls \
--set minioIngress.host=minio-test.common.global.bsf.tools \
--set minioIngress.tls.secretName=minio-test-tls \
--wait \
--atomic \
--timeout 10m
- kubectl rollout status deployment/orchard-test-server -n $NAMESPACE --timeout=10m
- *verify_deployment
environment:
name: test
url: https://orchard-test.common.global.bsf.tools
on_stop: cleanup_test
kubernetes:
agent: esv/bsf/bsf-integration/orchard/orchard-mvp:orchard-stage
rules:
- if: '$CI_COMMIT_BRANCH == "main"'
when: on_success
# Integration tests for ephemeral test deployment (main branch)
# Runs against orchard-test before promoting to long-running stage
integration_test_main:
<<: *integration_test_template
needs: [deploy_test]
variables:
ORCHARD_TEST_URL: https://orchard-test.common.global.bsf.tools
ORCHARD_TEST_PASSWORD: $STAGE_ADMIN_PASSWORD
rules:
- if: '$CI_COMMIT_BRANCH == "main"'
when: on_success
# Cleanup ephemeral test deployment after integration tests
cleanup_test:
stage: deploy
needs: [integration_test_main]
image: deps.global.bsf.tools/registry-1.docker.io/alpine/k8s:1.29.12
timeout: 5m
variables:
NAMESPACE: orch-stage-namespace
GIT_STRATEGY: none
before_script:
- kubectl config use-context esv/bsf/bsf-integration/orchard/orchard-mvp:orchard-stage
script:
- echo "Cleaning up ephemeral test deployment orchard-test"
- helm uninstall orchard-test --namespace $NAMESPACE || true
environment:
name: test
action: stop
kubernetes:
agent: esv/bsf/bsf-integration/orchard/orchard-mvp:orchard-stage
rules:
- if: '$CI_COMMIT_BRANCH == "main"'
when: on_success
allow_failure: true
# Deploy to long-running stage (main branch, after ephemeral tests pass)
deploy_stage:
stage: deploy
# Wait for ephemeral test to pass before promoting to long-running stage
needs: [cleanup_test]
image: deps.global.bsf.tools/registry-1.docker.io/alpine/k8s:1.29.12
variables:
NAMESPACE: orch-stage-namespace
VALUES_FILE: helm/orchard/values-stage.yaml
BASE_URL: $STAGE_URL
before_script:
- kubectl config use-context esv/bsf/bsf-integration/orchard/orchard-mvp:orchard-stage
- *helm_setup
script:
- echo "Deploying to long-running stage environment"
- cd $CI_PROJECT_DIR
- |
helm upgrade --install orchard-stage ./helm/orchard \
--namespace $NAMESPACE \
-f $VALUES_FILE \
--set image.tag=git.linux-amd64-$CI_COMMIT_SHA \
--set orchard.auth.adminPassword=$STAGE_ADMIN_PASSWORD \
--wait \
--atomic \
--timeout 5m
- kubectl rollout status deployment/orchard-stage-server -n $NAMESPACE --timeout=5m
--timeout 10m
- kubectl rollout status deployment/orchard-stage-server -n $NAMESPACE --timeout=10m
- *verify_deployment
environment:
name: stage
url: https://orchard-stage.common.global.bsf.tools
url: $STAGE_URL
kubernetes:
agent: esv/bsf/bsf-integration/orchard/orchard-mvp:orchard-stage
rules:
- if: '$CI_COMMIT_BRANCH == "main"'
when: always
when: on_success
# Smoke test for long-running stage (after promotion)
smoke_test_stage:
<<: *smoke_test_template
needs: [deploy_stage]
variables:
ORCHARD_TEST_URL: $STAGE_URL
rules:
- if: '$CI_COMMIT_BRANCH == "main"'
when: on_success
# Deploy feature branch to dev namespace
deploy_feature:
@@ -275,6 +518,7 @@ deploy_feature:
--namespace $NAMESPACE \
-f $VALUES_FILE \
--set image.tag=git.linux-amd64-$CI_COMMIT_SHA \
--set orchard.auth.adminPassword=$DEV_ADMIN_PASSWORD \
--set ingress.hosts[0].host=orchard-$CI_COMMIT_REF_SLUG.common.global.bsf.tools \
--set ingress.tls[0].hosts[0]=orchard-$CI_COMMIT_REF_SLUG.common.global.bsf.tools \
--set ingress.tls[0].secretName=orchard-$CI_COMMIT_REF_SLUG-tls \
@@ -282,8 +526,8 @@ deploy_feature:
--set minioIngress.tls.secretName=minio-$CI_COMMIT_REF_SLUG-tls \
--wait \
--atomic \
--timeout 5m
- kubectl rollout status deployment/orchard-$CI_COMMIT_REF_SLUG-server -n $NAMESPACE --timeout=5m
--timeout 10m
- kubectl rollout status deployment/orchard-$CI_COMMIT_REF_SLUG-server -n $NAMESPACE --timeout=10m
- export BASE_URL="https://orchard-$CI_COMMIT_REF_SLUG.common.global.bsf.tools"
- *verify_deployment
environment:
@@ -295,12 +539,14 @@ deploy_feature:
agent: esv/bsf/bsf-integration/orchard/orchard-mvp:orchard
rules:
- if: '$CI_COMMIT_BRANCH && $CI_COMMIT_BRANCH != "main"'
when: always
when: on_success
# Cleanup feature branch deployment
# Cleanup feature branch deployment (standalone - doesn't need deploy dependencies)
cleanup_feature:
<<: *deploy_template
stage: deploy
needs: []
image: deps.global.bsf.tools/registry-1.docker.io/alpine/k8s:1.29.12
timeout: 5m
variables:
NAMESPACE: orch-dev-namespace
GIT_STRATEGY: none # No source needed, branch may be deleted
@@ -318,3 +564,50 @@ cleanup_feature:
- if: '$CI_COMMIT_BRANCH && $CI_COMMIT_BRANCH != "main"'
when: manual
allow_failure: true
# Deploy to production (version tags only)
deploy_prod:
stage: deploy
# For tag pipelines, no other jobs run - image was already built when commit was on main
needs: []
image: deps.global.bsf.tools/registry-1.docker.io/alpine/k8s:1.29.12
variables:
NAMESPACE: orch-namespace
VALUES_FILE: helm/orchard/values-prod.yaml
BASE_URL: $PROD_URL
before_script:
- kubectl config use-context esv/bsf/bsf-integration/orchard/orchard-mvp:orchard-prod
- *helm_setup
script:
- echo "Deploying to PRODUCTION - version $CI_COMMIT_TAG"
- cd $CI_PROJECT_DIR
- |
helm upgrade --install orchard-prod ./helm/orchard \
--namespace $NAMESPACE \
-f $VALUES_FILE \
--set image.tag=git.linux-amd64-$CI_COMMIT_SHA \
--wait \
--atomic \
--timeout 10m
- kubectl rollout status deployment/orchard-prod-server -n $NAMESPACE --timeout=10m
- *verify_deployment
environment:
name: production
url: $PROD_URL
kubernetes:
agent: esv/bsf/bsf-integration/orchard/orchard-mvp:orchard-prod
rules:
# Only run on semantic version tags (v1.0.0, v1.2.3, etc.)
- if: '$CI_COMMIT_TAG =~ /^v\d+\.\d+\.\d+$/'
when: on_success
allow_failure: false
# Smoke tests for production deployment (read-only, no test data creation)
smoke_test_prod:
<<: *smoke_test_template
needs: [deploy_prod]
variables:
ORCHARD_TEST_URL: $PROD_URL
rules:
- if: '$CI_COMMIT_TAG =~ /^v\d+\.\d+\.\d+$/'
when: on_success

8
.gitleaks.toml Normal file
View File

@@ -0,0 +1,8 @@
# Gitleaks configuration
# https://github.com/gitleaks/gitleaks#configuration
[allowlist]
# Test files that contain variable names matching secret patterns (e.g., s3_key)
paths = [
'''backend/tests/.*\.py''',
]

View File

@@ -4,6 +4,7 @@
# False positive: s3_key is an attribute name in test assertions, not a secret
# These are historical commits - files have since been deleted or updated with inline comments
7e68baed0886a3c928644cd01aa3b39f92d4f976:backend/tests/test_duplicate_detection.py:generic-api-key:154
81458b3bcb5ace97109ba4c16f4afa6e55b1b8bd:backend/tests/test_duplicate_detection.py:generic-api-key:154
2f1891cf0126ec0e7d4c789d872a2cb2dd3a1745:backend/tests/unit/test_storage.py:generic-api-key:381
10d36947948de796f0bacea3827f4531529c405d:backend/tests/unit/test_storage.py:generic-api-key:381
bccbc71c13570d14b8b26a11335c45f102fe3072:backend/tests/unit/test_storage.py:generic-api-key:381
@@ -11,3 +12,8 @@ bccbc71c13570d14b8b26a11335c45f102fe3072:backend/tests/unit/test_storage.py:gene
90bb2a3a393d2361dc3136ee8d761debb0726d8a:backend/tests/unit/test_storage.py:generic-api-key:381
37666e41a72d2a4f34447c0d1a8728e1d7271d24:backend/tests/unit/test_storage.py:generic-api-key:381
0cc4f253621a9601c5193f6ae1e7ae33f0e7fc9b:backend/tests/unit/test_storage.py:generic-api-key:381
35fda65d381acc5ab59bc592ee3013f75906c197:backend/tests/unit/test_storage.py:generic-api-key:381
08dce6cbb836b687002751fed4159bfc2da61f8b:backend/tests/unit/test_storage.py:generic-api-key:381
617bcbe89cff9a009d77e4f1f1864efed1820e63:backend/tests/unit/test_storage.py:generic-api-key:381
1cbd33544388e0fe6db752fa8886fab33cf9ce7c:backend/tests/unit/test_storage.py:generic-api-key:381
7cfad28f678f5a5b8b927d694a17b9ba446b7138:backend/tests/unit/test_storage.py:generic-api-key:381

View File

@@ -6,7 +6,318 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
## [Unreleased]
## [0.6.0] - 2026-02-05
### Added
- Added S3 bucket provisioning terraform configuration (#59)
- Creates an S3 bucket to be used for anything Orchard
- Creates a log bucket for any logs tracking the S3 bucket
- Added auto-fetch capability to dependency resolution endpoint
- `GET /api/v1/project/{project}/{package}/+/{ref}/resolve?auto_fetch=true` fetches missing dependencies from upstream registries
- PyPI registry client queries PyPI JSON API to resolve version constraints
- Fetched artifacts are cached and included in response `fetched` field
- Missing dependencies show `fetch_attempted` and `fetch_error` status
- Configurable max fetch depth via `ORCHARD_AUTO_FETCH_MAX_DEPTH` (default: 3)
- Added `backend/app/registry_client.py` with extensible registry client abstraction
- `RegistryClient` ABC for implementing upstream registry clients
- `PyPIRegistryClient` implementation using PyPI JSON API
- `get_registry_client()` factory function for future npm/maven support
- Added `fetch_and_cache_pypi_package()` reusable function for PyPI package fetching
- Added HTTP connection pooling infrastructure for improved PyPI proxy performance
- `HttpClientManager` with configurable pool size, timeouts, and thread pool executor
- Eliminates per-request connection overhead (~100-500ms → ~5ms)
- Added Redis caching layer with category-aware TTL for hermetic builds
- `CacheService` with graceful fallback when Redis unavailable
- Immutable data (artifact metadata, dependencies) cached forever
- Mutable data (package index, versions) uses configurable TTL
- Added `ArtifactRepository` for batch database operations
- `batch_upsert_dependencies()` reduces N+1 queries to single INSERT
- `get_or_create_artifact()` uses atomic ON CONFLICT upsert
- Added infrastructure status to health endpoint (`/health`)
- Reports HTTP pool size and worker threads
- Reports Redis cache connection status
- Added new configuration settings for HTTP client, Redis, and cache TTL
- `ORCHARD_HTTP_MAX_CONNECTIONS`, `ORCHARD_HTTP_CONNECT_TIMEOUT`, etc.
- `ORCHARD_REDIS_HOST`, `ORCHARD_REDIS_PORT`, `ORCHARD_REDIS_ENABLED`
- `ORCHARD_CACHE_TTL_INDEX`, `ORCHARD_CACHE_TTL_VERSIONS`, etc.
- Added transparent PyPI proxy implementing PEP 503 Simple API (#108)
- `GET /pypi/simple/` - package index (proxied from upstream)
- `GET /pypi/simple/{package}/` - version list with rewritten download links
- `GET /pypi/simple/{package}/{filename}` - download with automatic caching
- Allows `pip install --index-url https://orchard.../pypi/simple/ <package>`
- Artifacts cached on first access through configured upstream sources
- Added `POST /api/v1/cache/resolve` endpoint to cache packages by coordinates instead of URL (#108)
- Added `ORCHARD_PURGE_SEED_DATA` environment variable support to stage helm values to remove seed data from long-running deployments (#107)
- Added frontend system projects visual distinction (#105)
- "Cache" badge for system projects in project list
- "System Cache" badge on project detail page
- Added `is_system` field to Project type
- Added frontend admin page for upstream sources and cache settings (#75)
- New `/admin/cache` page accessible from user menu (admin only)
- Upstream sources table with create/edit/delete/test connectivity
- Cache settings section with air-gap mode and auto-create system projects toggles
- Visual indicators for env-defined sources (locked, cannot be modified)
- Environment variable override badges when settings are overridden
- API client functions for all cache admin operations
- Added environment variable overrides for cache configuration (#74)
- `ORCHARD_CACHE_ALLOW_PUBLIC_INTERNET` - Override allow_public_internet (air-gap mode)
- `ORCHARD_CACHE_AUTO_CREATE_SYSTEM_PROJECTS` - Override auto_create_system_projects
- `ORCHARD_UPSTREAM__{NAME}__*` - Define upstream sources via env vars
- Env-defined sources appear in API with `source: "env"` marker
- Env-defined sources cannot be modified/deleted via API (400 error)
- Cache settings response includes `*_env_override` fields when overridden
- 7 unit tests for env var parsing and configuration
- Added Global Cache Settings Admin API (#73)
- `GET /api/v1/admin/cache-settings` - Retrieve current cache settings
- `PUT /api/v1/admin/cache-settings` - Update cache settings (partial updates)
- Admin-only access with audit logging
- Controls `allow_public_internet` (air-gap mode) and `auto_create_system_projects`
- 7 integration tests for settings management
- Added Upstream Sources Admin API for managing cache sources (#72)
- `GET /api/v1/admin/upstream-sources` - List sources with filtering
- `POST /api/v1/admin/upstream-sources` - Create source with auth configuration
- `GET /api/v1/admin/upstream-sources/{id}` - Get source details
- `PUT /api/v1/admin/upstream-sources/{id}` - Update source (partial updates)
- `DELETE /api/v1/admin/upstream-sources/{id}` - Delete source
- `POST /api/v1/admin/upstream-sources/{id}/test` - Test connectivity
- Admin-only access with audit logging
- Credentials never exposed (only has_password/has_headers flags)
- 13 integration tests for all CRUD operations
- Added system project restrictions and management (#71)
- System projects (`_npm`, `_pypi`, etc.) cannot be deleted (returns 403)
- System projects cannot be made private (must remain public)
- `GET /api/v1/system-projects` endpoint to list all system cache projects
- 5 integration tests for system project restrictions
- Added Cache API endpoint for fetching and storing artifacts from upstream URLs (#70)
- `POST /api/v1/cache` endpoint to cache artifacts from upstream registries
- URL parsing helpers to extract package name/version from npm, PyPI, Maven URLs
- Automatic system project creation (`_npm`, `_pypi`, `_maven`, etc.)
- URL-to-artifact provenance tracking via `cached_urls` table
- Optional user project cross-referencing for custom organization
- Cache hit returns existing artifact without re-fetching
- Air-gap mode enforcement (blocks public URLs when disabled)
- Hash verification for downloaded artifacts
- 21 unit tests for URL parsing and cache endpoint
- Added HTTP client for fetching artifacts from upstream sources (#69)
- `UpstreamClient` class in `backend/app/upstream.py` with streaming downloads
- SHA256 hash computation while streaming (doesn't load large files into memory)
- Auth support: none, basic auth, bearer token, API key (custom headers)
- URL-to-source matching by URL prefix with priority ordering
- Configuration options: timeouts, retries with exponential backoff, redirect limits, max file size
- Air-gap mode enforcement via `allow_public_internet` setting
- Response header capture for provenance tracking
- Proper error handling with custom exception types
- Connection test method for upstream source validation
- 33 unit tests for client functionality
- Added upstream artifact caching schema for hermetic builds (#68)
- `upstream_sources` table for configuring upstream registries (npm, PyPI, Maven, etc.)
- `cache_settings` table for global settings including air-gap mode
- `cached_urls` table for URL-to-artifact provenance tracking
- `is_system` column on projects for system cache projects (_npm, _pypi, etc.)
- Support for multiple auth types: none, basic auth, bearer token, API key
- Fernet encryption for credentials using `ORCHARD_CACHE_ENCRYPTION_KEY`
- Default upstream sources seeded (npm-public, pypi-public, maven-central, docker-hub) - disabled by default
- Migration `010_upstream_caching.sql`
- Added team-based multi-tenancy for organizing projects and collaboration (#88-#104)
- Teams serve as organizational containers for projects
- Users can belong to multiple teams with different roles (owner, admin, member)
- Projects can optionally belong to a team
- Added database schema for teams (#88):
- `teams` table with id, name, slug, description, settings, timestamps
- `team_memberships` table mapping users to teams with roles
- `team_id` column on projects table for team association
- Migrations `009_teams.sql` and `009b_migrate_projects.sql`
- Added Team and TeamMembership ORM models with relationships (#89)
- Added TeamAuthorizationService for team-level access control (#90):
- Team owner/admin gets admin access to all team projects
- Team member gets read access to team projects (upgradeable by explicit permission)
- Role hierarchy: owner > admin > member
- Added Team API endpoints (#92, #93, #94, #95):
- `GET /api/v1/teams` - List teams user belongs to (paginated)
- `POST /api/v1/teams` - Create team (creator becomes owner)
- `GET /api/v1/teams/{slug}` - Get team details
- `PUT /api/v1/teams/{slug}` - Update team (requires admin)
- `DELETE /api/v1/teams/{slug}` - Delete team (requires owner)
- `GET /api/v1/teams/{slug}/members` - List team members
- `POST /api/v1/teams/{slug}/members` - Add member (requires admin)
- `PUT /api/v1/teams/{slug}/members/{username}` - Update member role
- `DELETE /api/v1/teams/{slug}/members/{username}` - Remove member
- `GET /api/v1/teams/{slug}/projects` - List team projects (paginated)
- Updated project creation to support optional team assignment (#95)
- Updated project responses to include team info (team_id, team_slug, team_name)
- Added frontend team management (#97-#104):
- TeamContext provider for managing current team selection
- TeamSelector dropdown component (persists selection in localStorage)
- Teams list page at `/teams`
- Team dashboard page at `/teams/{slug}` with inline project creation
- Team settings page at `/teams/{slug}/settings`
- Team members page at `/teams/{slug}/members`
- Teams navigation link in header (authenticated users only)
- Updated seed data to create a "Demo Team" and assign all seed projects to it
- Added TypeScript types and API client functions for teams
- Access management now shows team-based permissions alongside explicit permissions
- Team-based access displayed as read-only with "Source" column indicating origin
- Team members with access show team slug and role
- Added integration tests for team CRUD, membership, and project operations
- Redesigned teams portal with modern card-based layout
- Card grid view with team avatar, name, slug, role badge, and stats
- Stats bar showing total teams, owned teams, and total projects
- Search functionality for filtering teams (appears when >3 teams)
- Empty states for no teams and no search results
- Added user autocomplete component for team member invitations
- `GET /api/v1/users/search` endpoint for username prefix search
- Dropdown shows matching users as you type
- Keyboard navigation support (arrow keys, enter, escape)
- Debounced search to reduce API calls
- Added unit tests for TeamAuthorizationService
- Added `ORCHARD_ADMIN_PASSWORD` environment variable to configure initial admin password (#87)
- When set, admin user is created with the specified password (no password change required)
- When not set, defaults to `changeme123` and requires password change on first login
- Added Helm chart support for admin password via multiple sources (#87):
- `orchard.auth.adminPassword` - plain value (creates K8s secret)
- `orchard.auth.existingSecret` - reference existing K8s secret
- `orchard.auth.secretsManager` - AWS Secrets Manager integration
- Added `.env.example` template for local development (#87)
- Added `.env` file support in docker-compose.local.yml (#87)
- Added Project Settings page accessible to project admins (#65)
- General settings section for editing description and visibility
- Access Management section (moved from project page)
- Danger Zone section with inline delete confirmation requiring project name
- Settings button (gear icon) on project page header for admins
- Added artifact dependency management system (#76, #77, #78, #79, #80, #81)
- `artifact_dependencies` table with version/tag constraints and check constraints
- `ArtifactDependency` SQLAlchemy model with indexes for fast lookups
- Ensure file parsing (`orchard.ensure` YAML format) during artifact upload
- Circular dependency detection at upload time (rejected with 400)
- Dependency conflict detection at resolution time (409 with conflict details)
- Added dependency API endpoints (#78, #79):
- `GET /api/v1/artifact/{artifact_id}/dependencies` - Get dependencies by artifact ID
- `GET /api/v1/project/{project}/{package}/+/{ref}/dependencies` - Get dependencies by ref
- `GET /api/v1/project/{project}/{package}/reverse-dependencies` - Get reverse dependencies (paginated)
- `GET /api/v1/project/{project}/{package}/+/{ref}/resolve` - Resolve full dependency tree
- Added dependency resolution with topological sorting (#79)
- Returns flat list of all artifacts needed in dependency order
- Includes download URLs, sizes, and version info for each artifact
- Added frontend dependency visualization (#84, #85, #86):
- Dependencies section on package page showing direct dependencies for selected tag
- Tag/version selector to switch between artifacts
- "Used By" section showing reverse dependencies with pagination
- Interactive dependency graph modal with:
- Tree visualization with collapsible nodes
- Zoom (mouse wheel + buttons) and pan (click-drag)
- Click to navigate to package
- Hover tooltip with package details
- Error display for circular dependencies and conflicts
- Added migration `008_artifact_dependencies.sql` for dependency schema
- Added `dependencies.py` module with parsing, validation, and resolution logic
- Added comprehensive integration tests for all dependency features
### Changed
- Removed Usage section from Package page (curl command examples)
- PyPI proxy now uses shared HTTP connection pool instead of per-request clients
- PyPI proxy now caches upstream source configuration in Redis
- Dependency storage now uses batch INSERT instead of individual queries
- Increased default database pool size from 5 to 20 connections
- Increased default database max overflow from 10 to 30 connections
- Enabled Redis in Helm chart values for dev, stage, and prod environments
- Upstream sources table text is now centered under column headers (#108)
- ENV badge now appears inline with source name instead of separate column (#108)
- Test and Edit buttons now have more prominent button styling (#108)
- Reduced footer padding for cleaner layout (#108)
- Upstream source connectivity test no longer follows redirects, fixing "Exceeded maximum allowed redirects" error with Artifactory proxies (#107)
- Test runs automatically after saving a new or updated upstream source (#107)
- Test status now shows as colored dots (green=success, red=error) instead of text badges (#107)
- Clicking red dot shows error details in a modal (#107)
- Source name column no longer wraps text for better table layout (#107)
- Renamed "Cache Management" page to "Upstream Sources" (#107)
- Moved Delete button from table row to edit modal for cleaner table layout (#107)
- Added pre-test stage reset to ensure known environment state before integration tests (#54)
- Upload endpoint now accepts optional `ensure` file parameter for declaring dependencies
- Updated upload API documentation with ensure file format and examples
- Converted teams list and team projects to use DataTable component for consistent styling
- Centered team members and team settings page content
- Added orchard logo icon and dot separator to footer
### Fixed
- Fixed purge_seed_data crash when deleting access permissions - was comparing UUID to VARCHAR column (#107)
- Fixed dark theme styling for team pages - modals, forms, and dropdowns now use correct theme variables
- Fixed UserAutocomplete and TeamSelector dropdown backgrounds for dark theme
- Fixed PyPI proxy filtering platform-specific dependencies (pyobjc on macOS, pywin32 on Windows)
- Fixed bare version constraints being treated as wildcards (e.g., `certifi@2025.10.5` now fetches exact version)
### Removed
- Removed `is_public` field from upstream sources - all sources are now treated as internal/private (#107)
- Removed `allow_public_internet` (air-gap mode) setting from cache settings - not needed for enterprise proxy use case (#107)
- Removed seeding of public registry URLs (npm-public, pypi-public, maven-central, docker-hub) (#107)
- Removed "Public" badge and checkbox from upstream sources UI (#107)
- Removed "Allow Public Internet" toggle from cache settings UI (#107)
- Removed "Global Settings" section from cache management UI - auto-create system projects is always enabled (#107)
- Removed unused CacheSettings frontend types and API functions (#107)
## [0.5.1] - 2026-01-23
### Changed
- Simplified tag pipeline to only run deploy and smoke tests (image already built on main) (#54)
### Fixed
- Fixed production CI deployment namespace to use correct `orch-namespace` (#54)
- Added gitleaks config to allowlist test files from secret scanning (#54)
## [0.5.0] - 2026-01-23
### Added
- Added factory reset endpoint `POST /api/v1/admin/factory-reset` for test environment cleanup (#54)
- Requires admin authentication and `X-Confirm-Reset: yes-delete-all-data` header
- Drops all database tables, clears S3 bucket, reinitializes schema, re-seeds default data
- CI pipeline automatically calls this after integration tests on stage
- Added `delete_all()` method to storage backend for bulk S3 object deletion (#54)
- Added AWS Secrets Manager CSI driver support for database credentials (#54)
- Added SecretProviderClass template for Secrets Manager integration (#54)
- Added IRSA service account annotations for prod and stage environments (#54)
- Added comprehensive upload/download tests for size boundaries (1B to 1GB) (#38)
- Added concurrent upload/download tests (2, 5, 10 parallel operations) (#38)
- Added data integrity tests (binary, text, unicode, compressed content) (#38)
- Added chunk boundary tests for edge cases (#38)
- Added `@pytest.mark.large` and `@pytest.mark.concurrent` test markers (#38)
- Added `generate_content()` and `generate_content_with_hash()` test helpers (#38)
- Added `sized_content` fixture for generating test content of specific sizes (#38)
- Added upload API tests: upload without tag, artifact creation verification, S3 object creation (#38)
- Added download API tests: tag: prefix resolution, 404 for nonexistent project/package/artifact (#38)
- Added download header tests: Content-Type, Content-Length, Content-Disposition, ETag, X-Checksum-SHA256 (#38)
- Added error handling tests: timeout behavior, checksum validation, resource cleanup, graceful error responses (#38)
- Added version API tests: version creation, auto-detection, listing, download by version prefix (#38)
- Added integrity verification tests: round-trip hash verification, client-side verification workflow, size variants (1KB-10MB) (#40)
- Added consistency check endpoint tests with response format validation (#40)
- Added corruption detection tests: bit flip, truncation, appended content, size mismatch, missing S3 objects (#40)
- Added Digest header tests (RFC 3230) and verification mode tests (#40)
- Added integrity verification documentation (`docs/integrity-verification.md`) (#40)
- Added conditional request support for downloads (If-None-Match, If-Modified-Since) returning 304 Not Modified (#42)
- Added caching headers to downloads: Cache-Control (immutable), Last-Modified (#42)
- Added 416 Range Not Satisfiable response for invalid range requests (#42)
- Added download completion logging with bytes transferred and throughput (#42)
- Added client disconnect handling during streaming downloads (#42)
- Added streaming download tests: range requests, conditional requests, caching headers, download resume (#42)
- Added upload duration and throughput metrics (`duration_ms`, `throughput_mbps`) to upload response (#43)
- Added upload progress logging for large files (hash computation and multipart upload phases) (#43)
- Added client disconnect handling during uploads with proper cleanup (#43)
- Added upload progress tracking endpoint `GET /upload/{upload_id}/progress` for resumable uploads (#43)
- Added large file upload tests (10MB, 100MB, 1GB) with multipart upload verification (#43)
- Added upload cancellation and timeout handling tests (#43)
- Added comprehensive API documentation for upload endpoints with curl, Python, and JavaScript examples (#43)
- Added `package_versions` table for immutable version tracking separate from mutable tags (#56)
- Versions are set at upload time via explicit `version` parameter or auto-detected from filename/metadata
- Version detection priority: explicit parameter > package metadata > filename pattern
- Versions are immutable once created (unlike tags which can be moved)
- Added version API endpoints (#56):
- `GET /api/v1/project/{project}/{package}/versions` - List all versions for a package
- `GET /api/v1/project/{project}/{package}/versions/{version}` - Get specific version details
- `DELETE /api/v1/project/{project}/{package}/versions/{version}` - Delete a version (admin only)
- Added version support to upload endpoint via `version` form parameter (#56)
- Added `version:X.Y.Z` prefix for explicit version resolution in download refs (#56)
- Added version field to tag responses (shows which version the artifact has, if any) (#56)
- Added migration `007_package_versions.sql` with ref_count triggers and data migration from semver tags (#56)
- Added production deployment job triggered by semantic version tags (v1.0.0) with manual approval gate (#63)
- Added production Helm values file with persistence enabled (20Gi PostgreSQL, 100Gi MinIO) (#63)
- Added integration tests for production deployment (#63)
- Added GitLab CI pipeline for feature branch deployments to dev namespace (#51)
- Added `deploy_feature` job with dynamic hostnames and unique release names (#51)
- Added `cleanup_feature` job with `on_stop` for automatic cleanup on merge (#51)
@@ -16,6 +327,22 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- Added internal proxy configuration for npm, pip, helm, and apt (#51)
### Changed
- Configured stage and prod to use AWS RDS instead of PostgreSQL subchart (#54)
- Configured stage and prod to use AWS S3 instead of MinIO subchart (#54)
- Changed prod deployment from manual to automatic on version tags (#54)
- Updated S3 client to support IRSA credentials when no explicit keys provided (#54)
- Changed prod image pullPolicy to Always (#54)
- Added proxy-body-size annotation to prod ingress for large uploads (#54)
- CI integration tests now run full pytest suite (~350 tests) against deployed environment instead of 3 smoke tests
- CI production deployment uses lightweight smoke tests only (no test data creation in prod)
- CI pipeline improvements: shared pip cache, `interruptible` flag on test jobs, retry on integration tests
- Simplified deploy verification to health check only (full checks done by integration tests)
- Extracted environment URLs to global variables for maintainability
- Made `cleanup_feature` job standalone (no longer inherits deploy template dependencies)
- Renamed `integration_test_prod` to `smoke_test_prod` for clarity
- Updated download ref resolution to check versions before tags (version → tag → artifact ID) (#56)
- Deploy jobs now require all security scans to pass before deployment (added test_image, app_deps_scan, cve_scan, cve_sbom_analysis, app_sbom_analysis to dependencies) (#63)
- Increased deploy job timeout from 5m to 10m (#63)
- Added `--atomic` flag to Helm deployments for automatic rollback on failure
- Adjusted dark mode color palette to use lighter background tones for better readability and reduced eye strain (#52)
- Replaced project card grid with sortable data table on Home page for better handling of large project lists
@@ -26,6 +353,23 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- Improved pod naming: Orchard pods now named `orchard-{env}-server-*` for clarity (#51)
### Fixed
- Fixed factory reset not creating default admin user after reset (#60)
- Admin user was only created at server startup, not after factory reset
- CI reset job would fail to login because admin user didn't exist
- Improved reset_stage CI job reliability (#60)
- Added application-level retry logic (3 attempts with 5s delay)
- Added job-level retry for transient failures
- Fixed httpx client to use proper context manager
- Increased timeout to 120s for reset operations
- Fixed CI integration test rate limiting: added configurable `ORCHARD_LOGIN_RATE_LIMIT` env var, relaxed to 1000/minute for dev/stage
- Fixed duplicate `TestSecurityEdgeCases` class definition in test_auth_api.py
- Fixed integration tests auth: session-scoped client, configurable credentials via env vars, fail-fast on auth errors
- Fixed 413 Request Entity Too Large errors on uploads by adding `proxy-body-size: "0"` nginx annotation to Orchard ingress
- Fixed CI tests that require direct S3 access: added `@pytest.mark.requires_direct_s3` marker and excluded from CI
- Fixed ref_count triggers not being created: added auto-migration for tags ref_count trigger functions
- Fixed Content-Disposition header encoding for non-ASCII filenames using RFC 5987 (#38)
- Fixed deploy jobs running even when tests or security scans fail (changed rules from `when: always` to `when: on_success`) (#63)
- Fixed python_tests job not using internal PyPI proxy (#63)
- Fixed `cleanup_feature` job failing when branch is deleted (`GIT_STRATEGY: none`) (#51)
- Fixed gitleaks false positives with fingerprints for historical commits (#51)
- Fixed integration tests running when deploy fails (`when: on_success`) (#51)
@@ -33,9 +377,12 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- Fixed deploy jobs running when secrets scan fails (added `secrets` to deploy dependencies)
- Fixed dev environment memory requests to equal limits per cluster Kyverno policy
- Fixed init containers missing resource limits (Kyverno policy compliance)
- Fixed Python SyntaxWarning for invalid escape sequence in database migration regex pattern
### Removed
- Removed unused `store_streaming()` method from storage.py (#51)
- Disabled PostgreSQL subchart for stage and prod environments (#54)
- Disabled MinIO subchart for stage and prod environments (#54)
## [0.4.0] - 2026-01-12
### Added

View File

@@ -22,6 +22,7 @@ Orchard is a centralized binary artifact storage system that provides content-ad
- **Package** - Named collection within a project
- **Artifact** - Specific content instance identified by SHA256
- **Tags** - Alias system for referencing artifacts by human-readable names (e.g., `v1.0.0`, `latest`, `stable`)
- **Versions** - Immutable version records set at upload time (explicit or auto-detected from filename/metadata), separate from mutable tags
- **Package Formats & Platforms** - Packages can be tagged with format (npm, pypi, docker, deb, rpm, etc.) and platform (linux, darwin, windows, etc.)
- **Rich Package Metadata** - Package listings include aggregated stats (tag count, artifact count, total size, latest tag)
- **S3-Compatible Backend** - Uses MinIO (or any S3-compatible storage) for artifact storage
@@ -73,6 +74,9 @@ Orchard is a centralized binary artifact storage system that provides content-ad
| `POST` | `/api/v1/project/:project/:package/tags` | Create a tag |
| `GET` | `/api/v1/project/:project/:package/tags/:tag_name` | Get single tag with artifact metadata |
| `GET` | `/api/v1/project/:project/:package/tags/:tag_name/history` | Get tag change history |
| `GET` | `/api/v1/project/:project/:package/versions` | List all versions for a package |
| `GET` | `/api/v1/project/:project/:package/versions/:version` | Get specific version details |
| `DELETE` | `/api/v1/project/:project/:package/versions/:version` | Delete a version (admin only) |
| `GET` | `/api/v1/project/:project/:package/artifacts` | List artifacts in package (with filtering) |
| `GET` | `/api/v1/project/:project/:package/consumers` | List consumers of a package |
| `GET` | `/api/v1/artifact/:id` | Get artifact metadata with referencing tags |
@@ -93,12 +97,14 @@ For large files, use the resumable upload API:
When downloading artifacts, the `:ref` parameter supports multiple formats:
- `latest` - Tag name directly
- `v1.0.0` - Version tag
- `latest` - Implicit lookup (checks version first, then tag, then artifact ID)
- `v1.0.0` - Implicit lookup (version takes precedence over tag with same name)
- `version:1.0.0` - Explicit version reference
- `tag:stable` - Explicit tag reference
- `version:2024.1` - Version reference
- `artifact:a3f5d8e12b4c6789...` - Direct SHA256 hash reference
**Resolution order for implicit refs:** version → tag → artifact ID
## Quick Start
### Prerequisites
@@ -230,9 +236,16 @@ curl "http://localhost:8080/api/v1/project/my-project/packages/releases?include_
### Upload an Artifact
```bash
# Upload with tag only (version auto-detected from filename)
curl -X POST http://localhost:8080/api/v1/project/my-project/releases/upload \
-F "file=@./build/app-v1.0.0.tar.gz" \
-F "tag=v1.0.0"
-F "tag=latest"
# Upload with explicit version and tag
curl -X POST http://localhost:8080/api/v1/project/my-project/releases/upload \
-F "file=@./build/app-v1.0.0.tar.gz" \
-F "tag=latest" \
-F "version=1.0.0"
```
Response:
@@ -242,7 +255,9 @@ Response:
"size": 1048576,
"project": "my-project",
"package": "releases",
"tag": "v1.0.0",
"tag": "latest",
"version": "1.0.0",
"version_source": "explicit",
"format_metadata": {
"format": "tarball",
"package_name": "app",
@@ -400,6 +415,38 @@ curl http://localhost:8080/api/v1/project/my-project/releases/tags/latest/histor
Returns list of artifact changes for the tag (most recent first).
### List Versions
```bash
# Basic listing
curl http://localhost:8080/api/v1/project/my-project/releases/versions
# With pagination and sorting
curl "http://localhost:8080/api/v1/project/my-project/releases/versions?sort=version&order=desc"
```
Response includes tags pointing to each version's artifact:
```json
{
"items": [
{
"id": "uuid",
"package_id": "uuid",
"version": "1.0.0",
"version_source": "explicit",
"artifact_id": "a3f5d8e...",
"size": 1048576,
"content_type": "application/gzip",
"original_name": "app-v1.0.0.tar.gz",
"created_at": "2025-01-01T00:00:00Z",
"created_by": "user",
"tags": ["latest", "stable"]
}
],
"pagination": {"page": 1, "limit": 20, "total": 1, "total_pages": 1}
}
```
### List Artifacts in Package
```bash
@@ -614,7 +661,8 @@ See `helm/orchard/values.yaml` for all configuration options.
- **projects** - Top-level organizational containers
- **packages** - Collections within projects
- **artifacts** - Content-addressable artifacts (SHA256)
- **tags** - Aliases pointing to artifacts
- **tags** - Mutable aliases pointing to artifacts
- **package_versions** - Immutable version records (set at upload time)
- **tag_history** - Audit trail for tag changes
- **uploads** - Upload event records
- **consumers** - Dependency tracking

View File

@@ -360,21 +360,36 @@ def create_default_admin(db: Session) -> Optional[User]:
"""Create the default admin user if no users exist.
Returns the created user, or None if users already exist.
The admin password can be set via ORCHARD_ADMIN_PASSWORD environment variable.
If not set, defaults to 'changeme123' and requires password change on first login.
"""
# Check if any users exist
user_count = db.query(User).count()
if user_count > 0:
return None
settings = get_settings()
# Use configured password or default
password = settings.admin_password if settings.admin_password else "changeme123"
# Only require password change if using the default password
must_change = not settings.admin_password
# Create default admin
auth_service = AuthService(db)
admin = auth_service.create_user(
username="admin",
password="changeme123",
password=password,
is_admin=True,
must_change_password=True,
must_change_password=must_change,
)
if settings.admin_password:
logger.info("Created default admin user with configured password")
else:
logger.info("Created default admin user with default password (changeme123)")
return admin
@@ -643,32 +658,51 @@ class AuthorizationService:
self, project_id: str, user: Optional[User]
) -> Optional[str]:
"""Get the user's access level for a project.
Returns the highest access level the user has, or None if no access.
Checks in order:
1. System admin - gets admin access to all projects
2. Project owner (created_by) - gets admin access
3. Explicit permission in access_permissions table
3. Team-based access (owner/admin gets admin, member gets read)
4. Explicit permission in access_permissions table
5. Public access
"""
from .models import Project, AccessPermission
from .models import Project, AccessPermission, TeamMembership
# Get the project
project = self.db.query(Project).filter(Project.id == project_id).first()
if not project:
return None
# Anonymous users only get access to public projects
if not user:
return "read" if project.is_public else None
# System admins get admin access everywhere
if user.is_admin:
return "admin"
# Project owner gets admin access
if project.created_by == user.username:
return "admin"
# Check team-based access if project belongs to a team
if project.team_id:
membership = (
self.db.query(TeamMembership)
.filter(
TeamMembership.team_id == project.team_id,
TeamMembership.user_id == user.id,
)
.first()
)
if membership:
# Team owner/admin gets admin on all team projects
if membership.role in ("owner", "admin"):
return "admin"
# Team member gets read access (upgradeable by explicit permission)
# Continue checking explicit permissions for potential upgrade
# Check explicit permissions
permission = (
self.db.query(AccessPermission)
@@ -678,13 +712,27 @@ class AuthorizationService:
)
.first()
)
if permission:
# Check expiration
if permission.expires_at and permission.expires_at < datetime.now(timezone.utc):
return "read" if project.is_public else None
return permission.level
pass # Permission expired, fall through
else:
return permission.level
# Team member gets read access if no explicit permission
if project.team_id:
membership = (
self.db.query(TeamMembership)
.filter(
TeamMembership.team_id == project.team_id,
TeamMembership.user_id == user.id,
)
.first()
)
if membership:
return "read"
# Fall back to public access
return "read" if project.is_public else None
@@ -869,6 +917,226 @@ def check_project_access(
return project
# --- Team Authorization ---
# Team roles in order of increasing privilege
TEAM_ROLES = ["member", "admin", "owner"]
def get_team_role_rank(role: str) -> int:
"""Get numeric rank for team role comparison."""
try:
return TEAM_ROLES.index(role)
except ValueError:
return -1
def has_sufficient_team_role(user_role: str, required_role: str) -> bool:
"""Check if user_role is sufficient for required_role.
Role hierarchy: owner > admin > member
"""
return get_team_role_rank(user_role) >= get_team_role_rank(required_role)
class TeamAuthorizationService:
"""Service for checking team-level authorization."""
def __init__(self, db: Session):
self.db = db
def get_user_team_role(
self, team_id: str, user: Optional[User]
) -> Optional[str]:
"""Get the user's role in a team.
Returns the role ('owner', 'admin', 'member') or None if not a member.
System admins who are not team members are treated as team admins.
"""
from .models import Team, TeamMembership
if not user:
return None
# Check actual membership first
membership = (
self.db.query(TeamMembership)
.filter(
TeamMembership.team_id == team_id,
TeamMembership.user_id == user.id,
)
.first()
)
if membership:
return membership.role
# System admins who are not members get admin access
if user.is_admin:
return "admin"
return None
def check_team_access(
self,
team_id: str,
user: Optional[User],
required_role: str = "member",
) -> bool:
"""Check if user has required role in team.
Args:
team_id: Team ID to check
user: User to check (None means no access)
required_role: Minimum required role ('member', 'admin', 'owner')
Returns:
True if user has sufficient role, False otherwise
"""
user_role = self.get_user_team_role(team_id, user)
if not user_role:
return False
return has_sufficient_team_role(user_role, required_role)
def can_create_project(self, team_id: str, user: Optional[User]) -> bool:
"""Check if user can create projects in team (requires admin+)."""
return self.check_team_access(team_id, user, "admin")
def can_manage_members(self, team_id: str, user: Optional[User]) -> bool:
"""Check if user can manage team members (requires admin+)."""
return self.check_team_access(team_id, user, "admin")
def can_delete_team(self, team_id: str, user: Optional[User]) -> bool:
"""Check if user can delete the team (requires owner)."""
return self.check_team_access(team_id, user, "owner")
def get_team_by_slug(self, slug: str) -> Optional["Team"]:
"""Get a team by its slug."""
from .models import Team
return self.db.query(Team).filter(Team.slug == slug).first()
def get_user_teams(self, user: User) -> list:
"""Get all teams a user is a member of."""
from .models import Team, TeamMembership
return (
self.db.query(Team)
.join(TeamMembership)
.filter(TeamMembership.user_id == user.id)
.order_by(Team.name)
.all()
)
def get_team_authorization_service(db: Session = Depends(get_db)) -> TeamAuthorizationService:
"""Get a TeamAuthorizationService instance."""
return TeamAuthorizationService(db)
class TeamAccessChecker:
"""Dependency for checking team access in route handlers."""
def __init__(self, required_role: str = "member"):
self.required_role = required_role
def __call__(
self,
slug: str,
db: Session = Depends(get_db),
current_user: Optional[User] = Depends(get_current_user_optional),
) -> User:
"""Check if user has required role in team.
Raises 404 if team not found, 401 if not authenticated, 403 if insufficient role.
Returns the current user.
"""
from .models import Team
# Find team by slug
team = db.query(Team).filter(Team.slug == slug).first()
if not team:
raise HTTPException(
status_code=status.HTTP_404_NOT_FOUND,
detail=f"Team '{slug}' not found",
)
if not current_user:
raise HTTPException(
status_code=status.HTTP_401_UNAUTHORIZED,
detail="Authentication required",
headers={"WWW-Authenticate": "Bearer"},
)
auth_service = TeamAuthorizationService(db)
if not auth_service.check_team_access(str(team.id), current_user, self.required_role):
raise HTTPException(
status_code=status.HTTP_403_FORBIDDEN,
detail=f"Insufficient team permissions. Required role: {self.required_role}",
)
return current_user
# Pre-configured team access checkers
require_team_member = TeamAccessChecker("member")
require_team_admin = TeamAccessChecker("admin")
require_team_owner = TeamAccessChecker("owner")
def check_team_access(
db: Session,
team_slug: str,
user: Optional[User],
required_role: str = "member",
) -> "Team":
"""Check if user has required role in team.
This is a helper function for use in route handlers.
Args:
db: Database session
team_slug: Slug of the team
user: Current user (can be None for no access)
required_role: Required team role (member, admin, owner)
Returns:
The Team object if access is granted
Raises:
HTTPException 404: Team not found
HTTPException 401: Authentication required
HTTPException 403: Insufficient permissions
"""
from .models import Team
# Find team by slug
team = db.query(Team).filter(Team.slug == team_slug).first()
if not team:
raise HTTPException(
status_code=status.HTTP_404_NOT_FOUND,
detail=f"Team '{team_slug}' not found",
)
if not user:
raise HTTPException(
status_code=status.HTTP_401_UNAUTHORIZED,
detail="Authentication required",
headers={"WWW-Authenticate": "Bearer"},
)
auth_service = TeamAuthorizationService(db)
if not auth_service.check_team_access(str(team.id), user, required_role):
raise HTTPException(
status_code=status.HTTP_403_FORBIDDEN,
detail=f"Insufficient team permissions. Required role: {required_role}",
)
return team
# --- OIDC Configuration Service ---

316
backend/app/cache.py Normal file
View File

@@ -0,0 +1,316 @@
"""
Cache service for upstream artifact caching.
Provides URL parsing, system project management, and caching logic
for the upstream caching feature.
"""
import logging
import re
from dataclasses import dataclass
from typing import Optional
from urllib.parse import urlparse, unquote
logger = logging.getLogger(__name__)
# System project names for each source type
SYSTEM_PROJECT_NAMES = {
"npm": "_npm",
"pypi": "_pypi",
"maven": "_maven",
"docker": "_docker",
"helm": "_helm",
"nuget": "_nuget",
"deb": "_deb",
"rpm": "_rpm",
"generic": "_generic",
}
# System project descriptions
SYSTEM_PROJECT_DESCRIPTIONS = {
"npm": "System cache for npm packages",
"pypi": "System cache for PyPI packages",
"maven": "System cache for Maven packages",
"docker": "System cache for Docker images",
"helm": "System cache for Helm charts",
"nuget": "System cache for NuGet packages",
"deb": "System cache for Debian packages",
"rpm": "System cache for RPM packages",
"generic": "System cache for generic artifacts",
}
@dataclass
class ParsedUrl:
"""Parsed URL information for caching."""
package_name: str
version: Optional[str] = None
filename: Optional[str] = None
def parse_npm_url(url: str) -> Optional[ParsedUrl]:
"""
Parse npm registry URL to extract package name and version.
Formats:
- https://registry.npmjs.org/{package}/-/{package}-{version}.tgz
- https://registry.npmjs.org/@{scope}/{package}/-/{package}-{version}.tgz
Examples:
- https://registry.npmjs.org/lodash/-/lodash-4.17.21.tgz
- https://registry.npmjs.org/@types/node/-/node-18.0.0.tgz
"""
parsed = urlparse(url)
path = unquote(parsed.path)
# Pattern for scoped packages: /@scope/package/-/package-version.tgz
scoped_pattern = r"^/@([^/]+)/([^/]+)/-/\2-(.+)\.tgz$"
match = re.match(scoped_pattern, path)
if match:
scope, name, version = match.groups()
return ParsedUrl(
package_name=f"@{scope}/{name}",
version=version,
filename=f"{name}-{version}.tgz",
)
# Pattern for unscoped packages: /package/-/package-version.tgz
unscoped_pattern = r"^/([^/@]+)/-/\1-(.+)\.tgz$"
match = re.match(unscoped_pattern, path)
if match:
name, version = match.groups()
return ParsedUrl(
package_name=name,
version=version,
filename=f"{name}-{version}.tgz",
)
return None
def parse_pypi_url(url: str) -> Optional[ParsedUrl]:
"""
Parse PyPI URL to extract package name and version.
Formats:
- https://files.pythonhosted.org/packages/.../package-version.tar.gz
- https://files.pythonhosted.org/packages/.../package-version-py3-none-any.whl
- https://pypi.org/packages/.../package-version.tar.gz
Examples:
- https://files.pythonhosted.org/packages/ab/cd/requests-2.28.0.tar.gz
- https://files.pythonhosted.org/packages/ab/cd/requests-2.28.0-py3-none-any.whl
"""
parsed = urlparse(url)
path = unquote(parsed.path)
# Get the filename from the path
filename = path.split("/")[-1]
if not filename:
return None
# Handle wheel files: package-version-py3-none-any.whl
wheel_pattern = r"^([a-zA-Z0-9_-]+)-(\d+[^-]*)-.*\.whl$"
match = re.match(wheel_pattern, filename)
if match:
name, version = match.groups()
# Normalize package name (PyPI uses underscores internally)
name = name.replace("_", "-").lower()
return ParsedUrl(
package_name=name,
version=version,
filename=filename,
)
# Handle source distributions: package-version.tar.gz or package-version.zip
sdist_pattern = r"^([a-zA-Z0-9_-]+)-(\d+(?:\.\d+)*(?:[a-zA-Z0-9_.+-]*)?)(?:\.tar\.gz|\.zip|\.tar\.bz2)$"
match = re.match(sdist_pattern, filename)
if match:
name, version = match.groups()
name = name.replace("_", "-").lower()
return ParsedUrl(
package_name=name,
version=version,
filename=filename,
)
return None
def parse_maven_url(url: str) -> Optional[ParsedUrl]:
"""
Parse Maven repository URL to extract artifact info.
Format:
- https://repo1.maven.org/maven2/{group}/{artifact}/{version}/{artifact}-{version}.jar
Examples:
- https://repo1.maven.org/maven2/org/apache/commons/commons-lang3/3.12.0/commons-lang3-3.12.0.jar
- https://repo1.maven.org/maven2/com/google/guava/guava/31.1-jre/guava-31.1-jre.jar
"""
parsed = urlparse(url)
path = unquote(parsed.path)
# Find /maven2/ or similar repository path
maven2_idx = path.find("/maven2/")
if maven2_idx >= 0:
path = path[maven2_idx + 8:] # Remove /maven2/
elif path.startswith("/"):
path = path[1:]
parts = path.split("/")
if len(parts) < 4:
return None
# Last part is filename, before that is version, before that is artifact
filename = parts[-1]
version = parts[-2]
artifact = parts[-3]
group = ".".join(parts[:-3])
# Verify filename matches expected pattern
if not filename.startswith(f"{artifact}-{version}"):
return None
return ParsedUrl(
package_name=f"{group}:{artifact}",
version=version,
filename=filename,
)
def parse_docker_url(url: str) -> Optional[ParsedUrl]:
"""
Parse Docker registry URL to extract image info.
Note: Docker registries are more complex (manifests, blobs, etc.)
This handles basic blob/manifest URLs.
Examples:
- https://registry-1.docker.io/v2/library/nginx/blobs/sha256:abc123
- https://registry-1.docker.io/v2/myuser/myimage/manifests/latest
"""
parsed = urlparse(url)
path = unquote(parsed.path)
# Pattern: /v2/{namespace}/{image}/blobs/{digest} or /manifests/{tag}
pattern = r"^/v2/([^/]+(?:/[^/]+)?)/([^/]+)/(blobs|manifests)/(.+)$"
match = re.match(pattern, path)
if match:
namespace, image, artifact_type, reference = match.groups()
if namespace == "library":
package_name = image
else:
package_name = f"{namespace}/{image}"
# For manifests, the reference is the tag
version = reference if artifact_type == "manifests" else None
return ParsedUrl(
package_name=package_name,
version=version,
filename=f"{image}-{reference}" if version else reference,
)
return None
def parse_generic_url(url: str) -> ParsedUrl:
"""
Parse a generic URL to extract filename.
Attempts to extract meaningful package name and version from filename.
Examples:
- https://example.com/downloads/myapp-1.2.3.tar.gz
- https://github.com/user/repo/releases/download/v1.0/release.zip
"""
parsed = urlparse(url)
path = unquote(parsed.path)
filename = path.split("/")[-1] or "artifact"
# List of known compound and simple extensions
known_extensions = [
".tar.gz", ".tar.bz2", ".tar.xz",
".zip", ".tgz", ".gz", ".jar", ".war", ".deb", ".rpm"
]
# Strip extension from filename first
base_name = filename
matched_ext = None
for ext in known_extensions:
if filename.endswith(ext):
base_name = filename[:-len(ext)]
matched_ext = ext
break
if matched_ext is None:
# Unknown extension, return filename as package name
return ParsedUrl(
package_name=filename,
version=None,
filename=filename,
)
# Try to extract version from base_name
# Pattern: name-version or name_version
# Version starts with digit(s) and can include dots, dashes, and alphanumeric suffixes
version_pattern = r"^(.+?)[-_](v?\d+(?:\.\d+)*(?:[-_][a-zA-Z0-9]+)?)$"
match = re.match(version_pattern, base_name)
if match:
name, version = match.groups()
return ParsedUrl(
package_name=name,
version=version,
filename=filename,
)
# No version found, use base_name as package name
return ParsedUrl(
package_name=base_name,
version=None,
filename=filename,
)
def parse_url(url: str, source_type: str) -> ParsedUrl:
"""
Parse URL to extract package name and version based on source type.
Args:
url: The URL to parse.
source_type: The source type (npm, pypi, maven, docker, etc.)
Returns:
ParsedUrl with extracted information.
"""
parsed = None
if source_type == "npm":
parsed = parse_npm_url(url)
elif source_type == "pypi":
parsed = parse_pypi_url(url)
elif source_type == "maven":
parsed = parse_maven_url(url)
elif source_type == "docker":
parsed = parse_docker_url(url)
# Fall back to generic parsing if type-specific parsing fails
if parsed is None:
parsed = parse_generic_url(url)
return parsed
def get_system_project_name(source_type: str) -> str:
"""Get the system project name for a source type."""
return SYSTEM_PROJECT_NAMES.get(source_type, "_generic")
def get_system_project_description(source_type: str) -> str:
"""Get the system project description for a source type."""
return SYSTEM_PROJECT_DESCRIPTIONS.get(
source_type, "System cache for artifacts"
)

View File

@@ -0,0 +1,262 @@
"""
Redis-backed caching service with category-aware TTL and invalidation.
Provides:
- Immutable caching for artifact data (hermetic builds)
- TTL-based caching for discovery data
- Event-driven invalidation for config changes
- Graceful fallback when Redis unavailable
"""
import logging
from enum import Enum
from typing import Optional
from .config import Settings
logger = logging.getLogger(__name__)
class CacheCategory(Enum):
"""
Cache categories with different TTL and invalidation rules.
Immutable (cache forever):
- ARTIFACT_METADATA: Artifact info by SHA256
- ARTIFACT_DEPENDENCIES: Extracted deps by SHA256
- DEPENDENCY_RESOLUTION: Resolution results by input hash
Mutable (TTL + event invalidation):
- UPSTREAM_SOURCES: Upstream config, invalidate on DB change
- PACKAGE_INDEX: PyPI/npm index pages, TTL only
- PACKAGE_VERSIONS: Version listings, TTL only
"""
# Immutable - cache forever (hermetic builds)
ARTIFACT_METADATA = "artifact"
ARTIFACT_DEPENDENCIES = "deps"
DEPENDENCY_RESOLUTION = "resolve"
# Mutable - TTL + event invalidation
UPSTREAM_SOURCES = "upstream"
PACKAGE_INDEX = "index"
PACKAGE_VERSIONS = "versions"
def get_category_ttl(category: CacheCategory, settings: Settings) -> Optional[int]:
"""
Get TTL for a cache category.
Returns:
TTL in seconds, or None for no expiry (immutable).
"""
ttl_map = {
# Immutable - no TTL
CacheCategory.ARTIFACT_METADATA: None,
CacheCategory.ARTIFACT_DEPENDENCIES: None,
CacheCategory.DEPENDENCY_RESOLUTION: None,
# Mutable - configurable TTL
CacheCategory.UPSTREAM_SOURCES: settings.cache_ttl_upstream,
CacheCategory.PACKAGE_INDEX: settings.cache_ttl_index,
CacheCategory.PACKAGE_VERSIONS: settings.cache_ttl_versions,
}
return ttl_map.get(category)
class CacheService:
"""
Redis-backed caching with category-aware TTL.
Key format: orchard:{category}:{protocol}:{identifier}
Example: orchard:deps:pypi:abc123def456
When Redis is disabled or unavailable, operations gracefully
return None/no-op to allow the application to function without caching.
"""
def __init__(self, settings: Settings):
self._settings = settings
self._enabled = settings.redis_enabled
self._redis: Optional["redis.asyncio.Redis"] = None
self._started = False
async def startup(self) -> None:
"""Initialize Redis connection. Called by FastAPI lifespan."""
if self._started:
return
if not self._enabled:
logger.info("CacheService disabled (redis_enabled=False)")
self._started = True
return
try:
import redis.asyncio as redis
logger.info(
f"Connecting to Redis at {self._settings.redis_host}:"
f"{self._settings.redis_port}/{self._settings.redis_db}"
)
self._redis = redis.Redis(
host=self._settings.redis_host,
port=self._settings.redis_port,
db=self._settings.redis_db,
password=self._settings.redis_password,
decode_responses=False, # We handle bytes
)
# Test connection
await self._redis.ping()
logger.info("CacheService connected to Redis")
except ImportError:
logger.warning("redis package not installed, caching disabled")
self._enabled = False
except Exception as e:
logger.warning(f"Redis connection failed, caching disabled: {e}")
self._enabled = False
self._redis = None
self._started = True
async def shutdown(self) -> None:
"""Close Redis connection. Called by FastAPI lifespan."""
if not self._started:
return
if self._redis:
await self._redis.aclose()
self._redis = None
self._started = False
logger.info("CacheService shutdown complete")
@staticmethod
def _make_key(category: CacheCategory, protocol: str, identifier: str) -> str:
"""Build namespaced cache key."""
return f"orchard:{category.value}:{protocol}:{identifier}"
async def get(
self,
category: CacheCategory,
key: str,
protocol: str = "default",
) -> Optional[bytes]:
"""
Get cached value.
Args:
category: Cache category for TTL rules
key: Unique identifier within category
protocol: Protocol namespace (pypi, npm, etc.)
Returns:
Cached bytes or None if not found/disabled.
"""
if not self._enabled or not self._redis:
return None
try:
full_key = self._make_key(category, protocol, key)
return await self._redis.get(full_key)
except Exception as e:
logger.warning(f"Cache get failed for {key}: {e}")
return None
async def set(
self,
category: CacheCategory,
key: str,
value: bytes,
protocol: str = "default",
) -> None:
"""
Set cached value with category-appropriate TTL.
Args:
category: Cache category for TTL rules
key: Unique identifier within category
value: Bytes to cache
protocol: Protocol namespace (pypi, npm, etc.)
"""
if not self._enabled or not self._redis:
return
try:
full_key = self._make_key(category, protocol, key)
ttl = get_category_ttl(category, self._settings)
if ttl is None:
await self._redis.set(full_key, value)
else:
await self._redis.setex(full_key, ttl, value)
except Exception as e:
logger.warning(f"Cache set failed for {key}: {e}")
async def delete(
self,
category: CacheCategory,
key: str,
protocol: str = "default",
) -> None:
"""Delete a specific cache entry."""
if not self._enabled or not self._redis:
return
try:
full_key = self._make_key(category, protocol, key)
await self._redis.delete(full_key)
except Exception as e:
logger.warning(f"Cache delete failed for {key}: {e}")
async def invalidate_pattern(
self,
category: CacheCategory,
pattern: str = "*",
protocol: str = "default",
) -> int:
"""
Invalidate all entries matching pattern.
Args:
category: Cache category
pattern: Glob pattern for keys (default "*" = all in category)
protocol: Protocol namespace
Returns:
Number of keys deleted.
"""
if not self._enabled or not self._redis:
return 0
try:
full_pattern = self._make_key(category, protocol, pattern)
keys = []
async for key in self._redis.scan_iter(match=full_pattern):
keys.append(key)
if keys:
return await self._redis.delete(*keys)
return 0
except Exception as e:
logger.warning(f"Cache invalidate failed for pattern {pattern}: {e}")
return 0
async def ping(self) -> bool:
"""Check if Redis is connected and responding."""
if not self._enabled or not self._redis:
return False
try:
await self._redis.ping()
return True
except Exception:
return False
@property
def enabled(self) -> bool:
"""Check if caching is enabled."""
return self._enabled

View File

@@ -1,5 +1,8 @@
from pydantic_settings import BaseSettings
from functools import lru_cache
from typing import Optional
import os
import re
class Settings(BaseSettings):
@@ -19,8 +22,8 @@ class Settings(BaseSettings):
database_sslmode: str = "disable"
# Database connection pool settings
database_pool_size: int = 5 # Number of connections to keep open
database_max_overflow: int = 10 # Max additional connections beyond pool_size
database_pool_size: int = 20 # Number of connections to keep open
database_max_overflow: int = 30 # Max additional connections beyond pool_size
database_pool_timeout: int = 30 # Seconds to wait for a connection from pool
database_pool_recycle: int = (
1800 # Recycle connections after this many seconds (30 min)
@@ -48,11 +51,48 @@ class Settings(BaseSettings):
presigned_url_expiry: int = (
3600 # Presigned URL expiry in seconds (default: 1 hour)
)
pypi_download_mode: str = "redirect" # "redirect" (to S3) or "proxy" (stream through Orchard)
# HTTP Client pool settings
http_max_connections: int = 100 # Max connections per pool
http_max_keepalive: int = 20 # Keep-alive connections
http_connect_timeout: float = 30.0 # Connection timeout seconds
http_read_timeout: float = 60.0 # Read timeout seconds
http_worker_threads: int = 32 # Thread pool for blocking ops
# Redis cache settings
redis_host: str = "localhost"
redis_port: int = 6379
redis_db: int = 0
redis_password: Optional[str] = None
redis_enabled: bool = True # Set False to disable caching
# Cache TTL settings (seconds, 0 = no expiry)
cache_ttl_index: int = 300 # Package index pages: 5 min
cache_ttl_versions: int = 300 # Version listings: 5 min
cache_ttl_upstream: int = 3600 # Upstream source config: 1 hour
# Logging settings
log_level: str = "INFO" # DEBUG, INFO, WARNING, ERROR, CRITICAL
log_format: str = "auto" # "json", "standard", or "auto" (json in production)
# Initial admin user settings
admin_password: str = "" # Initial admin password (if empty, uses 'changeme123')
# Cache settings
cache_encryption_key: str = "" # Fernet key for encrypting upstream credentials (auto-generated if empty)
# Global cache settings override (None = use DB value, True/False = override DB)
cache_auto_create_system_projects: Optional[bool] = None # Override auto_create_system_projects
# PyPI Cache Worker settings
pypi_cache_workers: int = 5 # Number of concurrent cache workers
pypi_cache_max_depth: int = 10 # Maximum recursion depth for dependency caching
pypi_cache_max_attempts: int = 3 # Maximum retry attempts for failed cache tasks
# Auto-fetch configuration for dependency resolution
auto_fetch_dependencies: bool = False # Server default for auto_fetch parameter
auto_fetch_timeout: int = 300 # Total timeout for auto-fetch resolution in seconds
# JWT Authentication settings (optional, for external identity providers)
jwt_enabled: bool = False # Enable JWT token validation
jwt_secret: str = "" # Secret key for HS256, or leave empty for RS256 with JWKS
@@ -77,6 +117,24 @@ class Settings(BaseSettings):
def is_production(self) -> bool:
return self.env.lower() == "production"
@property
def PORT(self) -> int:
"""Alias for server_port for compatibility."""
return self.server_port
# Uppercase aliases for PyPI cache settings (for backward compatibility)
@property
def PYPI_CACHE_WORKERS(self) -> int:
return self.pypi_cache_workers
@property
def PYPI_CACHE_MAX_DEPTH(self) -> int:
return self.pypi_cache_max_depth
@property
def PYPI_CACHE_MAX_ATTEMPTS(self) -> int:
return self.pypi_cache_max_attempts
class Config:
env_prefix = "ORCHARD_"
case_sensitive = False
@@ -85,3 +143,110 @@ class Settings(BaseSettings):
@lru_cache()
def get_settings() -> Settings:
return Settings()
class EnvUpstreamSource:
"""Represents an upstream source defined via environment variables."""
def __init__(
self,
name: str,
url: str,
source_type: str = "generic",
enabled: bool = True,
auth_type: str = "none",
username: Optional[str] = None,
password: Optional[str] = None,
priority: int = 100,
):
self.name = name
self.url = url
self.source_type = source_type
self.enabled = enabled
self.auth_type = auth_type
self.username = username
self.password = password
self.priority = priority
self.source = "env" # Mark as env-defined
def parse_upstream_sources_from_env() -> list[EnvUpstreamSource]:
"""
Parse upstream sources from environment variables.
Uses double underscore (__) as separator to allow source names with single underscores.
Pattern: ORCHARD_UPSTREAM__{NAME}__FIELD
Example:
ORCHARD_UPSTREAM__NPM_PRIVATE__URL=https://npm.corp.com
ORCHARD_UPSTREAM__NPM_PRIVATE__TYPE=npm
ORCHARD_UPSTREAM__NPM_PRIVATE__ENABLED=true
ORCHARD_UPSTREAM__NPM_PRIVATE__AUTH_TYPE=basic
ORCHARD_UPSTREAM__NPM_PRIVATE__USERNAME=reader
ORCHARD_UPSTREAM__NPM_PRIVATE__PASSWORD=secret
Returns:
List of EnvUpstreamSource objects parsed from environment variables.
"""
# Pattern: ORCHARD_UPSTREAM__{NAME}__{FIELD}
pattern = re.compile(r"^ORCHARD_UPSTREAM__([A-Z0-9_]+)__([A-Z_]+)$", re.IGNORECASE)
# Collect all env vars matching the pattern, grouped by source name
sources_data: dict[str, dict[str, str]] = {}
for key, value in os.environ.items():
match = pattern.match(key)
if match:
source_name = match.group(1).lower() # Normalize to lowercase
field = match.group(2).upper()
if source_name not in sources_data:
sources_data[source_name] = {}
sources_data[source_name][field] = value
# Build source objects from collected data
sources: list[EnvUpstreamSource] = []
for name, data in sources_data.items():
# URL is required
url = data.get("URL")
if not url:
continue # Skip sources without URL
# Parse boolean fields
def parse_bool(val: Optional[str], default: bool) -> bool:
if val is None:
return default
return val.lower() in ("true", "1", "yes", "on")
# Parse integer fields
def parse_int(val: Optional[str], default: int) -> int:
if val is None:
return default
try:
return int(val)
except ValueError:
return default
source = EnvUpstreamSource(
name=name.replace("_", "-"), # Convert underscores to hyphens for readability
url=url,
source_type=data.get("TYPE", "generic").lower(),
enabled=parse_bool(data.get("ENABLED"), True),
auth_type=data.get("AUTH_TYPE", "none").lower(),
username=data.get("USERNAME"),
password=data.get("PASSWORD"),
priority=parse_int(data.get("PRIORITY"), 100),
)
sources.append(source)
return sources
@lru_cache()
def get_env_upstream_sources() -> tuple[EnvUpstreamSource, ...]:
"""
Get cached list of upstream sources from environment variables.
Returns a tuple for hashability (required by lru_cache).
"""
return tuple(parse_upstream_sources_from_env())

View File

@@ -1,17 +1,34 @@
from sqlalchemy import create_engine, text, event
from sqlalchemy.orm import sessionmaker, Session
from sqlalchemy.pool import QueuePool
from typing import Generator
from typing import Generator, NamedTuple
from contextlib import contextmanager
import logging
import time
import hashlib
from .config import get_settings
from .models import Base
from .purge_seed_data import should_purge_seed_data, purge_seed_data
settings = get_settings()
logger = logging.getLogger(__name__)
class Migration(NamedTuple):
"""A database migration with a unique name and SQL to execute."""
name: str
sql: str
# PostgreSQL error codes that indicate "already exists" - safe to skip
SAFE_PG_ERROR_CODES = {
"42P07", # duplicate_table
"42701", # duplicate_column
"42710", # duplicate_object (index, constraint, etc.)
"42P16", # invalid_table_definition (e.g., column already exists)
}
# Build connect_args with query timeout if configured
connect_args = {}
if settings.database_query_timeout > 0:
@@ -64,102 +81,562 @@ def init_db():
# Run migrations for schema updates
_run_migrations()
# Purge seed data if requested (for transitioning to production-like environment)
if should_purge_seed_data():
db = SessionLocal()
try:
purge_seed_data(db)
finally:
db.close()
def _ensure_migrations_table(conn) -> None:
"""Create the migrations tracking table if it doesn't exist."""
conn.execute(text("""
CREATE TABLE IF NOT EXISTS _schema_migrations (
name VARCHAR(255) PRIMARY KEY,
checksum VARCHAR(64) NOT NULL,
applied_at TIMESTAMP WITH TIME ZONE DEFAULT NOW()
);
"""))
conn.commit()
def _get_applied_migrations(conn) -> dict[str, str]:
"""Get all applied migrations and their checksums."""
result = conn.execute(text(
"SELECT name, checksum FROM _schema_migrations"
))
return {row[0]: row[1] for row in result}
def _compute_checksum(sql: str) -> str:
"""Compute a checksum for migration SQL to detect changes."""
return hashlib.sha256(sql.strip().encode()).hexdigest()[:16]
def _is_safe_error(exception: Exception) -> bool:
"""Check if the error indicates the migration was already applied."""
# Check for psycopg2 errors with pgcode attribute
original = getattr(exception, "orig", None)
if original is not None:
pgcode = getattr(original, "pgcode", None)
if pgcode in SAFE_PG_ERROR_CODES:
return True
# Fallback: check error message for common "already exists" patterns
error_str = str(exception).lower()
safe_patterns = [
"already exists",
"duplicate key",
"relation .* already exists",
"column .* already exists",
]
return any(pattern in error_str for pattern in safe_patterns)
def _record_migration(conn, name: str, checksum: str) -> None:
"""Record a migration as applied."""
conn.execute(text(
"INSERT INTO _schema_migrations (name, checksum) VALUES (:name, :checksum)"
), {"name": name, "checksum": checksum})
conn.commit()
def _run_migrations():
"""Run manual migrations for schema updates"""
"""Run manual migrations for schema updates with tracking and error detection."""
migrations = [
# Add format_metadata column to artifacts table
"""
DO $$
BEGIN
IF NOT EXISTS (
SELECT 1 FROM information_schema.columns
WHERE table_name = 'artifacts' AND column_name = 'format_metadata'
) THEN
ALTER TABLE artifacts ADD COLUMN format_metadata JSONB DEFAULT '{}';
END IF;
END $$;
""",
# Add format column to packages table
"""
DO $$
BEGIN
IF NOT EXISTS (
SELECT 1 FROM information_schema.columns
WHERE table_name = 'packages' AND column_name = 'format'
) THEN
ALTER TABLE packages ADD COLUMN format VARCHAR(50) DEFAULT 'generic' NOT NULL;
CREATE INDEX IF NOT EXISTS idx_packages_format ON packages(format);
END IF;
END $$;
""",
# Add platform column to packages table
"""
DO $$
BEGIN
IF NOT EXISTS (
SELECT 1 FROM information_schema.columns
WHERE table_name = 'packages' AND column_name = 'platform'
) THEN
ALTER TABLE packages ADD COLUMN platform VARCHAR(50) DEFAULT 'any' NOT NULL;
CREATE INDEX IF NOT EXISTS idx_packages_platform ON packages(platform);
END IF;
END $$;
""",
# Add ref_count index and constraints for artifacts
"""
DO $$
BEGIN
-- Add ref_count index
IF NOT EXISTS (
SELECT 1 FROM pg_indexes WHERE indexname = 'idx_artifacts_ref_count'
) THEN
CREATE INDEX idx_artifacts_ref_count ON artifacts(ref_count);
END IF;
Migration(
name="001_add_format_metadata",
sql="""
DO $$
BEGIN
IF NOT EXISTS (
SELECT 1 FROM information_schema.columns
WHERE table_name = 'artifacts' AND column_name = 'format_metadata'
) THEN
ALTER TABLE artifacts ADD COLUMN format_metadata JSONB DEFAULT '{}';
END IF;
END $$;
""",
),
Migration(
name="002_add_package_format",
sql="""
DO $$
BEGIN
IF NOT EXISTS (
SELECT 1 FROM information_schema.columns
WHERE table_name = 'packages' AND column_name = 'format'
) THEN
ALTER TABLE packages ADD COLUMN format VARCHAR(50) DEFAULT 'generic' NOT NULL;
CREATE INDEX IF NOT EXISTS idx_packages_format ON packages(format);
END IF;
END $$;
""",
),
Migration(
name="003_add_package_platform",
sql="""
DO $$
BEGIN
IF NOT EXISTS (
SELECT 1 FROM information_schema.columns
WHERE table_name = 'packages' AND column_name = 'platform'
) THEN
ALTER TABLE packages ADD COLUMN platform VARCHAR(50) DEFAULT 'any' NOT NULL;
CREATE INDEX IF NOT EXISTS idx_packages_platform ON packages(platform);
END IF;
END $$;
""",
),
Migration(
name="004_add_ref_count_index_constraint",
sql="""
DO $$
BEGIN
IF NOT EXISTS (
SELECT 1 FROM pg_indexes WHERE indexname = 'idx_artifacts_ref_count'
) THEN
CREATE INDEX idx_artifacts_ref_count ON artifacts(ref_count);
END IF;
-- Add ref_count >= 0 constraint
IF NOT EXISTS (
SELECT 1 FROM pg_constraint WHERE conname = 'check_ref_count_non_negative'
) THEN
ALTER TABLE artifacts ADD CONSTRAINT check_ref_count_non_negative CHECK (ref_count >= 0);
END IF;
END $$;
""",
# Add composite indexes for packages and tags
"""
DO $$
BEGIN
-- Composite index for package lookup by project and name
IF NOT EXISTS (
SELECT 1 FROM pg_indexes WHERE indexname = 'idx_packages_project_name'
) THEN
CREATE UNIQUE INDEX idx_packages_project_name ON packages(project_id, name);
END IF;
IF NOT EXISTS (
SELECT 1 FROM pg_constraint WHERE conname = 'check_ref_count_non_negative'
) THEN
ALTER TABLE artifacts ADD CONSTRAINT check_ref_count_non_negative CHECK (ref_count >= 0);
END IF;
END $$;
""",
),
Migration(
name="005_add_composite_indexes",
sql="""
DO $$
BEGIN
IF NOT EXISTS (
SELECT 1 FROM pg_indexes WHERE indexname = 'idx_packages_project_name'
) THEN
CREATE UNIQUE INDEX idx_packages_project_name ON packages(project_id, name);
END IF;
-- Composite index for tag lookup by package and name
IF NOT EXISTS (
SELECT 1 FROM pg_indexes WHERE indexname = 'idx_tags_package_name'
) THEN
CREATE UNIQUE INDEX idx_tags_package_name ON tags(package_id, name);
END IF;
-- Tag indexes removed: tags table no longer exists (removed in tag system removal)
END $$;
""",
),
Migration(
name="006_add_package_versions_indexes",
sql="""
DO $$
BEGIN
IF EXISTS (SELECT 1 FROM information_schema.tables WHERE table_name = 'package_versions') THEN
IF NOT EXISTS (SELECT 1 FROM pg_indexes WHERE indexname = 'idx_package_versions_package_id') THEN
CREATE INDEX idx_package_versions_package_id ON package_versions(package_id);
END IF;
IF NOT EXISTS (SELECT 1 FROM pg_indexes WHERE indexname = 'idx_package_versions_artifact_id') THEN
CREATE INDEX idx_package_versions_artifact_id ON package_versions(artifact_id);
END IF;
IF NOT EXISTS (SELECT 1 FROM pg_indexes WHERE indexname = 'idx_package_versions_package_version') THEN
CREATE INDEX idx_package_versions_package_version ON package_versions(package_id, version);
END IF;
END IF;
END $$;
""",
),
Migration(
name="007_create_ref_count_trigger_functions",
sql="""
CREATE OR REPLACE FUNCTION increment_artifact_ref_count()
RETURNS TRIGGER AS $$
BEGIN
UPDATE artifacts SET ref_count = ref_count + 1 WHERE id = NEW.artifact_id;
RETURN NEW;
END;
$$ LANGUAGE plpgsql;
-- Composite index for recent tags queries
IF NOT EXISTS (
SELECT 1 FROM pg_indexes WHERE indexname = 'idx_tags_package_created_at'
) THEN
CREATE INDEX idx_tags_package_created_at ON tags(package_id, created_at);
END IF;
END $$;
""",
CREATE OR REPLACE FUNCTION decrement_artifact_ref_count()
RETURNS TRIGGER AS $$
BEGIN
UPDATE artifacts SET ref_count = ref_count - 1 WHERE id = OLD.artifact_id;
RETURN OLD;
END;
$$ LANGUAGE plpgsql;
CREATE OR REPLACE FUNCTION update_artifact_ref_count()
RETURNS TRIGGER AS $$
BEGIN
IF OLD.artifact_id != NEW.artifact_id THEN
UPDATE artifacts SET ref_count = ref_count - 1 WHERE id = OLD.artifact_id;
UPDATE artifacts SET ref_count = ref_count + 1 WHERE id = NEW.artifact_id;
END IF;
RETURN NEW;
END;
$$ LANGUAGE plpgsql;
""",
),
Migration(
name="008_create_tags_ref_count_triggers",
sql="""
-- Tags table removed: triggers no longer needed (tag system removed)
DO $$ BEGIN NULL; END $$;
""",
),
Migration(
name="009_create_version_ref_count_functions",
sql="""
CREATE OR REPLACE FUNCTION increment_version_ref_count()
RETURNS TRIGGER AS $$
BEGIN
UPDATE artifacts SET ref_count = ref_count + 1 WHERE id = NEW.artifact_id;
RETURN NEW;
END;
$$ LANGUAGE plpgsql;
CREATE OR REPLACE FUNCTION decrement_version_ref_count()
RETURNS TRIGGER AS $$
BEGIN
UPDATE artifacts SET ref_count = ref_count - 1 WHERE id = OLD.artifact_id;
RETURN OLD;
END;
$$ LANGUAGE plpgsql;
""",
),
Migration(
name="010_create_package_versions_triggers",
sql="""
DO $$
BEGIN
IF EXISTS (SELECT 1 FROM information_schema.tables WHERE table_name = 'package_versions') THEN
DROP TRIGGER IF EXISTS package_versions_ref_count_insert ON package_versions;
CREATE TRIGGER package_versions_ref_count_insert
AFTER INSERT ON package_versions
FOR EACH ROW
EXECUTE FUNCTION increment_version_ref_count();
DROP TRIGGER IF EXISTS package_versions_ref_count_delete ON package_versions;
CREATE TRIGGER package_versions_ref_count_delete
AFTER DELETE ON package_versions
FOR EACH ROW
EXECUTE FUNCTION decrement_version_ref_count();
END IF;
END $$;
""",
),
Migration(
name="011_migrate_semver_tags_to_versions",
sql=r"""
-- Migrate semver tags to versions (only if both tables exist - for existing databases)
DO $$
BEGIN
IF EXISTS (SELECT 1 FROM information_schema.tables WHERE table_name = 'package_versions')
AND EXISTS (SELECT 1 FROM information_schema.tables WHERE table_name = 'tags') THEN
INSERT INTO package_versions (id, package_id, artifact_id, version, version_source, created_by, created_at)
SELECT
gen_random_uuid(),
t.package_id,
t.artifact_id,
CASE WHEN t.name LIKE 'v%' THEN substring(t.name from 2) ELSE t.name END,
'migrated_from_tag',
t.created_by,
t.created_at
FROM tags t
WHERE t.name ~ '^v?[0-9]+\.[0-9]+(\.[0-9]+)?([-.][a-zA-Z0-9]+)?$'
ON CONFLICT (package_id, version) DO NOTHING;
END IF;
END $$;
""",
),
Migration(
name="012_create_teams_table",
sql="""
CREATE TABLE IF NOT EXISTS teams (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
name VARCHAR(255) NOT NULL,
slug VARCHAR(255) NOT NULL UNIQUE,
description TEXT,
created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
updated_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
created_by VARCHAR(255) NOT NULL,
settings JSONB DEFAULT '{}'
);
""",
),
Migration(
name="013_create_team_memberships_table",
sql="""
CREATE TABLE IF NOT EXISTS team_memberships (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
team_id UUID NOT NULL REFERENCES teams(id) ON DELETE CASCADE,
user_id UUID NOT NULL REFERENCES users(id) ON DELETE CASCADE,
role VARCHAR(50) NOT NULL DEFAULT 'member',
created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
invited_by VARCHAR(255),
CONSTRAINT team_memberships_unique UNIQUE (team_id, user_id),
CONSTRAINT team_memberships_role_check CHECK (role IN ('owner', 'admin', 'member'))
);
""",
),
Migration(
name="014_add_team_id_to_projects",
sql="""
DO $$
BEGIN
IF NOT EXISTS (
SELECT 1 FROM information_schema.columns
WHERE table_name = 'projects' AND column_name = 'team_id'
) THEN
ALTER TABLE projects ADD COLUMN team_id UUID REFERENCES teams(id) ON DELETE SET NULL;
CREATE INDEX IF NOT EXISTS idx_projects_team_id ON projects(team_id);
END IF;
END $$;
""",
),
Migration(
name="015_add_teams_indexes",
sql="""
DO $$
BEGIN
IF NOT EXISTS (SELECT 1 FROM pg_indexes WHERE indexname = 'idx_teams_slug') THEN
CREATE INDEX idx_teams_slug ON teams(slug);
END IF;
IF NOT EXISTS (SELECT 1 FROM pg_indexes WHERE indexname = 'idx_teams_created_by') THEN
CREATE INDEX idx_teams_created_by ON teams(created_by);
END IF;
IF NOT EXISTS (SELECT 1 FROM pg_indexes WHERE indexname = 'idx_team_memberships_team_id') THEN
CREATE INDEX idx_team_memberships_team_id ON team_memberships(team_id);
END IF;
IF NOT EXISTS (SELECT 1 FROM pg_indexes WHERE indexname = 'idx_team_memberships_user_id') THEN
CREATE INDEX idx_team_memberships_user_id ON team_memberships(user_id);
END IF;
END $$;
""",
),
Migration(
name="016_add_is_system_to_projects",
sql="""
DO $$
BEGIN
IF NOT EXISTS (
SELECT 1 FROM information_schema.columns
WHERE table_name = 'projects' AND column_name = 'is_system'
) THEN
ALTER TABLE projects ADD COLUMN is_system BOOLEAN NOT NULL DEFAULT FALSE;
CREATE INDEX IF NOT EXISTS idx_projects_is_system ON projects(is_system);
END IF;
END $$;
""",
),
Migration(
name="017_create_upstream_sources",
sql="""
CREATE TABLE IF NOT EXISTS upstream_sources (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
name VARCHAR(255) NOT NULL UNIQUE,
source_type VARCHAR(50) NOT NULL DEFAULT 'generic',
url VARCHAR(2048) NOT NULL,
enabled BOOLEAN NOT NULL DEFAULT FALSE,
auth_type VARCHAR(20) NOT NULL DEFAULT 'none',
username VARCHAR(255),
password_encrypted BYTEA,
headers_encrypted BYTEA,
priority INTEGER NOT NULL DEFAULT 100,
created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
updated_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
CONSTRAINT check_source_type CHECK (
source_type IN ('npm', 'pypi', 'maven', 'docker', 'helm', 'nuget', 'deb', 'rpm', 'generic')
),
CONSTRAINT check_auth_type CHECK (
auth_type IN ('none', 'basic', 'bearer', 'api_key')
),
CONSTRAINT check_priority_positive CHECK (priority > 0)
);
CREATE INDEX IF NOT EXISTS idx_upstream_sources_enabled ON upstream_sources(enabled);
CREATE INDEX IF NOT EXISTS idx_upstream_sources_source_type ON upstream_sources(source_type);
CREATE INDEX IF NOT EXISTS idx_upstream_sources_priority ON upstream_sources(priority);
""",
),
Migration(
name="018_create_cache_settings",
sql="""
CREATE TABLE IF NOT EXISTS cache_settings (
id INTEGER PRIMARY KEY DEFAULT 1,
auto_create_system_projects BOOLEAN NOT NULL DEFAULT TRUE,
created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
updated_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
CONSTRAINT check_cache_settings_singleton CHECK (id = 1)
);
INSERT INTO cache_settings (id, auto_create_system_projects)
VALUES (1, TRUE)
ON CONFLICT (id) DO NOTHING;
""",
),
Migration(
name="019_create_cached_urls",
sql="""
CREATE TABLE IF NOT EXISTS cached_urls (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
url VARCHAR(4096) NOT NULL,
url_hash VARCHAR(64) NOT NULL UNIQUE,
artifact_id VARCHAR(64) NOT NULL REFERENCES artifacts(id),
source_id UUID REFERENCES upstream_sources(id) ON DELETE SET NULL,
fetched_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT NOW(),
response_headers JSONB DEFAULT '{}',
created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW()
);
CREATE INDEX IF NOT EXISTS idx_cached_urls_url_hash ON cached_urls(url_hash);
CREATE INDEX IF NOT EXISTS idx_cached_urls_artifact_id ON cached_urls(artifact_id);
CREATE INDEX IF NOT EXISTS idx_cached_urls_source_id ON cached_urls(source_id);
CREATE INDEX IF NOT EXISTS idx_cached_urls_fetched_at ON cached_urls(fetched_at);
""",
),
Migration(
name="020_seed_default_upstream_sources",
sql="""
-- Originally seeded public sources, but these are no longer used.
-- Migration 023 deletes any previously seeded sources.
-- This migration is now a no-op for fresh installs.
SELECT 1;
""",
),
Migration(
name="021_remove_is_public_from_upstream_sources",
sql="""
DO $$
BEGIN
-- Drop the index if it exists
DROP INDEX IF EXISTS idx_upstream_sources_is_public;
-- Drop the column if it exists
IF EXISTS (
SELECT 1 FROM information_schema.columns
WHERE table_name = 'upstream_sources' AND column_name = 'is_public'
) THEN
ALTER TABLE upstream_sources DROP COLUMN is_public;
END IF;
END $$;
""",
),
Migration(
name="022_remove_allow_public_internet_from_cache_settings",
sql="""
DO $$
BEGIN
IF EXISTS (
SELECT 1 FROM information_schema.columns
WHERE table_name = 'cache_settings' AND column_name = 'allow_public_internet'
) THEN
ALTER TABLE cache_settings DROP COLUMN allow_public_internet;
END IF;
END $$;
""",
),
Migration(
name="023_delete_seeded_public_sources",
sql="""
-- Delete the seeded public sources that were added by migration 020
DELETE FROM upstream_sources
WHERE name IN ('npm-public', 'pypi-public', 'maven-central', 'docker-hub');
""",
),
Migration(
name="024_remove_tags",
sql="""
-- Remove tag system, keeping only versions for artifact references
DO $$
BEGIN
-- Drop triggers on tags table (if they exist)
DROP TRIGGER IF EXISTS tags_ref_count_insert_trigger ON tags;
DROP TRIGGER IF EXISTS tags_ref_count_delete_trigger ON tags;
DROP TRIGGER IF EXISTS tags_ref_count_update_trigger ON tags;
DROP TRIGGER IF EXISTS tags_updated_at_trigger ON tags;
DROP TRIGGER IF EXISTS tag_changes_trigger ON tags;
-- Drop the tag change tracking function
DROP FUNCTION IF EXISTS track_tag_changes();
-- Remove tag_constraint from artifact_dependencies
IF EXISTS (
SELECT 1 FROM information_schema.table_constraints
WHERE constraint_name = 'check_constraint_type'
AND table_name = 'artifact_dependencies'
) THEN
ALTER TABLE artifact_dependencies DROP CONSTRAINT check_constraint_type;
END IF;
-- Remove the tag_constraint column if it exists
IF EXISTS (
SELECT 1 FROM information_schema.columns
WHERE table_name = 'artifact_dependencies' AND column_name = 'tag_constraint'
) THEN
ALTER TABLE artifact_dependencies DROP COLUMN tag_constraint;
END IF;
-- Make version_constraint NOT NULL
UPDATE artifact_dependencies SET version_constraint = '*' WHERE version_constraint IS NULL;
ALTER TABLE artifact_dependencies ALTER COLUMN version_constraint SET NOT NULL;
-- Drop tag_history table first (depends on tags)
DROP TABLE IF EXISTS tag_history;
-- Drop tags table
DROP TABLE IF EXISTS tags;
-- Rename uploads.tag_name to version if it exists and version doesn't
IF EXISTS (
SELECT 1 FROM information_schema.columns
WHERE table_name = 'uploads' AND column_name = 'tag_name'
) AND NOT EXISTS (
SELECT 1 FROM information_schema.columns
WHERE table_name = 'uploads' AND column_name = 'version'
) THEN
ALTER TABLE uploads RENAME COLUMN tag_name TO version;
END IF;
END $$;
""",
),
]
with engine.connect() as conn:
# Ensure migrations tracking table exists
_ensure_migrations_table(conn)
# Get already-applied migrations
applied = _get_applied_migrations(conn)
for migration in migrations:
checksum = _compute_checksum(migration.sql)
# Check if migration was already applied
if migration.name in applied:
stored_checksum = applied[migration.name]
if stored_checksum != checksum:
logger.warning(
f"Migration '{migration.name}' has changed since it was applied! "
f"Stored checksum: {stored_checksum}, current: {checksum}"
)
continue
# Run the migration
try:
conn.execute(text(migration))
logger.info(f"Running migration: {migration.name}")
conn.execute(text(migration.sql))
conn.commit()
_record_migration(conn, migration.name, checksum)
logger.info(f"Migration '{migration.name}' applied successfully")
except Exception as e:
logger.warning(f"Migration failed (may already be applied): {e}")
conn.rollback()
if _is_safe_error(e):
# Migration was already applied (schema already exists)
logger.info(
f"Migration '{migration.name}' already applied (schema exists), recording as complete"
)
_record_migration(conn, migration.name, checksum)
else:
# Real error - fail hard
logger.error(f"Migration '{migration.name}' failed: {e}")
raise RuntimeError(
f"Migration '{migration.name}' failed with error: {e}"
) from e
def get_db() -> Generator[Session, None, None]:

175
backend/app/db_utils.py Normal file
View File

@@ -0,0 +1,175 @@
"""
Database utilities for optimized artifact operations.
Provides batch operations to eliminate N+1 queries.
"""
import logging
from typing import Optional
from sqlalchemy.dialects.postgresql import insert as pg_insert
from sqlalchemy.orm import Session
from .models import Artifact, ArtifactDependency, CachedUrl
logger = logging.getLogger(__name__)
class ArtifactRepository:
"""
Optimized database operations for artifact storage.
Key optimizations:
- Atomic upserts using ON CONFLICT
- Batch inserts for dependencies
- Joined queries to avoid N+1
"""
def __init__(self, db: Session):
self.db = db
@staticmethod
def _format_dependency_values(
artifact_id: str,
dependencies: list[tuple[str, str, str]],
) -> list[dict]:
"""
Format dependencies for batch insert.
Args:
artifact_id: SHA256 of the artifact
dependencies: List of (project, package, version_constraint)
Returns:
List of dicts ready for bulk insert.
"""
return [
{
"artifact_id": artifact_id,
"dependency_project": proj,
"dependency_package": pkg,
"version_constraint": ver,
}
for proj, pkg, ver in dependencies
]
def get_or_create_artifact(
self,
sha256: str,
size: int,
filename: str,
content_type: Optional[str] = None,
created_by: str = "system",
s3_key: Optional[str] = None,
) -> tuple[Artifact, bool]:
"""
Get existing artifact or create new one atomically.
Uses INSERT ... ON CONFLICT DO UPDATE to handle races.
If artifact exists, increments ref_count.
Args:
sha256: Content hash (primary key)
size: File size in bytes
filename: Original filename
content_type: MIME type
created_by: User who created the artifact
s3_key: S3 storage key (defaults to standard path)
Returns:
(artifact, created) tuple where created is True for new artifacts.
"""
if s3_key is None:
s3_key = f"fruits/{sha256[:2]}/{sha256[2:4]}/{sha256}"
stmt = pg_insert(Artifact).values(
id=sha256,
size=size,
original_name=filename,
content_type=content_type,
ref_count=1,
created_by=created_by,
s3_key=s3_key,
).on_conflict_do_update(
index_elements=['id'],
set_={'ref_count': Artifact.ref_count + 1}
).returning(Artifact)
result = self.db.execute(stmt)
artifact = result.scalar_one()
# Check if this was an insert or update by comparing ref_count
# ref_count=1 means new, >1 means existing
created = artifact.ref_count == 1
return artifact, created
def batch_upsert_dependencies(
self,
artifact_id: str,
dependencies: list[tuple[str, str, str]],
) -> int:
"""
Insert dependencies in a single batch operation.
Uses ON CONFLICT DO NOTHING to skip duplicates.
Args:
artifact_id: SHA256 of the artifact
dependencies: List of (project, package, version_constraint)
Returns:
Number of dependencies inserted.
"""
if not dependencies:
return 0
values = self._format_dependency_values(artifact_id, dependencies)
stmt = pg_insert(ArtifactDependency).values(values)
stmt = stmt.on_conflict_do_nothing(
index_elements=['artifact_id', 'dependency_project', 'dependency_package']
)
result = self.db.execute(stmt)
return result.rowcount
def get_cached_url_with_artifact(
self,
url_hash: str,
) -> Optional[tuple[CachedUrl, Artifact]]:
"""
Get cached URL and its artifact in a single query.
Args:
url_hash: SHA256 of the URL
Returns:
(CachedUrl, Artifact) tuple or None if not found.
"""
result = (
self.db.query(CachedUrl, Artifact)
.join(Artifact, CachedUrl.artifact_id == Artifact.id)
.filter(CachedUrl.url_hash == url_hash)
.first()
)
return result
def get_artifact_dependencies(
self,
artifact_id: str,
) -> list[ArtifactDependency]:
"""
Get all dependencies for an artifact in a single query.
Args:
artifact_id: SHA256 of the artifact
Returns:
List of ArtifactDependency objects.
"""
return (
self.db.query(ArtifactDependency)
.filter(ArtifactDependency.artifact_id == artifact_id)
.all()
)

1312
backend/app/dependencies.py Normal file

File diff suppressed because it is too large Load Diff

160
backend/app/encryption.py Normal file
View File

@@ -0,0 +1,160 @@
"""
Encryption utilities for sensitive data storage.
Uses Fernet symmetric encryption for credentials like upstream passwords.
The encryption key is sourced from ORCHARD_CACHE_ENCRYPTION_KEY environment variable.
If not set, a random key is generated on startup (with a warning).
"""
import base64
import logging
import os
import secrets
from functools import lru_cache
from typing import Optional
from cryptography.fernet import Fernet, InvalidToken
logger = logging.getLogger(__name__)
# Module-level storage for auto-generated key (only used if env var not set)
_generated_key: Optional[bytes] = None
def _get_key_from_env() -> Optional[bytes]:
"""Get encryption key from environment variable."""
key_str = os.environ.get("ORCHARD_CACHE_ENCRYPTION_KEY", "")
if not key_str:
return None
# Support both raw base64 and url-safe base64 formats
try:
# Try to decode as-is (Fernet keys are url-safe base64)
key_bytes = key_str.encode("utf-8")
# Validate it's a valid Fernet key by trying to create a Fernet instance
Fernet(key_bytes)
return key_bytes
except Exception:
pass
# Try base64 decoding if it's a raw 32-byte key encoded as base64
try:
decoded = base64.urlsafe_b64decode(key_str)
if len(decoded) == 32:
# Re-encode as url-safe base64 for Fernet
key_bytes = base64.urlsafe_b64encode(decoded)
Fernet(key_bytes)
return key_bytes
except Exception:
pass
logger.error(
"ORCHARD_CACHE_ENCRYPTION_KEY is set but invalid. "
"Must be a valid Fernet key (32 bytes, url-safe base64 encoded). "
"Generate one with: python -c \"from cryptography.fernet import Fernet; print(Fernet.generate_key().decode())\""
)
return None
def get_encryption_key() -> bytes:
"""
Get the Fernet encryption key.
Returns the key from ORCHARD_CACHE_ENCRYPTION_KEY if set and valid,
otherwise generates a random key (with a warning logged).
The generated key is cached for the lifetime of the process.
"""
global _generated_key
# Try to get from environment
env_key = _get_key_from_env()
if env_key:
return env_key
# Generate a new key if needed
if _generated_key is None:
_generated_key = Fernet.generate_key()
logger.warning(
"ORCHARD_CACHE_ENCRYPTION_KEY not set - using auto-generated key. "
"Encrypted credentials will be lost on restart! "
"Set ORCHARD_CACHE_ENCRYPTION_KEY for persistent encryption. "
"Generate a key with: python -c \"from cryptography.fernet import Fernet; print(Fernet.generate_key().decode())\""
)
return _generated_key
@lru_cache(maxsize=1)
def _get_fernet() -> Fernet:
"""Get a cached Fernet instance."""
return Fernet(get_encryption_key())
def encrypt_value(plaintext: str) -> bytes:
"""
Encrypt a string value using Fernet.
Args:
plaintext: The string to encrypt
Returns:
Encrypted bytes (includes Fernet token with timestamp)
"""
if not plaintext:
raise ValueError("Cannot encrypt empty value")
fernet = _get_fernet()
return fernet.encrypt(plaintext.encode("utf-8"))
def decrypt_value(ciphertext: bytes) -> str:
"""
Decrypt a Fernet-encrypted value.
Args:
ciphertext: The encrypted bytes
Returns:
Decrypted string
Raises:
InvalidToken: If decryption fails (wrong key or corrupted data)
"""
if not ciphertext:
raise ValueError("Cannot decrypt empty value")
fernet = _get_fernet()
return fernet.decrypt(ciphertext).decode("utf-8")
def can_decrypt(ciphertext: bytes) -> bool:
"""
Check if a value can be decrypted with the current key.
Useful for checking if credentials are still valid after key rotation.
Args:
ciphertext: The encrypted bytes
Returns:
True if decryption succeeds, False otherwise
"""
if not ciphertext:
return False
try:
decrypt_value(ciphertext)
return True
except (InvalidToken, ValueError):
return False
def generate_key() -> str:
"""
Generate a new Fernet encryption key.
Returns:
A valid Fernet key as a string (url-safe base64 encoded)
"""
return Fernet.generate_key().decode("utf-8")

179
backend/app/http_client.py Normal file
View File

@@ -0,0 +1,179 @@
"""
HTTP client manager with connection pooling and lifecycle management.
Provides:
- Shared connection pools for upstream requests
- Per-upstream client isolation when needed
- Thread pool for blocking I/O operations
- FastAPI lifespan integration
"""
import asyncio
import logging
from concurrent.futures import ThreadPoolExecutor
from typing import Any, Callable, Optional
import httpx
from .config import Settings
logger = logging.getLogger(__name__)
class HttpClientManager:
"""
Manages httpx.AsyncClient pools with FastAPI lifespan integration.
Features:
- Default shared pool for general requests
- Per-upstream pools for sources needing specific config/auth
- Dedicated thread pool for blocking operations
- Graceful shutdown
"""
def __init__(self, settings: Settings):
self.max_connections = settings.http_max_connections
self.max_keepalive = settings.http_max_keepalive
self.connect_timeout = settings.http_connect_timeout
self.read_timeout = settings.http_read_timeout
self.worker_threads = settings.http_worker_threads
self._default_client: Optional[httpx.AsyncClient] = None
self._upstream_clients: dict[str, httpx.AsyncClient] = {}
self._executor: Optional[ThreadPoolExecutor] = None
self._started = False
async def startup(self) -> None:
"""Initialize clients and thread pool. Called by FastAPI lifespan."""
if self._started:
return
logger.info(
f"Starting HttpClientManager: max_connections={self.max_connections}, "
f"worker_threads={self.worker_threads}"
)
# Create connection limits
limits = httpx.Limits(
max_connections=self.max_connections,
max_keepalive_connections=self.max_keepalive,
)
# Create timeout config
timeout = httpx.Timeout(
connect=self.connect_timeout,
read=self.read_timeout,
write=self.read_timeout,
pool=self.connect_timeout,
)
# Create default client
self._default_client = httpx.AsyncClient(
limits=limits,
timeout=timeout,
follow_redirects=False, # Handle redirects manually for auth
)
# Create thread pool for blocking operations
self._executor = ThreadPoolExecutor(
max_workers=self.worker_threads,
thread_name_prefix="orchard-blocking-",
)
self._started = True
logger.info("HttpClientManager started")
async def shutdown(self) -> None:
"""Close all clients and thread pool. Called by FastAPI lifespan."""
if not self._started:
return
logger.info("Shutting down HttpClientManager")
# Close default client
if self._default_client:
await self._default_client.aclose()
self._default_client = None
# Close upstream-specific clients
for name, client in self._upstream_clients.items():
logger.debug(f"Closing upstream client: {name}")
await client.aclose()
self._upstream_clients.clear()
# Shutdown thread pool
if self._executor:
self._executor.shutdown(wait=True)
self._executor = None
self._started = False
logger.info("HttpClientManager shutdown complete")
def get_client(self, upstream_name: Optional[str] = None) -> httpx.AsyncClient:
"""
Get HTTP client for making requests.
Args:
upstream_name: Optional upstream source name for dedicated pool.
If None, returns the default shared client.
Returns:
httpx.AsyncClient configured for the request.
Raises:
RuntimeError: If manager not started.
"""
if not self._started or not self._default_client:
raise RuntimeError("HttpClientManager not started. Call startup() first.")
if upstream_name and upstream_name in self._upstream_clients:
return self._upstream_clients[upstream_name]
return self._default_client
async def run_blocking(self, func: Callable[..., Any], *args: Any) -> Any:
"""
Run a blocking function in the thread pool.
Use this for:
- File I/O operations
- Archive extraction (zipfile, tarfile)
- Hash computation on large data
Args:
func: Synchronous function to execute
*args: Arguments to pass to the function
Returns:
The function's return value.
"""
if not self._executor:
raise RuntimeError("HttpClientManager not started. Call startup() first.")
loop = asyncio.get_running_loop()
return await loop.run_in_executor(self._executor, func, *args)
@property
def active_connections(self) -> int:
"""Get approximate number of active connections (for health checks)."""
if not self._default_client:
return 0
# httpx doesn't expose this directly, return pool size as approximation
return self.max_connections
@property
def pool_size(self) -> int:
"""Get configured pool size."""
return self.max_connections
@property
def executor_active(self) -> int:
"""Get number of active thread pool workers."""
if not self._executor:
return 0
return len(self._executor._threads)
@property
def executor_max(self) -> int:
"""Get max thread pool workers."""
return self.worker_threads

View File

@@ -11,9 +11,12 @@ from slowapi.errors import RateLimitExceeded
from .config import get_settings
from .database import init_db, SessionLocal
from .routes import router
from .pypi_proxy import router as pypi_router
from .seed import seed_database
from .auth import create_default_admin
from .rate_limit import limiter
from .http_client import HttpClientManager
from .cache_service import CacheService
settings = get_settings()
logging.basicConfig(level=logging.INFO)
@@ -37,6 +40,17 @@ async def lifespan(app: FastAPI):
finally:
db.close()
# Initialize infrastructure services
logger.info("Initializing infrastructure services...")
app.state.http_client = HttpClientManager(settings)
await app.state.http_client.startup()
app.state.cache = CacheService(settings)
await app.state.cache.startup()
logger.info("Infrastructure services ready")
# Seed test data in development mode
if settings.is_development:
logger.info(f"Running in {settings.env} mode - checking for seed data")
@@ -49,7 +63,12 @@ async def lifespan(app: FastAPI):
logger.info(f"Running in {settings.env} mode - skipping seed data")
yield
# Shutdown: cleanup if needed
# Shutdown infrastructure services
logger.info("Shutting down infrastructure services...")
await app.state.http_client.shutdown()
await app.state.cache.shutdown()
logger.info("Shutdown complete")
app = FastAPI(
@@ -65,6 +84,7 @@ app.add_exception_handler(RateLimitExceeded, _rate_limit_exceeded_handler)
# Include API routes
app.include_router(router)
app.include_router(pypi_router)
# Serve static files (React build) if the directory exists
static_dir = os.path.join(os.path.dirname(__file__), "..", "..", "frontend", "dist")

View File

@@ -245,9 +245,10 @@ def extract_tarball_metadata(file: BinaryIO, filename: str) -> Dict[str, Any]:
break
# Try to split name and version
# Handle optional 'v' prefix on version (e.g., package-v1.0.0)
patterns = [
r"^(.+)-(\d+\.\d+(?:\.\d+)?(?:[-._]\w+)?)$", # name-version
r"^(.+)_(\d+\.\d+(?:\.\d+)?(?:[-._]\w+)?)$", # name_version
r"^(.+)-v?(\d+\.\d+(?:\.\d+)?(?:[-_]\w+)?)$", # name-version or name-vversion
r"^(.+)_v?(\d+\.\d+(?:\.\d+)?(?:[-_]\w+)?)$", # name_version or name_vversion
]
for pattern in patterns:

View File

@@ -12,6 +12,7 @@ from sqlalchemy import (
Index,
JSON,
ARRAY,
LargeBinary,
)
from sqlalchemy.dialects.postgresql import UUID
from sqlalchemy.orm import relationship, declarative_base
@@ -27,11 +28,13 @@ class Project(Base):
name = Column(String(255), unique=True, nullable=False)
description = Column(Text)
is_public = Column(Boolean, default=True)
is_system = Column(Boolean, default=False, nullable=False)
created_at = Column(DateTime(timezone=True), default=datetime.utcnow)
updated_at = Column(
DateTime(timezone=True), default=datetime.utcnow, onupdate=datetime.utcnow
)
created_by = Column(String(255), nullable=False)
team_id = Column(UUID(as_uuid=True), ForeignKey("teams.id", ondelete="SET NULL"))
packages = relationship(
"Package", back_populates="project", cascade="all, delete-orphan"
@@ -39,10 +42,13 @@ class Project(Base):
permissions = relationship(
"AccessPermission", back_populates="project", cascade="all, delete-orphan"
)
team = relationship("Team", back_populates="projects")
__table_args__ = (
Index("idx_projects_name", "name"),
Index("idx_projects_created_by", "created_by"),
Index("idx_projects_team_id", "team_id"),
Index("idx_projects_is_system", "is_system"),
)
@@ -65,13 +71,15 @@ class Package(Base):
)
project = relationship("Project", back_populates="packages")
tags = relationship("Tag", back_populates="package", cascade="all, delete-orphan")
uploads = relationship(
"Upload", back_populates="package", cascade="all, delete-orphan"
)
consumers = relationship(
"Consumer", back_populates="package", cascade="all, delete-orphan"
)
versions = relationship(
"PackageVersion", back_populates="package", cascade="all, delete-orphan"
)
__table_args__ = (
Index("idx_packages_project_id", "project_id"),
@@ -111,8 +119,11 @@ class Artifact(Base):
ref_count = Column(Integer, default=1)
s3_key = Column(String(1024), nullable=False)
tags = relationship("Tag", back_populates="artifact")
uploads = relationship("Upload", back_populates="artifact")
versions = relationship("PackageVersion", back_populates="artifact")
dependencies = relationship(
"ArtifactDependency", back_populates="artifact", cascade="all, delete-orphan"
)
@property
def sha256(self) -> str:
@@ -138,8 +149,14 @@ class Artifact(Base):
)
class Tag(Base):
__tablename__ = "tags"
class PackageVersion(Base):
"""Immutable version record for a package-artifact relationship.
Separates versions (immutable, set at upload) from tags (mutable labels).
Each artifact in a package can have at most one version.
"""
__tablename__ = "package_versions"
id = Column(UUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
package_id = Column(
@@ -147,53 +164,20 @@ class Tag(Base):
ForeignKey("packages.id", ondelete="CASCADE"),
nullable=False,
)
name = Column(String(255), nullable=False)
artifact_id = Column(String(64), ForeignKey("artifacts.id"), nullable=False)
version = Column(String(255), nullable=False)
version_source = Column(String(50)) # 'explicit', 'filename', 'metadata', 'migrated_from_tag'
created_at = Column(DateTime(timezone=True), default=datetime.utcnow)
updated_at = Column(
DateTime(timezone=True), default=datetime.utcnow, onupdate=datetime.utcnow
)
created_by = Column(String(255), nullable=False)
package = relationship("Package", back_populates="tags")
artifact = relationship("Artifact", back_populates="tags")
history = relationship(
"TagHistory", back_populates="tag", cascade="all, delete-orphan"
)
package = relationship("Package", back_populates="versions")
artifact = relationship("Artifact", back_populates="versions")
__table_args__ = (
Index("idx_tags_package_id", "package_id"),
Index("idx_tags_artifact_id", "artifact_id"),
Index(
"idx_tags_package_name", "package_id", "name", unique=True
), # Composite unique index
Index(
"idx_tags_package_created_at", "package_id", "created_at"
), # For recent tags queries
)
class TagHistory(Base):
__tablename__ = "tag_history"
id = Column(UUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
tag_id = Column(
UUID(as_uuid=True), ForeignKey("tags.id", ondelete="CASCADE"), nullable=False
)
old_artifact_id = Column(String(64), ForeignKey("artifacts.id"))
new_artifact_id = Column(String(64), ForeignKey("artifacts.id"), nullable=False)
change_type = Column(String(20), nullable=False, default="update")
changed_at = Column(DateTime(timezone=True), default=datetime.utcnow)
changed_by = Column(String(255), nullable=False)
tag = relationship("Tag", back_populates="history")
__table_args__ = (
Index("idx_tag_history_tag_id", "tag_id"),
Index("idx_tag_history_changed_at", "changed_at"),
CheckConstraint(
"change_type IN ('create', 'update', 'delete')", name="check_change_type"
),
Index("idx_package_versions_package_id", "package_id"),
Index("idx_package_versions_artifact_id", "artifact_id"),
Index("idx_package_versions_package_version", "package_id", "version", unique=True),
Index("idx_package_versions_package_artifact", "package_id", "artifact_id", unique=True),
)
@@ -204,7 +188,7 @@ class Upload(Base):
artifact_id = Column(String(64), ForeignKey("artifacts.id"), nullable=False)
package_id = Column(UUID(as_uuid=True), ForeignKey("packages.id"), nullable=False)
original_name = Column(String(1024))
tag_name = Column(String(255)) # Tag assigned during upload
version = Column(String(255)) # Version assigned during upload
user_agent = Column(String(512)) # Client identification
duration_ms = Column(Integer) # Upload timing in milliseconds
deduplicated = Column(Boolean, default=False) # Whether artifact was deduplicated
@@ -330,6 +314,9 @@ class User(Base):
sessions = relationship(
"Session", back_populates="user", cascade="all, delete-orphan"
)
team_memberships = relationship(
"TeamMembership", back_populates="user", cascade="all, delete-orphan"
)
__table_args__ = (
Index("idx_users_username", "username"),
@@ -471,3 +458,280 @@ class PackageHistory(Base):
Index("idx_package_history_changed_at", "changed_at"),
Index("idx_package_history_package_changed_at", "package_id", "changed_at"),
)
class ArtifactDependency(Base):
"""Dependency declared by an artifact on another package.
Each artifact can declare dependencies on other packages, specifying a version.
This enables recursive dependency resolution.
"""
__tablename__ = "artifact_dependencies"
id = Column(UUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
artifact_id = Column(
String(64),
ForeignKey("artifacts.id", ondelete="CASCADE"),
nullable=False,
)
dependency_project = Column(String(255), nullable=False)
dependency_package = Column(String(255), nullable=False)
version_constraint = Column(String(255), nullable=False)
created_at = Column(DateTime(timezone=True), default=datetime.utcnow)
# Relationship to the artifact that declares this dependency
artifact = relationship("Artifact", back_populates="dependencies")
__table_args__ = (
# Each artifact can only depend on a specific project/package once
Index(
"idx_artifact_dependencies_artifact_id",
"artifact_id",
),
Index(
"idx_artifact_dependencies_target",
"dependency_project",
"dependency_package",
),
Index(
"idx_artifact_dependencies_unique",
"artifact_id",
"dependency_project",
"dependency_package",
unique=True,
),
)
class Team(Base):
"""Team for organizing projects and users."""
__tablename__ = "teams"
id = Column(UUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
name = Column(String(255), nullable=False)
slug = Column(String(255), unique=True, nullable=False)
description = Column(Text)
created_at = Column(DateTime(timezone=True), default=datetime.utcnow)
updated_at = Column(
DateTime(timezone=True), default=datetime.utcnow, onupdate=datetime.utcnow
)
created_by = Column(String(255), nullable=False)
settings = Column(JSON, default=dict)
# Relationships
memberships = relationship(
"TeamMembership", back_populates="team", cascade="all, delete-orphan"
)
projects = relationship("Project", back_populates="team")
__table_args__ = (
Index("idx_teams_slug", "slug"),
Index("idx_teams_created_by", "created_by"),
Index("idx_teams_created_at", "created_at"),
CheckConstraint(
"slug ~ '^[a-z0-9][a-z0-9-]*[a-z0-9]$' OR slug ~ '^[a-z0-9]$'",
name="check_team_slug_format",
),
)
class TeamMembership(Base):
"""Maps users to teams with their roles."""
__tablename__ = "team_memberships"
id = Column(UUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
team_id = Column(
UUID(as_uuid=True),
ForeignKey("teams.id", ondelete="CASCADE"),
nullable=False,
)
user_id = Column(
UUID(as_uuid=True),
ForeignKey("users.id", ondelete="CASCADE"),
nullable=False,
)
role = Column(String(20), nullable=False, default="member")
created_at = Column(DateTime(timezone=True), default=datetime.utcnow)
invited_by = Column(String(255))
# Relationships
team = relationship("Team", back_populates="memberships")
user = relationship("User", back_populates="team_memberships")
__table_args__ = (
Index("idx_team_memberships_team_id", "team_id"),
Index("idx_team_memberships_user_id", "user_id"),
Index("idx_team_memberships_role", "role"),
Index("idx_team_memberships_team_role", "team_id", "role"),
Index("idx_team_memberships_unique", "team_id", "user_id", unique=True),
CheckConstraint(
"role IN ('owner', 'admin', 'member')",
name="check_team_role",
),
)
# =============================================================================
# Upstream Caching Models
# =============================================================================
# Valid source types for upstream registries
SOURCE_TYPES = ["npm", "pypi", "maven", "docker", "helm", "nuget", "deb", "rpm", "generic"]
# Valid authentication types
AUTH_TYPES = ["none", "basic", "bearer", "api_key"]
class UpstreamSource(Base):
"""Configuration for an upstream artifact registry.
Stores connection details and authentication for upstream registries
like npm, PyPI, Maven Central, or private Artifactory instances.
"""
__tablename__ = "upstream_sources"
id = Column(UUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
name = Column(String(255), unique=True, nullable=False)
source_type = Column(String(50), default="generic", nullable=False)
url = Column(String(2048), nullable=False)
enabled = Column(Boolean, default=False, nullable=False)
auth_type = Column(String(20), default="none", nullable=False)
username = Column(String(255))
password_encrypted = Column(LargeBinary)
headers_encrypted = Column(LargeBinary)
priority = Column(Integer, default=100, nullable=False)
created_at = Column(DateTime(timezone=True), default=datetime.utcnow)
updated_at = Column(
DateTime(timezone=True), default=datetime.utcnow, onupdate=datetime.utcnow
)
# Relationships
cached_urls = relationship("CachedUrl", back_populates="source")
__table_args__ = (
Index("idx_upstream_sources_enabled", "enabled"),
Index("idx_upstream_sources_source_type", "source_type"),
Index("idx_upstream_sources_priority", "priority"),
CheckConstraint(
"source_type IN ('npm', 'pypi', 'maven', 'docker', 'helm', 'nuget', 'deb', 'rpm', 'generic')",
name="check_source_type",
),
CheckConstraint(
"auth_type IN ('none', 'basic', 'bearer', 'api_key')",
name="check_auth_type",
),
CheckConstraint("priority > 0", name="check_priority_positive"),
)
def set_password(self, password: str) -> None:
"""Encrypt and store a password/token."""
from .encryption import encrypt_value
if password:
self.password_encrypted = encrypt_value(password)
else:
self.password_encrypted = None
def get_password(self) -> str | None:
"""Decrypt and return the stored password/token."""
from .encryption import decrypt_value
if self.password_encrypted:
try:
return decrypt_value(self.password_encrypted)
except Exception:
return None
return None
def has_password(self) -> bool:
"""Check if a password/token is stored."""
return self.password_encrypted is not None
def set_headers(self, headers: dict) -> None:
"""Encrypt and store custom headers as JSON."""
from .encryption import encrypt_value
import json
if headers:
self.headers_encrypted = encrypt_value(json.dumps(headers))
else:
self.headers_encrypted = None
def get_headers(self) -> dict | None:
"""Decrypt and return custom headers."""
from .encryption import decrypt_value
import json
if self.headers_encrypted:
try:
return json.loads(decrypt_value(self.headers_encrypted))
except Exception:
return None
return None
class CacheSettings(Base):
"""Global cache settings (singleton table).
Controls behavior of the upstream caching system.
"""
__tablename__ = "cache_settings"
id = Column(Integer, primary_key=True, default=1)
auto_create_system_projects = Column(Boolean, default=True, nullable=False)
created_at = Column(DateTime(timezone=True), default=datetime.utcnow)
updated_at = Column(
DateTime(timezone=True), default=datetime.utcnow, onupdate=datetime.utcnow
)
__table_args__ = (
CheckConstraint("id = 1", name="check_cache_settings_singleton"),
)
class CachedUrl(Base):
"""Tracks URL to artifact mappings for provenance.
Records which URLs have been cached and maps them to their stored artifacts.
Enables "is this URL already cached?" lookups and audit trails.
"""
__tablename__ = "cached_urls"
id = Column(UUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
url = Column(String(4096), nullable=False)
url_hash = Column(String(64), unique=True, nullable=False)
artifact_id = Column(
String(64), ForeignKey("artifacts.id"), nullable=False
)
source_id = Column(
UUID(as_uuid=True),
ForeignKey("upstream_sources.id", ondelete="SET NULL"),
)
fetched_at = Column(DateTime(timezone=True), default=datetime.utcnow, nullable=False)
response_headers = Column(JSON, default=dict)
created_at = Column(DateTime(timezone=True), default=datetime.utcnow)
# Relationships
artifact = relationship("Artifact")
source = relationship("UpstreamSource", back_populates="cached_urls")
__table_args__ = (
Index("idx_cached_urls_url_hash", "url_hash"),
Index("idx_cached_urls_artifact_id", "artifact_id"),
Index("idx_cached_urls_source_id", "source_id"),
Index("idx_cached_urls_fetched_at", "fetched_at"),
)
@staticmethod
def compute_url_hash(url: str) -> str:
"""Compute SHA256 hash of a URL for fast lookups."""
import hashlib
return hashlib.sha256(url.encode("utf-8")).hexdigest()

View File

@@ -0,0 +1,202 @@
"""
Purge seed/demo data from the database.
This is used when transitioning an environment from dev/test to production-like.
Triggered by setting ORCHARD_PURGE_SEED_DATA=true environment variable.
"""
import logging
import os
from sqlalchemy.orm import Session
from .models import (
Project,
Package,
Artifact,
Upload,
PackageVersion,
ArtifactDependency,
Team,
TeamMembership,
User,
AccessPermission,
)
from .storage import get_storage
logger = logging.getLogger(__name__)
# Seed data identifiers (from seed.py)
SEED_PROJECT_NAMES = [
"frontend-libs",
"backend-services",
"mobile-apps",
"internal-tools",
]
SEED_TEAM_SLUG = "demo-team"
SEED_USERNAMES = [
"alice",
"bob",
"charlie",
"diana",
"eve",
"frank",
]
def should_purge_seed_data() -> bool:
"""Check if seed data should be purged based on environment variable."""
return os.environ.get("ORCHARD_PURGE_SEED_DATA", "").lower() == "true"
def purge_seed_data(db: Session) -> dict:
"""
Purge all seed/demo data from the database.
Returns a dict with counts of deleted items.
"""
logger.warning("PURGING SEED DATA - This will delete demo projects, users, and teams")
results = {
"dependencies_deleted": 0,
"versions_deleted": 0,
"uploads_deleted": 0,
"artifacts_deleted": 0,
"packages_deleted": 0,
"projects_deleted": 0,
"permissions_deleted": 0,
"team_memberships_deleted": 0,
"users_deleted": 0,
"teams_deleted": 0,
"s3_objects_deleted": 0,
}
storage = get_storage()
# Find seed projects
seed_projects = db.query(Project).filter(Project.name.in_(SEED_PROJECT_NAMES)).all()
seed_project_ids = [p.id for p in seed_projects]
if not seed_projects:
logger.info("No seed projects found, nothing to purge")
return results
logger.info(f"Found {len(seed_projects)} seed projects to purge")
# Find packages in seed projects
seed_packages = db.query(Package).filter(Package.project_id.in_(seed_project_ids)).all()
seed_package_ids = [p.id for p in seed_packages]
# Find artifacts in seed packages (via uploads)
seed_uploads = db.query(Upload).filter(Upload.package_id.in_(seed_package_ids)).all()
seed_artifact_ids = list(set(u.artifact_id for u in seed_uploads))
# Delete in order (respecting foreign keys)
# 1. Delete artifact dependencies
if seed_artifact_ids:
count = db.query(ArtifactDependency).filter(
ArtifactDependency.artifact_id.in_(seed_artifact_ids)
).delete(synchronize_session=False)
results["dependencies_deleted"] = count
logger.info(f"Deleted {count} artifact dependencies")
# 2. Delete package versions
if seed_package_ids:
count = db.query(PackageVersion).filter(
PackageVersion.package_id.in_(seed_package_ids)
).delete(synchronize_session=False)
results["versions_deleted"] = count
logger.info(f"Deleted {count} package versions")
# 3. Delete uploads
if seed_package_ids:
count = db.query(Upload).filter(Upload.package_id.in_(seed_package_ids)).delete(
synchronize_session=False
)
results["uploads_deleted"] = count
logger.info(f"Deleted {count} uploads")
# 4. Delete S3 objects for seed artifacts
if seed_artifact_ids:
seed_artifacts = db.query(Artifact).filter(Artifact.id.in_(seed_artifact_ids)).all()
for artifact in seed_artifacts:
if artifact.s3_key:
try:
storage.client.delete_object(Bucket=storage.bucket, Key=artifact.s3_key)
results["s3_objects_deleted"] += 1
except Exception as e:
logger.warning(f"Failed to delete S3 object {artifact.s3_key}: {e}")
logger.info(f"Deleted {results['s3_objects_deleted']} S3 objects")
# 5. Delete artifacts (only those with ref_count that would be 0 after our deletions)
# Since we deleted all versions pointing to these artifacts, we can delete them
if seed_artifact_ids:
count = db.query(Artifact).filter(Artifact.id.in_(seed_artifact_ids)).delete(
synchronize_session=False
)
results["artifacts_deleted"] = count
logger.info(f"Deleted {count} artifacts")
# 6. Delete packages
if seed_package_ids:
count = db.query(Package).filter(Package.id.in_(seed_package_ids)).delete(
synchronize_session=False
)
results["packages_deleted"] = count
logger.info(f"Deleted {count} packages")
# 7. Delete access permissions for seed projects
if seed_project_ids:
count = db.query(AccessPermission).filter(
AccessPermission.project_id.in_(seed_project_ids)
).delete(synchronize_session=False)
results["permissions_deleted"] = count
logger.info(f"Deleted {count} access permissions")
# 8. Delete seed projects
count = db.query(Project).filter(Project.name.in_(SEED_PROJECT_NAMES)).delete(
synchronize_session=False
)
results["projects_deleted"] = count
logger.info(f"Deleted {count} projects")
# 9. Find and delete seed team
seed_team = db.query(Team).filter(Team.slug == SEED_TEAM_SLUG).first()
if seed_team:
# Delete team memberships first
count = db.query(TeamMembership).filter(
TeamMembership.team_id == seed_team.id
).delete(synchronize_session=False)
results["team_memberships_deleted"] = count
logger.info(f"Deleted {count} team memberships")
# Delete the team
db.delete(seed_team)
results["teams_deleted"] = 1
logger.info(f"Deleted team: {SEED_TEAM_SLUG}")
# 10. Delete seed users (but NOT admin)
seed_users = db.query(User).filter(User.username.in_(SEED_USERNAMES)).all()
for user in seed_users:
# Delete any remaining team memberships for this user
db.query(TeamMembership).filter(TeamMembership.user_id == user.id).delete(
synchronize_session=False
)
# Delete any access permissions for this user
# Note: AccessPermission.user_id is VARCHAR (username), not UUID
db.query(AccessPermission).filter(AccessPermission.user_id == user.username).delete(
synchronize_session=False
)
db.delete(user)
results["users_deleted"] += 1
if results["users_deleted"] > 0:
logger.info(f"Deleted {results['users_deleted']} seed users")
db.commit()
logger.warning("SEED DATA PURGE COMPLETE")
logger.info(f"Purge results: {results}")
return results

1176
backend/app/pypi_proxy.py Normal file

File diff suppressed because it is too large Load Diff

View File

@@ -0,0 +1,426 @@
"""
Registry client abstraction for upstream package registries.
Provides a pluggable interface for fetching packages from upstream registries
(PyPI, npm, Maven, etc.) during dependency resolution with auto-fetch enabled.
"""
import hashlib
import logging
import os
import re
import tempfile
from abc import ABC, abstractmethod
from dataclasses import dataclass
from typing import List, Optional, TYPE_CHECKING
from urllib.parse import urljoin, urlparse
import httpx
from packaging.specifiers import SpecifierSet, InvalidSpecifier
from packaging.version import Version, InvalidVersion
from sqlalchemy.orm import Session
if TYPE_CHECKING:
from .storage import S3Storage
from .http_client import HttpClientManager
logger = logging.getLogger(__name__)
@dataclass
class VersionInfo:
"""Information about a package version from an upstream registry."""
version: str
download_url: str
filename: str
sha256: Optional[str] = None
size: Optional[int] = None
content_type: Optional[str] = None
@dataclass
class FetchResult:
"""Result of fetching a package from upstream."""
artifact_id: str # SHA256 hash
size: int
version: str
filename: str
already_cached: bool = False
class RegistryClient(ABC):
"""Abstract base class for upstream registry clients."""
@property
@abstractmethod
def source_type(self) -> str:
"""Return the source type this client handles (e.g., 'pypi', 'npm')."""
pass
@abstractmethod
async def get_available_versions(self, package_name: str) -> List[str]:
"""
Get all available versions of a package from upstream.
Args:
package_name: The normalized package name
Returns:
List of version strings, sorted from oldest to newest
"""
pass
@abstractmethod
async def resolve_constraint(
self, package_name: str, constraint: str
) -> Optional[VersionInfo]:
"""
Find the best version matching a constraint.
Args:
package_name: The normalized package name
constraint: Version constraint (e.g., '>=1.9', '<2.0,>=1.5', '*')
Returns:
VersionInfo with download URL, or None if no matching version found
"""
pass
@abstractmethod
async def fetch_package(
self,
package_name: str,
version_info: VersionInfo,
db: Session,
storage: "S3Storage",
) -> Optional[FetchResult]:
"""
Fetch and cache a package from upstream.
Args:
package_name: The normalized package name
version_info: Version details including download URL
db: Database session for creating records
storage: S3 storage for caching the artifact
Returns:
FetchResult with artifact_id, or None if fetch failed
"""
pass
class PyPIRegistryClient(RegistryClient):
"""PyPI registry client using the JSON API."""
# Timeout configuration for PyPI requests
CONNECT_TIMEOUT = 30.0
READ_TIMEOUT = 60.0
DOWNLOAD_TIMEOUT = 300.0 # Longer timeout for file downloads
def __init__(
self,
http_client: httpx.AsyncClient,
upstream_sources: List,
pypi_api_url: str = "https://pypi.org/pypi",
):
"""
Initialize PyPI registry client.
Args:
http_client: Shared async HTTP client
upstream_sources: List of configured upstream sources for auth
pypi_api_url: Base URL for PyPI JSON API
"""
self.client = http_client
self.sources = upstream_sources
self.api_url = pypi_api_url
@property
def source_type(self) -> str:
return "pypi"
def _normalize_package_name(self, name: str) -> str:
"""Normalize a PyPI package name per PEP 503."""
return re.sub(r"[-_.]+", "-", name).lower()
def _get_auth_headers(self) -> dict:
"""Get authentication headers from configured sources."""
headers = {"User-Agent": "Orchard-Registry-Client/1.0"}
if self.sources:
source = self.sources[0]
if hasattr(source, "auth_type"):
if source.auth_type == "bearer":
password = (
source.get_password()
if hasattr(source, "get_password")
else getattr(source, "password", None)
)
if password:
headers["Authorization"] = f"Bearer {password}"
elif source.auth_type == "api_key":
custom_headers = (
source.get_headers()
if hasattr(source, "get_headers")
else {}
)
if custom_headers:
headers.update(custom_headers)
return headers
def _get_basic_auth(self) -> Optional[tuple]:
"""Get basic auth credentials if configured."""
if self.sources:
source = self.sources[0]
if hasattr(source, "auth_type") and source.auth_type == "basic":
username = getattr(source, "username", None)
if username:
password = (
source.get_password()
if hasattr(source, "get_password")
else getattr(source, "password", "")
)
return (username, password or "")
return None
async def get_available_versions(self, package_name: str) -> List[str]:
"""Get all available versions from PyPI JSON API."""
normalized = self._normalize_package_name(package_name)
url = f"{self.api_url}/{normalized}/json"
headers = self._get_auth_headers()
auth = self._get_basic_auth()
timeout = httpx.Timeout(self.READ_TIMEOUT, connect=self.CONNECT_TIMEOUT)
try:
response = await self.client.get(
url, headers=headers, auth=auth, timeout=timeout
)
if response.status_code == 404:
logger.debug(f"Package {normalized} not found on PyPI")
return []
if response.status_code != 200:
logger.warning(
f"PyPI API returned {response.status_code} for {normalized}"
)
return []
data = response.json()
releases = data.get("releases", {})
# Filter to valid versions and sort
versions = []
for v in releases.keys():
try:
Version(v)
versions.append(v)
except InvalidVersion:
continue
versions.sort(key=lambda x: Version(x))
return versions
except httpx.RequestError as e:
logger.warning(f"Failed to query PyPI for {normalized}: {e}")
return []
except Exception as e:
logger.warning(f"Error parsing PyPI response for {normalized}: {e}")
return []
async def resolve_constraint(
self, package_name: str, constraint: str
) -> Optional[VersionInfo]:
"""Find best version matching constraint from PyPI."""
normalized = self._normalize_package_name(package_name)
url = f"{self.api_url}/{normalized}/json"
headers = self._get_auth_headers()
auth = self._get_basic_auth()
timeout = httpx.Timeout(self.READ_TIMEOUT, connect=self.CONNECT_TIMEOUT)
try:
response = await self.client.get(
url, headers=headers, auth=auth, timeout=timeout
)
if response.status_code == 404:
logger.debug(f"Package {normalized} not found on PyPI")
return None
if response.status_code != 200:
logger.warning(
f"PyPI API returned {response.status_code} for {normalized}"
)
return None
data = response.json()
releases = data.get("releases", {})
# Handle wildcard - return latest version
if constraint == "*":
latest_version = data.get("info", {}).get("version")
if latest_version and latest_version in releases:
return self._get_version_info(
normalized, latest_version, releases[latest_version]
)
return None
# Parse constraint
# If constraint looks like a bare version (no operator), treat as exact match
# e.g., "2025.10.5" -> "==2025.10.5"
effective_constraint = constraint
if constraint and constraint[0].isdigit():
effective_constraint = f"=={constraint}"
logger.debug(
f"Bare version '{constraint}' for {normalized}, "
f"treating as exact match '{effective_constraint}'"
)
try:
specifier = SpecifierSet(effective_constraint)
except InvalidSpecifier:
# Invalid constraint - treat as wildcard
logger.warning(
f"Invalid version constraint '{constraint}' for {normalized}, "
"treating as wildcard"
)
latest_version = data.get("info", {}).get("version")
if latest_version and latest_version in releases:
return self._get_version_info(
normalized, latest_version, releases[latest_version]
)
return None
# Find matching versions
matching = []
for v_str, files in releases.items():
if not files: # Skip versions with no files
continue
try:
v = Version(v_str)
if v in specifier:
matching.append((v_str, v, files))
except InvalidVersion:
continue
if not matching:
logger.debug(
f"No versions of {normalized} match constraint '{constraint}'"
)
return None
# Sort by version and return highest match
matching.sort(key=lambda x: x[1], reverse=True)
best_version, _, best_files = matching[0]
return self._get_version_info(normalized, best_version, best_files)
except httpx.RequestError as e:
logger.warning(f"Failed to query PyPI for {normalized}: {e}")
return None
except Exception as e:
logger.warning(f"Error resolving {normalized}@{constraint}: {e}")
return None
def _get_version_info(
self, package_name: str, version: str, files: List[dict]
) -> Optional[VersionInfo]:
"""Extract download info from PyPI release files."""
if not files:
return None
# Prefer wheel over sdist
wheel_file = None
sdist_file = None
for f in files:
filename = f.get("filename", "")
if filename.endswith(".whl"):
# Prefer platform-agnostic wheels
if "py3-none-any" in filename or wheel_file is None:
wheel_file = f
elif filename.endswith(".tar.gz") and sdist_file is None:
sdist_file = f
selected = wheel_file or sdist_file
if not selected:
# Fall back to first available file
selected = files[0]
return VersionInfo(
version=version,
download_url=selected.get("url", ""),
filename=selected.get("filename", ""),
sha256=selected.get("digests", {}).get("sha256"),
size=selected.get("size"),
content_type="application/zip"
if selected.get("filename", "").endswith(".whl")
else "application/gzip",
)
async def fetch_package(
self,
package_name: str,
version_info: VersionInfo,
db: Session,
storage: "S3Storage",
) -> Optional[FetchResult]:
"""Fetch and cache a PyPI package."""
# Import here to avoid circular imports
from .pypi_proxy import fetch_and_cache_pypi_package
normalized = self._normalize_package_name(package_name)
logger.info(
f"Fetching {normalized}=={version_info.version} from upstream PyPI"
)
result = await fetch_and_cache_pypi_package(
db=db,
storage=storage,
http_client=self.client,
package_name=normalized,
filename=version_info.filename,
download_url=version_info.download_url,
expected_sha256=version_info.sha256,
)
if result is None:
return None
return FetchResult(
artifact_id=result["artifact_id"],
size=result["size"],
version=version_info.version,
filename=version_info.filename,
already_cached=result.get("already_cached", False),
)
def get_registry_client(
source_type: str,
http_client: httpx.AsyncClient,
upstream_sources: List,
) -> Optional[RegistryClient]:
"""
Factory function to get a registry client for a source type.
Args:
source_type: The registry type ('pypi', 'npm', etc.)
http_client: Shared async HTTP client
upstream_sources: List of configured upstream sources
Returns:
RegistryClient for the source type, or None if not supported
"""
if source_type == "pypi":
# Filter to PyPI sources
pypi_sources = [s for s in upstream_sources if getattr(s, "source_type", "") == "pypi"]
return PyPIRegistryClient(http_client, pypi_sources)
# Future: Add npm, maven, etc.
logger.debug(f"No registry client available for source type: {source_type}")
return None

View File

@@ -9,7 +9,6 @@ from .base import BaseRepository
from .project import ProjectRepository
from .package import PackageRepository
from .artifact import ArtifactRepository
from .tag import TagRepository
from .upload import UploadRepository
__all__ = [
@@ -17,6 +16,5 @@ __all__ = [
"ProjectRepository",
"PackageRepository",
"ArtifactRepository",
"TagRepository",
"UploadRepository",
]

View File

@@ -8,7 +8,7 @@ from sqlalchemy import func, or_
from uuid import UUID
from .base import BaseRepository
from ..models import Artifact, Tag, Upload, Package, Project
from ..models import Artifact, PackageVersion, Upload, Package, Project
class ArtifactRepository(BaseRepository[Artifact]):
@@ -77,14 +77,14 @@ class ArtifactRepository(BaseRepository[Artifact]):
.all()
)
def get_artifacts_without_tags(self, limit: int = 100) -> List[Artifact]:
"""Get artifacts that have no tags pointing to them."""
# Subquery to find artifact IDs that have tags
tagged_artifacts = self.db.query(Tag.artifact_id).distinct().subquery()
def get_artifacts_without_versions(self, limit: int = 100) -> List[Artifact]:
"""Get artifacts that have no versions pointing to them."""
# Subquery to find artifact IDs that have versions
versioned_artifacts = self.db.query(PackageVersion.artifact_id).distinct().subquery()
return (
self.db.query(Artifact)
.filter(~Artifact.id.in_(tagged_artifacts))
.filter(~Artifact.id.in_(versioned_artifacts))
.limit(limit)
.all()
)
@@ -115,34 +115,34 @@ class ArtifactRepository(BaseRepository[Artifact]):
return artifacts, total
def get_referencing_tags(self, artifact_id: str) -> List[Tuple[Tag, Package, Project]]:
"""Get all tags referencing this artifact with package and project info."""
def get_referencing_versions(self, artifact_id: str) -> List[Tuple[PackageVersion, Package, Project]]:
"""Get all versions referencing this artifact with package and project info."""
return (
self.db.query(Tag, Package, Project)
.join(Package, Tag.package_id == Package.id)
self.db.query(PackageVersion, Package, Project)
.join(Package, PackageVersion.package_id == Package.id)
.join(Project, Package.project_id == Project.id)
.filter(Tag.artifact_id == artifact_id)
.filter(PackageVersion.artifact_id == artifact_id)
.all()
)
def search(self, query_str: str, limit: int = 10) -> List[Tuple[Tag, Artifact, str, str]]:
def search(self, query_str: str, limit: int = 10) -> List[Tuple[PackageVersion, Artifact, str, str]]:
"""
Search artifacts by tag name or original filename.
Returns (tag, artifact, package_name, project_name) tuples.
Search artifacts by version or original filename.
Returns (version, artifact, package_name, project_name) tuples.
"""
search_lower = query_str.lower()
return (
self.db.query(Tag, Artifact, Package.name, Project.name)
.join(Artifact, Tag.artifact_id == Artifact.id)
.join(Package, Tag.package_id == Package.id)
self.db.query(PackageVersion, Artifact, Package.name, Project.name)
.join(Artifact, PackageVersion.artifact_id == Artifact.id)
.join(Package, PackageVersion.package_id == Package.id)
.join(Project, Package.project_id == Project.id)
.filter(
or_(
func.lower(Tag.name).contains(search_lower),
func.lower(PackageVersion.version).contains(search_lower),
func.lower(Artifact.original_name).contains(search_lower)
)
)
.order_by(Tag.name)
.order_by(PackageVersion.version)
.limit(limit)
.all()
)

View File

@@ -8,7 +8,7 @@ from sqlalchemy import func, or_, asc, desc
from uuid import UUID
from .base import BaseRepository
from ..models import Package, Project, Tag, Upload, Artifact
from ..models import Package, Project, PackageVersion, Upload, Artifact
class PackageRepository(BaseRepository[Package]):
@@ -136,10 +136,10 @@ class PackageRepository(BaseRepository[Package]):
return self.update(package, **updates)
def get_stats(self, package_id: UUID) -> dict:
"""Get package statistics (tag count, artifact count, total size)."""
tag_count = (
self.db.query(func.count(Tag.id))
.filter(Tag.package_id == package_id)
"""Get package statistics (version count, artifact count, total size)."""
version_count = (
self.db.query(func.count(PackageVersion.id))
.filter(PackageVersion.package_id == package_id)
.scalar() or 0
)
@@ -154,7 +154,7 @@ class PackageRepository(BaseRepository[Package]):
)
return {
"tag_count": tag_count,
"version_count": version_count,
"artifact_count": artifact_stats[0] if artifact_stats else 0,
"total_size": artifact_stats[1] if artifact_stats else 0,
}

View File

@@ -1,168 +0,0 @@
"""
Tag repository for data access operations.
"""
from typing import Optional, List, Tuple
from sqlalchemy.orm import Session
from sqlalchemy import func, or_, asc, desc
from uuid import UUID
from .base import BaseRepository
from ..models import Tag, TagHistory, Artifact, Package, Project
class TagRepository(BaseRepository[Tag]):
"""Repository for Tag entity operations."""
model = Tag
def get_by_name(self, package_id: UUID, name: str) -> Optional[Tag]:
"""Get tag by name within a package."""
return (
self.db.query(Tag)
.filter(Tag.package_id == package_id, Tag.name == name)
.first()
)
def get_with_artifact(self, package_id: UUID, name: str) -> Optional[Tuple[Tag, Artifact]]:
"""Get tag with its artifact."""
return (
self.db.query(Tag, Artifact)
.join(Artifact, Tag.artifact_id == Artifact.id)
.filter(Tag.package_id == package_id, Tag.name == name)
.first()
)
def exists_by_name(self, package_id: UUID, name: str) -> bool:
"""Check if tag with name exists in package."""
return self.db.query(
self.db.query(Tag)
.filter(Tag.package_id == package_id, Tag.name == name)
.exists()
).scalar()
def list_by_package(
self,
package_id: UUID,
page: int = 1,
limit: int = 20,
search: Optional[str] = None,
sort: str = "name",
order: str = "asc",
) -> Tuple[List[Tuple[Tag, Artifact]], int]:
"""
List tags in a package with artifact metadata.
Returns tuple of ((tag, artifact) tuples, total_count).
"""
query = (
self.db.query(Tag, Artifact)
.join(Artifact, Tag.artifact_id == Artifact.id)
.filter(Tag.package_id == package_id)
)
# Apply search filter (tag name or artifact original filename)
if search:
search_lower = search.lower()
query = query.filter(
or_(
func.lower(Tag.name).contains(search_lower),
func.lower(Artifact.original_name).contains(search_lower)
)
)
# Get total count
total = query.count()
# Apply sorting
sort_columns = {
"name": Tag.name,
"created_at": Tag.created_at,
}
sort_column = sort_columns.get(sort, Tag.name)
if order == "desc":
query = query.order_by(desc(sort_column))
else:
query = query.order_by(asc(sort_column))
# Apply pagination
offset = (page - 1) * limit
results = query.offset(offset).limit(limit).all()
return results, total
def create_tag(
self,
package_id: UUID,
name: str,
artifact_id: str,
created_by: str,
) -> Tag:
"""Create a new tag."""
return self.create(
package_id=package_id,
name=name,
artifact_id=artifact_id,
created_by=created_by,
)
def update_artifact(
self,
tag: Tag,
new_artifact_id: str,
changed_by: str,
record_history: bool = True,
) -> Tag:
"""
Update tag to point to a different artifact.
Optionally records change in tag history.
"""
old_artifact_id = tag.artifact_id
if record_history and old_artifact_id != new_artifact_id:
history = TagHistory(
tag_id=tag.id,
old_artifact_id=old_artifact_id,
new_artifact_id=new_artifact_id,
changed_by=changed_by,
)
self.db.add(history)
tag.artifact_id = new_artifact_id
tag.created_by = changed_by
self.db.flush()
return tag
def get_history(self, tag_id: UUID) -> List[TagHistory]:
"""Get tag change history."""
return (
self.db.query(TagHistory)
.filter(TagHistory.tag_id == tag_id)
.order_by(TagHistory.changed_at.desc())
.all()
)
def get_latest_in_package(self, package_id: UUID) -> Optional[Tag]:
"""Get the most recently created/updated tag in a package."""
return (
self.db.query(Tag)
.filter(Tag.package_id == package_id)
.order_by(Tag.created_at.desc())
.first()
)
def get_by_artifact(self, artifact_id: str) -> List[Tag]:
"""Get all tags pointing to an artifact."""
return (
self.db.query(Tag)
.filter(Tag.artifact_id == artifact_id)
.all()
)
def count_by_artifact(self, artifact_id: str) -> int:
"""Count tags pointing to an artifact."""
return (
self.db.query(func.count(Tag.id))
.filter(Tag.artifact_id == artifact_id)
.scalar() or 0
)

File diff suppressed because it is too large Load Diff

View File

@@ -25,6 +25,7 @@ class ProjectCreate(BaseModel):
name: str
description: Optional[str] = None
is_public: bool = True
team_id: Optional[UUID] = None
class ProjectResponse(BaseModel):
@@ -32,9 +33,13 @@ class ProjectResponse(BaseModel):
name: str
description: Optional[str]
is_public: bool
is_system: bool = False
created_at: datetime
updated_at: datetime
created_by: str
team_id: Optional[UUID] = None
team_slug: Optional[str] = None
team_name: Optional[str] = None
class Config:
from_attributes = True
@@ -109,14 +114,6 @@ class PackageUpdate(BaseModel):
platform: Optional[str] = None
class TagSummary(BaseModel):
"""Lightweight tag info for embedding in package responses"""
name: str
artifact_id: str
created_at: datetime
class PackageDetailResponse(BaseModel):
"""Package with aggregated metadata"""
@@ -129,13 +126,9 @@ class PackageDetailResponse(BaseModel):
created_at: datetime
updated_at: datetime
# Aggregated fields
tag_count: int = 0
artifact_count: int = 0
total_size: int = 0
latest_tag: Optional[str] = None
latest_upload_at: Optional[datetime] = None
# Recent tags (limit 5)
recent_tags: List[TagSummary] = []
class Config:
from_attributes = True
@@ -160,77 +153,6 @@ class ArtifactResponse(BaseModel):
from_attributes = True
# Tag schemas
class TagCreate(BaseModel):
name: str
artifact_id: str
class TagResponse(BaseModel):
id: UUID
package_id: UUID
name: str
artifact_id: str
created_at: datetime
created_by: str
class Config:
from_attributes = True
class TagDetailResponse(BaseModel):
"""Tag with embedded artifact metadata"""
id: UUID
package_id: UUID
name: str
artifact_id: str
created_at: datetime
created_by: str
# Artifact metadata
artifact_size: int
artifact_content_type: Optional[str]
artifact_original_name: Optional[str]
artifact_created_at: datetime
artifact_format_metadata: Optional[Dict[str, Any]] = None
class Config:
from_attributes = True
class TagHistoryResponse(BaseModel):
"""History entry for tag changes"""
id: UUID
tag_id: UUID
old_artifact_id: Optional[str]
new_artifact_id: str
changed_at: datetime
changed_by: str
class Config:
from_attributes = True
class TagHistoryDetailResponse(BaseModel):
"""Tag history with artifact metadata for each version"""
id: UUID
tag_id: UUID
tag_name: str
old_artifact_id: Optional[str]
new_artifact_id: str
changed_at: datetime
changed_by: str
# Artifact metadata for new artifact
artifact_size: int
artifact_original_name: Optional[str]
artifact_content_type: Optional[str]
class Config:
from_attributes = True
# Audit log schemas
class AuditLogResponse(BaseModel):
"""Audit log entry response"""
@@ -257,7 +179,7 @@ class UploadHistoryResponse(BaseModel):
package_name: str
project_name: str
original_name: Optional[str]
tag_name: Optional[str]
version: Optional[str]
uploaded_at: datetime
uploaded_by: str
source_ip: Optional[str]
@@ -288,10 +210,10 @@ class ArtifactProvenanceResponse(BaseModel):
# Usage statistics
upload_count: int
# References
packages: List[Dict[str, Any]] # List of {project_name, package_name, tag_names}
tags: List[
packages: List[Dict[str, Any]] # List of {project_name, package_name, versions}
versions: List[
Dict[str, Any]
] # List of {project_name, package_name, tag_name, created_at}
] # List of {project_name, package_name, version, created_at}
# Upload history
uploads: List[Dict[str, Any]] # List of upload events
@@ -299,18 +221,8 @@ class ArtifactProvenanceResponse(BaseModel):
from_attributes = True
class ArtifactTagInfo(BaseModel):
"""Tag info for embedding in artifact responses"""
id: UUID
name: str
package_id: UUID
package_name: str
project_name: str
class ArtifactDetailResponse(BaseModel):
"""Artifact with list of tags/packages referencing it"""
"""Artifact with metadata"""
id: str
sha256: str # Explicit SHA256 field (same as id)
@@ -324,14 +236,14 @@ class ArtifactDetailResponse(BaseModel):
created_by: str
ref_count: int
format_metadata: Optional[Dict[str, Any]] = None
tags: List[ArtifactTagInfo] = []
versions: List[Dict[str, Any]] = [] # List of {version, package_name, project_name}
class Config:
from_attributes = True
class PackageArtifactResponse(BaseModel):
"""Artifact with tags for package artifact listing"""
"""Artifact for package artifact listing"""
id: str
sha256: str # Explicit SHA256 field (same as id)
@@ -344,7 +256,7 @@ class PackageArtifactResponse(BaseModel):
created_at: datetime
created_by: str
format_metadata: Optional[Dict[str, Any]] = None
tags: List[str] = [] # Tag names pointing to this artifact
version: Optional[str] = None # Version from PackageVersion if exists
class Config:
from_attributes = True
@@ -362,27 +274,9 @@ class GlobalArtifactResponse(BaseModel):
created_by: str
format_metadata: Optional[Dict[str, Any]] = None
ref_count: int = 0
# Context from tags/packages
# Context from versions/packages
projects: List[str] = [] # List of project names containing this artifact
packages: List[str] = [] # List of "project/package" paths
tags: List[str] = [] # List of "project/package:tag" references
class Config:
from_attributes = True
class GlobalTagResponse(BaseModel):
"""Tag with project/package context for global listing"""
id: UUID
name: str
artifact_id: str
created_at: datetime
created_by: str
project_name: str
package_name: str
artifact_size: Optional[int] = None
artifact_content_type: Optional[str] = None
class Config:
from_attributes = True
@@ -395,7 +289,8 @@ class UploadResponse(BaseModel):
size: int
project: str
package: str
tag: Optional[str]
version: Optional[str] = None # Version assigned to this artifact
version_source: Optional[str] = None # How version was determined: 'explicit', 'filename', 'metadata'
checksum_md5: Optional[str] = None
checksum_sha1: Optional[str] = None
s3_etag: Optional[str] = None
@@ -407,6 +302,9 @@ class UploadResponse(BaseModel):
content_type: Optional[str] = None
original_name: Optional[str] = None
created_at: Optional[datetime] = None
# Upload metrics (Issue #43)
duration_ms: Optional[int] = None # Upload duration in milliseconds
throughput_mbps: Optional[float] = None # Upload throughput in MB/s
# Resumable upload schemas
@@ -417,7 +315,7 @@ class ResumableUploadInitRequest(BaseModel):
filename: str
content_type: Optional[str] = None
size: int
tag: Optional[str] = None
version: Optional[str] = None # Explicit version (auto-detected if not provided)
@field_validator("expected_hash")
@classmethod
@@ -451,7 +349,7 @@ class ResumableUploadPartResponse(BaseModel):
class ResumableUploadCompleteRequest(BaseModel):
"""Request to complete a resumable upload"""
tag: Optional[str] = None
pass
class ResumableUploadCompleteResponse(BaseModel):
@@ -461,7 +359,6 @@ class ResumableUploadCompleteResponse(BaseModel):
size: int
project: str
package: str
tag: Optional[str]
class ResumableUploadStatusResponse(BaseModel):
@@ -472,6 +369,21 @@ class ResumableUploadStatusResponse(BaseModel):
total_uploaded_bytes: int
class UploadProgressResponse(BaseModel):
"""Progress information for an in-flight upload"""
upload_id: str
status: str # 'in_progress', 'completed', 'failed', 'not_found'
bytes_uploaded: int = 0
bytes_total: Optional[int] = None
percent_complete: Optional[float] = None
parts_uploaded: int = 0
parts_total: Optional[int] = None
started_at: Optional[datetime] = None
elapsed_seconds: Optional[float] = None
throughput_mbps: Optional[float] = None
# Consumer schemas
class ConsumerResponse(BaseModel):
id: UUID
@@ -484,6 +396,34 @@ class ConsumerResponse(BaseModel):
from_attributes = True
# Package version schemas
class PackageVersionResponse(BaseModel):
"""Immutable version record for an artifact in a package"""
id: UUID
package_id: UUID
artifact_id: str
version: str
version_source: Optional[str] = None # 'explicit', 'filename', 'metadata', 'migrated_from_tag'
created_at: datetime
created_by: str
# Enriched fields from joins
size: Optional[int] = None
content_type: Optional[str] = None
original_name: Optional[str] = None
class Config:
from_attributes = True
class PackageVersionDetailResponse(PackageVersionResponse):
"""Version with full artifact metadata"""
format_metadata: Optional[Dict[str, Any]] = None
checksum_md5: Optional[str] = None
checksum_sha1: Optional[str] = None
# Global search schemas
class SearchResultProject(BaseModel):
"""Project result for global search"""
@@ -512,11 +452,10 @@ class SearchResultPackage(BaseModel):
class SearchResultArtifact(BaseModel):
"""Artifact/tag result for global search"""
"""Artifact result for global search"""
tag_id: UUID
tag_name: str
artifact_id: str
version: Optional[str]
package_id: UUID
package_name: str
project_name: str
@@ -554,6 +493,8 @@ class HealthResponse(BaseModel):
version: str = "1.0.0"
storage_healthy: Optional[bool] = None
database_healthy: Optional[bool] = None
http_pool: Optional[Dict[str, Any]] = None
cache: Optional[Dict[str, Any]] = None
# Garbage collection schemas
@@ -629,7 +570,7 @@ class ProjectStatsResponse(BaseModel):
project_id: str
project_name: str
package_count: int
tag_count: int
version_count: int
artifact_count: int
total_size_bytes: int
upload_count: int
@@ -644,7 +585,7 @@ class PackageStatsResponse(BaseModel):
package_id: str
package_name: str
project_name: str
tag_count: int
version_count: int
artifact_count: int
total_size_bytes: int
upload_count: int
@@ -661,9 +602,9 @@ class ArtifactStatsResponse(BaseModel):
size: int
ref_count: int
storage_savings: int # (ref_count - 1) * size
tags: List[Dict[str, Any]] # Tags referencing this artifact
projects: List[str] # Projects using this artifact
packages: List[str] # Packages using this artifact
versions: List[Dict[str, Any]] = [] # List of {version, package_name, project_name}
first_uploaded: Optional[datetime] = None
last_referenced: Optional[datetime] = None
@@ -854,6 +795,9 @@ class AccessPermissionResponse(BaseModel):
level: str
created_at: datetime
expires_at: Optional[datetime]
source: Optional[str] = "explicit" # "explicit" or "team"
team_slug: Optional[str] = None # Team slug if source is "team"
team_role: Optional[str] = None # Team role if source is "team"
class Config:
from_attributes = True
@@ -863,3 +807,532 @@ class ProjectWithAccessResponse(ProjectResponse):
"""Project response with user's access level"""
user_access_level: Optional[str] = None
# Artifact Dependency schemas
class DependencyCreate(BaseModel):
"""Schema for creating a dependency"""
project: str
package: str
version: str
class DependencyResponse(BaseModel):
"""Schema for dependency response"""
id: UUID
artifact_id: str
project: str
package: str
version: str
created_at: datetime
class Config:
from_attributes = True
@classmethod
def from_orm_model(cls, dep) -> "DependencyResponse":
"""Create from ORM model with field mapping"""
return cls(
id=dep.id,
artifact_id=dep.artifact_id,
project=dep.dependency_project,
package=dep.dependency_package,
version=dep.version_constraint,
created_at=dep.created_at,
)
class ArtifactDependenciesResponse(BaseModel):
"""Response containing all dependencies for an artifact"""
artifact_id: str
dependencies: List[DependencyResponse]
class DependentInfo(BaseModel):
"""Information about an artifact that depends on a package"""
artifact_id: str
project: str
package: str
version: Optional[str] = None
constraint_value: str
class ReverseDependenciesResponse(BaseModel):
"""Response containing packages that depend on a given package"""
project: str
package: str
dependents: List[DependentInfo]
pagination: PaginationMeta
class EnsureFileDependency(BaseModel):
"""Dependency entry from orchard.ensure file"""
project: str
package: str
version: str
class EnsureFileContent(BaseModel):
"""Parsed content of orchard.ensure file"""
dependencies: List[EnsureFileDependency] = []
class ResolvedArtifact(BaseModel):
"""A resolved artifact in the dependency tree"""
artifact_id: str
project: str
package: str
version: Optional[str] = None
size: int
download_url: str
class MissingDependency(BaseModel):
"""A dependency that could not be resolved (not cached on server)"""
project: str
package: str
constraint: Optional[str] = None
required_by: Optional[str] = None
fetch_attempted: bool = False # True if auto-fetch was attempted
fetch_error: Optional[str] = None # Error message if fetch failed
class DependencyResolutionResponse(BaseModel):
"""Response from dependency resolution endpoint"""
requested: Dict[str, str] # project, package, ref
resolved: List[ResolvedArtifact]
missing: List[MissingDependency] = []
fetched: List[ResolvedArtifact] = [] # Artifacts fetched from upstream during resolution
total_size: int
artifact_count: int
class DependencyConflict(BaseModel):
"""Details about a dependency conflict"""
project: str
package: str
requirements: List[Dict[str, Any]] # version and required_by info
class DependencyConflictError(BaseModel):
"""Error response for dependency conflicts"""
error: str = "dependency_conflict"
message: str
conflicts: List[DependencyConflict]
class CircularDependencyError(BaseModel):
"""Error response for circular dependencies"""
error: str = "circular_dependency"
message: str
cycle: List[str] # List of "project/package" strings showing the cycle
# Team schemas
TEAM_ROLES = ["owner", "admin", "member"]
RESERVED_TEAM_SLUGS = {"new", "api", "admin", "settings", "members", "projects", "search"}
class TeamCreate(BaseModel):
"""Create a new team"""
name: str
slug: str
description: Optional[str] = None
@field_validator('name')
@classmethod
def validate_name(cls, v: str) -> str:
"""Validate team name."""
if not v or not v.strip():
raise ValueError("Name cannot be empty")
if len(v) > 255:
raise ValueError("Name must be 255 characters or less")
return v.strip()
@field_validator('slug')
@classmethod
def validate_slug(cls, v: str) -> str:
"""Validate team slug format (lowercase alphanumeric with hyphens)."""
import re
if not v:
raise ValueError("Slug cannot be empty")
if len(v) < 2:
raise ValueError("Slug must be at least 2 characters")
if len(v) > 255:
raise ValueError("Slug must be 255 characters or less")
if not re.match(r'^[a-z0-9][a-z0-9-]*[a-z0-9]$', v) and not re.match(r'^[a-z0-9]$', v):
raise ValueError(
"Slug must be lowercase alphanumeric with hyphens, "
"starting and ending with alphanumeric characters"
)
if '--' in v:
raise ValueError("Slug cannot contain consecutive hyphens")
if v in RESERVED_TEAM_SLUGS:
raise ValueError(f"Slug '{v}' is reserved and cannot be used")
return v
@field_validator('description')
@classmethod
def validate_description(cls, v: Optional[str]) -> Optional[str]:
"""Validate team description."""
if v is not None and len(v) > 2000:
raise ValueError("Description must be 2000 characters or less")
return v
class TeamUpdate(BaseModel):
"""Update team details"""
name: Optional[str] = None
description: Optional[str] = None
@field_validator('name')
@classmethod
def validate_name(cls, v: Optional[str]) -> Optional[str]:
"""Validate team name."""
if v is not None:
if not v.strip():
raise ValueError("Name cannot be empty")
if len(v) > 255:
raise ValueError("Name must be 255 characters or less")
return v.strip()
return v
@field_validator('description')
@classmethod
def validate_description(cls, v: Optional[str]) -> Optional[str]:
"""Validate team description."""
if v is not None and len(v) > 2000:
raise ValueError("Description must be 2000 characters or less")
return v
class TeamResponse(BaseModel):
"""Team response with basic info"""
id: UUID
name: str
slug: str
description: Optional[str]
created_at: datetime
updated_at: datetime
member_count: int = 0
project_count: int = 0
class Config:
from_attributes = True
class TeamDetailResponse(TeamResponse):
"""Team response with user's role"""
user_role: Optional[str] = None # 'owner', 'admin', 'member', or None
class TeamMemberCreate(BaseModel):
"""Add a member to a team"""
username: str
role: str = "member"
@field_validator('role')
@classmethod
def validate_role(cls, v: str) -> str:
if v not in TEAM_ROLES:
raise ValueError(f"Role must be one of: {', '.join(TEAM_ROLES)}")
return v
class TeamMemberUpdate(BaseModel):
"""Update a team member's role"""
role: str
@field_validator('role')
@classmethod
def validate_role(cls, v: str) -> str:
if v not in TEAM_ROLES:
raise ValueError(f"Role must be one of: {', '.join(TEAM_ROLES)}")
return v
class TeamMemberResponse(BaseModel):
"""Team member response"""
id: UUID
user_id: UUID
username: str
email: Optional[str]
role: str
created_at: datetime
class Config:
from_attributes = True
# =============================================================================
# Upstream Caching Schemas
# =============================================================================
# Valid source types
SOURCE_TYPES = ["npm", "pypi", "maven", "docker", "helm", "nuget", "deb", "rpm", "generic"]
# Valid auth types
AUTH_TYPES = ["none", "basic", "bearer", "api_key"]
class UpstreamSourceCreate(BaseModel):
"""Create a new upstream source"""
name: str
source_type: str = "generic"
url: str
enabled: bool = False
auth_type: str = "none"
username: Optional[str] = None
password: Optional[str] = None # Write-only
headers: Optional[dict] = None # Write-only, custom headers
priority: int = 100
@field_validator('name')
@classmethod
def validate_name(cls, v: str) -> str:
v = v.strip()
if not v:
raise ValueError("name cannot be empty")
if len(v) > 255:
raise ValueError("name must be 255 characters or less")
return v
@field_validator('source_type')
@classmethod
def validate_source_type(cls, v: str) -> str:
if v not in SOURCE_TYPES:
raise ValueError(f"source_type must be one of: {', '.join(SOURCE_TYPES)}")
return v
@field_validator('url')
@classmethod
def validate_url(cls, v: str) -> str:
v = v.strip()
if not v:
raise ValueError("url cannot be empty")
if not (v.startswith('http://') or v.startswith('https://')):
raise ValueError("url must start with http:// or https://")
if len(v) > 2048:
raise ValueError("url must be 2048 characters or less")
return v
@field_validator('auth_type')
@classmethod
def validate_auth_type(cls, v: str) -> str:
if v not in AUTH_TYPES:
raise ValueError(f"auth_type must be one of: {', '.join(AUTH_TYPES)}")
return v
@field_validator('priority')
@classmethod
def validate_priority(cls, v: int) -> int:
if v <= 0:
raise ValueError("priority must be greater than 0")
return v
class UpstreamSourceUpdate(BaseModel):
"""Update an upstream source (partial)"""
name: Optional[str] = None
source_type: Optional[str] = None
url: Optional[str] = None
enabled: Optional[bool] = None
auth_type: Optional[str] = None
username: Optional[str] = None
password: Optional[str] = None # Write-only, None = keep existing, empty string = clear
headers: Optional[dict] = None # Write-only
priority: Optional[int] = None
@field_validator('name')
@classmethod
def validate_name(cls, v: Optional[str]) -> Optional[str]:
if v is not None:
v = v.strip()
if not v:
raise ValueError("name cannot be empty")
if len(v) > 255:
raise ValueError("name must be 255 characters or less")
return v
@field_validator('source_type')
@classmethod
def validate_source_type(cls, v: Optional[str]) -> Optional[str]:
if v is not None and v not in SOURCE_TYPES:
raise ValueError(f"source_type must be one of: {', '.join(SOURCE_TYPES)}")
return v
@field_validator('url')
@classmethod
def validate_url(cls, v: Optional[str]) -> Optional[str]:
if v is not None:
v = v.strip()
if not v:
raise ValueError("url cannot be empty")
if not (v.startswith('http://') or v.startswith('https://')):
raise ValueError("url must start with http:// or https://")
if len(v) > 2048:
raise ValueError("url must be 2048 characters or less")
return v
@field_validator('auth_type')
@classmethod
def validate_auth_type(cls, v: Optional[str]) -> Optional[str]:
if v is not None and v not in AUTH_TYPES:
raise ValueError(f"auth_type must be one of: {', '.join(AUTH_TYPES)}")
return v
@field_validator('priority')
@classmethod
def validate_priority(cls, v: Optional[int]) -> Optional[int]:
if v is not None and v <= 0:
raise ValueError("priority must be greater than 0")
return v
class UpstreamSourceResponse(BaseModel):
"""Upstream source response (credentials never included)"""
id: UUID
name: str
source_type: str
url: str
enabled: bool
auth_type: str
username: Optional[str]
has_password: bool # True if password is set
has_headers: bool # True if custom headers are set
priority: int
source: str = "database" # "database" or "env" (env = defined via environment variables)
created_at: Optional[datetime] = None # May be None for legacy/env data
updated_at: Optional[datetime] = None # May be None for legacy/env data
class Config:
from_attributes = True
class CacheSettingsResponse(BaseModel):
"""Global cache settings response"""
auto_create_system_projects: bool
auto_create_system_projects_env_override: Optional[bool] = None # Set if overridden by env var
created_at: Optional[datetime] = None # May be None for legacy data
updated_at: Optional[datetime] = None # May be None for legacy data
class Config:
from_attributes = True
class CacheSettingsUpdate(BaseModel):
"""Update cache settings (partial)"""
auto_create_system_projects: Optional[bool] = None
class CachedUrlResponse(BaseModel):
"""Cached URL response"""
id: UUID
url: str
url_hash: str
artifact_id: str
source_id: Optional[UUID]
source_name: Optional[str] = None # Populated from join
fetched_at: datetime
created_at: datetime
class Config:
from_attributes = True
class CacheRequest(BaseModel):
"""Request to cache an artifact from an upstream URL"""
url: str
source_type: str
package_name: Optional[str] = None # Auto-derived from URL if not provided
version: Optional[str] = None # Auto-derived from URL if not provided
user_project: Optional[str] = None # Cross-reference to user project
user_package: Optional[str] = None
user_version: Optional[str] = None
expected_hash: Optional[str] = None # Verify downloaded content
@field_validator('url')
@classmethod
def validate_url(cls, v: str) -> str:
v = v.strip()
if not v:
raise ValueError("url cannot be empty")
if not (v.startswith('http://') or v.startswith('https://')):
raise ValueError("url must start with http:// or https://")
if len(v) > 4096:
raise ValueError("url must be 4096 characters or less")
return v
@field_validator('source_type')
@classmethod
def validate_source_type(cls, v: str) -> str:
if v not in SOURCE_TYPES:
raise ValueError(f"source_type must be one of: {', '.join(SOURCE_TYPES)}")
return v
@field_validator('expected_hash')
@classmethod
def validate_expected_hash(cls, v: Optional[str]) -> Optional[str]:
if v is not None:
v = v.strip().lower()
# Remove sha256: prefix if present
if v.startswith('sha256:'):
v = v[7:]
# Validate hex format
if len(v) != 64 or not all(c in '0123456789abcdef' for c in v):
raise ValueError("expected_hash must be a 64-character hex string (SHA256)")
return v
class CacheResponse(BaseModel):
"""Response from caching an artifact"""
artifact_id: str
sha256: str
size: int
content_type: Optional[str]
already_cached: bool
source_url: str
source_name: Optional[str]
system_project: str
system_package: str
system_version: Optional[str]
user_reference: Optional[str] = None # e.g., "my-app/npm-deps/+/4.17.21"
class CacheResolveRequest(BaseModel):
"""Request to cache an artifact by package coordinates (no URL required).
The server will construct the appropriate URL based on source_type and
configured upstream sources.
"""
source_type: str
package: str
version: str
user_project: Optional[str] = None
user_package: Optional[str] = None
user_version: Optional[str] = None
@field_validator('source_type')
@classmethod
def validate_source_type(cls, v: str) -> str:
if v not in SOURCE_TYPES:
raise ValueError(f"source_type must be one of: {', '.join(SOURCE_TYPES)}")
return v
@field_validator('package')
@classmethod
def validate_package(cls, v: str) -> str:
v = v.strip()
if not v:
raise ValueError("package cannot be empty")
return v
@field_validator('version')
@classmethod
def validate_version(cls, v: str) -> str:
v = v.strip()
if not v:
raise ValueError("version cannot be empty")
return v

View File

@@ -5,8 +5,9 @@ import hashlib
import logging
from sqlalchemy.orm import Session
from .models import Project, Package, Artifact, Tag, Upload
from .models import Project, Package, Artifact, Upload, PackageVersion, ArtifactDependency, Team, TeamMembership, User
from .storage import get_storage
from .auth import hash_password
logger = logging.getLogger(__name__)
@@ -74,7 +75,7 @@ TEST_PROJECTS = [
},
]
# Sample artifacts to create (content, tags)
# Sample artifacts to create (content, tags, version)
TEST_ARTIFACTS = [
{
"project": "frontend-libs",
@@ -83,6 +84,7 @@ TEST_ARTIFACTS = [
"filename": "ui-components-1.0.0.js",
"content_type": "application/javascript",
"tags": ["v1.0.0", "latest"],
"version": "1.0.0",
},
{
"project": "frontend-libs",
@@ -91,6 +93,7 @@ TEST_ARTIFACTS = [
"filename": "ui-components-1.1.0.js",
"content_type": "application/javascript",
"tags": ["v1.1.0"],
"version": "1.1.0",
},
{
"project": "frontend-libs",
@@ -99,6 +102,7 @@ TEST_ARTIFACTS = [
"filename": "tokens.json",
"content_type": "application/json",
"tags": ["v1.0.0", "latest"],
"version": "1.0.0",
},
{
"project": "backend-services",
@@ -107,6 +111,7 @@ TEST_ARTIFACTS = [
"filename": "utils-2.0.0.py",
"content_type": "text/x-python",
"tags": ["v2.0.0", "stable", "latest"],
"version": "2.0.0",
},
{
"project": "backend-services",
@@ -115,9 +120,21 @@ TEST_ARTIFACTS = [
"filename": "auth-lib-1.0.0.go",
"content_type": "text/x-go",
"tags": ["v1.0.0", "latest"],
"version": "1.0.0",
},
]
# Dependencies to create (source artifact -> dependency)
# Format: (source_project, source_package, source_version, dep_project, dep_package, version_constraint)
TEST_DEPENDENCIES = [
# ui-components v1.1.0 depends on design-tokens v1.0.0
("frontend-libs", "ui-components", "1.1.0", "frontend-libs", "design-tokens", "1.0.0"),
# auth-lib v1.0.0 depends on common-utils v2.0.0
("backend-services", "auth-lib", "1.0.0", "backend-services", "common-utils", "2.0.0"),
# auth-lib v1.0.0 also depends on design-tokens v1.0.0
("backend-services", "auth-lib", "1.0.0", "frontend-libs", "design-tokens", "1.0.0"),
]
def is_database_empty(db: Session) -> bool:
"""Check if the database has any projects."""
@@ -133,6 +150,80 @@ def seed_database(db: Session) -> None:
logger.info("Seeding database with test data...")
storage = get_storage()
# Find or use admin user for team ownership
admin_user = db.query(User).filter(User.username == "admin").first()
team_owner_username = admin_user.username if admin_user else "seed-user"
# Create a demo team
demo_team = Team(
name="Demo Team",
slug="demo-team",
description="A demonstration team with sample projects",
created_by=team_owner_username,
)
db.add(demo_team)
db.flush()
# Add admin user as team owner if they exist
if admin_user:
membership = TeamMembership(
team_id=demo_team.id,
user_id=admin_user.id,
role="owner",
invited_by=team_owner_username,
)
db.add(membership)
db.flush()
logger.info(f"Created team: {demo_team.name} ({demo_team.slug})")
# Create test users with various roles
test_users = [
{"username": "alice", "email": "alice@example.com", "role": "admin"},
{"username": "bob", "email": "bob@example.com", "role": "admin"},
{"username": "charlie", "email": "charlie@example.com", "role": "member"},
{"username": "diana", "email": "diana@example.com", "role": "member"},
{"username": "eve", "email": "eve@example.com", "role": "member"},
{"username": "frank", "email": None, "role": "member"},
]
for user_data in test_users:
# Check if user already exists
existing_user = db.query(User).filter(User.username == user_data["username"]).first()
if existing_user:
test_user = existing_user
else:
# Create the user with password same as username
test_user = User(
username=user_data["username"],
email=user_data["email"],
password_hash=hash_password(user_data["username"]),
is_admin=False,
is_active=True,
must_change_password=False,
)
db.add(test_user)
db.flush()
logger.info(f"Created test user: {user_data['username']}")
# Add to demo team with specified role
existing_membership = db.query(TeamMembership).filter(
TeamMembership.team_id == demo_team.id,
TeamMembership.user_id == test_user.id,
).first()
if not existing_membership:
membership = TeamMembership(
team_id=demo_team.id,
user_id=test_user.id,
role=user_data["role"],
invited_by=team_owner_username,
)
db.add(membership)
logger.info(f"Added {user_data['username']} to {demo_team.slug} as {user_data['role']}")
db.flush()
# Create projects and packages
project_map = {}
package_map = {}
@@ -142,7 +233,8 @@ def seed_database(db: Session) -> None:
name=project_data["name"],
description=project_data["description"],
is_public=project_data["is_public"],
created_by="seed-user",
created_by=team_owner_username,
team_id=demo_team.id, # Assign to demo team
)
db.add(project)
db.flush() # Get the ID
@@ -158,11 +250,11 @@ def seed_database(db: Session) -> None:
db.flush()
package_map[(project_data["name"], package_data["name"])] = package
logger.info(f"Created {len(project_map)} projects and {len(package_map)} packages")
logger.info(f"Created {len(project_map)} projects and {len(package_map)} packages (assigned to {demo_team.slug})")
# Create artifacts and tags
# Create artifacts and versions
artifact_count = 0
tag_count = 0
version_count = 0
for artifact_data in TEST_ARTIFACTS:
project = project_map[artifact_data["project"]]
@@ -184,15 +276,20 @@ def seed_database(db: Session) -> None:
logger.warning(f"Failed to store artifact in S3: {e}")
continue
# Calculate ref_count: tags + version (if present)
ref_count = len(artifact_data["tags"])
if artifact_data.get("version"):
ref_count += 1
# Create artifact record
artifact = Artifact(
id=sha256_hash,
size=size,
content_type=artifact_data["content_type"],
original_name=artifact_data["filename"],
created_by="seed-user",
created_by=team_owner_username,
s3_key=s3_key,
ref_count=len(artifact_data["tags"]),
ref_count=ref_count,
)
db.add(artifact)
@@ -206,17 +303,51 @@ def seed_database(db: Session) -> None:
db.add(upload)
artifact_count += 1
# Create tags
for tag_name in artifact_data["tags"]:
tag = Tag(
# Create version record if specified
if artifact_data.get("version"):
version = PackageVersion(
package_id=package.id,
name=tag_name,
artifact_id=sha256_hash,
created_by="seed-user",
version=artifact_data["version"],
version_source="explicit",
created_by=team_owner_username,
)
db.add(tag)
tag_count += 1
db.add(version)
version_count += 1
db.flush()
# Create dependencies
dependency_count = 0
for dep_data in TEST_DEPENDENCIES:
src_project, src_package, src_version, dep_project, dep_package, version_constraint = dep_data
# Find the source artifact by looking up its version
src_pkg = package_map.get((src_project, src_package))
if not src_pkg:
logger.warning(f"Source package not found: {src_project}/{src_package}")
continue
# Find the artifact for this version
src_version_record = db.query(PackageVersion).filter(
PackageVersion.package_id == src_pkg.id,
PackageVersion.version == src_version,
).first()
if not src_version_record:
logger.warning(f"Source version not found: {src_project}/{src_package}@{src_version}")
continue
# Create the dependency
dependency = ArtifactDependency(
artifact_id=src_version_record.artifact_id,
dependency_project=dep_project,
dependency_package=dep_package,
version_constraint=version_constraint,
)
db.add(dependency)
dependency_count += 1
db.commit()
logger.info(f"Created {artifact_count} artifacts and {tag_count} tags")
logger.info(f"Created {artifact_count} artifacts, {version_count} versions, and {dependency_count} dependencies")
logger.info("Database seeding complete")

View File

@@ -6,9 +6,8 @@ from typing import List, Optional, Tuple
from sqlalchemy.orm import Session
import logging
from ..models import Artifact, Tag
from ..models import Artifact, PackageVersion
from ..repositories.artifact import ArtifactRepository
from ..repositories.tag import TagRepository
from ..storage import S3Storage
logger = logging.getLogger(__name__)
@@ -21,8 +20,8 @@ class ArtifactCleanupService:
Reference counting rules:
- ref_count starts at 1 when artifact is first uploaded
- ref_count increments when the same artifact is uploaded again (deduplication)
- ref_count decrements when a tag is deleted or updated to point elsewhere
- ref_count decrements when a package is deleted (for each tag pointing to artifact)
- ref_count decrements when a version is deleted or updated to point elsewhere
- ref_count decrements when a package is deleted (for each version pointing to artifact)
- When ref_count reaches 0, artifact is a candidate for deletion from S3
"""
@@ -30,12 +29,11 @@ class ArtifactCleanupService:
self.db = db
self.storage = storage
self.artifact_repo = ArtifactRepository(db)
self.tag_repo = TagRepository(db)
def on_tag_deleted(self, artifact_id: str) -> Artifact:
def on_version_deleted(self, artifact_id: str) -> Artifact:
"""
Called when a tag is deleted.
Decrements ref_count for the artifact the tag was pointing to.
Called when a version is deleted.
Decrements ref_count for the artifact the version was pointing to.
"""
artifact = self.artifact_repo.get_by_sha256(artifact_id)
if artifact:
@@ -45,11 +43,11 @@ class ArtifactCleanupService:
)
return artifact
def on_tag_updated(
def on_version_updated(
self, old_artifact_id: str, new_artifact_id: str
) -> Tuple[Optional[Artifact], Optional[Artifact]]:
"""
Called when a tag is updated to point to a different artifact.
Called when a version is updated to point to a different artifact.
Decrements ref_count for old artifact, increments for new (if different).
Returns (old_artifact, new_artifact) tuple.
@@ -79,21 +77,21 @@ class ArtifactCleanupService:
def on_package_deleted(self, package_id) -> List[str]:
"""
Called when a package is deleted.
Decrements ref_count for all artifacts that had tags in the package.
Decrements ref_count for all artifacts that had versions in the package.
Returns list of artifact IDs that were affected.
"""
# Get all tags in the package before deletion
tags = self.db.query(Tag).filter(Tag.package_id == package_id).all()
# Get all versions in the package before deletion
versions = self.db.query(PackageVersion).filter(PackageVersion.package_id == package_id).all()
affected_artifacts = []
for tag in tags:
artifact = self.artifact_repo.get_by_sha256(tag.artifact_id)
for version in versions:
artifact = self.artifact_repo.get_by_sha256(version.artifact_id)
if artifact:
self.artifact_repo.decrement_ref_count(artifact)
affected_artifacts.append(tag.artifact_id)
affected_artifacts.append(version.artifact_id)
logger.info(
f"Decremented ref_count for artifact {tag.artifact_id} (package delete)"
f"Decremented ref_count for artifact {version.artifact_id} (package delete)"
)
return affected_artifacts
@@ -152,7 +150,7 @@ class ArtifactCleanupService:
def verify_ref_counts(self, fix: bool = False) -> List[dict]:
"""
Verify that ref_counts match actual tag references.
Verify that ref_counts match actual version references.
Args:
fix: If True, fix any mismatched ref_counts
@@ -162,28 +160,28 @@ class ArtifactCleanupService:
"""
from sqlalchemy import func
# Get actual tag counts per artifact
tag_counts = (
self.db.query(Tag.artifact_id, func.count(Tag.id).label("tag_count"))
.group_by(Tag.artifact_id)
# Get actual version counts per artifact
version_counts = (
self.db.query(PackageVersion.artifact_id, func.count(PackageVersion.id).label("version_count"))
.group_by(PackageVersion.artifact_id)
.all()
)
tag_count_map = {artifact_id: count for artifact_id, count in tag_counts}
version_count_map = {artifact_id: count for artifact_id, count in version_counts}
# Check all artifacts
artifacts = self.db.query(Artifact).all()
mismatches = []
for artifact in artifacts:
actual_count = tag_count_map.get(artifact.id, 0)
actual_count = version_count_map.get(artifact.id, 0)
# ref_count should be at least 1 (initial upload) + additional uploads
# But tags are the primary reference, so we check against tag count
# But versions are the primary reference, so we check against version count
if artifact.ref_count < actual_count:
mismatch = {
"artifact_id": artifact.id,
"stored_ref_count": artifact.ref_count,
"actual_tag_count": actual_count,
"actual_version_count": actual_count,
}
mismatches.append(mismatch)

View File

@@ -242,15 +242,19 @@ class S3Storage:
},
)
self.client = boto3.client(
"s3",
endpoint_url=settings.s3_endpoint if settings.s3_endpoint else None,
region_name=settings.s3_region,
aws_access_key_id=settings.s3_access_key_id,
aws_secret_access_key=settings.s3_secret_access_key,
config=config,
verify=settings.s3_verify_ssl, # SSL/TLS verification
)
# Build client kwargs - only include credentials if explicitly provided
# This allows IRSA/IAM role credentials to be used when no explicit creds are set
client_kwargs = {
"endpoint_url": settings.s3_endpoint if settings.s3_endpoint else None,
"region_name": settings.s3_region,
"config": config,
"verify": settings.s3_verify_ssl,
}
if settings.s3_access_key_id and settings.s3_secret_access_key:
client_kwargs["aws_access_key_id"] = settings.s3_access_key_id
client_kwargs["aws_secret_access_key"] = settings.s3_secret_access_key
self.client = boto3.client("s3", **client_kwargs)
self.bucket = settings.s3_bucket
# Store active multipart uploads for resumable support
self._active_uploads: Dict[str, Dict[str, Any]] = {}
@@ -378,10 +382,16 @@ class S3Storage:
"""
# First pass: compute all hashes by streaming through file
try:
import time
sha256_hasher = hashlib.sha256()
md5_hasher = hashlib.md5()
sha1_hasher = hashlib.sha1()
size = 0
hash_start_time = time.time()
last_log_time = hash_start_time
log_interval_seconds = 5 # Log progress every 5 seconds
logger.info(f"Computing hashes for large file: expected_size={content_length}")
# Read file in chunks to compute hashes
while True:
@@ -393,6 +403,18 @@ class S3Storage:
sha1_hasher.update(chunk)
size += len(chunk)
# Log hash computation progress periodically
current_time = time.time()
if current_time - last_log_time >= log_interval_seconds:
elapsed = current_time - hash_start_time
percent = (size / content_length) * 100 if content_length > 0 else 0
throughput = (size / (1024 * 1024)) / elapsed if elapsed > 0 else 0
logger.info(
f"Hash computation progress: bytes={size}/{content_length} ({percent:.1f}%) "
f"throughput={throughput:.2f}MB/s"
)
last_log_time = current_time
# Enforce file size limit during streaming (protection against spoofing)
if size > settings.max_file_size:
raise FileSizeExceededError(
@@ -405,6 +427,14 @@ class S3Storage:
sha256_hash = sha256_hasher.hexdigest()
md5_hash = md5_hasher.hexdigest()
sha1_hash = sha1_hasher.hexdigest()
# Log hash computation completion
hash_elapsed = time.time() - hash_start_time
hash_throughput = (size / (1024 * 1024)) / hash_elapsed if hash_elapsed > 0 else 0
logger.info(
f"Hash computation completed: hash={sha256_hash[:16]}... "
f"size={size} duration={hash_elapsed:.2f}s throughput={hash_throughput:.2f}MB/s"
)
except (HashComputationError, FileSizeExceededError):
raise
except Exception as e:
@@ -458,8 +488,19 @@ class S3Storage:
upload_id = mpu["UploadId"]
try:
import time
parts = []
part_number = 1
bytes_uploaded = 0
upload_start_time = time.time()
last_log_time = upload_start_time
log_interval_seconds = 5 # Log progress every 5 seconds
total_parts = (content_length + MULTIPART_CHUNK_SIZE - 1) // MULTIPART_CHUNK_SIZE
logger.info(
f"Starting multipart upload: hash={sha256_hash[:16]}... "
f"size={content_length} parts={total_parts}"
)
while True:
chunk = file.read(MULTIPART_CHUNK_SIZE)
@@ -479,8 +520,32 @@ class S3Storage:
"ETag": response["ETag"],
}
)
bytes_uploaded += len(chunk)
# Log progress periodically
current_time = time.time()
if current_time - last_log_time >= log_interval_seconds:
elapsed = current_time - upload_start_time
percent = (bytes_uploaded / content_length) * 100
throughput = (bytes_uploaded / (1024 * 1024)) / elapsed if elapsed > 0 else 0
logger.info(
f"Upload progress: hash={sha256_hash[:16]}... "
f"part={part_number}/{total_parts} "
f"bytes={bytes_uploaded}/{content_length} ({percent:.1f}%) "
f"throughput={throughput:.2f}MB/s"
)
last_log_time = current_time
part_number += 1
# Log completion
total_elapsed = time.time() - upload_start_time
final_throughput = (content_length / (1024 * 1024)) / total_elapsed if total_elapsed > 0 else 0
logger.info(
f"Multipart upload completed: hash={sha256_hash[:16]}... "
f"size={content_length} duration={total_elapsed:.2f}s throughput={final_throughput:.2f}MB/s"
)
# Complete multipart upload
complete_response = self.client.complete_multipart_upload(
Bucket=self.bucket,
@@ -502,12 +567,28 @@ class S3Storage:
except Exception as e:
# Abort multipart upload on failure
logger.error(f"Multipart upload failed: {e}")
self.client.abort_multipart_upload(
Bucket=self.bucket,
Key=s3_key,
UploadId=upload_id,
error_str = str(e).lower()
is_client_disconnect = (
isinstance(e, (ConnectionResetError, BrokenPipeError)) or
"connection" in error_str or "broken pipe" in error_str or "reset" in error_str
)
if is_client_disconnect:
logger.warning(
f"Multipart upload aborted (client disconnect): hash={sha256_hash[:16]}... "
f"parts_uploaded={len(parts)} bytes_uploaded={bytes_uploaded}"
)
else:
logger.error(f"Multipart upload failed: hash={sha256_hash[:16]}... error={e}")
try:
self.client.abort_multipart_upload(
Bucket=self.bucket,
Key=s3_key,
UploadId=upload_id,
)
logger.info(f"Multipart upload aborted and cleaned up: upload_id={upload_id[:16]}...")
except Exception as abort_error:
logger.error(f"Failed to abort multipart upload: {abort_error}")
raise
def initiate_resumable_upload(self, expected_hash: str) -> Dict[str, Any]:
@@ -529,12 +610,17 @@ class S3Storage:
mpu = self.client.create_multipart_upload(Bucket=self.bucket, Key=s3_key)
upload_id = mpu["UploadId"]
import time
session = {
"upload_id": upload_id,
"s3_key": s3_key,
"already_exists": False,
"parts": [],
"expected_hash": expected_hash,
"started_at": time.time(),
"bytes_uploaded": 0,
"expected_size": None, # Set when init provides size
"status": "in_progress",
}
self._active_uploads[upload_id] = session
return session
@@ -561,10 +647,57 @@ class S3Storage:
part_info = {
"PartNumber": part_number,
"ETag": response["ETag"],
"size": len(data),
}
session["parts"].append(part_info)
session["bytes_uploaded"] = session.get("bytes_uploaded", 0) + len(data)
return part_info
def get_upload_progress(self, upload_id: str) -> Optional[Dict[str, Any]]:
"""
Get progress information for a resumable upload.
Returns None if upload not found.
"""
import time
session = self._active_uploads.get(upload_id)
if not session:
return None
bytes_uploaded = session.get("bytes_uploaded", 0)
expected_size = session.get("expected_size")
started_at = session.get("started_at")
progress = {
"upload_id": upload_id,
"status": session.get("status", "in_progress"),
"bytes_uploaded": bytes_uploaded,
"bytes_total": expected_size,
"parts_uploaded": len(session.get("parts", [])),
"parts_total": None,
"started_at": started_at,
"elapsed_seconds": None,
"percent_complete": None,
"throughput_mbps": None,
}
if expected_size and expected_size > 0:
progress["percent_complete"] = round((bytes_uploaded / expected_size) * 100, 2)
progress["parts_total"] = (expected_size + MULTIPART_CHUNK_SIZE - 1) // MULTIPART_CHUNK_SIZE
if started_at:
elapsed = time.time() - started_at
progress["elapsed_seconds"] = round(elapsed, 2)
if elapsed > 0 and bytes_uploaded > 0:
progress["throughput_mbps"] = round((bytes_uploaded / (1024 * 1024)) / elapsed, 2)
return progress
def set_upload_expected_size(self, upload_id: str, size: int):
"""Set the expected size for an upload (for progress tracking)."""
session = self._active_uploads.get(upload_id)
if session:
session["expected_size"] = size
def complete_resumable_upload(self, upload_id: str) -> Tuple[str, str]:
"""
Complete a resumable upload.
@@ -702,6 +835,36 @@ class S3Storage:
except ClientError:
return False
def delete_all(self) -> int:
"""
Delete all objects in the bucket.
Returns:
Number of objects deleted
"""
deleted_count = 0
try:
paginator = self.client.get_paginator("list_objects_v2")
for page in paginator.paginate(Bucket=self.bucket):
objects = page.get("Contents", [])
if not objects:
continue
# Delete objects in batches of 1000 (S3 limit)
delete_keys = [{"Key": obj["Key"]} for obj in objects]
if delete_keys:
self.client.delete_objects(
Bucket=self.bucket, Delete={"Objects": delete_keys}
)
deleted_count += len(delete_keys)
logger.info(f"Deleted {len(delete_keys)} objects from S3")
logger.info(f"Total objects deleted from S3: {deleted_count}")
return deleted_count
except ClientError as e:
logger.error(f"Failed to delete all S3 objects: {e}")
raise
def generate_presigned_url(
self,
s3_key: str,

565
backend/app/upstream.py Normal file
View File

@@ -0,0 +1,565 @@
"""
HTTP client for fetching artifacts from upstream sources.
Provides streaming downloads with SHA256 computation, authentication support,
and automatic source matching based on URL prefixes.
"""
from __future__ import annotations
import hashlib
import logging
import tempfile
import time
from dataclasses import dataclass, field
from pathlib import Path
from typing import BinaryIO, Optional, TYPE_CHECKING
from urllib.parse import urlparse
import httpx
if TYPE_CHECKING:
from .models import CacheSettings, UpstreamSource
logger = logging.getLogger(__name__)
class UpstreamError(Exception):
"""Base exception for upstream client errors."""
pass
class UpstreamConnectionError(UpstreamError):
"""Connection to upstream failed (network error, DNS, etc.)."""
pass
class UpstreamTimeoutError(UpstreamError):
"""Request to upstream timed out."""
pass
class UpstreamHTTPError(UpstreamError):
"""Upstream returned an HTTP error response."""
def __init__(self, message: str, status_code: int, response_headers: dict = None):
super().__init__(message)
self.status_code = status_code
self.response_headers = response_headers or {}
class UpstreamSSLError(UpstreamError):
"""SSL/TLS error when connecting to upstream."""
pass
class FileSizeExceededError(UpstreamError):
"""File size exceeds the maximum allowed."""
def __init__(self, message: str, content_length: int, max_size: int):
super().__init__(message)
self.content_length = content_length
self.max_size = max_size
class SourceNotFoundError(UpstreamError):
"""No matching upstream source found for URL."""
pass
class SourceDisabledError(UpstreamError):
"""The matching upstream source is disabled."""
pass
@dataclass
class FetchResult:
"""Result of fetching an artifact from upstream."""
content: BinaryIO # File-like object with content
sha256: str # SHA256 hash of content
size: int # Size in bytes
content_type: Optional[str] # Content-Type header
response_headers: dict # All response headers for provenance
source_name: Optional[str] = None # Name of matched upstream source
temp_path: Optional[Path] = None # Path to temp file (for cleanup)
def close(self):
"""Close and clean up resources."""
if self.content:
try:
self.content.close()
except Exception:
pass
if self.temp_path and self.temp_path.exists():
try:
self.temp_path.unlink()
except Exception:
pass
@dataclass
class UpstreamClientConfig:
"""Configuration for the upstream client."""
connect_timeout: float = 30.0 # Connection timeout in seconds
read_timeout: float = 300.0 # Read timeout in seconds (5 minutes for large files)
max_retries: int = 3 # Maximum number of retry attempts
retry_backoff_base: float = 1.0 # Base delay for exponential backoff
retry_backoff_max: float = 30.0 # Maximum delay between retries
follow_redirects: bool = True # Whether to follow redirects
max_redirects: int = 5 # Maximum number of redirects to follow
max_file_size: Optional[int] = None # Maximum file size (None = unlimited)
verify_ssl: bool = True # Verify SSL certificates
user_agent: str = "Orchard-UpstreamClient/1.0"
class UpstreamClient:
"""
HTTP client for fetching artifacts from upstream sources.
Supports streaming downloads, multiple authentication methods,
automatic source matching, and air-gap mode enforcement.
"""
def __init__(
self,
sources: list[UpstreamSource] = None,
cache_settings: CacheSettings = None,
config: UpstreamClientConfig = None,
):
"""
Initialize the upstream client.
Args:
sources: List of upstream sources for URL matching and auth.
Should be sorted by priority (lowest first).
cache_settings: Global cache settings including air-gap mode.
config: Client configuration options.
"""
self.sources = sources or []
self.cache_settings = cache_settings
self.config = config or UpstreamClientConfig()
# Sort sources by priority (lower = higher priority)
self.sources = sorted(self.sources, key=lambda s: s.priority)
def _match_source(self, url: str) -> Optional[UpstreamSource]:
"""
Find the upstream source that matches the given URL.
Matches by URL prefix, returns the highest priority match.
Args:
url: The URL to match.
Returns:
The matching UpstreamSource or None if no match.
"""
for source in self.sources:
# Check if URL starts with source URL (prefix match)
if url.startswith(source.url.rstrip("/")):
return source
return None
def _build_auth_headers(self, source: UpstreamSource) -> dict:
"""
Build authentication headers for the given source.
Args:
source: The upstream source with auth configuration.
Returns:
Dictionary of headers to add to the request.
"""
headers = {}
if source.auth_type == "none":
pass
elif source.auth_type == "basic":
# httpx handles basic auth via auth parameter, but we can also
# do it manually if needed. We'll use the auth parameter instead.
pass
elif source.auth_type == "bearer":
password = source.get_password()
if password:
headers["Authorization"] = f"Bearer {password}"
elif source.auth_type == "api_key":
# API key auth uses custom headers
custom_headers = source.get_headers()
if custom_headers:
headers.update(custom_headers)
return headers
def _get_basic_auth(self, source: UpstreamSource) -> Optional[tuple[str, str]]:
"""
Get basic auth credentials if applicable.
Args:
source: The upstream source.
Returns:
Tuple of (username, password) or None.
"""
if source.auth_type == "basic" and source.username:
password = source.get_password() or ""
return (source.username, password)
return None
def _should_retry(self, error: Exception, attempt: int) -> bool:
"""
Determine if a request should be retried.
Args:
error: The exception that occurred.
attempt: Current attempt number (0-indexed).
Returns:
True if the request should be retried.
"""
if attempt >= self.config.max_retries - 1:
return False
# Retry on connection errors and timeouts
if isinstance(error, (httpx.ConnectError, httpx.ConnectTimeout)):
return True
# Retry on read timeouts
if isinstance(error, httpx.ReadTimeout):
return True
# Retry on certain HTTP errors (502, 503, 504)
if isinstance(error, httpx.HTTPStatusError):
return error.response.status_code in (502, 503, 504)
return False
def _calculate_backoff(self, attempt: int) -> float:
"""
Calculate backoff delay for retry.
Uses exponential backoff with jitter.
Args:
attempt: Current attempt number (0-indexed).
Returns:
Delay in seconds.
"""
import random
delay = self.config.retry_backoff_base * (2**attempt)
# Add jitter (±25%)
delay *= 0.75 + random.random() * 0.5
return min(delay, self.config.retry_backoff_max)
def fetch(self, url: str, expected_hash: Optional[str] = None) -> FetchResult:
"""
Fetch an artifact from the given URL.
Streams the response to a temp file while computing the SHA256 hash.
Handles authentication, retries, and error cases.
Args:
url: The URL to fetch.
expected_hash: Optional expected SHA256 hash for verification.
Returns:
FetchResult with content, hash, size, and headers.
Raises:
SourceDisabledError: If the matching source is disabled.
UpstreamConnectionError: On connection failures.
UpstreamTimeoutError: On timeout.
UpstreamHTTPError: On HTTP error responses.
UpstreamSSLError: On SSL/TLS errors.
FileSizeExceededError: If Content-Length exceeds max_file_size.
"""
start_time = time.time()
# Match URL to source
source = self._match_source(url)
# Check if source is enabled (if we have a match)
if source is not None and not source.enabled:
raise SourceDisabledError(
f"Upstream source '{source.name}' is disabled"
)
source_name = source.name if source else None
logger.info(
f"Fetching URL: {url} (source: {source_name or 'none'})"
)
# Build request parameters
headers = {"User-Agent": self.config.user_agent}
auth = None
if source:
headers.update(self._build_auth_headers(source))
auth = self._get_basic_auth(source)
timeout = httpx.Timeout(
connect=self.config.connect_timeout,
read=self.config.read_timeout,
write=30.0,
pool=10.0,
)
# Attempt fetch with retries
last_error = None
for attempt in range(self.config.max_retries):
try:
return self._do_fetch(
url=url,
headers=headers,
auth=auth,
timeout=timeout,
source_name=source_name,
start_time=start_time,
expected_hash=expected_hash,
)
except (
httpx.ConnectError,
httpx.ConnectTimeout,
httpx.ReadTimeout,
httpx.HTTPStatusError,
) as e:
last_error = e
if self._should_retry(e, attempt):
delay = self._calculate_backoff(attempt)
logger.warning(
f"Fetch failed (attempt {attempt + 1}/{self.config.max_retries}), "
f"retrying in {delay:.1f}s: {e}"
)
time.sleep(delay)
else:
break
# Convert final error to our exception types
self._raise_upstream_error(last_error, url)
def _do_fetch(
self,
url: str,
headers: dict,
auth: Optional[tuple[str, str]],
timeout: httpx.Timeout,
source_name: Optional[str],
start_time: float,
expected_hash: Optional[str] = None,
) -> FetchResult:
"""
Perform the actual fetch operation.
Args:
url: URL to fetch.
headers: Request headers.
auth: Basic auth credentials or None.
timeout: Request timeout configuration.
source_name: Name of matched source for logging.
start_time: Request start time for timing.
expected_hash: Optional expected hash for verification.
Returns:
FetchResult with content and metadata.
"""
with httpx.Client(
timeout=timeout,
follow_redirects=self.config.follow_redirects,
max_redirects=self.config.max_redirects,
verify=self.config.verify_ssl,
) as client:
with client.stream("GET", url, headers=headers, auth=auth) as response:
# Check for HTTP errors
response.raise_for_status()
# Check Content-Length against max size
content_length = response.headers.get("content-length")
if content_length:
content_length = int(content_length)
if (
self.config.max_file_size
and content_length > self.config.max_file_size
):
raise FileSizeExceededError(
f"File size {content_length} exceeds maximum {self.config.max_file_size}",
content_length,
self.config.max_file_size,
)
# Stream to temp file while computing hash
hasher = hashlib.sha256()
size = 0
# Create temp file
temp_file = tempfile.NamedTemporaryFile(
delete=False, prefix="orchard_upstream_"
)
temp_path = Path(temp_file.name)
try:
for chunk in response.iter_bytes(chunk_size=65536):
temp_file.write(chunk)
hasher.update(chunk)
size += len(chunk)
# Check size while streaming if max_file_size is set
if self.config.max_file_size and size > self.config.max_file_size:
temp_file.close()
temp_path.unlink()
raise FileSizeExceededError(
f"Downloaded size {size} exceeds maximum {self.config.max_file_size}",
size,
self.config.max_file_size,
)
temp_file.close()
sha256 = hasher.hexdigest()
# Verify hash if expected
if expected_hash and sha256 != expected_hash.lower():
temp_path.unlink()
raise UpstreamError(
f"Hash mismatch: expected {expected_hash}, got {sha256}"
)
# Capture response headers
response_headers = dict(response.headers)
# Get content type
content_type = response.headers.get("content-type")
elapsed = time.time() - start_time
logger.info(
f"Fetched {url}: {size} bytes, sha256={sha256[:12]}..., "
f"source={source_name}, time={elapsed:.2f}s"
)
# Return file handle positioned at start
content = open(temp_path, "rb")
return FetchResult(
content=content,
sha256=sha256,
size=size,
content_type=content_type,
response_headers=response_headers,
source_name=source_name,
temp_path=temp_path,
)
except Exception:
# Clean up on error
try:
temp_file.close()
except Exception:
pass
if temp_path.exists():
temp_path.unlink()
raise
def _raise_upstream_error(self, error: Exception, url: str):
"""
Convert httpx exception to appropriate UpstreamError.
Args:
error: The httpx exception.
url: The URL that was being fetched.
Raises:
Appropriate UpstreamError subclass.
"""
if error is None:
raise UpstreamError(f"Unknown error fetching {url}")
if isinstance(error, httpx.ConnectError):
raise UpstreamConnectionError(
f"Failed to connect to upstream: {error}"
) from error
if isinstance(error, (httpx.ConnectTimeout, httpx.ReadTimeout)):
raise UpstreamTimeoutError(
f"Request timed out: {error}"
) from error
if isinstance(error, httpx.HTTPStatusError):
raise UpstreamHTTPError(
f"HTTP {error.response.status_code}: {error}",
error.response.status_code,
dict(error.response.headers),
) from error
# Check for SSL errors in the error chain
if "ssl" in str(error).lower() or "certificate" in str(error).lower():
raise UpstreamSSLError(f"SSL/TLS error: {error}") from error
raise UpstreamError(f"Error fetching {url}: {error}") from error
def test_connection(self, source: UpstreamSource) -> tuple[bool, Optional[str], Optional[int]]:
"""
Test connectivity to an upstream source.
Performs a HEAD request to the source URL to verify connectivity
and authentication. Does not follow redirects - a 3xx response
is considered successful since it proves the server is reachable.
Args:
source: The upstream source to test.
Returns:
Tuple of (success, error_message, status_code).
"""
headers = {"User-Agent": self.config.user_agent}
headers.update(self._build_auth_headers(source))
auth = self._get_basic_auth(source)
timeout = httpx.Timeout(
connect=self.config.connect_timeout,
read=30.0,
write=30.0,
pool=10.0,
)
try:
with httpx.Client(
timeout=timeout,
verify=self.config.verify_ssl,
) as client:
response = client.head(
source.url,
headers=headers,
auth=auth,
follow_redirects=False,
)
# Consider 2xx and 3xx as success, also 405 (Method Not Allowed)
# since some servers don't support HEAD
if response.status_code < 400 or response.status_code == 405:
return (True, None, response.status_code)
else:
return (
False,
f"HTTP {response.status_code}",
response.status_code,
)
except httpx.ConnectError as e:
return (False, f"Connection failed: {e}", None)
except httpx.ConnectTimeout as e:
return (False, f"Connection timed out: {e}", None)
except httpx.ReadTimeout as e:
return (False, f"Read timed out: {e}", None)
except httpx.TooManyRedirects as e:
return (False, f"Too many redirects: {e}", None)
except Exception as e:
return (False, f"Error: {e}", None)

View File

@@ -12,6 +12,8 @@ markers =
unit: Unit tests (no external dependencies)
integration: Integration tests (require database/storage)
slow: Slow tests (skip with -m "not slow")
large: Large file tests (100MB+, skip with -m "not large")
concurrent: Concurrent operation tests
# Coverage configuration
[coverage:run]

View File

@@ -11,10 +11,11 @@ python-jose[cryptography]==3.3.0
passlib[bcrypt]==1.7.4
bcrypt==4.0.1
slowapi==0.1.9
httpx>=0.25.0
redis>=5.0.0
# Test dependencies
pytest>=7.4.0
pytest-asyncio>=0.21.0
pytest-cov>=4.1.0
httpx>=0.25.0
moto[s3]>=4.2.0

View File

@@ -0,0 +1 @@
# Scripts package

View File

@@ -0,0 +1,262 @@
#!/usr/bin/env python3
"""
Backfill script to extract dependencies from cached PyPI packages.
This script scans all artifacts in the _pypi project and extracts
Requires-Dist metadata from wheel and sdist files that don't already
have dependencies recorded.
Usage:
# From within the container:
python -m scripts.backfill_pypi_dependencies
# Or with docker exec:
docker exec orchard_orchard-server_1 python -m scripts.backfill_pypi_dependencies
# Dry run (preview only):
docker exec orchard_orchard-server_1 python -m scripts.backfill_pypi_dependencies --dry-run
"""
import argparse
import logging
import re
import sys
import tarfile
import zipfile
from io import BytesIO
from typing import List, Optional, Tuple
# Add parent directory to path for imports
sys.path.insert(0, "/app")
from sqlalchemy import create_engine
from sqlalchemy.orm import sessionmaker
from backend.app.config import get_settings
from backend.app.models import (
Artifact,
ArtifactDependency,
Package,
Project,
Tag,
)
from backend.app.storage import get_storage
logging.basicConfig(
level=logging.INFO,
format="%(asctime)s - %(levelname)s - %(message)s",
)
logger = logging.getLogger(__name__)
def parse_requires_dist(requires_dist: str) -> Tuple[Optional[str], Optional[str]]:
"""Parse a Requires-Dist line into (package_name, version_constraint)."""
# Remove any environment markers (after semicolon)
if ";" in requires_dist:
requires_dist = requires_dist.split(";")[0].strip()
# Match patterns like "package (>=1.0)" or "package>=1.0" or "package"
match = re.match(
r"^([a-zA-Z0-9][-a-zA-Z0-9._]*)\s*(?:\(([^)]+)\)|([<>=!~][^\s;]+))?",
requires_dist.strip(),
)
if not match:
return None, None
package_name = match.group(1)
version_constraint = match.group(2) or match.group(3)
# Normalize package name (PEP 503)
normalized_name = re.sub(r"[-_.]+", "-", package_name).lower()
if version_constraint:
version_constraint = version_constraint.strip()
return normalized_name, version_constraint
def extract_requires_from_metadata(metadata_content: str) -> List[Tuple[str, Optional[str]]]:
"""Extract all Requires-Dist entries from METADATA/PKG-INFO content."""
dependencies = []
for line in metadata_content.split("\n"):
if line.startswith("Requires-Dist:"):
value = line[len("Requires-Dist:"):].strip()
pkg_name, version = parse_requires_dist(value)
if pkg_name:
dependencies.append((pkg_name, version))
return dependencies
def extract_metadata_from_wheel(content: bytes) -> Optional[str]:
"""Extract METADATA file content from a wheel (zip) file."""
try:
with zipfile.ZipFile(BytesIO(content)) as zf:
for name in zf.namelist():
if name.endswith(".dist-info/METADATA"):
return zf.read(name).decode("utf-8", errors="replace")
except Exception as e:
logger.warning(f"Failed to extract metadata from wheel: {e}")
return None
def extract_metadata_from_sdist(content: bytes) -> Optional[str]:
"""Extract PKG-INFO file content from a source distribution (.tar.gz)."""
try:
with tarfile.open(fileobj=BytesIO(content), mode="r:gz") as tf:
for member in tf.getmembers():
if member.name.endswith("/PKG-INFO") and member.name.count("/") == 1:
f = tf.extractfile(member)
if f:
return f.read().decode("utf-8", errors="replace")
except Exception as e:
logger.warning(f"Failed to extract metadata from sdist: {e}")
return None
def extract_dependencies(content: bytes, filename: str) -> List[Tuple[str, Optional[str]]]:
"""Extract dependencies from a PyPI package file."""
metadata = None
if filename.endswith(".whl"):
metadata = extract_metadata_from_wheel(content)
elif filename.endswith(".tar.gz"):
metadata = extract_metadata_from_sdist(content)
if metadata:
return extract_requires_from_metadata(metadata)
return []
def backfill_dependencies(dry_run: bool = False):
"""Main backfill function."""
settings = get_settings()
# Create database connection
engine = create_engine(settings.database_url)
Session = sessionmaker(bind=engine)
db = Session()
# Create storage client
storage = get_storage()
try:
# Find the _pypi project
pypi_project = db.query(Project).filter(Project.name == "_pypi").first()
if not pypi_project:
logger.info("No _pypi project found. Nothing to backfill.")
return
# Get all packages in _pypi
packages = db.query(Package).filter(Package.project_id == pypi_project.id).all()
logger.info(f"Found {len(packages)} packages in _pypi project")
total_artifacts = 0
artifacts_with_deps = 0
artifacts_processed = 0
dependencies_added = 0
for package in packages:
# Get all tags (each tag points to an artifact)
tags = db.query(Tag).filter(Tag.package_id == package.id).all()
for tag in tags:
total_artifacts += 1
filename = tag.name
# Skip non-package files (like .metadata files)
if not (filename.endswith(".whl") or filename.endswith(".tar.gz")):
continue
# Check if this artifact already has dependencies
existing_deps = db.query(ArtifactDependency).filter(
ArtifactDependency.artifact_id == tag.artifact_id
).count()
if existing_deps > 0:
artifacts_with_deps += 1
continue
# Get the artifact
artifact = db.query(Artifact).filter(Artifact.id == tag.artifact_id).first()
if not artifact:
logger.warning(f"Artifact {tag.artifact_id} not found for tag {filename}")
continue
logger.info(f"Processing {package.name}/{filename}...")
if dry_run:
logger.info(f" [DRY RUN] Would extract dependencies from {filename}")
artifacts_processed += 1
continue
# Download the artifact from S3
try:
content = storage.get(artifact.s3_key)
except Exception as e:
logger.error(f" Failed to download {filename}: {e}")
continue
# Extract dependencies
deps = extract_dependencies(content, filename)
if deps:
logger.info(f" Found {len(deps)} dependencies")
for dep_name, dep_version in deps:
# Check if already exists (race condition protection)
existing = db.query(ArtifactDependency).filter(
ArtifactDependency.artifact_id == tag.artifact_id,
ArtifactDependency.dependency_project == "_pypi",
ArtifactDependency.dependency_package == dep_name,
).first()
if not existing:
dep = ArtifactDependency(
artifact_id=tag.artifact_id,
dependency_project="_pypi",
dependency_package=dep_name,
version_constraint=dep_version if dep_version else "*",
)
db.add(dep)
dependencies_added += 1
logger.info(f" + {dep_name} {dep_version or '*'}")
db.commit()
else:
logger.info(f" No dependencies found")
artifacts_processed += 1
logger.info("")
logger.info("=" * 50)
logger.info("Backfill complete!")
logger.info(f" Total artifacts: {total_artifacts}")
logger.info(f" Already had deps: {artifacts_with_deps}")
logger.info(f" Processed: {artifacts_processed}")
logger.info(f" Dependencies added: {dependencies_added}")
if dry_run:
logger.info(" (DRY RUN - no changes made)")
finally:
db.close()
def main():
parser = argparse.ArgumentParser(
description="Backfill dependencies for cached PyPI packages"
)
parser.add_argument(
"--dry-run",
action="store_true",
help="Preview what would be done without making changes",
)
args = parser.parse_args()
backfill_dependencies(dry_run=args.dry_run)
if __name__ == "__main__":
main()

View File

@@ -9,6 +9,37 @@ This module provides:
import os
import pytest
# =============================================================================
# Pytest Markers
# =============================================================================
def pytest_configure(config):
"""Register custom pytest markers."""
config.addinivalue_line(
"markers",
"auth_intensive: marks tests that make many login requests (excluded from CI integration tests due to rate limiting)",
)
config.addinivalue_line(
"markers",
"integration: marks tests as integration tests",
)
config.addinivalue_line(
"markers",
"large: marks tests that handle large files (slow)",
)
config.addinivalue_line(
"markers",
"slow: marks tests as slow running",
)
config.addinivalue_line(
"markers",
"requires_direct_s3: marks tests that require direct S3/MinIO access (skipped in CI where S3 is not directly accessible)",
)
import io
from typing import Generator
from unittest.mock import MagicMock
@@ -25,6 +56,26 @@ os.environ.setdefault("ORCHARD_S3_BUCKET", "test-bucket")
os.environ.setdefault("ORCHARD_S3_ACCESS_KEY_ID", "test")
os.environ.setdefault("ORCHARD_S3_SECRET_ACCESS_KEY", "test")
# =============================================================================
# Admin Credentials Helper
# =============================================================================
def get_admin_password() -> str:
"""Get the admin password for test authentication.
Returns the password from ORCHARD_TEST_PASSWORD environment variable,
or 'changeme123' as the default for local development.
"""
return os.environ.get("ORCHARD_TEST_PASSWORD", "changeme123")
def get_admin_username() -> str:
"""Get the admin username for test authentication."""
return os.environ.get("ORCHARD_TEST_USERNAME", "admin")
# Re-export factory functions for backward compatibility
from tests.factories import (
create_test_file,
@@ -32,6 +83,8 @@ from tests.factories import (
compute_md5,
compute_sha1,
upload_test_file,
generate_content,
generate_content_with_hash,
TEST_CONTENT_HELLO,
TEST_HASH_HELLO,
TEST_MD5_HELLO,
@@ -179,29 +232,64 @@ def test_app():
# =============================================================================
@pytest.fixture
@pytest.fixture(scope="session")
def integration_client():
"""
Create an authenticated test client for integration tests.
Uses the real database and MinIO from docker-compose.local.yml.
Authenticates as admin for write operations.
Uses the real database and MinIO from docker-compose.local.yml or deployed environment.
Authenticates as admin for write operations. Session-scoped to reuse login across tests.
Environment variables:
ORCHARD_TEST_URL: Base URL of the Orchard server (default: http://localhost:8080)
ORCHARD_TEST_USERNAME: Admin username for authentication (default: admin)
ORCHARD_TEST_PASSWORD: Admin password for authentication (default: changeme123)
"""
from httpx import Client
import httpx
# Connect to the running orchard-server container
# Connect to the running orchard-server container or deployed environment
base_url = os.environ.get("ORCHARD_TEST_URL", "http://localhost:8080")
username = os.environ.get("ORCHARD_TEST_USERNAME", "admin")
password = os.environ.get("ORCHARD_TEST_PASSWORD", "changeme123")
with Client(base_url=base_url, timeout=30.0) as client:
with httpx.Client(base_url=base_url, timeout=30.0) as client:
# Login as admin to enable write operations
login_response = client.post(
"/api/v1/auth/login",
json={"username": "admin", "password": "changeme123"},
json={"username": username, "password": password},
)
# If login fails, tests will fail - that's expected if auth is broken
if login_response.status_code != 200:
# Try to continue without auth for backward compatibility
pass
pytest.fail(
f"Authentication failed against {base_url}: {login_response.status_code} - {login_response.text}. "
f"Set ORCHARD_TEST_USERNAME and ORCHARD_TEST_PASSWORD environment variables if using non-default credentials."
)
# Verify cookie was set
if not client.cookies:
pytest.fail(
f"Login succeeded but no session cookie was set. Response headers: {login_response.headers}"
)
yield client
@pytest.fixture
def auth_client():
"""
Create a function-scoped test client for authentication tests.
Unlike integration_client (session-scoped), this creates a fresh client
for each test. Use this for tests that manipulate authentication state
(login, logout, cookie clearing) to avoid polluting other tests.
Environment variables:
ORCHARD_TEST_URL: Base URL of the Orchard server (default: http://localhost:8080)
"""
import httpx
base_url = os.environ.get("ORCHARD_TEST_URL", "http://localhost:8080")
with httpx.Client(base_url=base_url, timeout=30.0) as client:
yield client
@@ -271,3 +359,41 @@ def test_content():
content = f"test-content-{uuid.uuid4().hex}".encode()
sha256 = compute_sha256(content)
return (content, sha256)
@pytest.fixture
def sized_content():
"""
Factory fixture for generating content of specific sizes.
Usage:
def test_example(sized_content):
content, hash = sized_content(1024) # 1KB
content, hash = sized_content(1024 * 1024) # 1MB
"""
def _generate(size: int, seed: int = None):
return generate_content_with_hash(size, seed)
return _generate
# =============================================================================
# Size Constants for Tests
# =============================================================================
# Common file sizes for boundary testing
SIZE_1B = 1
SIZE_1KB = 1024
SIZE_10KB = 10 * 1024
SIZE_100KB = 100 * 1024
SIZE_1MB = 1024 * 1024
SIZE_5MB = 5 * 1024 * 1024
SIZE_10MB = 10 * 1024 * 1024
SIZE_50MB = 50 * 1024 * 1024
SIZE_100MB = 100 * 1024 * 1024
SIZE_250MB = 250 * 1024 * 1024
SIZE_500MB = 500 * 1024 * 1024
SIZE_1GB = 1024 * 1024 * 1024
# Chunk size boundaries (based on typical S3 multipart chunk sizes)
CHUNK_SIZE = 64 * 1024 # 64KB typical chunk
MULTIPART_THRESHOLD = 100 * 1024 * 1024 # 100MB multipart threshold

View File

@@ -96,7 +96,7 @@ def upload_test_file(
package: str,
content: bytes,
filename: str = "test.bin",
tag: Optional[str] = None,
version: Optional[str] = None,
) -> dict:
"""
Helper function to upload a test file via the API.
@@ -107,15 +107,15 @@ def upload_test_file(
package: Package name
content: File content as bytes
filename: Original filename
tag: Optional tag to assign
version: Optional version to assign
Returns:
The upload response as a dict
"""
files = {"file": (filename, io.BytesIO(content), "application/octet-stream")}
data = {}
if tag:
data["tag"] = tag
if version:
data["version"] = version
response = client.post(
f"/api/v1/project/{project}/{package}/upload",
@@ -126,6 +126,41 @@ def upload_test_file(
return response.json()
def generate_content(size: int, seed: Optional[int] = None) -> bytes:
"""
Generate deterministic or random content of a specified size.
Args:
size: Size of content in bytes
seed: Optional seed for reproducible content (None for random)
Returns:
Bytes of the specified size
"""
if size == 0:
return b""
if seed is not None:
import random
rng = random.Random(seed)
return bytes(rng.randint(0, 255) for _ in range(size))
return os.urandom(size)
def generate_content_with_hash(size: int, seed: Optional[int] = None) -> tuple[bytes, str]:
"""
Generate content of specified size and compute its SHA256 hash.
Args:
size: Size of content in bytes
seed: Optional seed for reproducible content
Returns:
Tuple of (content_bytes, sha256_hash)
"""
content = generate_content(size, seed)
return content, compute_sha256(content)
# =============================================================================
# Project/Package Factories
# =============================================================================

View File

@@ -25,7 +25,7 @@ class TestArtifactRetrieval:
expected_hash = compute_sha256(content)
upload_test_file(
integration_client, project_name, package_name, content, tag="v1"
integration_client, project_name, package_name, content, version="v1"
)
response = integration_client.get(f"/api/v1/artifact/{expected_hash}")
@@ -46,27 +46,27 @@ class TestArtifactRetrieval:
assert response.status_code == 404
@pytest.mark.integration
def test_artifact_includes_tags(self, integration_client, test_package):
"""Test artifact response includes tags pointing to it."""
def test_artifact_includes_versions(self, integration_client, test_package):
"""Test artifact response includes versions pointing to it."""
project_name, package_name = test_package
content = b"artifact with tags test"
content = b"artifact with versions test"
expected_hash = compute_sha256(content)
upload_test_file(
integration_client, project_name, package_name, content, tag="tagged-v1"
integration_client, project_name, package_name, content, version="1.0.0"
)
response = integration_client.get(f"/api/v1/artifact/{expected_hash}")
assert response.status_code == 200
data = response.json()
assert "tags" in data
assert len(data["tags"]) >= 1
assert "versions" in data
assert len(data["versions"]) >= 1
tag = data["tags"][0]
assert "name" in tag
assert "package_name" in tag
assert "project_name" in tag
version = data["versions"][0]
assert "version" in version
assert "package_name" in version
assert "project_name" in version
class TestArtifactStats:
@@ -82,7 +82,7 @@ class TestArtifactStats:
expected_hash = compute_sha256(content)
upload_test_file(
integration_client, project, package, content, tag=f"art-{unique_test_id}"
integration_client, project, package, content, version=f"art-{unique_test_id}"
)
response = integration_client.get(f"/api/v1/artifact/{expected_hash}/stats")
@@ -94,7 +94,7 @@ class TestArtifactStats:
assert "size" in data
assert "ref_count" in data
assert "storage_savings" in data
assert "tags" in data
assert "versions" in data
assert "projects" in data
assert "packages" in data
@@ -136,8 +136,8 @@ class TestArtifactStats:
)
# Upload same content to both projects
upload_test_file(integration_client, proj1, "pkg", content, tag="v1")
upload_test_file(integration_client, proj2, "pkg", content, tag="v1")
upload_test_file(integration_client, proj1, "pkg", content, version="v1")
upload_test_file(integration_client, proj2, "pkg", content, version="v1")
# Check artifact stats
response = integration_client.get(f"/api/v1/artifact/{expected_hash}/stats")
@@ -203,7 +203,7 @@ class TestArtifactProvenance:
assert "first_uploaded_by" in data
assert "upload_count" in data
assert "packages" in data
assert "tags" in data
assert "versions" in data
assert "uploads" in data
@pytest.mark.integration
@@ -214,17 +214,17 @@ class TestArtifactProvenance:
assert response.status_code == 404
@pytest.mark.integration
def test_artifact_history_with_tag(self, integration_client, test_package):
"""Test artifact history includes tag information when tagged."""
def test_artifact_history_with_version(self, integration_client, test_package):
"""Test artifact history includes version information when versioned."""
project_name, package_name = test_package
upload_result = upload_test_file(
integration_client,
project_name,
package_name,
b"tagged provenance test",
"tagged.txt",
tag="v1.0.0",
b"versioned provenance test",
"versioned.txt",
version="v1.0.0",
)
artifact_id = upload_result["artifact_id"]
@@ -232,12 +232,12 @@ class TestArtifactProvenance:
assert response.status_code == 200
data = response.json()
assert len(data["tags"]) >= 1
assert len(data["versions"]) >= 1
tag = data["tags"][0]
assert "project_name" in tag
assert "package_name" in tag
assert "tag_name" in tag
version = data["versions"][0]
assert "project_name" in version
assert "package_name" in version
assert "version" in version
class TestArtifactUploads:
@@ -306,24 +306,24 @@ class TestOrphanedArtifacts:
assert len(response.json()) <= 5
@pytest.mark.integration
def test_artifact_becomes_orphaned_when_tag_deleted(
def test_artifact_becomes_orphaned_when_version_deleted(
self, integration_client, test_package, unique_test_id
):
"""Test artifact appears in orphaned list after tag is deleted."""
"""Test artifact appears in orphaned list after version is deleted."""
project, package = test_package
content = f"orphan test {unique_test_id}".encode()
expected_hash = compute_sha256(content)
# Upload with tag
upload_test_file(integration_client, project, package, content, tag="temp-tag")
# Upload with version
upload_test_file(integration_client, project, package, content, version="1.0.0-temp")
# Verify not in orphaned list
response = integration_client.get("/api/v1/admin/orphaned-artifacts?limit=1000")
orphaned_ids = [a["id"] for a in response.json()]
assert expected_hash not in orphaned_ids
# Delete the tag
integration_client.delete(f"/api/v1/project/{project}/{package}/tags/temp-tag")
# Delete the version
integration_client.delete(f"/api/v1/project/{project}/{package}/versions/1.0.0-temp")
# Verify now in orphaned list
response = integration_client.get("/api/v1/admin/orphaned-artifacts?limit=1000")
@@ -356,9 +356,9 @@ class TestGarbageCollection:
content = f"dry run test {unique_test_id}".encode()
expected_hash = compute_sha256(content)
# Upload and delete tag to create orphan
upload_test_file(integration_client, project, package, content, tag="dry-run")
integration_client.delete(f"/api/v1/project/{project}/{package}/tags/dry-run")
# Upload and delete version to create orphan
upload_test_file(integration_client, project, package, content, version="1.0.0-dryrun")
integration_client.delete(f"/api/v1/project/{project}/{package}/versions/1.0.0-dryrun")
# Verify artifact exists
response = integration_client.get(f"/api/v1/artifact/{expected_hash}")
@@ -385,7 +385,7 @@ class TestGarbageCollection:
expected_hash = compute_sha256(content)
# Upload with tag (ref_count=1)
upload_test_file(integration_client, project, package, content, tag="keep-this")
upload_test_file(integration_client, project, package, content, version="keep-this")
# Verify artifact exists with ref_count=1
response = integration_client.get(f"/api/v1/artifact/{expected_hash}")
@@ -534,50 +534,6 @@ class TestGlobalArtifacts:
assert response.status_code == 400
class TestGlobalTags:
"""Tests for global tags endpoint."""
@pytest.mark.integration
def test_global_tags_returns_200(self, integration_client):
"""Test global tags endpoint returns 200."""
response = integration_client.get("/api/v1/tags")
assert response.status_code == 200
data = response.json()
assert "items" in data
assert "pagination" in data
@pytest.mark.integration
def test_global_tags_pagination(self, integration_client):
"""Test global tags endpoint respects pagination."""
response = integration_client.get("/api/v1/tags?limit=5&page=1")
assert response.status_code == 200
data = response.json()
assert len(data["items"]) <= 5
assert data["pagination"]["limit"] == 5
@pytest.mark.integration
def test_global_tags_has_project_context(self, integration_client):
"""Test global tags response includes project/package context."""
response = integration_client.get("/api/v1/tags?limit=1")
assert response.status_code == 200
data = response.json()
if len(data["items"]) > 0:
item = data["items"][0]
assert "project_name" in item
assert "package_name" in item
assert "artifact_id" in item
@pytest.mark.integration
def test_global_tags_search_with_wildcard(self, integration_client):
"""Test global tags search supports wildcards."""
response = integration_client.get("/api/v1/tags?search=v*")
assert response.status_code == 200
# Just verify it doesn't error; results may vary
class TestAuditLogs:
"""Tests for global audit logs endpoint."""

View File

@@ -1,39 +1,50 @@
"""Integration tests for authentication API endpoints."""
"""Integration tests for authentication API endpoints.
Note: These tests are marked as auth_intensive because they make many login
requests. Dev/stage deployments have relaxed rate limits (1000/minute) to
allow these tests to run. Production uses strict rate limits (5/minute).
"""
import pytest
from uuid import uuid4
from tests.conftest import get_admin_password, get_admin_username
# Mark all tests in this module as auth_intensive (informational, not excluded from CI)
pytestmark = pytest.mark.auth_intensive
class TestAuthLogin:
"""Tests for login endpoint."""
@pytest.mark.integration
def test_login_success(self, integration_client):
def test_login_success(self, auth_client):
"""Test successful login with default admin credentials."""
response = integration_client.post(
response = auth_client.post(
"/api/v1/auth/login",
json={"username": "admin", "password": "changeme123"},
json={"username": get_admin_username(), "password": get_admin_password()},
)
assert response.status_code == 200
data = response.json()
assert data["username"] == "admin"
assert data["username"] == get_admin_username()
assert data["is_admin"] is True
assert "orchard_session" in response.cookies
@pytest.mark.integration
def test_login_invalid_password(self, integration_client):
def test_login_invalid_password(self, auth_client):
"""Test login with wrong password."""
response = integration_client.post(
response = auth_client.post(
"/api/v1/auth/login",
json={"username": "admin", "password": "wrongpassword"},
json={"username": get_admin_username(), "password": "wrongpassword"},
)
assert response.status_code == 401
assert "Invalid username or password" in response.json()["detail"]
@pytest.mark.integration
def test_login_nonexistent_user(self, integration_client):
def test_login_nonexistent_user(self, auth_client):
"""Test login with non-existent user."""
response = integration_client.post(
response = auth_client.post(
"/api/v1/auth/login",
json={"username": "nonexistent", "password": "password"},
)
@@ -44,24 +55,24 @@ class TestAuthLogout:
"""Tests for logout endpoint."""
@pytest.mark.integration
def test_logout_success(self, integration_client):
def test_logout_success(self, auth_client):
"""Test successful logout."""
# First login
login_response = integration_client.post(
login_response = auth_client.post(
"/api/v1/auth/login",
json={"username": "admin", "password": "changeme123"},
json={"username": get_admin_username(), "password": get_admin_password()},
)
assert login_response.status_code == 200
# Then logout
logout_response = integration_client.post("/api/v1/auth/logout")
logout_response = auth_client.post("/api/v1/auth/logout")
assert logout_response.status_code == 200
assert "Logged out successfully" in logout_response.json()["message"]
@pytest.mark.integration
def test_logout_without_session(self, integration_client):
def test_logout_without_session(self, auth_client):
"""Test logout without being logged in."""
response = integration_client.post("/api/v1/auth/logout")
response = auth_client.post("/api/v1/auth/logout")
# Should succeed even without session
assert response.status_code == 200
@@ -70,84 +81,105 @@ class TestAuthMe:
"""Tests for get current user endpoint."""
@pytest.mark.integration
def test_get_me_authenticated(self, integration_client):
def test_get_me_authenticated(self, auth_client):
"""Test getting current user when authenticated."""
# Login first
integration_client.post(
auth_client.post(
"/api/v1/auth/login",
json={"username": "admin", "password": "changeme123"},
json={"username": get_admin_username(), "password": get_admin_password()},
)
response = integration_client.get("/api/v1/auth/me")
response = auth_client.get("/api/v1/auth/me")
assert response.status_code == 200
data = response.json()
assert data["username"] == "admin"
assert data["username"] == get_admin_username()
assert data["is_admin"] is True
assert "id" in data
assert "created_at" in data
@pytest.mark.integration
def test_get_me_unauthenticated(self, integration_client):
def test_get_me_unauthenticated(self, auth_client):
"""Test getting current user without authentication."""
# Clear any existing cookies
integration_client.cookies.clear()
auth_client.cookies.clear()
response = integration_client.get("/api/v1/auth/me")
response = auth_client.get("/api/v1/auth/me")
assert response.status_code == 401
assert "Not authenticated" in response.json()["detail"]
class TestAuthChangePassword:
"""Tests for change password endpoint."""
"""Tests for change password endpoint.
Note: These tests use dedicated test users instead of admin to avoid
invalidating the integration_client session (which uses admin).
"""
@pytest.mark.integration
def test_change_password_success(self, integration_client):
def test_change_password_success(self, auth_client):
"""Test successful password change."""
# Login first
integration_client.post(
# Login as admin to create a test user
auth_client.post(
"/api/v1/auth/login",
json={"username": "admin", "password": "changeme123"},
json={"username": get_admin_username(), "password": get_admin_password()},
)
test_username = f"pwchange_{uuid4().hex[:8]}"
auth_client.post(
"/api/v1/admin/users",
json={"username": test_username, "password": "oldpassword123"},
)
# Login as test user
auth_client.cookies.clear()
auth_client.post(
"/api/v1/auth/login",
json={"username": test_username, "password": "oldpassword123"},
)
# Change password
response = integration_client.post(
response = auth_client.post(
"/api/v1/auth/change-password",
json={"current_password": "changeme123", "new_password": "newpassword123"},
json={"current_password": "oldpassword123", "new_password": "newpassword123"},
)
assert response.status_code == 200
# Verify old password no longer works
integration_client.cookies.clear()
response = integration_client.post(
auth_client.cookies.clear()
response = auth_client.post(
"/api/v1/auth/login",
json={"username": "admin", "password": "changeme123"},
json={"username": test_username, "password": "oldpassword123"},
)
assert response.status_code == 401
# Verify new password works
response = integration_client.post(
response = auth_client.post(
"/api/v1/auth/login",
json={"username": "admin", "password": "newpassword123"},
json={"username": test_username, "password": "newpassword123"},
)
assert response.status_code == 200
# Reset password back to original for other tests
reset_response = integration_client.post(
"/api/v1/auth/change-password",
json={"current_password": "newpassword123", "new_password": "changeme123"},
)
assert reset_response.status_code == 200, "Failed to reset admin password back to default"
@pytest.mark.integration
def test_change_password_wrong_current(self, integration_client):
def test_change_password_wrong_current(self, auth_client):
"""Test password change with wrong current password."""
# Login first
integration_client.post(
# Login as admin to create a test user
auth_client.post(
"/api/v1/auth/login",
json={"username": "admin", "password": "changeme123"},
json={"username": get_admin_username(), "password": get_admin_password()},
)
test_username = f"pwwrong_{uuid4().hex[:8]}"
auth_client.post(
"/api/v1/admin/users",
json={"username": test_username, "password": "password123"},
)
response = integration_client.post(
# Login as test user
auth_client.cookies.clear()
auth_client.post(
"/api/v1/auth/login",
json={"username": test_username, "password": "password123"},
)
response = auth_client.post(
"/api/v1/auth/change-password",
json={"current_password": "wrongpassword", "new_password": "newpassword"},
)
@@ -159,16 +191,16 @@ class TestAPIKeys:
"""Tests for API key management endpoints."""
@pytest.mark.integration
def test_create_and_list_api_key(self, integration_client):
def test_create_and_list_api_key(self, auth_client):
"""Test creating and listing API keys."""
# Login first
integration_client.post(
auth_client.post(
"/api/v1/auth/login",
json={"username": "admin", "password": "changeme123"},
json={"username": get_admin_username(), "password": get_admin_password()},
)
# Create API key
create_response = integration_client.post(
create_response = auth_client.post(
"/api/v1/auth/keys",
json={"name": "test-key", "description": "Test API key"},
)
@@ -182,23 +214,23 @@ class TestAPIKeys:
api_key = data["key"]
# List API keys
list_response = integration_client.get("/api/v1/auth/keys")
list_response = auth_client.get("/api/v1/auth/keys")
assert list_response.status_code == 200
keys = list_response.json()
assert any(k["id"] == key_id for k in keys)
# Clean up - delete the key
integration_client.delete(f"/api/v1/auth/keys/{key_id}")
auth_client.delete(f"/api/v1/auth/keys/{key_id}")
@pytest.mark.integration
def test_use_api_key_for_auth(self, integration_client):
def test_use_api_key_for_auth(self, auth_client):
"""Test using API key for authentication."""
# Login and create API key
integration_client.post(
auth_client.post(
"/api/v1/auth/login",
json={"username": "admin", "password": "changeme123"},
json={"username": get_admin_username(), "password": get_admin_password()},
)
create_response = integration_client.post(
create_response = auth_client.post(
"/api/v1/auth/keys",
json={"name": "auth-test-key"},
)
@@ -206,30 +238,30 @@ class TestAPIKeys:
key_id = create_response.json()["id"]
# Clear cookies and use API key
integration_client.cookies.clear()
response = integration_client.get(
auth_client.cookies.clear()
response = auth_client.get(
"/api/v1/auth/me",
headers={"Authorization": f"Bearer {api_key}"},
)
assert response.status_code == 200
assert response.json()["username"] == "admin"
assert response.json()["username"] == get_admin_username()
# Clean up
integration_client.post(
auth_client.post(
"/api/v1/auth/login",
json={"username": "admin", "password": "changeme123"},
json={"username": get_admin_username(), "password": get_admin_password()},
)
integration_client.delete(f"/api/v1/auth/keys/{key_id}")
auth_client.delete(f"/api/v1/auth/keys/{key_id}")
@pytest.mark.integration
def test_delete_api_key(self, integration_client):
def test_delete_api_key(self, auth_client):
"""Test revoking an API key."""
# Login and create API key
integration_client.post(
auth_client.post(
"/api/v1/auth/login",
json={"username": "admin", "password": "changeme123"},
json={"username": get_admin_username(), "password": get_admin_password()},
)
create_response = integration_client.post(
create_response = auth_client.post(
"/api/v1/auth/keys",
json={"name": "delete-test-key"},
)
@@ -237,12 +269,12 @@ class TestAPIKeys:
api_key = create_response.json()["key"]
# Delete the key
delete_response = integration_client.delete(f"/api/v1/auth/keys/{key_id}")
delete_response = auth_client.delete(f"/api/v1/auth/keys/{key_id}")
assert delete_response.status_code == 200
# Verify key no longer works
integration_client.cookies.clear()
response = integration_client.get(
auth_client.cookies.clear()
response = auth_client.get(
"/api/v1/auth/me",
headers={"Authorization": f"Bearer {api_key}"},
)
@@ -253,32 +285,32 @@ class TestAdminUserManagement:
"""Tests for admin user management endpoints."""
@pytest.mark.integration
def test_list_users(self, integration_client):
def test_list_users(self, auth_client):
"""Test listing users as admin."""
# Login as admin
integration_client.post(
auth_client.post(
"/api/v1/auth/login",
json={"username": "admin", "password": "changeme123"},
json={"username": get_admin_username(), "password": get_admin_password()},
)
response = integration_client.get("/api/v1/admin/users")
response = auth_client.get("/api/v1/admin/users")
assert response.status_code == 200
users = response.json()
assert len(users) >= 1
assert any(u["username"] == "admin" for u in users)
assert any(u["username"] == get_admin_username() for u in users)
@pytest.mark.integration
def test_create_user(self, integration_client):
def test_create_user(self, auth_client):
"""Test creating a new user as admin."""
# Login as admin
integration_client.post(
auth_client.post(
"/api/v1/auth/login",
json={"username": "admin", "password": "changeme123"},
json={"username": get_admin_username(), "password": get_admin_password()},
)
# Create new user
test_username = f"testuser_{uuid4().hex[:8]}"
response = integration_client.post(
response = auth_client.post(
"/api/v1/admin/users",
json={
"username": test_username,
@@ -293,31 +325,31 @@ class TestAdminUserManagement:
assert data["is_admin"] is False
# Verify new user can login
integration_client.cookies.clear()
login_response = integration_client.post(
auth_client.cookies.clear()
login_response = auth_client.post(
"/api/v1/auth/login",
json={"username": test_username, "password": "testpassword"},
)
assert login_response.status_code == 200
@pytest.mark.integration
def test_update_user(self, integration_client):
def test_update_user(self, auth_client):
"""Test updating a user as admin."""
# Login as admin
integration_client.post(
auth_client.post(
"/api/v1/auth/login",
json={"username": "admin", "password": "changeme123"},
json={"username": get_admin_username(), "password": get_admin_password()},
)
# Create a test user
test_username = f"updateuser_{uuid4().hex[:8]}"
integration_client.post(
auth_client.post(
"/api/v1/admin/users",
json={"username": test_username, "password": "password"},
)
# Update the user
response = integration_client.put(
response = auth_client.put(
f"/api/v1/admin/users/{test_username}",
json={"email": "updated@example.com", "is_admin": True},
)
@@ -327,59 +359,59 @@ class TestAdminUserManagement:
assert data["is_admin"] is True
@pytest.mark.integration
def test_reset_user_password(self, integration_client):
def test_reset_user_password(self, auth_client):
"""Test resetting a user's password as admin."""
# Login as admin
integration_client.post(
auth_client.post(
"/api/v1/auth/login",
json={"username": "admin", "password": "changeme123"},
json={"username": get_admin_username(), "password": get_admin_password()},
)
# Create a test user
test_username = f"resetuser_{uuid4().hex[:8]}"
integration_client.post(
auth_client.post(
"/api/v1/admin/users",
json={"username": test_username, "password": "oldpassword"},
)
# Reset password
response = integration_client.post(
response = auth_client.post(
f"/api/v1/admin/users/{test_username}/reset-password",
json={"new_password": "newpassword"},
)
assert response.status_code == 200
# Verify new password works
integration_client.cookies.clear()
login_response = integration_client.post(
auth_client.cookies.clear()
login_response = auth_client.post(
"/api/v1/auth/login",
json={"username": test_username, "password": "newpassword"},
)
assert login_response.status_code == 200
@pytest.mark.integration
def test_non_admin_cannot_access_admin_endpoints(self, integration_client):
def test_non_admin_cannot_access_admin_endpoints(self, auth_client):
"""Test that non-admin users cannot access admin endpoints."""
# Login as admin and create non-admin user
integration_client.post(
auth_client.post(
"/api/v1/auth/login",
json={"username": "admin", "password": "changeme123"},
json={"username": get_admin_username(), "password": get_admin_password()},
)
test_username = f"nonadmin_{uuid4().hex[:8]}"
integration_client.post(
auth_client.post(
"/api/v1/admin/users",
json={"username": test_username, "password": "password", "is_admin": False},
)
# Login as non-admin
integration_client.cookies.clear()
integration_client.post(
auth_client.cookies.clear()
auth_client.post(
"/api/v1/auth/login",
json={"username": test_username, "password": "password"},
)
# Try to access admin endpoints
response = integration_client.get("/api/v1/admin/users")
response = auth_client.get("/api/v1/admin/users")
assert response.status_code == 403
assert "Admin privileges required" in response.json()["detail"]
@@ -388,28 +420,28 @@ class TestSecurityEdgeCases:
"""Tests for security edge cases and validation."""
@pytest.mark.integration
def test_login_inactive_user(self, integration_client):
def test_login_inactive_user(self, auth_client):
"""Test that inactive users cannot login."""
# Login as admin and create a user
integration_client.post(
auth_client.post(
"/api/v1/auth/login",
json={"username": "admin", "password": "changeme123"},
json={"username": get_admin_username(), "password": get_admin_password()},
)
test_username = f"inactive_{uuid4().hex[:8]}"
integration_client.post(
auth_client.post(
"/api/v1/admin/users",
json={"username": test_username, "password": "password123"},
)
# Deactivate the user
integration_client.put(
auth_client.put(
f"/api/v1/admin/users/{test_username}",
json={"is_active": False},
)
# Try to login as inactive user
integration_client.cookies.clear()
response = integration_client.post(
auth_client.cookies.clear()
response = auth_client.post(
"/api/v1/auth/login",
json={"username": test_username, "password": "password123"},
)
@@ -417,14 +449,14 @@ class TestSecurityEdgeCases:
assert "Invalid username or password" in response.json()["detail"]
@pytest.mark.integration
def test_password_too_short_on_create(self, integration_client):
def test_password_too_short_on_create(self, auth_client):
"""Test that short passwords are rejected when creating users."""
integration_client.post(
auth_client.post(
"/api/v1/auth/login",
json={"username": "admin", "password": "changeme123"},
json={"username": get_admin_username(), "password": get_admin_password()},
)
response = integration_client.post(
response = auth_client.post(
"/api/v1/admin/users",
json={"username": f"shortpw_{uuid4().hex[:8]}", "password": "short"},
)
@@ -432,36 +464,49 @@ class TestSecurityEdgeCases:
assert "at least 8 characters" in response.json()["detail"]
@pytest.mark.integration
def test_password_too_short_on_change(self, integration_client):
def test_password_too_short_on_change(self, auth_client):
"""Test that short passwords are rejected when changing password."""
integration_client.post(
# Create test user
auth_client.post(
"/api/v1/auth/login",
json={"username": "admin", "password": "changeme123"},
json={"username": get_admin_username(), "password": get_admin_password()},
)
test_username = f"shortchange_{uuid4().hex[:8]}"
auth_client.post(
"/api/v1/admin/users",
json={"username": test_username, "password": "password123"},
)
response = integration_client.post(
# Login as test user
auth_client.cookies.clear()
auth_client.post(
"/api/v1/auth/login",
json={"username": test_username, "password": "password123"},
)
response = auth_client.post(
"/api/v1/auth/change-password",
json={"current_password": "changeme123", "new_password": "short"},
json={"current_password": "password123", "new_password": "short"},
)
assert response.status_code == 400
assert "at least 8 characters" in response.json()["detail"]
@pytest.mark.integration
def test_password_too_short_on_reset(self, integration_client):
def test_password_too_short_on_reset(self, auth_client):
"""Test that short passwords are rejected when resetting password."""
integration_client.post(
auth_client.post(
"/api/v1/auth/login",
json={"username": "admin", "password": "changeme123"},
json={"username": get_admin_username(), "password": get_admin_password()},
)
# Create a test user first
test_username = f"resetshort_{uuid4().hex[:8]}"
integration_client.post(
auth_client.post(
"/api/v1/admin/users",
json={"username": test_username, "password": "password123"},
)
response = integration_client.post(
response = auth_client.post(
f"/api/v1/admin/users/{test_username}/reset-password",
json={"new_password": "short"},
)
@@ -469,23 +514,23 @@ class TestSecurityEdgeCases:
assert "at least 8 characters" in response.json()["detail"]
@pytest.mark.integration
def test_duplicate_username_rejected(self, integration_client):
def test_duplicate_username_rejected(self, auth_client):
"""Test that duplicate usernames are rejected."""
integration_client.post(
auth_client.post(
"/api/v1/auth/login",
json={"username": "admin", "password": "changeme123"},
json={"username": get_admin_username(), "password": get_admin_password()},
)
test_username = f"duplicate_{uuid4().hex[:8]}"
# Create user first time
response1 = integration_client.post(
response1 = auth_client.post(
"/api/v1/admin/users",
json={"username": test_username, "password": "password123"},
)
assert response1.status_code == 200
# Try to create same username again
response2 = integration_client.post(
response2 = auth_client.post(
"/api/v1/admin/users",
json={"username": test_username, "password": "password456"},
)
@@ -493,14 +538,14 @@ class TestSecurityEdgeCases:
assert "already exists" in response2.json()["detail"]
@pytest.mark.integration
def test_cannot_delete_other_users_api_key(self, integration_client):
def test_cannot_delete_other_users_api_key(self, auth_client):
"""Test that users cannot delete API keys owned by other users."""
# Login as admin and create an API key
integration_client.post(
auth_client.post(
"/api/v1/auth/login",
json={"username": "admin", "password": "changeme123"},
json={"username": get_admin_username(), "password": get_admin_password()},
)
create_response = integration_client.post(
create_response = auth_client.post(
"/api/v1/auth/keys",
json={"name": "admin-key"},
)
@@ -508,253 +553,65 @@ class TestSecurityEdgeCases:
# Create a non-admin user
test_username = f"nonadmin_{uuid4().hex[:8]}"
integration_client.post(
auth_client.post(
"/api/v1/admin/users",
json={"username": test_username, "password": "password123"},
)
# Login as non-admin
integration_client.cookies.clear()
integration_client.post(
auth_client.cookies.clear()
auth_client.post(
"/api/v1/auth/login",
json={"username": test_username, "password": "password123"},
)
# Try to delete admin's API key
response = integration_client.delete(f"/api/v1/auth/keys/{admin_key_id}")
response = auth_client.delete(f"/api/v1/auth/keys/{admin_key_id}")
assert response.status_code == 403
assert "Cannot delete another user's API key" in response.json()["detail"]
# Cleanup: login as admin and delete the key
integration_client.cookies.clear()
integration_client.post(
auth_client.cookies.clear()
auth_client.post(
"/api/v1/auth/login",
json={"username": "admin", "password": "changeme123"},
json={"username": get_admin_username(), "password": get_admin_password()},
)
integration_client.delete(f"/api/v1/auth/keys/{admin_key_id}")
auth_client.delete(f"/api/v1/auth/keys/{admin_key_id}")
@pytest.mark.integration
def test_sessions_invalidated_on_password_change(self, integration_client):
def test_sessions_invalidated_on_password_change(self, auth_client):
"""Test that all sessions are invalidated when password is changed."""
# Create a test user
integration_client.post(
auth_client.post(
"/api/v1/auth/login",
json={"username": "admin", "password": "changeme123"},
json={"username": get_admin_username(), "password": get_admin_password()},
)
test_username = f"sessiontest_{uuid4().hex[:8]}"
integration_client.post(
auth_client.post(
"/api/v1/admin/users",
json={"username": test_username, "password": "password123"},
)
# Login as test user
integration_client.cookies.clear()
login_response = integration_client.post(
auth_client.cookies.clear()
login_response = auth_client.post(
"/api/v1/auth/login",
json={"username": test_username, "password": "password123"},
)
assert login_response.status_code == 200
# Verify session works
me_response = integration_client.get("/api/v1/auth/me")
me_response = auth_client.get("/api/v1/auth/me")
assert me_response.status_code == 200
# Change password
integration_client.post(
auth_client.post(
"/api/v1/auth/change-password",
json={"current_password": "password123", "new_password": "newpassword123"},
)
# Old session should be invalidated - try to access /me
# (note: the change-password call itself may have cleared the session cookie)
me_response2 = integration_client.get("/api/v1/auth/me")
# This should fail because all sessions were invalidated
assert me_response2.status_code == 401
class TestSecurityEdgeCases:
"""Tests for security edge cases and validation."""
@pytest.mark.integration
def test_login_inactive_user(self, integration_client):
"""Test that inactive users cannot login."""
# Login as admin and create a user
integration_client.post(
"/api/v1/auth/login",
json={"username": "admin", "password": "changeme123"},
)
test_username = f"inactive_{uuid4().hex[:8]}"
integration_client.post(
"/api/v1/admin/users",
json={"username": test_username, "password": "password123"},
)
# Deactivate the user
integration_client.put(
f"/api/v1/admin/users/{test_username}",
json={"is_active": False},
)
# Try to login as inactive user
integration_client.cookies.clear()
response = integration_client.post(
"/api/v1/auth/login",
json={"username": test_username, "password": "password123"},
)
assert response.status_code == 401
assert "Invalid username or password" in response.json()["detail"]
@pytest.mark.integration
def test_password_too_short_on_create(self, integration_client):
"""Test that short passwords are rejected when creating users."""
integration_client.post(
"/api/v1/auth/login",
json={"username": "admin", "password": "changeme123"},
)
response = integration_client.post(
"/api/v1/admin/users",
json={"username": f"shortpw_{uuid4().hex[:8]}", "password": "short"},
)
assert response.status_code == 400
assert "at least 8 characters" in response.json()["detail"]
@pytest.mark.integration
def test_password_too_short_on_change(self, integration_client):
"""Test that short passwords are rejected when changing password."""
integration_client.post(
"/api/v1/auth/login",
json={"username": "admin", "password": "changeme123"},
)
response = integration_client.post(
"/api/v1/auth/change-password",
json={"current_password": "changeme123", "new_password": "short"},
)
assert response.status_code == 400
assert "at least 8 characters" in response.json()["detail"]
@pytest.mark.integration
def test_password_too_short_on_reset(self, integration_client):
"""Test that short passwords are rejected when resetting password."""
integration_client.post(
"/api/v1/auth/login",
json={"username": "admin", "password": "changeme123"},
)
# Create a test user first
test_username = f"resetshort_{uuid4().hex[:8]}"
integration_client.post(
"/api/v1/admin/users",
json={"username": test_username, "password": "password123"},
)
response = integration_client.post(
f"/api/v1/admin/users/{test_username}/reset-password",
json={"new_password": "short"},
)
assert response.status_code == 400
assert "at least 8 characters" in response.json()["detail"]
@pytest.mark.integration
def test_duplicate_username_rejected(self, integration_client):
"""Test that duplicate usernames are rejected."""
integration_client.post(
"/api/v1/auth/login",
json={"username": "admin", "password": "changeme123"},
)
test_username = f"duplicate_{uuid4().hex[:8]}"
# Create user first time
response1 = integration_client.post(
"/api/v1/admin/users",
json={"username": test_username, "password": "password123"},
)
assert response1.status_code == 200
# Try to create same username again
response2 = integration_client.post(
"/api/v1/admin/users",
json={"username": test_username, "password": "password456"},
)
assert response2.status_code == 409
assert "already exists" in response2.json()["detail"]
@pytest.mark.integration
def test_cannot_delete_other_users_api_key(self, integration_client):
"""Test that users cannot delete API keys owned by other users."""
# Login as admin and create an API key
integration_client.post(
"/api/v1/auth/login",
json={"username": "admin", "password": "changeme123"},
)
create_response = integration_client.post(
"/api/v1/auth/keys",
json={"name": "admin-key"},
)
admin_key_id = create_response.json()["id"]
# Create a non-admin user
test_username = f"nonadmin_{uuid4().hex[:8]}"
integration_client.post(
"/api/v1/admin/users",
json={"username": test_username, "password": "password123"},
)
# Login as non-admin
integration_client.cookies.clear()
integration_client.post(
"/api/v1/auth/login",
json={"username": test_username, "password": "password123"},
)
# Try to delete admin's API key
response = integration_client.delete(f"/api/v1/auth/keys/{admin_key_id}")
assert response.status_code == 403
assert "Cannot delete another user's API key" in response.json()["detail"]
# Cleanup: login as admin and delete the key
integration_client.cookies.clear()
integration_client.post(
"/api/v1/auth/login",
json={"username": "admin", "password": "changeme123"},
)
integration_client.delete(f"/api/v1/auth/keys/{admin_key_id}")
@pytest.mark.integration
def test_sessions_invalidated_on_password_change(self, integration_client):
"""Test that all sessions are invalidated when password is changed."""
# Create a test user
integration_client.post(
"/api/v1/auth/login",
json={"username": "admin", "password": "changeme123"},
)
test_username = f"sessiontest_{uuid4().hex[:8]}"
integration_client.post(
"/api/v1/admin/users",
json={"username": test_username, "password": "password123"},
)
# Login as test user
integration_client.cookies.clear()
login_response = integration_client.post(
"/api/v1/auth/login",
json={"username": test_username, "password": "password123"},
)
assert login_response.status_code == 200
# Verify session works
me_response = integration_client.get("/api/v1/auth/me")
assert me_response.status_code == 200
# Change password
integration_client.post(
"/api/v1/auth/change-password",
json={"current_password": "password123", "new_password": "newpassword123"},
)
# Old session should be invalidated - try to access /me
# (note: the change-password call itself may have cleared the session cookie)
me_response2 = integration_client.get("/api/v1/auth/me")
me_response2 = auth_client.get("/api/v1/auth/me")
# This should fail because all sessions were invalidated
assert me_response2.status_code == 401

View File

@@ -0,0 +1,759 @@
"""
Integration tests for concurrent upload and download operations.
Tests cover:
- Concurrent uploads of different files
- Concurrent uploads of same file (deduplication race)
- Concurrent downloads of same artifact
- Concurrent downloads of different artifacts
- Mixed concurrent uploads and downloads
- Data corruption prevention under concurrency
"""
import pytest
import io
import os
from concurrent.futures import ThreadPoolExecutor, as_completed
from tests.factories import (
compute_sha256,
upload_test_file,
generate_content_with_hash,
)
def get_api_key(integration_client):
"""Create an API key for concurrent test workers."""
import uuid
response = integration_client.post(
"/api/v1/auth/keys",
json={"name": f"concurrent-test-{uuid.uuid4().hex[:8]}"},
)
if response.status_code == 200:
return response.json()["key"]
return None
class TestConcurrentUploads:
"""Tests for concurrent upload operations."""
@pytest.mark.integration
@pytest.mark.concurrent
def test_2_concurrent_uploads_different_files(self, integration_client, test_package):
"""Test 2 concurrent uploads of different files."""
project, package = test_package
api_key = get_api_key(integration_client)
assert api_key, "Failed to create API key"
files_data = [
generate_content_with_hash(1024, seed=i) for i in range(2)
]
results = []
errors = []
def upload_worker(idx, content, expected_hash):
try:
from httpx import Client
base_url = os.environ.get("ORCHARD_TEST_URL", "http://localhost:8080")
with Client(base_url=base_url, timeout=60.0) as client:
files = {
"file": (f"file-{idx}.bin", io.BytesIO(content), "application/octet-stream")
}
response = client.post(
f"/api/v1/project/{project}/{package}/upload",
files=files,
data={"version": f"concurrent-{idx}"},
headers={"Authorization": f"Bearer {api_key}"},
)
if response.status_code == 200:
result = response.json()
results.append((idx, result, expected_hash))
else:
errors.append(f"Worker {idx}: Status {response.status_code}: {response.text}")
except Exception as e:
errors.append(f"Worker {idx}: {str(e)}")
with ThreadPoolExecutor(max_workers=2) as executor:
futures = [
executor.submit(upload_worker, i, content, hash)
for i, (content, hash) in enumerate(files_data)
]
for future in as_completed(futures):
pass
assert len(errors) == 0, f"Errors: {errors}"
assert len(results) == 2
# Verify each upload returned correct artifact_id
for idx, result, expected_hash in results:
assert result["artifact_id"] == expected_hash
@pytest.mark.integration
@pytest.mark.concurrent
def test_5_concurrent_uploads_different_files(self, integration_client, test_package):
"""Test 5 concurrent uploads of different files."""
project, package = test_package
api_key = get_api_key(integration_client)
assert api_key, "Failed to create API key"
num_files = 5
files_data = [
generate_content_with_hash(2048, seed=100 + i) for i in range(num_files)
]
results = []
errors = []
def upload_worker(idx, content, expected_hash):
try:
from httpx import Client
base_url = os.environ.get("ORCHARD_TEST_URL", "http://localhost:8080")
with Client(base_url=base_url, timeout=60.0) as client:
files = {
"file": (f"file-{idx}.bin", io.BytesIO(content), "application/octet-stream")
}
response = client.post(
f"/api/v1/project/{project}/{package}/upload",
files=files,
data={"version": f"concurrent5-{idx}"},
headers={"Authorization": f"Bearer {api_key}"},
)
if response.status_code == 200:
result = response.json()
results.append((idx, result, expected_hash))
else:
errors.append(f"Worker {idx}: Status {response.status_code}")
except Exception as e:
errors.append(f"Worker {idx}: {str(e)}")
with ThreadPoolExecutor(max_workers=num_files) as executor:
futures = [
executor.submit(upload_worker, i, content, hash)
for i, (content, hash) in enumerate(files_data)
]
for future in as_completed(futures):
pass
assert len(errors) == 0, f"Errors: {errors}"
assert len(results) == num_files
# Verify all uploads have unique artifact_ids
artifact_ids = set(r[1]["artifact_id"] for r in results)
assert len(artifact_ids) == num_files
@pytest.mark.integration
@pytest.mark.concurrent
def test_10_concurrent_uploads_different_files(self, integration_client, test_package):
"""Test 10 concurrent uploads of different files."""
project, package = test_package
api_key = get_api_key(integration_client)
assert api_key, "Failed to create API key"
num_files = 10
files_data = [
generate_content_with_hash(1024, seed=200 + i) for i in range(num_files)
]
results = []
errors = []
def upload_worker(idx, content, expected_hash):
try:
from httpx import Client
base_url = os.environ.get("ORCHARD_TEST_URL", "http://localhost:8080")
with Client(base_url=base_url, timeout=60.0) as client:
files = {
"file": (f"file-{idx}.bin", io.BytesIO(content), "application/octet-stream")
}
response = client.post(
f"/api/v1/project/{project}/{package}/upload",
files=files,
data={"version": f"concurrent10-{idx}"},
headers={"Authorization": f"Bearer {api_key}"},
)
if response.status_code == 200:
result = response.json()
results.append((idx, result, expected_hash))
else:
errors.append(f"Worker {idx}: Status {response.status_code}")
except Exception as e:
errors.append(f"Worker {idx}: {str(e)}")
with ThreadPoolExecutor(max_workers=num_files) as executor:
futures = [
executor.submit(upload_worker, i, content, hash)
for i, (content, hash) in enumerate(files_data)
]
for future in as_completed(futures):
pass
assert len(errors) == 0, f"Errors: {errors}"
assert len(results) == num_files
@pytest.mark.integration
@pytest.mark.concurrent
def test_concurrent_uploads_same_file_deduplication(
self, integration_client, test_project, unique_test_id
):
"""Test concurrent uploads of same file handle deduplication correctly.
Same content uploaded to different packages should result in:
- Same artifact_id (content-addressable)
- ref_count = number of packages (one version per package)
"""
project = test_project
api_key = get_api_key(integration_client)
assert api_key, "Failed to create API key"
num_concurrent = 5
package_names = []
# Create multiple packages for concurrent uploads
for i in range(num_concurrent):
pkg_name = f"dedup-pkg-{unique_test_id}-{i}"
response = integration_client.post(
f"/api/v1/project/{project}/packages",
json={"name": pkg_name, "description": f"Dedup test package {i}"},
)
assert response.status_code == 200
package_names.append(pkg_name)
content, expected_hash = generate_content_with_hash(4096, seed=999)
results = []
errors = []
def upload_worker(idx, package):
try:
from httpx import Client
base_url = os.environ.get("ORCHARD_TEST_URL", "http://localhost:8080")
with Client(base_url=base_url, timeout=60.0) as client:
files = {
"file": (f"same-{idx}.bin", io.BytesIO(content), "application/octet-stream")
}
response = client.post(
f"/api/v1/project/{project}/{package}/upload",
files=files,
data={"version": "1.0.0"},
headers={"Authorization": f"Bearer {api_key}"},
)
if response.status_code == 200:
results.append(response.json())
else:
errors.append(f"Worker {idx}: Status {response.status_code}")
except Exception as e:
errors.append(f"Worker {idx}: {str(e)}")
with ThreadPoolExecutor(max_workers=num_concurrent) as executor:
futures = [
executor.submit(upload_worker, i, package_names[i])
for i in range(num_concurrent)
]
for future in as_completed(futures):
pass
assert len(errors) == 0, f"Errors: {errors}"
assert len(results) == num_concurrent
# All should have same artifact_id
artifact_ids = set(r["artifact_id"] for r in results)
assert len(artifact_ids) == 1
assert expected_hash in artifact_ids
# Verify final ref_count equals number of packages
response = integration_client.get(f"/api/v1/artifact/{expected_hash}")
assert response.status_code == 200
assert response.json()["ref_count"] == num_concurrent
@pytest.mark.integration
@pytest.mark.concurrent
def test_concurrent_uploads_to_different_packages(self, integration_client, test_project, unique_test_id):
"""Test concurrent uploads to different packages."""
project = test_project
api_key = get_api_key(integration_client)
assert api_key, "Failed to create API key"
num_packages = 3
package_names = []
# Create multiple packages
for i in range(num_packages):
pkg_name = f"pkg-{unique_test_id}-{i}"
response = integration_client.post(
f"/api/v1/project/{project}/packages",
json={"name": pkg_name, "description": f"Package {i}"},
)
assert response.status_code == 200
package_names.append(pkg_name)
files_data = [
generate_content_with_hash(1024, seed=300 + i) for i in range(num_packages)
]
results = []
errors = []
def upload_worker(idx, package, content, expected_hash):
try:
from httpx import Client
base_url = os.environ.get("ORCHARD_TEST_URL", "http://localhost:8080")
with Client(base_url=base_url, timeout=60.0) as client:
files = {
"file": (f"file-{idx}.bin", io.BytesIO(content), "application/octet-stream")
}
response = client.post(
f"/api/v1/project/{project}/{package}/upload",
files=files,
data={"version": "latest"},
headers={"Authorization": f"Bearer {api_key}"},
)
if response.status_code == 200:
result = response.json()
results.append((package, result, expected_hash))
else:
errors.append(f"Worker {idx}: Status {response.status_code}")
except Exception as e:
errors.append(f"Worker {idx}: {str(e)}")
with ThreadPoolExecutor(max_workers=num_packages) as executor:
futures = [
executor.submit(upload_worker, i, package_names[i], content, hash)
for i, (content, hash) in enumerate(files_data)
]
for future in as_completed(futures):
pass
assert len(errors) == 0, f"Errors: {errors}"
assert len(results) == num_packages
class TestConcurrentDownloads:
"""Tests for concurrent download operations."""
@pytest.mark.integration
@pytest.mark.concurrent
def test_2_concurrent_downloads_same_artifact(self, integration_client, test_package):
"""Test 2 concurrent downloads of same artifact."""
project, package = test_package
content, expected_hash = generate_content_with_hash(2048, seed=400)
# Upload first
upload_test_file(integration_client, project, package, content, version="download-test")
results = []
errors = []
def download_worker(idx):
try:
from httpx import Client
base_url = os.environ.get("ORCHARD_TEST_URL", "http://localhost:8080")
with Client(base_url=base_url, timeout=60.0) as client:
response = client.get(
f"/api/v1/project/{project}/{package}/+/download-test",
params={"mode": "proxy"},
)
if response.status_code == 200:
results.append((idx, response.content))
else:
errors.append(f"Worker {idx}: Status {response.status_code}")
except Exception as e:
errors.append(f"Worker {idx}: {str(e)}")
with ThreadPoolExecutor(max_workers=2) as executor:
futures = [executor.submit(download_worker, i) for i in range(2)]
for future in as_completed(futures):
pass
assert len(errors) == 0, f"Errors: {errors}"
assert len(results) == 2
# All downloads should match original
for idx, downloaded in results:
assert downloaded == content
@pytest.mark.integration
@pytest.mark.concurrent
def test_5_concurrent_downloads_same_artifact(self, integration_client, test_package):
"""Test 5 concurrent downloads of same artifact."""
project, package = test_package
content, expected_hash = generate_content_with_hash(4096, seed=500)
upload_test_file(integration_client, project, package, content, version="download5-test")
num_downloads = 5
results = []
errors = []
def download_worker(idx):
try:
from httpx import Client
base_url = os.environ.get("ORCHARD_TEST_URL", "http://localhost:8080")
with Client(base_url=base_url, timeout=60.0) as client:
response = client.get(
f"/api/v1/project/{project}/{package}/+/download5-test",
params={"mode": "proxy"},
)
if response.status_code == 200:
results.append((idx, response.content))
else:
errors.append(f"Worker {idx}: Status {response.status_code}")
except Exception as e:
errors.append(f"Worker {idx}: {str(e)}")
with ThreadPoolExecutor(max_workers=num_downloads) as executor:
futures = [executor.submit(download_worker, i) for i in range(num_downloads)]
for future in as_completed(futures):
pass
assert len(errors) == 0, f"Errors: {errors}"
assert len(results) == num_downloads
for idx, downloaded in results:
assert downloaded == content
@pytest.mark.integration
@pytest.mark.concurrent
def test_10_concurrent_downloads_same_artifact(self, integration_client, test_package):
"""Test 10 concurrent downloads of same artifact."""
project, package = test_package
content, expected_hash = generate_content_with_hash(8192, seed=600)
upload_test_file(integration_client, project, package, content, version="download10-test")
num_downloads = 10
results = []
errors = []
def download_worker(idx):
try:
from httpx import Client
base_url = os.environ.get("ORCHARD_TEST_URL", "http://localhost:8080")
with Client(base_url=base_url, timeout=60.0) as client:
response = client.get(
f"/api/v1/project/{project}/{package}/+/download10-test",
params={"mode": "proxy"},
)
if response.status_code == 200:
results.append((idx, response.content))
else:
errors.append(f"Worker {idx}: Status {response.status_code}")
except Exception as e:
errors.append(f"Worker {idx}: {str(e)}")
with ThreadPoolExecutor(max_workers=num_downloads) as executor:
futures = [executor.submit(download_worker, i) for i in range(num_downloads)]
for future in as_completed(futures):
pass
assert len(errors) == 0, f"Errors: {errors}"
assert len(results) == num_downloads
for idx, downloaded in results:
assert downloaded == content
@pytest.mark.integration
@pytest.mark.concurrent
def test_concurrent_downloads_different_artifacts(self, integration_client, test_package):
"""Test concurrent downloads of different artifacts."""
project, package = test_package
# Upload multiple files
num_files = 5
uploads = []
for i in range(num_files):
content, expected_hash = generate_content_with_hash(1024, seed=700 + i)
upload_test_file(
integration_client, project, package, content,
version=f"multi-download-{i}"
)
uploads.append((f"multi-download-{i}", content))
results = []
errors = []
def download_worker(tag, expected_content):
try:
from httpx import Client
base_url = os.environ.get("ORCHARD_TEST_URL", "http://localhost:8080")
with Client(base_url=base_url, timeout=60.0) as client:
response = client.get(
f"/api/v1/project/{project}/{package}/+/{tag}",
params={"mode": "proxy"},
)
if response.status_code == 200:
results.append((tag, response.content, expected_content))
else:
errors.append(f"Tag {tag}: Status {response.status_code}")
except Exception as e:
errors.append(f"Tag {tag}: {str(e)}")
with ThreadPoolExecutor(max_workers=num_files) as executor:
futures = [
executor.submit(download_worker, tag, content)
for tag, content in uploads
]
for future in as_completed(futures):
pass
assert len(errors) == 0, f"Errors: {errors}"
assert len(results) == num_files
for tag, downloaded, expected in results:
assert downloaded == expected, f"Content mismatch for {tag}"
class TestMixedConcurrentOperations:
"""Tests for mixed concurrent upload and download operations."""
@pytest.mark.integration
@pytest.mark.concurrent
def test_upload_while_download_in_progress(self, integration_client, test_package):
"""Test uploading while a download is in progress."""
project, package = test_package
api_key = get_api_key(integration_client)
assert api_key, "Failed to create API key"
# Upload initial content
content1, hash1 = generate_content_with_hash(10240, seed=800) # 10KB
upload_test_file(integration_client, project, package, content1, version="initial")
# New content for upload during download
content2, hash2 = generate_content_with_hash(10240, seed=801)
results = {"downloads": [], "uploads": []}
errors = []
def download_worker():
try:
from httpx import Client
base_url = os.environ.get("ORCHARD_TEST_URL", "http://localhost:8080")
with Client(base_url=base_url, timeout=60.0) as client:
response = client.get(
f"/api/v1/project/{project}/{package}/+/initial",
params={"mode": "proxy"},
)
if response.status_code == 200:
results["downloads"].append(response.content)
else:
errors.append(f"Download: Status {response.status_code}")
except Exception as e:
errors.append(f"Download: {str(e)}")
def upload_worker():
try:
from httpx import Client
base_url = os.environ.get("ORCHARD_TEST_URL", "http://localhost:8080")
with Client(base_url=base_url, timeout=60.0) as client:
files = {
"file": ("new.bin", io.BytesIO(content2), "application/octet-stream")
}
response = client.post(
f"/api/v1/project/{project}/{package}/upload",
files=files,
data={"version": "during-download"},
headers={"Authorization": f"Bearer {api_key}"},
)
if response.status_code == 200:
results["uploads"].append(response.json())
else:
errors.append(f"Upload: Status {response.status_code}")
except Exception as e:
errors.append(f"Upload: {str(e)}")
with ThreadPoolExecutor(max_workers=2) as executor:
futures = [
executor.submit(download_worker),
executor.submit(upload_worker),
]
for future in as_completed(futures):
pass
assert len(errors) == 0, f"Errors: {errors}"
assert len(results["downloads"]) == 1
assert len(results["uploads"]) == 1
# Verify download got correct content
assert results["downloads"][0] == content1
# Verify upload succeeded
assert results["uploads"][0]["artifact_id"] == hash2
@pytest.mark.integration
@pytest.mark.concurrent
def test_multiple_uploads_and_downloads_simultaneously(self, integration_client, test_package):
"""Test multiple uploads and downloads running simultaneously."""
project, package = test_package
api_key = get_api_key(integration_client)
assert api_key, "Failed to create API key"
# Pre-upload some files for downloading
existing_files = []
for i in range(3):
content, hash = generate_content_with_hash(2048, seed=900 + i)
upload_test_file(integration_client, project, package, content, version=f"existing-{i}")
existing_files.append((f"existing-{i}", content))
# New files for uploading
new_files = [
generate_content_with_hash(2048, seed=910 + i) for i in range(3)
]
results = {"downloads": [], "uploads": []}
errors = []
def download_worker(tag, expected):
try:
from httpx import Client
base_url = os.environ.get("ORCHARD_TEST_URL", "http://localhost:8080")
with Client(base_url=base_url, timeout=60.0) as client:
response = client.get(
f"/api/v1/project/{project}/{package}/+/{tag}",
params={"mode": "proxy"},
)
if response.status_code == 200:
results["downloads"].append((tag, response.content, expected))
else:
errors.append(f"Download {tag}: Status {response.status_code}")
except Exception as e:
errors.append(f"Download {tag}: {str(e)}")
def upload_worker(idx, content, expected_hash):
try:
from httpx import Client
base_url = os.environ.get("ORCHARD_TEST_URL", "http://localhost:8080")
with Client(base_url=base_url, timeout=60.0) as client:
files = {
"file": (f"new-{idx}.bin", io.BytesIO(content), "application/octet-stream")
}
response = client.post(
f"/api/v1/project/{project}/{package}/upload",
files=files,
data={"version": f"new-{idx}"},
headers={"Authorization": f"Bearer {api_key}"},
)
if response.status_code == 200:
results["uploads"].append((idx, response.json(), expected_hash))
else:
errors.append(f"Upload {idx}: Status {response.status_code}")
except Exception as e:
errors.append(f"Upload {idx}: {str(e)}")
with ThreadPoolExecutor(max_workers=6) as executor:
futures = []
# Submit downloads
for tag, content in existing_files:
futures.append(executor.submit(download_worker, tag, content))
# Submit uploads
for i, (content, hash) in enumerate(new_files):
futures.append(executor.submit(upload_worker, i, content, hash))
for future in as_completed(futures):
pass
assert len(errors) == 0, f"Errors: {errors}"
assert len(results["downloads"]) == 3
assert len(results["uploads"]) == 3
# Verify downloads
for tag, downloaded, expected in results["downloads"]:
assert downloaded == expected, f"Download mismatch for {tag}"
# Verify uploads
for idx, result, expected_hash in results["uploads"]:
assert result["artifact_id"] == expected_hash
@pytest.mark.integration
@pytest.mark.concurrent
def test_no_data_corruption_under_concurrency(self, integration_client, test_package):
"""Test that no data corruption occurs under concurrent operations."""
project, package = test_package
api_key = get_api_key(integration_client)
assert api_key, "Failed to create API key"
# Create content with recognizable patterns
num_files = 5
files_data = []
for i in range(num_files):
# Each file has unique repeating pattern for easy corruption detection
pattern = bytes([i] * 256)
content = pattern * 40 # 10KB each
hash = compute_sha256(content)
files_data.append((content, hash))
results = []
errors = []
def upload_and_verify(idx, content, expected_hash):
try:
from httpx import Client
base_url = os.environ.get("ORCHARD_TEST_URL", "http://localhost:8080")
with Client(base_url=base_url, timeout=60.0) as client:
# Upload
files = {
"file": (f"pattern-{idx}.bin", io.BytesIO(content), "application/octet-stream")
}
upload_resp = client.post(
f"/api/v1/project/{project}/{package}/upload",
files=files,
data={"version": f"pattern-{idx}"},
headers={"Authorization": f"Bearer {api_key}"},
)
if upload_resp.status_code != 200:
errors.append(f"Upload {idx}: Status {upload_resp.status_code}")
return
upload_result = upload_resp.json()
if upload_result["artifact_id"] != expected_hash:
errors.append(f"Upload {idx}: Hash mismatch")
return
# Immediately download and verify
download_resp = client.get(
f"/api/v1/project/{project}/{package}/+/pattern-{idx}",
params={"mode": "proxy"},
)
if download_resp.status_code != 200:
errors.append(f"Download {idx}: Status {download_resp.status_code}")
return
if download_resp.content != content:
errors.append(f"Worker {idx}: DATA CORRUPTION DETECTED")
return
# Verify the downloaded content hash
downloaded_hash = compute_sha256(download_resp.content)
if downloaded_hash != expected_hash:
errors.append(f"Worker {idx}: Hash verification failed")
return
results.append(idx)
except Exception as e:
errors.append(f"Worker {idx}: {str(e)}")
with ThreadPoolExecutor(max_workers=num_files) as executor:
futures = [
executor.submit(upload_and_verify, i, content, hash)
for i, (content, hash) in enumerate(files_data)
]
for future in as_completed(futures):
pass
assert len(errors) == 0, f"Errors: {errors}"
assert len(results) == num_files

View File

@@ -0,0 +1,322 @@
"""
Integration tests for error handling in upload and download operations.
Tests cover:
- Timeout handling
- Invalid request handling
- Resource cleanup on failures
- Graceful error responses
"""
import pytest
import io
import time
from tests.factories import (
compute_sha256,
upload_test_file,
generate_content_with_hash,
)
class TestUploadErrorHandling:
"""Tests for upload error handling."""
@pytest.mark.integration
def test_upload_to_nonexistent_project_returns_404(
self, integration_client, unique_test_id
):
"""Test upload to nonexistent project returns 404."""
content = b"test content for nonexistent project"
files = {"file": ("test.bin", io.BytesIO(content), "application/octet-stream")}
response = integration_client.post(
f"/api/v1/project/nonexistent-project-{unique_test_id}/nonexistent-pkg/upload",
files=files,
)
assert response.status_code == 404
@pytest.mark.integration
def test_upload_to_nonexistent_package_returns_404(
self, integration_client, test_project, unique_test_id
):
"""Test upload to nonexistent package returns 404."""
content = b"test content for nonexistent package"
files = {"file": ("test.bin", io.BytesIO(content), "application/octet-stream")}
response = integration_client.post(
f"/api/v1/project/{test_project}/nonexistent-package-{unique_test_id}/upload",
files=files,
)
assert response.status_code == 404
@pytest.mark.integration
def test_upload_empty_file_rejected(self, integration_client, test_package):
"""Test empty file upload is rejected."""
project, package = test_package
files = {"file": ("empty.bin", io.BytesIO(b""), "application/octet-stream")}
response = integration_client.post(
f"/api/v1/project/{project}/{package}/upload",
files=files,
)
assert response.status_code in [400, 422]
@pytest.mark.integration
def test_upload_missing_file_returns_422(self, integration_client, test_package):
"""Test upload without file field returns 422."""
project, package = test_package
response = integration_client.post(
f"/api/v1/project/{project}/{package}/upload",
data={"version": "no-file-provided"},
)
assert response.status_code == 422
@pytest.mark.integration
def test_upload_invalid_checksum_format_returns_400(
self, integration_client, test_package
):
"""Test upload with invalid checksum format returns 400."""
project, package = test_package
content = b"checksum format test"
files = {"file": ("test.bin", io.BytesIO(content), "application/octet-stream")}
response = integration_client.post(
f"/api/v1/project/{project}/{package}/upload",
files=files,
headers={"X-Checksum-SHA256": "invalid-hash-format"},
)
assert response.status_code == 400
@pytest.mark.integration
def test_upload_checksum_mismatch_returns_422(
self, integration_client, test_package
):
"""Test upload with mismatched checksum returns 422."""
project, package = test_package
content = b"checksum mismatch test"
wrong_hash = "0" * 64 # Valid format but wrong hash
files = {"file": ("test.bin", io.BytesIO(content), "application/octet-stream")}
response = integration_client.post(
f"/api/v1/project/{project}/{package}/upload",
files=files,
headers={"X-Checksum-SHA256": wrong_hash},
)
assert response.status_code == 422
@pytest.mark.integration
def test_upload_with_correct_checksum_succeeds(
self, integration_client, test_package
):
"""Test upload with correct checksum succeeds."""
project, package = test_package
content = b"correct checksum test"
correct_hash = compute_sha256(content)
files = {"file": ("test.bin", io.BytesIO(content), "application/octet-stream")}
response = integration_client.post(
f"/api/v1/project/{project}/{package}/upload",
files=files,
headers={"X-Checksum-SHA256": correct_hash},
)
assert response.status_code == 200
assert response.json()["artifact_id"] == correct_hash
class TestDownloadErrorHandling:
"""Tests for download error handling."""
@pytest.mark.integration
def test_download_nonexistent_tag_returns_404(
self, integration_client, test_package
):
"""Test download of nonexistent tag returns 404."""
project, package = test_package
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/nonexistent-tag-xyz"
)
assert response.status_code == 404
@pytest.mark.integration
def test_download_nonexistent_artifact_returns_404(
self, integration_client, test_package
):
"""Test download of nonexistent artifact ID returns 404."""
project, package = test_package
fake_hash = "a" * 64
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/artifact:{fake_hash}"
)
assert response.status_code == 404
@pytest.mark.integration
def test_download_invalid_artifact_id_format(
self, integration_client, test_package
):
"""Test download with invalid artifact ID format."""
project, package = test_package
# Too short
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/artifact:abc123"
)
assert response.status_code == 404
@pytest.mark.integration
def test_download_from_nonexistent_project_returns_404(
self, integration_client, unique_test_id
):
"""Test download from nonexistent project returns 404."""
response = integration_client.get(
f"/api/v1/project/nonexistent-{unique_test_id}/pkg/+/tag"
)
assert response.status_code == 404
@pytest.mark.integration
def test_download_from_nonexistent_package_returns_404(
self, integration_client, test_project, unique_test_id
):
"""Test download from nonexistent package returns 404."""
response = integration_client.get(
f"/api/v1/project/{test_project}/nonexistent-{unique_test_id}/+/tag"
)
assert response.status_code == 404
class TestTimeoutBehavior:
"""Tests for timeout behavior (integration level)."""
@pytest.mark.integration
@pytest.mark.slow
def test_large_upload_completes_within_reasonable_time(
self, integration_client, test_package, sized_content
):
"""Test that a 10MB upload completes within reasonable time."""
project, package = test_package
content, expected_hash = sized_content(10 * 1024 * 1024, seed=999) # 10MB
start_time = time.time()
result = upload_test_file(
integration_client, project, package, content, version="timeout-test"
)
elapsed = time.time() - start_time
assert result["artifact_id"] == expected_hash
# Should complete within 60 seconds for 10MB on local docker
assert elapsed < 60, f"Upload took too long: {elapsed:.2f}s"
@pytest.mark.integration
@pytest.mark.slow
def test_large_download_completes_within_reasonable_time(
self, integration_client, test_package, sized_content
):
"""Test that a 10MB download completes within reasonable time."""
project, package = test_package
content, expected_hash = sized_content(10 * 1024 * 1024, seed=998) # 10MB
# First upload
upload_test_file(
integration_client, project, package, content, version="download-timeout-test"
)
# Then download and time it
start_time = time.time()
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/download-timeout-test",
params={"mode": "proxy"},
)
elapsed = time.time() - start_time
assert response.status_code == 200
assert len(response.content) == len(content)
# Should complete within 60 seconds for 10MB on local docker
assert elapsed < 60, f"Download took too long: {elapsed:.2f}s"
class TestResourceCleanup:
"""Tests for proper resource cleanup on failures.
Note: More comprehensive cleanup tests are in test_upload_download_api.py
(TestUploadFailureCleanup class) including S3 object cleanup verification.
"""
@pytest.mark.integration
def test_checksum_mismatch_no_orphaned_artifact(
self, integration_client, test_package, unique_test_id
):
"""Test checksum mismatch doesn't leave orphaned artifact."""
project, package = test_package
# Use unique content to ensure artifact doesn't exist from prior tests
content = f"checksum mismatch orphan test {unique_test_id}".encode()
wrong_hash = "0" * 64
actual_hash = compute_sha256(content)
# Verify artifact doesn't exist before test
pre_check = integration_client.get(f"/api/v1/artifact/{actual_hash}")
assert pre_check.status_code == 404, "Artifact should not exist before test"
files = {"file": ("test.bin", io.BytesIO(content), "application/octet-stream")}
response = integration_client.post(
f"/api/v1/project/{project}/{package}/upload",
files=files,
headers={"X-Checksum-SHA256": wrong_hash},
)
assert response.status_code == 422
# Verify no artifact was created with either hash
response1 = integration_client.get(f"/api/v1/artifact/{wrong_hash}")
response2 = integration_client.get(f"/api/v1/artifact/{actual_hash}")
assert response1.status_code == 404
assert response2.status_code == 404
class TestGracefulErrorResponses:
"""Tests for graceful and informative error responses."""
@pytest.mark.integration
def test_404_response_has_detail_message(
self, integration_client, test_package
):
"""Test 404 responses include a detail message."""
project, package = test_package
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/nonexistent-tag"
)
assert response.status_code == 404
data = response.json()
assert "detail" in data
assert len(data["detail"]) > 0
@pytest.mark.integration
def test_422_response_has_detail_message(self, integration_client, test_package):
"""Test 422 responses include a detail message."""
project, package = test_package
# Upload with mismatched checksum
content = b"detail message test"
wrong_hash = "0" * 64
files = {"file": ("test.bin", io.BytesIO(content), "application/octet-stream")}
response = integration_client.post(
f"/api/v1/project/{project}/{package}/upload",
files=files,
headers={"X-Checksum-SHA256": wrong_hash},
)
assert response.status_code == 422
data = response.json()
assert "detail" in data
@pytest.mark.integration
def test_error_response_is_json(self, integration_client, unique_test_id):
"""Test error responses are valid JSON."""
response = integration_client.get(
f"/api/v1/project/nonexistent-{unique_test_id}/pkg/+/tag"
)
assert response.status_code == 404
# Should not raise exception - valid JSON
data = response.json()
assert isinstance(data, dict)

View File

@@ -0,0 +1,775 @@
"""
Integration tests for artifact integrity verification.
Tests cover:
- Round-trip verification (upload -> download -> verify hash)
- Consistency check endpoint
- Header-based verification
- Integrity verification across file sizes
- Client-side verification workflow
"""
import pytest
import io
import hashlib
from tests.factories import (
compute_sha256,
upload_test_file,
generate_content_with_hash,
s3_object_exists,
get_s3_client,
get_s3_bucket,
)
from tests.conftest import (
SIZE_1KB,
SIZE_10KB,
SIZE_100KB,
SIZE_1MB,
SIZE_10MB,
)
class TestRoundTripVerification:
"""Tests for complete round-trip integrity verification."""
@pytest.mark.integration
def test_upload_download_hash_matches(self, integration_client, test_package):
"""Test that upload -> download round trip preserves content integrity."""
project, package = test_package
content = b"Round trip integrity test content"
expected_hash = compute_sha256(content)
# Upload and capture returned hash
result = upload_test_file(
integration_client, project, package, content, version="roundtrip"
)
uploaded_hash = result["artifact_id"]
# Verify upload returned correct hash
assert uploaded_hash == expected_hash
# Download artifact
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/roundtrip",
params={"mode": "proxy"},
)
assert response.status_code == 200
# Compute hash of downloaded content
downloaded_hash = compute_sha256(response.content)
# All three hashes should match
assert downloaded_hash == expected_hash
assert downloaded_hash == uploaded_hash
@pytest.mark.integration
def test_upload_response_contains_hash(self, integration_client, test_package):
"""Test upload response contains artifact_id which is the SHA256 hash."""
project, package = test_package
content = b"Upload response hash test"
expected_hash = compute_sha256(content)
result = upload_test_file(integration_client, project, package, content)
assert "artifact_id" in result
assert result["artifact_id"] == expected_hash
assert len(result["artifact_id"]) == 64
assert all(c in "0123456789abcdef" for c in result["artifact_id"])
@pytest.mark.integration
def test_download_header_matches_artifact_id(self, integration_client, test_package):
"""Test X-Checksum-SHA256 header matches artifact ID."""
project, package = test_package
content = b"Header verification test"
expected_hash = compute_sha256(content)
upload_test_file(
integration_client, project, package, content, version="header-check"
)
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/header-check",
params={"mode": "proxy"},
)
assert response.status_code == 200
assert response.headers.get("X-Checksum-SHA256") == expected_hash
@pytest.mark.integration
def test_etag_matches_artifact_id(self, integration_client, test_package):
"""Test ETag header matches artifact ID."""
project, package = test_package
content = b"ETag verification test"
expected_hash = compute_sha256(content)
upload_test_file(
integration_client, project, package, content, version="etag-check"
)
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/etag-check",
params={"mode": "proxy"},
)
assert response.status_code == 200
etag = response.headers.get("ETag", "").strip('"')
assert etag == expected_hash
@pytest.mark.integration
def test_artifact_endpoint_returns_correct_hash(self, integration_client, test_package):
"""Test artifact endpoint returns correct hash/ID."""
project, package = test_package
content = b"Artifact endpoint hash test"
expected_hash = compute_sha256(content)
upload_test_file(integration_client, project, package, content)
# Query artifact directly
response = integration_client.get(f"/api/v1/artifact/{expected_hash}")
assert response.status_code == 200
data = response.json()
assert data["id"] == expected_hash
assert data.get("sha256") == expected_hash
class TestClientSideVerificationWorkflow:
"""Tests for client-side verification workflow."""
@pytest.mark.integration
def test_client_can_verify_before_upload(self, integration_client, test_package):
"""Test client can compute hash before upload and verify response matches."""
project, package = test_package
content = b"Client pre-upload verification test"
# Client computes hash locally before upload
client_hash = compute_sha256(content)
# Upload
result = upload_test_file(integration_client, project, package, content)
# Client verifies server returned the same hash
assert result["artifact_id"] == client_hash
@pytest.mark.integration
def test_client_can_provide_checksum_header(self, integration_client, test_package):
"""Test client can provide X-Checksum-SHA256 header for verification."""
project, package = test_package
content = b"Client checksum header test"
client_hash = compute_sha256(content)
files = {"file": ("test.bin", io.BytesIO(content), "application/octet-stream")}
response = integration_client.post(
f"/api/v1/project/{project}/{package}/upload",
files=files,
headers={"X-Checksum-SHA256": client_hash},
)
assert response.status_code == 200
assert response.json()["artifact_id"] == client_hash
@pytest.mark.integration
def test_checksum_mismatch_rejected(self, integration_client, test_package):
"""Test upload with wrong client checksum is rejected."""
project, package = test_package
content = b"Checksum mismatch test"
wrong_hash = "0" * 64
files = {"file": ("test.bin", io.BytesIO(content), "application/octet-stream")}
response = integration_client.post(
f"/api/v1/project/{project}/{package}/upload",
files=files,
headers={"X-Checksum-SHA256": wrong_hash},
)
assert response.status_code == 422
@pytest.mark.integration
def test_client_can_verify_after_download(self, integration_client, test_package):
"""Test client can verify downloaded content matches header hash."""
project, package = test_package
content = b"Client post-download verification"
upload_test_file(
integration_client, project, package, content, version="verify-after"
)
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/verify-after",
params={"mode": "proxy"},
)
assert response.status_code == 200
# Client gets hash from header
header_hash = response.headers.get("X-Checksum-SHA256")
# Client computes hash of downloaded content
downloaded_hash = compute_sha256(response.content)
# Client verifies they match
assert downloaded_hash == header_hash
class TestIntegritySizeVariants:
"""Tests for integrity verification across different file sizes."""
@pytest.mark.integration
def test_integrity_1kb(self, integration_client, test_package, sized_content):
"""Test integrity verification for 1KB file."""
project, package = test_package
content, expected_hash = sized_content(SIZE_1KB, seed=100)
result = upload_test_file(
integration_client, project, package, content, version="int-1kb"
)
assert result["artifact_id"] == expected_hash
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/int-1kb",
params={"mode": "proxy"},
)
assert response.status_code == 200
assert compute_sha256(response.content) == expected_hash
assert response.headers.get("X-Checksum-SHA256") == expected_hash
@pytest.mark.integration
def test_integrity_100kb(self, integration_client, test_package, sized_content):
"""Test integrity verification for 100KB file."""
project, package = test_package
content, expected_hash = sized_content(SIZE_100KB, seed=101)
result = upload_test_file(
integration_client, project, package, content, version="int-100kb"
)
assert result["artifact_id"] == expected_hash
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/int-100kb",
params={"mode": "proxy"},
)
assert response.status_code == 200
assert compute_sha256(response.content) == expected_hash
assert response.headers.get("X-Checksum-SHA256") == expected_hash
@pytest.mark.integration
def test_integrity_1mb(self, integration_client, test_package, sized_content):
"""Test integrity verification for 1MB file."""
project, package = test_package
content, expected_hash = sized_content(SIZE_1MB, seed=102)
result = upload_test_file(
integration_client, project, package, content, version="int-1mb"
)
assert result["artifact_id"] == expected_hash
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/int-1mb",
params={"mode": "proxy"},
)
assert response.status_code == 200
assert compute_sha256(response.content) == expected_hash
assert response.headers.get("X-Checksum-SHA256") == expected_hash
@pytest.mark.integration
@pytest.mark.slow
def test_integrity_10mb(self, integration_client, test_package, sized_content):
"""Test integrity verification for 10MB file."""
project, package = test_package
content, expected_hash = sized_content(SIZE_10MB, seed=103)
result = upload_test_file(
integration_client, project, package, content, version="int-10mb"
)
assert result["artifact_id"] == expected_hash
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/int-10mb",
params={"mode": "proxy"},
)
assert response.status_code == 200
assert compute_sha256(response.content) == expected_hash
assert response.headers.get("X-Checksum-SHA256") == expected_hash
class TestConsistencyCheck:
"""Tests for the admin consistency check endpoint."""
@pytest.mark.integration
def test_consistency_check_returns_200(self, integration_client):
"""Test consistency check endpoint returns 200."""
response = integration_client.get("/api/v1/admin/consistency-check")
assert response.status_code == 200
@pytest.mark.integration
def test_consistency_check_response_format(self, integration_client):
"""Test consistency check returns expected response format."""
response = integration_client.get("/api/v1/admin/consistency-check")
assert response.status_code == 200
data = response.json()
# Check expected fields
assert "total_artifacts_checked" in data
assert "orphaned_s3_objects" in data
assert "missing_s3_objects" in data
assert "size_mismatches" in data
assert "healthy" in data
assert "orphaned_s3_keys" in data
assert "missing_s3_keys" in data
assert "size_mismatch_artifacts" in data
# Verify types
assert isinstance(data["total_artifacts_checked"], int)
assert isinstance(data["orphaned_s3_objects"], int)
assert isinstance(data["missing_s3_objects"], int)
assert isinstance(data["size_mismatches"], int)
assert isinstance(data["healthy"], bool)
assert isinstance(data["orphaned_s3_keys"], list)
assert isinstance(data["missing_s3_keys"], list)
assert isinstance(data["size_mismatch_artifacts"], list)
@pytest.mark.integration
def test_consistency_check_after_upload(self, integration_client, test_package):
"""Test consistency check runs successfully after a valid upload.
Note: We don't assert healthy=True because other tests (especially
corruption detection tests) may leave orphaned S3 objects behind.
This test validates the consistency check endpoint works and the
uploaded artifact is included in the check count.
"""
project, package = test_package
content = b"Consistency check test content"
# Upload artifact
upload_test_file(integration_client, project, package, content)
# Run consistency check
response = integration_client.get("/api/v1/admin/consistency-check")
assert response.status_code == 200
data = response.json()
# Verify check ran - at least 1 artifact was checked
assert data["total_artifacts_checked"] >= 1
# Verify no missing S3 objects (uploaded artifact should exist)
assert data["missing_s3_objects"] == 0
@pytest.mark.integration
def test_consistency_check_limit_parameter(self, integration_client):
"""Test consistency check respects limit parameter."""
response = integration_client.get(
"/api/v1/admin/consistency-check",
params={"limit": 10}
)
assert response.status_code == 200
data = response.json()
# Lists should not exceed limit
assert len(data["orphaned_s3_keys"]) <= 10
assert len(data["missing_s3_keys"]) <= 10
assert len(data["size_mismatch_artifacts"]) <= 10
class TestDigestHeader:
"""Tests for RFC 3230 Digest header."""
@pytest.mark.integration
def test_download_includes_digest_header(self, integration_client, test_package):
"""Test download includes Digest header in RFC 3230 format."""
project, package = test_package
content = b"Digest header test"
expected_hash = compute_sha256(content)
upload_test_file(
integration_client, project, package, content, version="digest-test"
)
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/digest-test",
params={"mode": "proxy"},
)
assert response.status_code == 200
assert "Digest" in response.headers
# Verify Digest format (sha-256=base64hash)
digest = response.headers["Digest"]
assert digest.startswith("sha-256=")
@pytest.mark.integration
def test_digest_header_base64_valid(self, integration_client, test_package):
"""Test Digest header contains valid base64 encoding."""
import base64
project, package = test_package
content = b"Digest base64 test"
expected_hash = compute_sha256(content)
upload_test_file(
integration_client, project, package, content, version="digest-b64"
)
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/digest-b64",
params={"mode": "proxy"},
)
assert response.status_code == 200
digest = response.headers["Digest"]
base64_part = digest.split("=", 1)[1]
# Should be valid base64
try:
decoded = base64.b64decode(base64_part)
assert len(decoded) == 32 # SHA256 is 32 bytes
except Exception as e:
pytest.fail(f"Invalid base64 in Digest header: {e}")
class TestVerificationModes:
"""Tests for download verification modes."""
@pytest.mark.integration
def test_pre_verification_mode(self, integration_client, test_package):
"""Test pre-verification mode verifies before streaming."""
project, package = test_package
content = b"Pre-verification mode test"
upload_test_file(
integration_client, project, package, content, version="pre-verify"
)
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/pre-verify",
params={"mode": "proxy", "verify": "true", "verify_mode": "pre"},
)
assert response.status_code == 200
assert response.content == content
# X-Verified header should be true
assert response.headers.get("X-Verified") == "true"
@pytest.mark.integration
def test_stream_verification_mode(self, integration_client, test_package):
"""Test streaming verification mode."""
project, package = test_package
content = b"Stream verification mode test"
upload_test_file(
integration_client, project, package, content, version="stream-verify"
)
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/stream-verify",
params={"mode": "proxy", "verify": "true", "verify_mode": "stream"},
)
assert response.status_code == 200
assert response.content == content
class TestArtifactIntegrityEndpoint:
"""Tests for artifact-specific integrity operations."""
@pytest.mark.integration
def test_artifact_size_matches(self, integration_client, test_package):
"""Test artifact endpoint returns correct size."""
project, package = test_package
content = b"Artifact size test content"
expected_size = len(content)
result = upload_test_file(integration_client, project, package, content)
artifact_id = result["artifact_id"]
response = integration_client.get(f"/api/v1/artifact/{artifact_id}")
assert response.status_code == 200
data = response.json()
assert data["size"] == expected_size
@pytest.mark.integration
def test_content_length_header_matches_size(self, integration_client, test_package):
"""Test Content-Length header matches artifact size."""
project, package = test_package
content = b"Content-Length header test"
expected_size = len(content)
upload_test_file(
integration_client, project, package, content, version="content-len"
)
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/content-len",
params={"mode": "proxy"},
)
assert response.status_code == 200
assert int(response.headers.get("Content-Length", 0)) == expected_size
assert len(response.content) == expected_size
@pytest.mark.requires_direct_s3
class TestCorruptionDetection:
"""Tests for detecting corrupted S3 objects.
These tests directly manipulate S3 objects to simulate corruption
and verify that the system can detect hash mismatches.
Note: These tests require direct S3/MinIO access and are skipped in CI
where S3 is not directly accessible from the test runner.
"""
@pytest.mark.integration
def test_detection_of_corrupted_content(self, integration_client, test_package):
"""Test that corrupted S3 content is detected via hash mismatch.
Uploads content, then directly modifies the S3 object, then
verifies that the downloaded content hash doesn't match.
"""
project, package = test_package
content = b"Original content for corruption test"
expected_hash = compute_sha256(content)
# Upload original content
result = upload_test_file(
integration_client, project, package, content, version="corrupt-test"
)
assert result["artifact_id"] == expected_hash
# Get the S3 object and corrupt it
s3_client = get_s3_client()
bucket = get_s3_bucket()
s3_key = f"fruits/{expected_hash[:2]}/{expected_hash[2:4]}/{expected_hash}"
# Replace with corrupted content
corrupted_content = b"Corrupted content - different from original!"
s3_client.put_object(Bucket=bucket, Key=s3_key, Body=corrupted_content)
# Download via proxy (bypasses hash verification)
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/corrupt-test",
params={"mode": "proxy"},
)
assert response.status_code == 200
# Verify the downloaded content doesn't match original hash
downloaded_hash = compute_sha256(response.content)
assert downloaded_hash != expected_hash, "Corruption was not detected - hashes match"
assert response.content == corrupted_content
# The X-Checksum-SHA256 header should still show the original hash (from DB)
# but the actual content hash is different
header_hash = response.headers.get("X-Checksum-SHA256")
assert header_hash == expected_hash # Header shows expected hash
assert downloaded_hash != header_hash # But content is corrupted
# Restore original content for cleanup
s3_client.put_object(Bucket=bucket, Key=s3_key, Body=content)
@pytest.mark.integration
def test_detection_of_single_bit_flip(self, integration_client, test_package):
"""Test detection of a single bit flip in S3 object content."""
project, package = test_package
content = b"Content for single bit flip detection test"
expected_hash = compute_sha256(content)
result = upload_test_file(
integration_client, project, package, content, version="bitflip-test"
)
assert result["artifact_id"] == expected_hash
# Get S3 object and flip a single bit
s3_client = get_s3_client()
bucket = get_s3_bucket()
s3_key = f"fruits/{expected_hash[:2]}/{expected_hash[2:4]}/{expected_hash}"
# Flip the first bit of the first byte
corrupted_content = bytearray(content)
corrupted_content[0] ^= 0x01
corrupted_content = bytes(corrupted_content)
s3_client.put_object(Bucket=bucket, Key=s3_key, Body=corrupted_content)
# Download and verify hash mismatch
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/bitflip-test",
params={"mode": "proxy"},
)
assert response.status_code == 200
downloaded_hash = compute_sha256(response.content)
assert downloaded_hash != expected_hash, "Single bit flip not detected"
# Restore original
s3_client.put_object(Bucket=bucket, Key=s3_key, Body=content)
@pytest.mark.integration
def test_detection_of_truncated_content(self, integration_client, test_package):
"""Test detection of truncated S3 object."""
project, package = test_package
content = b"This is content that will be truncated for testing purposes"
expected_hash = compute_sha256(content)
result = upload_test_file(
integration_client, project, package, content, version="truncate-test"
)
assert result["artifact_id"] == expected_hash
# Get S3 object and truncate it
s3_client = get_s3_client()
bucket = get_s3_bucket()
s3_key = f"fruits/{expected_hash[:2]}/{expected_hash[2:4]}/{expected_hash}"
# Truncate to half the original size
truncated_content = content[: len(content) // 2]
s3_client.put_object(Bucket=bucket, Key=s3_key, Body=truncated_content)
# Download and verify hash mismatch
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/truncate-test",
params={"mode": "proxy"},
)
assert response.status_code == 200
downloaded_hash = compute_sha256(response.content)
assert downloaded_hash != expected_hash, "Truncation not detected"
assert len(response.content) < len(content), "Content was not truncated"
# Restore original
s3_client.put_object(Bucket=bucket, Key=s3_key, Body=content)
@pytest.mark.integration
def test_detection_of_appended_content(self, integration_client, test_package):
"""Test detection of content with extra bytes appended."""
project, package = test_package
content = b"Original content"
expected_hash = compute_sha256(content)
result = upload_test_file(
integration_client, project, package, content, version="append-test"
)
assert result["artifact_id"] == expected_hash
# Get S3 object and append extra bytes
s3_client = get_s3_client()
bucket = get_s3_bucket()
s3_key = f"fruits/{expected_hash[:2]}/{expected_hash[2:4]}/{expected_hash}"
appended_content = content + b" - extra bytes appended"
s3_client.put_object(Bucket=bucket, Key=s3_key, Body=appended_content)
# Download and verify hash mismatch
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/append-test",
params={"mode": "proxy"},
)
assert response.status_code == 200
downloaded_hash = compute_sha256(response.content)
assert downloaded_hash != expected_hash, "Appended content not detected"
assert len(response.content) > len(content), "Content was not extended"
# Restore original
s3_client.put_object(Bucket=bucket, Key=s3_key, Body=content)
@pytest.mark.integration
def test_client_detects_hash_mismatch_post_download(
self, integration_client, test_package
):
"""Test that a client can detect hash mismatch after downloading corrupted content.
This simulates the full client verification workflow:
1. Download content
2. Get expected hash from header
3. Compute actual hash of content
4. Verify they match (or detect corruption)
"""
project, package = test_package
content = b"Content for client-side corruption detection"
expected_hash = compute_sha256(content)
result = upload_test_file(
integration_client, project, package, content, version="client-detect"
)
# Corrupt the S3 object
s3_client = get_s3_client()
bucket = get_s3_bucket()
s3_key = f"fruits/{expected_hash[:2]}/{expected_hash[2:4]}/{expected_hash}"
corrupted = b"This is completely different content"
s3_client.put_object(Bucket=bucket, Key=s3_key, Body=corrupted)
# Simulate client download and verification
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/client-detect",
params={"mode": "proxy"},
)
assert response.status_code == 200
# Client gets expected hash from header
header_hash = response.headers.get("X-Checksum-SHA256")
# Client computes hash of downloaded content
actual_hash = compute_sha256(response.content)
# Client detects the mismatch
corruption_detected = actual_hash != header_hash
assert corruption_detected, "Client should detect hash mismatch"
# Restore original
s3_client.put_object(Bucket=bucket, Key=s3_key, Body=content)
@pytest.mark.integration
def test_consistency_check_detects_size_mismatch(
self, integration_client, test_package, unique_test_id
):
"""Test that consistency check detects size mismatches.
Uploads content, modifies S3 object size, then runs consistency check.
"""
project, package = test_package
content = b"Content for size mismatch consistency check test " + unique_test_id.encode()
expected_hash = compute_sha256(content)
result = upload_test_file(
integration_client, project, package, content, version="size-mismatch"
)
# Modify S3 object to have different size
s3_client = get_s3_client()
bucket = get_s3_bucket()
s3_key = f"fruits/{expected_hash[:2]}/{expected_hash[2:4]}/{expected_hash}"
different_size_content = content + b"extra extra extra"
s3_client.put_object(Bucket=bucket, Key=s3_key, Body=different_size_content)
# Run consistency check
response = integration_client.get("/api/v1/admin/consistency-check")
assert response.status_code == 200
data = response.json()
# Should detect the size mismatch
assert data["size_mismatches"] >= 1 or len(data["size_mismatch_artifacts"]) >= 1
# Restore original
s3_client.put_object(Bucket=bucket, Key=s3_key, Body=content)
@pytest.mark.integration
def test_consistency_check_detects_missing_s3_object(
self, integration_client, test_package, unique_test_id
):
"""Test that consistency check detects missing S3 objects.
Uploads content, deletes S3 object, then runs consistency check.
"""
project, package = test_package
content = b"Content for missing S3 object test " + unique_test_id.encode()
expected_hash = compute_sha256(content)
result = upload_test_file(
integration_client, project, package, content, version="missing-s3"
)
# Delete the S3 object
s3_client = get_s3_client()
bucket = get_s3_bucket()
s3_key = f"fruits/{expected_hash[:2]}/{expected_hash[2:4]}/{expected_hash}"
s3_client.delete_object(Bucket=bucket, Key=s3_key)
# Run consistency check
response = integration_client.get("/api/v1/admin/consistency-check")
assert response.status_code == 200
data = response.json()
# Should detect the missing S3 object
assert data["missing_s3_objects"] >= 1 or len(data["missing_s3_keys"]) >= 1
# Restore the object for cleanup
s3_client.put_object(Bucket=bucket, Key=s3_key, Body=content)

View File

@@ -0,0 +1,552 @@
"""
Integration tests for large file upload functionality.
Tests cover:
- Large file uploads (100MB, 1GB)
- Multipart upload behavior
- Upload metrics (duration, throughput)
- Memory efficiency during uploads
- Upload progress tracking
Note: Large tests are marked with @pytest.mark.slow and will be skipped
by default. Run with `pytest --run-slow` to include them.
"""
import os
import pytest
import io
import time
from tests.factories import (
compute_sha256,
upload_test_file,
s3_object_exists,
)
from tests.conftest import (
SIZE_1KB,
SIZE_100KB,
SIZE_1MB,
SIZE_10MB,
SIZE_100MB,
SIZE_1GB,
)
class TestUploadMetrics:
"""Tests for upload duration and throughput metrics."""
@pytest.mark.integration
def test_upload_response_includes_duration_ms(self, integration_client, test_package):
"""Test upload response includes duration_ms field."""
project, package = test_package
content = b"duration test content"
result = upload_test_file(
integration_client, project, package, content, version="duration-test"
)
assert "duration_ms" in result
assert result["duration_ms"] is not None
assert result["duration_ms"] >= 0
@pytest.mark.integration
def test_upload_response_includes_throughput(self, integration_client, test_package):
"""Test upload response includes throughput_mbps field."""
project, package = test_package
content = b"throughput test content"
result = upload_test_file(
integration_client, project, package, content, version="throughput-test"
)
assert "throughput_mbps" in result
# For small files throughput may be very high or None
# Just verify the field exists
@pytest.mark.integration
def test_upload_duration_reasonable(
self, integration_client, test_package, sized_content
):
"""Test upload duration is reasonable for file size."""
project, package = test_package
content, _ = sized_content(SIZE_1MB, seed=100)
start = time.time()
result = upload_test_file(
integration_client, project, package, content, version="duration-check"
)
actual_duration = (time.time() - start) * 1000 # ms
# Reported duration should be close to actual
assert result["duration_ms"] is not None
# Allow some variance (network overhead)
assert result["duration_ms"] <= actual_duration + 1000 # Within 1s
class TestLargeFileUploads:
"""Tests for large file uploads using multipart."""
@pytest.mark.integration
def test_upload_10mb_file(self, integration_client, test_package, sized_content):
"""Test uploading a 10MB file."""
project, package = test_package
content, expected_hash = sized_content(SIZE_10MB, seed=200)
result = upload_test_file(
integration_client, project, package, content, version="large-10mb"
)
assert result["artifact_id"] == expected_hash
assert result["size"] == SIZE_10MB
assert result["duration_ms"] is not None
assert result["throughput_mbps"] is not None
@pytest.mark.integration
@pytest.mark.slow
@pytest.mark.requires_direct_s3
def test_upload_100mb_file(self, integration_client, test_package, sized_content):
"""Test uploading a 100MB file (triggers multipart upload)."""
project, package = test_package
content, expected_hash = sized_content(SIZE_100MB, seed=300)
result = upload_test_file(
integration_client, project, package, content, version="large-100mb"
)
assert result["artifact_id"] == expected_hash
assert result["size"] == SIZE_100MB
# Verify S3 object exists
assert s3_object_exists(expected_hash)
@pytest.mark.integration
@pytest.mark.slow
@pytest.mark.large
def test_upload_1gb_file(self, integration_client, test_package, sized_content):
"""Test uploading a 1GB file."""
project, package = test_package
content, expected_hash = sized_content(SIZE_1GB, seed=400)
result = upload_test_file(
integration_client, project, package, content, version="large-1gb"
)
assert result["artifact_id"] == expected_hash
assert result["size"] == SIZE_1GB
# Should have measurable throughput
assert result["throughput_mbps"] is not None
assert result["throughput_mbps"] > 0
@pytest.mark.integration
def test_large_file_deduplication(
self, integration_client, test_package, sized_content, unique_test_id
):
"""Test deduplication works for large files."""
project, package = test_package
# Use unique_test_id to ensure unique content per test run
seed = hash(unique_test_id) % 10000
content, expected_hash = sized_content(SIZE_10MB, seed=seed)
# First upload
result1 = upload_test_file(
integration_client, project, package, content, version=f"dedup-{unique_test_id}-1"
)
# Note: may be True if previous test uploaded same content
first_dedupe = result1["deduplicated"]
# Second upload of same content
result2 = upload_test_file(
integration_client, project, package, content, version=f"dedup-{unique_test_id}-2"
)
assert result2["artifact_id"] == expected_hash
# Second upload MUST be deduplicated
assert result2["deduplicated"] is True
class TestUploadProgress:
"""Tests for upload progress tracking endpoint."""
@pytest.mark.integration
def test_progress_endpoint_returns_not_found_for_invalid_id(
self, integration_client, test_package
):
"""Test progress endpoint returns not_found status for invalid upload ID."""
project, package = test_package
response = integration_client.get(
f"/api/v1/project/{project}/{package}/upload/invalid-upload-id/progress"
)
assert response.status_code == 200
data = response.json()
assert data["status"] == "not_found"
assert data["upload_id"] == "invalid-upload-id"
@pytest.mark.integration
def test_progress_endpoint_requires_valid_project(
self, integration_client, unique_test_id
):
"""Test progress endpoint validates project exists."""
response = integration_client.get(
f"/api/v1/project/nonexistent-{unique_test_id}/pkg/upload/upload-id/progress"
)
assert response.status_code == 404
@pytest.mark.integration
def test_progress_endpoint_requires_valid_package(
self, integration_client, test_project, unique_test_id
):
"""Test progress endpoint validates package exists."""
response = integration_client.get(
f"/api/v1/project/{test_project}/nonexistent-{unique_test_id}/upload/upload-id/progress"
)
assert response.status_code == 404
class TestResumableUploadProgress:
"""Tests for progress tracking during resumable uploads."""
@pytest.mark.integration
def test_resumable_upload_init_and_progress(
self, integration_client, test_package, sized_content
):
"""Test initializing resumable upload and checking progress."""
project, package = test_package
content, expected_hash = sized_content(SIZE_100KB, seed=600)
# Get API key for auth
api_key_response = integration_client.post(
"/api/v1/auth/keys",
json={"name": "progress-test-key"},
)
assert api_key_response.status_code == 200
api_key = api_key_response.json()["key"]
# Initialize resumable upload
init_response = integration_client.post(
f"/api/v1/project/{project}/{package}/upload/init",
json={
"expected_hash": expected_hash,
"filename": "progress-test.bin",
"size": SIZE_100KB,
},
headers={"Authorization": f"Bearer {api_key}"},
)
assert init_response.status_code == 200
upload_id = init_response.json().get("upload_id")
if upload_id:
# Check initial progress
progress_response = integration_client.get(
f"/api/v1/project/{project}/{package}/upload/{upload_id}/progress",
headers={"Authorization": f"Bearer {api_key}"},
)
assert progress_response.status_code == 200
progress = progress_response.json()
assert progress["status"] == "in_progress"
assert progress["bytes_uploaded"] == 0
assert progress["bytes_total"] == SIZE_100KB
# Abort to clean up
integration_client.delete(
f"/api/v1/project/{project}/{package}/upload/{upload_id}",
headers={"Authorization": f"Bearer {api_key}"},
)
class TestUploadSizeLimits:
"""Tests for upload size limit enforcement."""
@pytest.mark.integration
def test_empty_file_rejected(self, integration_client, test_package):
"""Test empty files are rejected."""
project, package = test_package
files = {"file": ("empty.txt", io.BytesIO(b""), "application/octet-stream")}
response = integration_client.post(
f"/api/v1/project/{project}/{package}/upload",
files=files,
)
assert response.status_code in [400, 422]
@pytest.mark.integration
def test_minimum_size_accepted(self, integration_client, test_package):
"""Test 1-byte file is accepted."""
project, package = test_package
content = b"X"
result = upload_test_file(
integration_client, project, package, content, version="min-size"
)
assert result["size"] == 1
@pytest.mark.integration
def test_content_length_header_used_in_response(self, integration_client, test_package):
"""Test that upload response size matches Content-Length."""
project, package = test_package
content = b"content length verification test"
result = upload_test_file(
integration_client, project, package, content, version="content-length-test"
)
# Size in response should match actual content length
assert result["size"] == len(content)
class TestUploadErrorHandling:
"""Tests for upload error handling."""
@pytest.mark.integration
def test_upload_to_nonexistent_project_returns_404(
self, integration_client, unique_test_id
):
"""Test upload to nonexistent project returns 404."""
content = b"test content"
files = {"file": ("test.bin", io.BytesIO(content), "application/octet-stream")}
response = integration_client.post(
f"/api/v1/project/nonexistent-{unique_test_id}/pkg/upload",
files=files,
)
assert response.status_code == 404
@pytest.mark.integration
def test_upload_to_nonexistent_package_returns_404(
self, integration_client, test_project, unique_test_id
):
"""Test upload to nonexistent package returns 404."""
content = b"test content"
files = {"file": ("test.bin", io.BytesIO(content), "application/octet-stream")}
response = integration_client.post(
f"/api/v1/project/{test_project}/nonexistent-{unique_test_id}/upload",
files=files,
)
assert response.status_code == 404
@pytest.mark.integration
def test_upload_without_file_returns_422(self, integration_client, test_package):
"""Test upload without file field returns 422."""
project, package = test_package
response = integration_client.post(
f"/api/v1/project/{project}/{package}/upload",
data={"version": "no-file"},
)
assert response.status_code == 422
@pytest.mark.integration
def test_upload_with_invalid_checksum_rejected(
self, integration_client, test_package
):
"""Test upload with invalid checksum header format is rejected."""
project, package = test_package
content = b"checksum test"
files = {"file": ("test.bin", io.BytesIO(content), "application/octet-stream")}
response = integration_client.post(
f"/api/v1/project/{project}/{package}/upload",
files=files,
headers={"X-Checksum-SHA256": "invalid-checksum"},
)
assert response.status_code == 400
@pytest.mark.integration
def test_upload_with_mismatched_checksum_rejected(
self, integration_client, test_package
):
"""Test upload with wrong checksum is rejected."""
project, package = test_package
content = b"mismatch test"
wrong_hash = "0" * 64
files = {"file": ("test.bin", io.BytesIO(content), "application/octet-stream")}
response = integration_client.post(
f"/api/v1/project/{project}/{package}/upload",
files=files,
headers={"X-Checksum-SHA256": wrong_hash},
)
assert response.status_code == 422
assert "verification failed" in response.json().get("detail", "").lower()
class TestResumableUploadCancellation:
"""Tests for resumable upload cancellation."""
@pytest.mark.integration
def test_abort_resumable_upload(self, integration_client, test_package, sized_content):
"""Test aborting a resumable upload cleans up properly."""
project, package = test_package
content, expected_hash = sized_content(SIZE_100KB, seed=700)
# Get API key for auth
api_key_response = integration_client.post(
"/api/v1/auth/keys",
json={"name": "abort-test-key"},
)
assert api_key_response.status_code == 200
api_key = api_key_response.json()["key"]
# Initialize resumable upload
init_response = integration_client.post(
f"/api/v1/project/{project}/{package}/upload/init",
json={
"expected_hash": expected_hash,
"filename": "abort-test.bin",
"size": SIZE_100KB,
},
headers={"Authorization": f"Bearer {api_key}"},
)
assert init_response.status_code == 200
upload_id = init_response.json().get("upload_id")
if upload_id:
# Abort the upload (without uploading any parts)
abort_response = integration_client.delete(
f"/api/v1/project/{project}/{package}/upload/{upload_id}",
headers={"Authorization": f"Bearer {api_key}"},
)
assert abort_response.status_code in [200, 204]
# Verify progress shows not_found after abort
progress_response = integration_client.get(
f"/api/v1/project/{project}/{package}/upload/{upload_id}/progress",
headers={"Authorization": f"Bearer {api_key}"},
)
assert progress_response.status_code == 200
assert progress_response.json()["status"] == "not_found"
@pytest.mark.integration
def test_abort_nonexistent_upload(self, integration_client, test_package):
"""Test aborting nonexistent upload returns appropriate error."""
project, package = test_package
# Get API key for auth
api_key_response = integration_client.post(
"/api/v1/auth/keys",
json={"name": "abort-nonexistent-key"},
)
assert api_key_response.status_code == 200
api_key = api_key_response.json()["key"]
response = integration_client.delete(
f"/api/v1/project/{project}/{package}/upload/nonexistent-upload-id",
headers={"Authorization": f"Bearer {api_key}"},
)
# Should return 404 or 200 (idempotent delete)
assert response.status_code in [200, 204, 404]
class TestUploadTimeout:
"""Tests for upload timeout handling."""
@pytest.mark.integration
def test_upload_with_short_timeout_succeeds_for_small_file(
self, integration_client, test_package
):
"""Test small file upload succeeds with reasonable timeout."""
project, package = test_package
content = b"small timeout test"
# httpx client should handle this quickly
result = upload_test_file(
integration_client, project, package, content, version="timeout-small"
)
assert result["artifact_id"] is not None
@pytest.mark.integration
def test_upload_response_duration_under_timeout(
self, integration_client, test_package, sized_content
):
"""Test upload completes within reasonable time."""
project, package = test_package
content, _ = sized_content(SIZE_1MB, seed=800)
start = time.time()
result = upload_test_file(
integration_client, project, package, content, version="timeout-check"
)
duration = time.time() - start
# 1MB should upload in well under 60 seconds on local
assert duration < 60
assert result["artifact_id"] is not None
class TestConcurrentUploads:
"""Tests for concurrent upload handling."""
@pytest.mark.integration
def test_concurrent_different_files(
self, integration_client, test_package, sized_content
):
"""Test concurrent uploads of different files succeed."""
from concurrent.futures import ThreadPoolExecutor, as_completed
project, package = test_package
# Get API key for auth
api_key_response = integration_client.post(
"/api/v1/auth/keys",
json={"name": "concurrent-diff-key"},
)
assert api_key_response.status_code == 200
api_key = api_key_response.json()["key"]
num_uploads = 3
results = []
errors = []
def upload_unique_file(idx):
try:
from httpx import Client
content, expected_hash = sized_content(SIZE_100KB, seed=900 + idx)
base_url = os.environ.get("ORCHARD_TEST_URL", "http://localhost:8080")
with Client(base_url=base_url, timeout=30.0) as client:
files = {
"file": (
f"concurrent-{idx}.bin",
io.BytesIO(content),
"application/octet-stream",
)
}
response = client.post(
f"/api/v1/project/{project}/{package}/upload",
files=files,
data={"version": f"concurrent-diff-{idx}"},
headers={"Authorization": f"Bearer {api_key}"},
)
if response.status_code == 200:
results.append((idx, response.json(), expected_hash))
else:
errors.append(f"Upload {idx}: {response.status_code} - {response.text}")
except Exception as e:
errors.append(f"Upload {idx}: {str(e)}")
with ThreadPoolExecutor(max_workers=num_uploads) as executor:
futures = [executor.submit(upload_unique_file, i) for i in range(num_uploads)]
for future in as_completed(futures):
pass
assert len(errors) == 0, f"Concurrent upload errors: {errors}"
assert len(results) == num_uploads
# Each upload should have unique artifact ID
artifact_ids = set(r[1]["artifact_id"] for r in results)
assert len(artifact_ids) == num_uploads
# Each should match expected hash
for idx, result, expected_hash in results:
assert result["artifact_id"] == expected_hash

View File

@@ -175,7 +175,7 @@ class TestPackageStats:
assert "package_id" in data
assert "package_name" in data
assert "project_name" in data
assert "tag_count" in data
assert "version_count" in data
assert "artifact_count" in data
assert "total_size_bytes" in data
assert "upload_count" in data
@@ -234,7 +234,11 @@ class TestPackageCascadeDelete:
def test_ref_count_decrements_on_package_delete(
self, integration_client, unique_test_id
):
"""Test ref_count decrements for all tags when package is deleted."""
"""Test ref_count decrements when package is deleted.
Each package can only have one version per artifact (same content = same version).
This test verifies that deleting a package decrements the artifact's ref_count.
"""
project_name = f"cascade-pkg-{unique_test_id}"
package_name = f"test-pkg-{unique_test_id}"
@@ -256,23 +260,17 @@ class TestPackageCascadeDelete:
)
assert response.status_code == 200
# Upload content with multiple tags
# Upload content with version
content = f"cascade delete test {unique_test_id}".encode()
expected_hash = compute_sha256(content)
upload_test_file(
integration_client, project_name, package_name, content, tag="v1"
)
upload_test_file(
integration_client, project_name, package_name, content, tag="v2"
)
upload_test_file(
integration_client, project_name, package_name, content, tag="v3"
integration_client, project_name, package_name, content, version="1.0.0"
)
# Verify ref_count is 3
# Verify ref_count is 1
response = integration_client.get(f"/api/v1/artifact/{expected_hash}")
assert response.json()["ref_count"] == 3
assert response.json()["ref_count"] == 1
# Delete the package
delete_response = integration_client.delete(

View File

@@ -128,7 +128,9 @@ class TestProjectListingFilters:
assert response.status_code == 200
data = response.json()
names = [p["name"] for p in data["items"]]
# Filter out system projects (names starting with "_") as they may have
# collation-specific sort behavior and aren't part of the test data
names = [p["name"] for p in data["items"] if not p["name"].startswith("_")]
assert names == sorted(names)
@@ -147,7 +149,7 @@ class TestProjectStats:
assert "project_id" in data
assert "project_name" in data
assert "package_count" in data
assert "tag_count" in data
assert "version_count" in data
assert "artifact_count" in data
assert "total_size_bytes" in data
assert "upload_count" in data
@@ -227,7 +229,11 @@ class TestProjectCascadeDelete:
def test_ref_count_decrements_on_project_delete(
self, integration_client, unique_test_id
):
"""Test ref_count decrements for all tags when project is deleted."""
"""Test ref_count decrements for all versions when project is deleted.
Each package can only have one version per artifact (same content = same version).
With 2 packages, ref_count should be 2, and go to 0 when project is deleted.
"""
project_name = f"cascade-proj-{unique_test_id}"
package1_name = f"pkg1-{unique_test_id}"
package2_name = f"pkg2-{unique_test_id}"
@@ -251,26 +257,20 @@ class TestProjectCascadeDelete:
)
assert response.status_code == 200
# Upload same content with tags in both packages
# Upload same content to both packages
content = f"project cascade test {unique_test_id}".encode()
expected_hash = compute_sha256(content)
upload_test_file(
integration_client, project_name, package1_name, content, tag="v1"
integration_client, project_name, package1_name, content, version="1.0.0"
)
upload_test_file(
integration_client, project_name, package1_name, content, tag="v2"
)
upload_test_file(
integration_client, project_name, package2_name, content, tag="latest"
)
upload_test_file(
integration_client, project_name, package2_name, content, tag="stable"
integration_client, project_name, package2_name, content, version="1.0.0"
)
# Verify ref_count is 4 (2 tags in each of 2 packages)
# Verify ref_count is 2 (1 version in each of 2 packages)
response = integration_client.get(f"/api/v1/artifact/{expected_hash}")
assert response.json()["ref_count"] == 4
assert response.json()["ref_count"] == 2
# Delete the project
delete_response = integration_client.delete(f"/api/v1/projects/{project_name}")

View File

@@ -0,0 +1,153 @@
"""Integration tests for PyPI transparent proxy."""
import os
import pytest
import httpx
def get_base_url():
"""Get the base URL for the Orchard server from environment."""
return os.environ.get("ORCHARD_TEST_URL", "http://localhost:8080")
class TestPyPIProxyEndpoints:
"""Tests for PyPI proxy endpoints.
These endpoints are public (no auth required) since pip needs to use them.
"""
@pytest.mark.integration
def test_pypi_simple_index(self):
"""Test that /pypi/simple/ returns HTML response."""
with httpx.Client(base_url=get_base_url(), timeout=30.0) as client:
response = client.get("/pypi/simple/")
# Returns 200 if sources configured, 503 if not
assert response.status_code in (200, 503)
if response.status_code == 200:
assert "text/html" in response.headers.get("content-type", "")
else:
assert "No PyPI upstream sources configured" in response.json()["detail"]
@pytest.mark.integration
def test_pypi_package_endpoint(self):
"""Test that /pypi/simple/{package}/ returns appropriate response."""
with httpx.Client(base_url=get_base_url(), timeout=30.0) as client:
response = client.get("/pypi/simple/requests/")
# Returns 200 if sources configured and package found,
# 404 if package not found, 503 if no sources
assert response.status_code in (200, 404, 503)
if response.status_code == 200:
assert "text/html" in response.headers.get("content-type", "")
elif response.status_code == 404:
assert "not found" in response.json()["detail"].lower()
else: # 503
assert "No PyPI upstream sources configured" in response.json()["detail"]
@pytest.mark.integration
def test_pypi_download_missing_upstream_param(self):
"""Test that /pypi/simple/{package}/{filename} requires upstream param."""
with httpx.Client(base_url=get_base_url(), timeout=30.0) as client:
response = client.get("/pypi/simple/requests/requests-2.31.0.tar.gz")
assert response.status_code == 400
assert "upstream" in response.json()["detail"].lower()
class TestPyPILinkRewriting:
"""Tests for URL rewriting in PyPI proxy responses."""
def test_rewrite_package_links(self):
"""Test that download links are rewritten to go through proxy."""
from app.pypi_proxy import _rewrite_package_links
html = '''
<html>
<body>
<a href="https://files.pythonhosted.org/packages/ab/cd/requests-2.31.0.tar.gz#sha256=abc123">requests-2.31.0.tar.gz</a>
<a href="https://files.pythonhosted.org/packages/ef/gh/requests-2.31.0-py3-none-any.whl#sha256=def456">requests-2.31.0-py3-none-any.whl</a>
</body>
</html>
'''
# upstream_base_url is used to resolve relative URLs (not needed here since URLs are absolute)
result = _rewrite_package_links(
html,
"http://localhost:8080",
"requests",
"https://pypi.org/simple/requests/"
)
# Links should be rewritten to go through our proxy
assert "/pypi/simple/requests/requests-2.31.0.tar.gz?upstream=" in result
assert "/pypi/simple/requests/requests-2.31.0-py3-none-any.whl?upstream=" in result
# Original URLs should be encoded in upstream param
assert "files.pythonhosted.org" in result
# Hash fragments should be preserved
assert "#sha256=abc123" in result
assert "#sha256=def456" in result
def test_rewrite_relative_links(self):
"""Test that relative URLs are resolved to absolute URLs."""
from app.pypi_proxy import _rewrite_package_links
# Artifactory-style relative URLs
html = '''
<html>
<body>
<a href="../../packages/ab/cd/requests-2.31.0.tar.gz#sha256=abc123">requests-2.31.0.tar.gz</a>
</body>
</html>
'''
result = _rewrite_package_links(
html,
"https://orchard.example.com",
"requests",
"https://artifactory.example.com/api/pypi/pypi-remote/simple/requests/"
)
# The relative URL should be resolved to absolute
# ../../packages/ab/cd/... from /api/pypi/pypi-remote/simple/requests/ resolves to /api/pypi/pypi-remote/packages/ab/cd/...
assert "upstream=https%3A%2F%2Fartifactory.example.com%2Fapi%2Fpypi%2Fpypi-remote%2Fpackages" in result
# Hash fragment should be preserved
assert "#sha256=abc123" in result
class TestPyPIPackageNormalization:
"""Tests for PyPI package name normalization."""
@pytest.mark.integration
def test_package_name_normalized(self):
"""Test that package names are normalized per PEP 503.
Different capitalizations/separators should all be valid paths.
The endpoint normalizes to lowercase with hyphens before lookup.
"""
with httpx.Client(base_url=get_base_url(), timeout=30.0) as client:
# Test various name formats - all should be valid endpoint paths
for package_name in ["Requests", "some_package", "some-package"]:
response = client.get(f"/pypi/simple/{package_name}/")
# 200 = found, 404 = not found, 503 = no sources configured
assert response.status_code in (200, 404, 503), \
f"Unexpected status {response.status_code} for {package_name}"
# Verify response is appropriate for the status code
if response.status_code == 200:
assert "text/html" in response.headers.get("content-type", "")
elif response.status_code == 503:
assert "No PyPI upstream sources configured" in response.json()["detail"]
class TestPyPIProxyInfrastructure:
"""Tests for PyPI proxy infrastructure integration."""
@pytest.mark.integration
def test_health_endpoint_includes_infrastructure(self, integration_client):
"""Health endpoint should report infrastructure status."""
response = integration_client.get("/health")
assert response.status_code == 200
data = response.json()
assert data["status"] == "ok"
# Infrastructure status should be present
assert "http_pool" in data
assert "cache" in data

View File

@@ -0,0 +1,583 @@
"""
Integration tests for upload/download with various file sizes.
Tests cover:
- Small files (0B - 100KB)
- Medium files (1MB - 50MB)
- Large files (100MB - 1GB) - marked as slow/large
- Exact chunk boundaries
- Data integrity verification across all sizes
"""
import pytest
import io
import time
from concurrent.futures import ThreadPoolExecutor, as_completed
from tests.factories import (
compute_sha256,
upload_test_file,
generate_content,
generate_content_with_hash,
)
from tests.conftest import (
SIZE_1B,
SIZE_1KB,
SIZE_10KB,
SIZE_100KB,
SIZE_1MB,
SIZE_5MB,
SIZE_10MB,
SIZE_50MB,
SIZE_100MB,
SIZE_250MB,
SIZE_500MB,
SIZE_1GB,
CHUNK_SIZE,
MULTIPART_THRESHOLD,
)
class TestSmallFileSizes:
"""Tests for small file uploads/downloads (0B - 100KB)."""
@pytest.mark.integration
def test_upload_download_1_byte(self, integration_client, test_package, sized_content):
"""Test upload/download of 1 byte file."""
project, package = test_package
content, expected_hash = sized_content(SIZE_1B, seed=1)
result = upload_test_file(
integration_client, project, package, content,
filename="1byte.bin", version="1byte"
)
assert result["artifact_id"] == expected_hash
assert result["size"] == SIZE_1B
# Download and verify
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/1byte",
params={"mode": "proxy"},
)
assert response.status_code == 200
assert response.content == content
assert len(response.content) == SIZE_1B
@pytest.mark.integration
def test_upload_download_1kb(self, integration_client, test_package, sized_content):
"""Test upload/download of 1KB file."""
project, package = test_package
content, expected_hash = sized_content(SIZE_1KB, seed=2)
result = upload_test_file(
integration_client, project, package, content,
filename="1kb.bin", version="1kb"
)
assert result["artifact_id"] == expected_hash
assert result["size"] == SIZE_1KB
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/1kb",
params={"mode": "proxy"},
)
assert response.status_code == 200
assert response.content == content
@pytest.mark.integration
def test_upload_download_10kb(self, integration_client, test_package, sized_content):
"""Test upload/download of 10KB file."""
project, package = test_package
content, expected_hash = sized_content(SIZE_10KB, seed=3)
result = upload_test_file(
integration_client, project, package, content,
filename="10kb.bin", version="10kb"
)
assert result["artifact_id"] == expected_hash
assert result["size"] == SIZE_10KB
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/10kb",
params={"mode": "proxy"},
)
assert response.status_code == 200
assert response.content == content
@pytest.mark.integration
def test_upload_download_100kb(self, integration_client, test_package, sized_content):
"""Test upload/download of 100KB file."""
project, package = test_package
content, expected_hash = sized_content(SIZE_100KB, seed=4)
result = upload_test_file(
integration_client, project, package, content,
filename="100kb.bin", version="100kb"
)
assert result["artifact_id"] == expected_hash
assert result["size"] == SIZE_100KB
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/100kb",
params={"mode": "proxy"},
)
assert response.status_code == 200
assert response.content == content
class TestMediumFileSizes:
"""Tests for medium file uploads/downloads (1MB - 50MB)."""
@pytest.mark.integration
def test_upload_download_1mb(self, integration_client, test_package, sized_content):
"""Test upload/download of 1MB file."""
project, package = test_package
content, expected_hash = sized_content(SIZE_1MB, seed=10)
result = upload_test_file(
integration_client, project, package, content,
filename="1mb.bin", version="1mb"
)
assert result["artifact_id"] == expected_hash
assert result["size"] == SIZE_1MB
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/1mb",
params={"mode": "proxy"},
)
assert response.status_code == 200
assert len(response.content) == SIZE_1MB
assert compute_sha256(response.content) == expected_hash
@pytest.mark.integration
def test_upload_download_5mb(self, integration_client, test_package, sized_content):
"""Test upload/download of 5MB file (multipart threshold boundary area)."""
project, package = test_package
content, expected_hash = sized_content(SIZE_5MB, seed=11)
result = upload_test_file(
integration_client, project, package, content,
filename="5mb.bin", version="5mb"
)
assert result["artifact_id"] == expected_hash
assert result["size"] == SIZE_5MB
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/5mb",
params={"mode": "proxy"},
)
assert response.status_code == 200
assert len(response.content) == SIZE_5MB
assert compute_sha256(response.content) == expected_hash
@pytest.mark.integration
@pytest.mark.slow
def test_upload_download_10mb(self, integration_client, test_package, sized_content):
"""Test upload/download of 10MB file."""
project, package = test_package
content, expected_hash = sized_content(SIZE_10MB, seed=12)
result = upload_test_file(
integration_client, project, package, content,
filename="10mb.bin", version="10mb"
)
assert result["artifact_id"] == expected_hash
assert result["size"] == SIZE_10MB
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/10mb",
params={"mode": "proxy"},
)
assert response.status_code == 200
assert len(response.content) == SIZE_10MB
assert compute_sha256(response.content) == expected_hash
@pytest.mark.integration
@pytest.mark.slow
def test_upload_download_50mb(self, integration_client, test_package, sized_content):
"""Test upload/download of 50MB file."""
project, package = test_package
content, expected_hash = sized_content(SIZE_50MB, seed=13)
start_time = time.time()
result = upload_test_file(
integration_client, project, package, content,
filename="50mb.bin", version="50mb"
)
upload_time = time.time() - start_time
assert result["artifact_id"] == expected_hash
assert result["size"] == SIZE_50MB
start_time = time.time()
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/50mb",
params={"mode": "proxy"},
)
download_time = time.time() - start_time
assert response.status_code == 200
assert len(response.content) == SIZE_50MB
assert compute_sha256(response.content) == expected_hash
# Log timing for performance tracking
print(f"\n50MB upload: {upload_time:.2f}s, download: {download_time:.2f}s")
class TestLargeFileSizes:
"""Tests for large file uploads/downloads (100MB - 1GB).
These tests are marked as slow and large, skipped by default.
Run with: pytest -m "large" to include these tests.
"""
@pytest.mark.integration
@pytest.mark.slow
@pytest.mark.large
def test_upload_download_100mb(self, integration_client, test_package, sized_content):
"""Test upload/download of 100MB file (multipart threshold)."""
project, package = test_package
content, expected_hash = sized_content(SIZE_100MB, seed=100)
start_time = time.time()
result = upload_test_file(
integration_client, project, package, content,
filename="100mb.bin", version="100mb"
)
upload_time = time.time() - start_time
assert result["artifact_id"] == expected_hash
assert result["size"] == SIZE_100MB
start_time = time.time()
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/100mb",
params={"mode": "proxy"},
)
download_time = time.time() - start_time
assert response.status_code == 200
assert len(response.content) == SIZE_100MB
assert compute_sha256(response.content) == expected_hash
print(f"\n100MB upload: {upload_time:.2f}s, download: {download_time:.2f}s")
@pytest.mark.integration
@pytest.mark.slow
@pytest.mark.large
def test_upload_download_250mb(self, integration_client, test_package, sized_content):
"""Test upload/download of 250MB file."""
project, package = test_package
content, expected_hash = sized_content(SIZE_250MB, seed=250)
start_time = time.time()
result = upload_test_file(
integration_client, project, package, content,
filename="250mb.bin", version="250mb"
)
upload_time = time.time() - start_time
assert result["artifact_id"] == expected_hash
assert result["size"] == SIZE_250MB
start_time = time.time()
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/250mb",
params={"mode": "proxy"},
)
download_time = time.time() - start_time
assert response.status_code == 200
assert len(response.content) == SIZE_250MB
assert compute_sha256(response.content) == expected_hash
print(f"\n250MB upload: {upload_time:.2f}s, download: {download_time:.2f}s")
@pytest.mark.integration
@pytest.mark.slow
@pytest.mark.large
def test_upload_download_500mb(self, integration_client, test_package, sized_content):
"""Test upload/download of 500MB file."""
project, package = test_package
content, expected_hash = sized_content(SIZE_500MB, seed=500)
start_time = time.time()
result = upload_test_file(
integration_client, project, package, content,
filename="500mb.bin", version="500mb"
)
upload_time = time.time() - start_time
assert result["artifact_id"] == expected_hash
assert result["size"] == SIZE_500MB
start_time = time.time()
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/500mb",
params={"mode": "proxy"},
)
download_time = time.time() - start_time
assert response.status_code == 200
assert len(response.content) == SIZE_500MB
assert compute_sha256(response.content) == expected_hash
print(f"\n500MB upload: {upload_time:.2f}s, download: {download_time:.2f}s")
@pytest.mark.integration
@pytest.mark.slow
@pytest.mark.large
def test_upload_download_1gb(self, integration_client, test_package, sized_content):
"""Test upload/download of 1GB file.
This test may take several minutes depending on network/disk speed.
"""
project, package = test_package
content, expected_hash = sized_content(SIZE_1GB, seed=1024)
start_time = time.time()
result = upload_test_file(
integration_client, project, package, content,
filename="1gb.bin", version="1gb"
)
upload_time = time.time() - start_time
assert result["artifact_id"] == expected_hash
assert result["size"] == SIZE_1GB
start_time = time.time()
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/1gb",
params={"mode": "proxy"},
)
download_time = time.time() - start_time
assert response.status_code == 200
assert len(response.content) == SIZE_1GB
assert compute_sha256(response.content) == expected_hash
print(f"\n1GB upload: {upload_time:.2f}s, download: {download_time:.2f}s")
class TestChunkBoundaries:
"""Tests for exact chunk size boundaries."""
@pytest.mark.integration
def test_upload_download_at_chunk_size(self, integration_client, test_package, sized_content):
"""Test upload/download at exact chunk size (64KB)."""
project, package = test_package
content, expected_hash = sized_content(CHUNK_SIZE, seed=64)
result = upload_test_file(
integration_client, project, package, content,
filename="chunk.bin", version="chunk-exact"
)
assert result["artifact_id"] == expected_hash
assert result["size"] == CHUNK_SIZE
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/chunk-exact",
params={"mode": "proxy"},
)
assert response.status_code == 200
assert response.content == content
@pytest.mark.integration
def test_upload_download_chunk_size_plus_1(self, integration_client, test_package, sized_content):
"""Test upload/download at chunk size + 1 byte."""
project, package = test_package
size = CHUNK_SIZE + 1
content, expected_hash = sized_content(size, seed=65)
result = upload_test_file(
integration_client, project, package, content,
filename="chunk_plus.bin", version="chunk-plus"
)
assert result["artifact_id"] == expected_hash
assert result["size"] == size
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/chunk-plus",
params={"mode": "proxy"},
)
assert response.status_code == 200
assert response.content == content
@pytest.mark.integration
def test_upload_download_chunk_size_minus_1(self, integration_client, test_package, sized_content):
"""Test upload/download at chunk size - 1 byte."""
project, package = test_package
size = CHUNK_SIZE - 1
content, expected_hash = sized_content(size, seed=63)
result = upload_test_file(
integration_client, project, package, content,
filename="chunk_minus.bin", version="chunk-minus"
)
assert result["artifact_id"] == expected_hash
assert result["size"] == size
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/chunk-minus",
params={"mode": "proxy"},
)
assert response.status_code == 200
assert response.content == content
@pytest.mark.integration
def test_upload_download_multiple_chunks(self, integration_client, test_package, sized_content):
"""Test upload/download spanning multiple chunks."""
project, package = test_package
size = CHUNK_SIZE * 3 + 1000 # 3 full chunks + partial
content, expected_hash = sized_content(size, seed=300)
result = upload_test_file(
integration_client, project, package, content,
filename="multi_chunk.bin", version="multi-chunk"
)
assert result["artifact_id"] == expected_hash
assert result["size"] == size
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/multi-chunk",
params={"mode": "proxy"},
)
assert response.status_code == 200
assert response.content == content
class TestDataIntegrity:
"""Tests for data integrity with various content types."""
@pytest.mark.integration
def test_binary_content_integrity(self, integration_client, test_package):
"""Test binary content (all byte values 0-255) integrity."""
project, package = test_package
# Content with all 256 possible byte values
content = bytes(range(256)) * 100 # 25.6KB
expected_hash = compute_sha256(content)
result = upload_test_file(
integration_client, project, package, content,
filename="binary.bin", version="binary"
)
assert result["artifact_id"] == expected_hash
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/binary",
params={"mode": "proxy"},
)
assert response.status_code == 200
assert response.content == content
@pytest.mark.integration
def test_text_content_integrity(self, integration_client, test_package):
"""Test UTF-8 text content integrity."""
project, package = test_package
content = "Hello, World! 你好世界 🌍 مرحبا العالم".encode("utf-8")
expected_hash = compute_sha256(content)
result = upload_test_file(
integration_client, project, package, content,
filename="text.txt", version="text"
)
assert result["artifact_id"] == expected_hash
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/text",
params={"mode": "proxy"},
)
assert response.status_code == 200
assert response.content == content
assert response.content.decode("utf-8") == "Hello, World! 你好世界 🌍 مرحبا العالم"
@pytest.mark.integration
def test_null_bytes_content_integrity(self, integration_client, test_package):
"""Test content with null bytes."""
project, package = test_package
content = b"before\x00null\x00bytes\x00after"
expected_hash = compute_sha256(content)
result = upload_test_file(
integration_client, project, package, content,
filename="nulls.bin", version="nulls"
)
assert result["artifact_id"] == expected_hash
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/nulls",
params={"mode": "proxy"},
)
assert response.status_code == 200
assert response.content == content
assert b"\x00" in response.content
@pytest.mark.integration
def test_unicode_filename_integrity(self, integration_client, test_package):
"""Test file with unicode filename."""
project, package = test_package
content = b"unicode filename test"
expected_hash = compute_sha256(content)
result = upload_test_file(
integration_client, project, package, content,
filename="文件名.txt", version="unicode-name"
)
assert result["artifact_id"] == expected_hash
assert result["original_name"] == "文件名.txt"
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/unicode-name",
params={"mode": "proxy"},
)
assert response.status_code == 200
assert response.content == content
@pytest.mark.integration
def test_compressed_content_integrity(self, integration_client, test_package):
"""Test gzip-compressed content integrity."""
import gzip
project, package = test_package
original = b"This is some text that will be compressed " * 100
content = gzip.compress(original)
expected_hash = compute_sha256(content)
result = upload_test_file(
integration_client, project, package, content,
filename="data.gz", version="compressed"
)
assert result["artifact_id"] == expected_hash
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/compressed",
params={"mode": "proxy"},
)
assert response.status_code == 200
assert response.content == content
# Verify we can decompress
assert gzip.decompress(response.content) == original
@pytest.mark.integration
def test_hash_verification_matches(self, integration_client, test_package, sized_content):
"""Test that computed hash matches artifact_id for various sizes."""
project, package = test_package
sizes = [SIZE_1B, SIZE_1KB, SIZE_10KB, SIZE_100KB, SIZE_1MB]
for i, size in enumerate(sizes):
content, expected_hash = sized_content(size, seed=1000 + i)
result = upload_test_file(
integration_client, project, package, content,
filename=f"hash_test_{size}.bin", version=f"hash-{size}"
)
# Verify artifact_id matches expected hash
assert result["artifact_id"] == expected_hash
# Download and verify hash of downloaded content
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/hash-{size}",
params={"mode": "proxy"},
)
downloaded_hash = compute_sha256(response.content)
assert downloaded_hash == expected_hash

View File

@@ -0,0 +1,535 @@
"""
Integration tests for streaming download functionality.
Tests cover:
- HTTP Range requests (partial downloads, resume)
- Conditional requests (If-None-Match, If-Modified-Since)
- Caching headers (Cache-Control, Last-Modified, Accept-Ranges)
- Large file streaming
- Download modes (proxy, redirect, presigned)
"""
import pytest
import io
import time
from email.utils import formatdate
from tests.factories import (
compute_sha256,
upload_test_file,
)
from tests.conftest import (
SIZE_1KB,
SIZE_100KB,
SIZE_1MB,
)
class TestRangeRequests:
"""Tests for HTTP Range request support (partial downloads)."""
@pytest.mark.integration
def test_range_request_first_bytes(self, integration_client, test_package):
"""Test range request for first N bytes."""
project, package = test_package
content = b"0123456789" * 100 # 1000 bytes
upload_test_file(integration_client, project, package, content, version="range-test")
# Request first 10 bytes
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/range-test",
params={"mode": "proxy"},
headers={"Range": "bytes=0-9"},
)
assert response.status_code == 206 # Partial Content
assert response.content == b"0123456789"
assert "Content-Range" in response.headers
assert response.headers["Content-Range"].startswith("bytes 0-9/")
@pytest.mark.integration
def test_range_request_middle_bytes(self, integration_client, test_package):
"""Test range request for bytes in the middle."""
project, package = test_package
content = b"ABCDEFGHIJKLMNOPQRSTUVWXYZ"
upload_test_file(integration_client, project, package, content, version="range-mid")
# Request bytes 10-19 (KLMNOPQRST)
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/range-mid",
params={"mode": "proxy"},
headers={"Range": "bytes=10-19"},
)
assert response.status_code == 206
assert response.content == b"KLMNOPQRST"
@pytest.mark.integration
def test_range_request_suffix_bytes(self, integration_client, test_package):
"""Test range request for last N bytes (suffix range)."""
project, package = test_package
content = b"0123456789ABCDEF" # 16 bytes
upload_test_file(integration_client, project, package, content, version="range-suffix")
# Request last 4 bytes
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/range-suffix",
params={"mode": "proxy"},
headers={"Range": "bytes=-4"},
)
assert response.status_code == 206
assert response.content == b"CDEF"
@pytest.mark.integration
def test_range_request_open_ended(self, integration_client, test_package):
"""Test range request from offset to end."""
project, package = test_package
content = b"0123456789"
upload_test_file(integration_client, project, package, content, version="range-open")
# Request from byte 5 to end
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/range-open",
params={"mode": "proxy"},
headers={"Range": "bytes=5-"},
)
assert response.status_code == 206
assert response.content == b"56789"
@pytest.mark.integration
def test_range_request_includes_accept_ranges_header(
self, integration_client, test_package
):
"""Test that range requests include Accept-Ranges header."""
project, package = test_package
content = b"test content"
upload_test_file(integration_client, project, package, content, version="accept-ranges")
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/accept-ranges",
params={"mode": "proxy"},
headers={"Range": "bytes=0-4"},
)
assert response.status_code == 206
assert response.headers.get("Accept-Ranges") == "bytes"
@pytest.mark.integration
def test_full_download_advertises_accept_ranges(
self, integration_client, test_package
):
"""Test that full downloads advertise range support."""
project, package = test_package
content = b"test content"
upload_test_file(integration_client, project, package, content, version="full-accept")
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/full-accept",
params={"mode": "proxy"},
)
assert response.status_code == 200
assert response.headers.get("Accept-Ranges") == "bytes"
class TestConditionalRequests:
"""Tests for conditional request handling (304 Not Modified)."""
@pytest.mark.integration
def test_if_none_match_returns_304(self, integration_client, test_package):
"""Test If-None-Match with matching ETag returns 304."""
project, package = test_package
content = b"conditional request test content"
expected_hash = compute_sha256(content)
upload_test_file(integration_client, project, package, content, version="cond-etag")
# Request with matching ETag
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/cond-etag",
params={"mode": "proxy"},
headers={"If-None-Match": f'"{expected_hash}"'},
)
assert response.status_code == 304
assert response.content == b"" # No body for 304
@pytest.mark.integration
def test_if_none_match_without_quotes(self, integration_client, test_package):
"""Test If-None-Match works with or without quotes."""
project, package = test_package
content = b"etag no quotes test"
expected_hash = compute_sha256(content)
upload_test_file(integration_client, project, package, content, version="cond-noquote")
# Request with ETag without quotes
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/cond-noquote",
params={"mode": "proxy"},
headers={"If-None-Match": expected_hash},
)
assert response.status_code == 304
@pytest.mark.integration
def test_if_none_match_mismatch_returns_200(self, integration_client, test_package):
"""Test If-None-Match with non-matching ETag returns 200."""
project, package = test_package
content = b"etag mismatch test"
upload_test_file(integration_client, project, package, content, version="cond-mismatch")
# Request with different ETag
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/cond-mismatch",
params={"mode": "proxy"},
headers={"If-None-Match": '"different-etag-value"'},
)
assert response.status_code == 200
assert response.content == content
@pytest.mark.integration
def test_if_modified_since_returns_304(self, integration_client, test_package):
"""Test If-Modified-Since with future date returns 304."""
project, package = test_package
content = b"modified since test"
upload_test_file(integration_client, project, package, content, version="cond-modified")
# Request with future date (artifact was definitely created before this)
future_date = formatdate(time.time() + 86400, usegmt=True) # Tomorrow
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/cond-modified",
params={"mode": "proxy"},
headers={"If-Modified-Since": future_date},
)
assert response.status_code == 304
@pytest.mark.integration
def test_if_modified_since_old_date_returns_200(
self, integration_client, test_package
):
"""Test If-Modified-Since with old date returns 200."""
project, package = test_package
content = b"old date test"
upload_test_file(integration_client, project, package, content, version="cond-old")
# Request with old date (2020-01-01)
old_date = "Wed, 01 Jan 2020 00:00:00 GMT"
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/cond-old",
params={"mode": "proxy"},
headers={"If-Modified-Since": old_date},
)
assert response.status_code == 200
assert response.content == content
@pytest.mark.integration
def test_304_includes_etag(self, integration_client, test_package):
"""Test 304 response includes ETag header."""
project, package = test_package
content = b"304 etag test"
expected_hash = compute_sha256(content)
upload_test_file(integration_client, project, package, content, version="304-etag")
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/304-etag",
params={"mode": "proxy"},
headers={"If-None-Match": f'"{expected_hash}"'},
)
assert response.status_code == 304
assert response.headers.get("ETag") == f'"{expected_hash}"'
@pytest.mark.integration
def test_304_includes_cache_control(self, integration_client, test_package):
"""Test 304 response includes Cache-Control header."""
project, package = test_package
content = b"304 cache test"
expected_hash = compute_sha256(content)
upload_test_file(integration_client, project, package, content, version="304-cache")
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/304-cache",
params={"mode": "proxy"},
headers={"If-None-Match": f'"{expected_hash}"'},
)
assert response.status_code == 304
assert "immutable" in response.headers.get("Cache-Control", "")
class TestCachingHeaders:
"""Tests for caching headers on download responses."""
@pytest.mark.integration
def test_download_includes_cache_control(self, integration_client, test_package):
"""Test download response includes Cache-Control header."""
project, package = test_package
content = b"cache control test"
upload_test_file(integration_client, project, package, content, version="cache-ctl")
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/cache-ctl",
params={"mode": "proxy"},
)
assert response.status_code == 200
cache_control = response.headers.get("Cache-Control", "")
assert "public" in cache_control
assert "immutable" in cache_control
assert "max-age" in cache_control
@pytest.mark.integration
def test_download_includes_last_modified(self, integration_client, test_package):
"""Test download response includes Last-Modified header."""
project, package = test_package
content = b"last modified test"
upload_test_file(integration_client, project, package, content, version="last-mod")
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/last-mod",
params={"mode": "proxy"},
)
assert response.status_code == 200
assert "Last-Modified" in response.headers
# Should be in RFC 7231 format
last_modified = response.headers["Last-Modified"]
assert "GMT" in last_modified
@pytest.mark.integration
def test_download_includes_etag(self, integration_client, test_package):
"""Test download response includes ETag header."""
project, package = test_package
content = b"etag header test"
expected_hash = compute_sha256(content)
upload_test_file(integration_client, project, package, content, version="etag-hdr")
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/etag-hdr",
params={"mode": "proxy"},
)
assert response.status_code == 200
assert response.headers.get("ETag") == f'"{expected_hash}"'
class TestDownloadResume:
"""Tests for download resume functionality using range requests."""
@pytest.mark.integration
def test_resume_download_after_partial(self, integration_client, test_package):
"""Test resuming download from where it left off."""
project, package = test_package
content = b"ABCDEFGHIJ" * 100 # 1000 bytes
upload_test_file(integration_client, project, package, content, version="resume-test")
# Simulate partial download (first 500 bytes)
response1 = integration_client.get(
f"/api/v1/project/{project}/{package}/+/resume-test",
params={"mode": "proxy"},
headers={"Range": "bytes=0-499"},
)
assert response1.status_code == 206
first_half = response1.content
assert len(first_half) == 500
# Resume from byte 500
response2 = integration_client.get(
f"/api/v1/project/{project}/{package}/+/resume-test",
params={"mode": "proxy"},
headers={"Range": "bytes=500-"},
)
assert response2.status_code == 206
second_half = response2.content
assert len(second_half) == 500
# Combine and verify
combined = first_half + second_half
assert combined == content
@pytest.mark.integration
def test_resume_with_etag_verification(self, integration_client, test_package):
"""Test that resumed download can verify content hasn't changed."""
project, package = test_package
content = b"resume etag verification test content"
expected_hash = compute_sha256(content)
upload_test_file(integration_client, project, package, content, version="resume-etag")
# Get ETag from first request
response1 = integration_client.get(
f"/api/v1/project/{project}/{package}/+/resume-etag",
params={"mode": "proxy"},
headers={"Range": "bytes=0-9"},
)
assert response1.status_code == 206
etag = response1.headers.get("ETag")
assert etag == f'"{expected_hash}"'
# Resume with If-Match to ensure content hasn't changed
# (Note: If-Match would fail and return 412 if content changed)
response2 = integration_client.get(
f"/api/v1/project/{project}/{package}/+/resume-etag",
params={"mode": "proxy"},
headers={"Range": "bytes=10-"},
)
assert response2.status_code == 206
# ETag should be the same
assert response2.headers.get("ETag") == etag
class TestLargeFileStreaming:
"""Tests for streaming large files."""
@pytest.mark.integration
def test_stream_1mb_file(self, integration_client, test_package, sized_content):
"""Test streaming a 1MB file."""
project, package = test_package
content, expected_hash = sized_content(SIZE_1MB, seed=500)
upload_test_file(integration_client, project, package, content, version="stream-1mb")
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/stream-1mb",
params={"mode": "proxy"},
)
assert response.status_code == 200
assert len(response.content) == SIZE_1MB
assert compute_sha256(response.content) == expected_hash
@pytest.mark.integration
def test_stream_large_file_has_correct_headers(
self, integration_client, test_package, sized_content
):
"""Test that large file streaming has correct headers."""
project, package = test_package
content, expected_hash = sized_content(SIZE_100KB, seed=501)
upload_test_file(integration_client, project, package, content, version="stream-hdr")
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/stream-hdr",
params={"mode": "proxy"},
)
assert response.status_code == 200
assert int(response.headers.get("Content-Length", 0)) == SIZE_100KB
assert response.headers.get("X-Checksum-SHA256") == expected_hash
assert response.headers.get("Accept-Ranges") == "bytes"
@pytest.mark.integration
def test_range_request_on_large_file(
self, integration_client, test_package, sized_content
):
"""Test range request on a larger file."""
project, package = test_package
content, _ = sized_content(SIZE_100KB, seed=502)
upload_test_file(integration_client, project, package, content, version="range-large")
# Request a slice from the middle
start = 50000
end = 50999
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/range-large",
params={"mode": "proxy"},
headers={"Range": f"bytes={start}-{end}"},
)
assert response.status_code == 206
assert len(response.content) == 1000
assert response.content == content[start : end + 1]
class TestDownloadModes:
"""Tests for different download modes."""
@pytest.mark.integration
def test_proxy_mode_streams_content(self, integration_client, test_package):
"""Test proxy mode streams content through backend."""
project, package = test_package
content = b"proxy mode test content"
upload_test_file(integration_client, project, package, content, version="mode-proxy")
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/mode-proxy",
params={"mode": "proxy"},
)
assert response.status_code == 200
assert response.content == content
@pytest.mark.integration
def test_presigned_mode_returns_url(self, integration_client, test_package):
"""Test presigned mode returns JSON with URL."""
project, package = test_package
content = b"presigned mode test"
upload_test_file(integration_client, project, package, content, version="mode-presign")
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/mode-presign",
params={"mode": "presigned"},
)
assert response.status_code == 200
data = response.json()
assert "url" in data
assert "expires_at" in data
assert data["url"].startswith("http")
@pytest.mark.integration
def test_redirect_mode_returns_302(self, integration_client, test_package):
"""Test redirect mode returns 302 to presigned URL."""
project, package = test_package
content = b"redirect mode test"
upload_test_file(integration_client, project, package, content, version="mode-redir")
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/mode-redir",
params={"mode": "redirect"},
follow_redirects=False,
)
assert response.status_code == 302
assert "Location" in response.headers
class TestIntegrityDuringStreaming:
"""Tests for data integrity during streaming downloads."""
@pytest.mark.integration
def test_checksum_header_matches_content(self, integration_client, test_package):
"""Test X-Checksum-SHA256 header matches actual downloaded content."""
project, package = test_package
content = b"integrity check content"
expected_hash = compute_sha256(content)
upload_test_file(integration_client, project, package, content, version="integrity")
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/integrity",
params={"mode": "proxy"},
)
assert response.status_code == 200
header_hash = response.headers.get("X-Checksum-SHA256")
actual_hash = compute_sha256(response.content)
assert header_hash == expected_hash
assert actual_hash == expected_hash
assert header_hash == actual_hash
@pytest.mark.integration
def test_etag_matches_content_hash(self, integration_client, test_package):
"""Test ETag header matches content hash."""
project, package = test_package
content = b"etag integrity test"
expected_hash = compute_sha256(content)
upload_test_file(integration_client, project, package, content, version="etag-int")
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/etag-int",
params={"mode": "proxy"},
)
assert response.status_code == 200
etag = response.headers.get("ETag", "").strip('"')
actual_hash = compute_sha256(response.content)
assert etag == expected_hash
assert actual_hash == expected_hash
@pytest.mark.integration
def test_digest_header_present(self, integration_client, test_package):
"""Test Digest header is present in RFC 3230 format."""
project, package = test_package
content = b"digest header test"
upload_test_file(integration_client, project, package, content, version="digest")
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/digest",
params={"mode": "proxy"},
)
assert response.status_code == 200
assert "Digest" in response.headers
assert response.headers["Digest"].startswith("sha-256=")

View File

@@ -1,403 +0,0 @@
"""
Integration tests for tag API endpoints.
Tests cover:
- Tag CRUD operations
- Tag listing with pagination and search
- Tag history tracking
- ref_count behavior with tag operations
"""
import pytest
from tests.factories import compute_sha256, upload_test_file
class TestTagCRUD:
"""Tests for tag create, read, delete operations."""
@pytest.mark.integration
def test_create_tag_via_upload(self, integration_client, test_package):
"""Test creating a tag via upload endpoint."""
project_name, package_name = test_package
result = upload_test_file(
integration_client,
project_name,
package_name,
b"tag create test",
tag="v1.0.0",
)
assert result["tag"] == "v1.0.0"
assert result["artifact_id"]
@pytest.mark.integration
def test_create_tag_via_post(
self, integration_client, test_package, unique_test_id
):
"""Test creating a tag via POST /tags endpoint."""
project_name, package_name = test_package
# First upload an artifact
result = upload_test_file(
integration_client,
project_name,
package_name,
b"artifact for tag",
)
artifact_id = result["artifact_id"]
# Create tag via POST
tag_name = f"post-tag-{unique_test_id}"
response = integration_client.post(
f"/api/v1/project/{project_name}/{package_name}/tags",
json={"name": tag_name, "artifact_id": artifact_id},
)
assert response.status_code == 200
data = response.json()
assert data["name"] == tag_name
assert data["artifact_id"] == artifact_id
@pytest.mark.integration
def test_get_tag(self, integration_client, test_package):
"""Test getting a tag by name."""
project_name, package_name = test_package
upload_test_file(
integration_client,
project_name,
package_name,
b"get tag test",
tag="get-tag",
)
response = integration_client.get(
f"/api/v1/project/{project_name}/{package_name}/tags/get-tag"
)
assert response.status_code == 200
data = response.json()
assert data["name"] == "get-tag"
assert "artifact_id" in data
assert "artifact_size" in data
assert "artifact_content_type" in data
@pytest.mark.integration
def test_list_tags(self, integration_client, test_package):
"""Test listing tags for a package."""
project_name, package_name = test_package
# Create some tags
upload_test_file(
integration_client,
project_name,
package_name,
b"list tags test",
tag="list-v1",
)
response = integration_client.get(
f"/api/v1/project/{project_name}/{package_name}/tags"
)
assert response.status_code == 200
data = response.json()
assert "items" in data
assert "pagination" in data
tag_names = [t["name"] for t in data["items"]]
assert "list-v1" in tag_names
@pytest.mark.integration
def test_delete_tag(self, integration_client, test_package):
"""Test deleting a tag."""
project_name, package_name = test_package
upload_test_file(
integration_client,
project_name,
package_name,
b"delete tag test",
tag="to-delete",
)
# Delete tag
response = integration_client.delete(
f"/api/v1/project/{project_name}/{package_name}/tags/to-delete"
)
assert response.status_code == 204
# Verify deleted
response = integration_client.get(
f"/api/v1/project/{project_name}/{package_name}/tags/to-delete"
)
assert response.status_code == 404
class TestTagListingFilters:
"""Tests for tag listing with filters and search."""
@pytest.mark.integration
def test_tags_pagination(self, integration_client, test_package):
"""Test tag listing respects pagination."""
project_name, package_name = test_package
response = integration_client.get(
f"/api/v1/project/{project_name}/{package_name}/tags?limit=5"
)
assert response.status_code == 200
data = response.json()
assert len(data["items"]) <= 5
assert data["pagination"]["limit"] == 5
@pytest.mark.integration
def test_tags_search(self, integration_client, test_package, unique_test_id):
"""Test tag search by name."""
project_name, package_name = test_package
tag_name = f"searchable-{unique_test_id}"
upload_test_file(
integration_client,
project_name,
package_name,
b"search test",
tag=tag_name,
)
response = integration_client.get(
f"/api/v1/project/{project_name}/{package_name}/tags?search=searchable"
)
assert response.status_code == 200
data = response.json()
tag_names = [t["name"] for t in data["items"]]
assert tag_name in tag_names
class TestTagHistory:
"""Tests for tag history tracking."""
@pytest.mark.integration
def test_tag_history_on_create(self, integration_client, test_package):
"""Test tag history is created when tag is created."""
project_name, package_name = test_package
upload_test_file(
integration_client,
project_name,
package_name,
b"history create test",
tag="history-create",
)
response = integration_client.get(
f"/api/v1/project/{project_name}/{package_name}/tags/history-create/history"
)
assert response.status_code == 200
data = response.json()
assert len(data) >= 1
@pytest.mark.integration
def test_tag_history_on_update(
self, integration_client, test_package, unique_test_id
):
"""Test tag history is created when tag is updated."""
project_name, package_name = test_package
tag_name = f"history-update-{unique_test_id}"
# Create tag with first artifact
upload_test_file(
integration_client,
project_name,
package_name,
b"first content",
tag=tag_name,
)
# Update tag with second artifact
upload_test_file(
integration_client,
project_name,
package_name,
b"second content",
tag=tag_name,
)
response = integration_client.get(
f"/api/v1/project/{project_name}/{package_name}/tags/{tag_name}/history"
)
assert response.status_code == 200
data = response.json()
# Should have at least 2 history entries (create + update)
assert len(data) >= 2
class TestTagRefCount:
"""Tests for ref_count behavior with tag operations."""
@pytest.mark.integration
def test_ref_count_decrements_on_tag_delete(self, integration_client, test_package):
"""Test ref_count decrements when a tag is deleted."""
project_name, package_name = test_package
content = b"ref count delete test"
expected_hash = compute_sha256(content)
# Upload with two tags
upload_test_file(
integration_client, project_name, package_name, content, tag="rc-v1"
)
upload_test_file(
integration_client, project_name, package_name, content, tag="rc-v2"
)
# Verify ref_count is 2
response = integration_client.get(f"/api/v1/artifact/{expected_hash}")
assert response.json()["ref_count"] == 2
# Delete one tag
delete_response = integration_client.delete(
f"/api/v1/project/{project_name}/{package_name}/tags/rc-v1"
)
assert delete_response.status_code == 204
# Verify ref_count is now 1
response = integration_client.get(f"/api/v1/artifact/{expected_hash}")
assert response.json()["ref_count"] == 1
@pytest.mark.integration
def test_ref_count_zero_after_all_tags_deleted(
self, integration_client, test_package
):
"""Test ref_count goes to 0 when all tags are deleted."""
project_name, package_name = test_package
content = b"orphan test content"
expected_hash = compute_sha256(content)
# Upload with one tag
upload_test_file(
integration_client, project_name, package_name, content, tag="only-tag"
)
# Delete the tag
integration_client.delete(
f"/api/v1/project/{project_name}/{package_name}/tags/only-tag"
)
# Verify ref_count is 0
response = integration_client.get(f"/api/v1/artifact/{expected_hash}")
assert response.json()["ref_count"] == 0
@pytest.mark.integration
def test_ref_count_adjusts_on_tag_update(
self, integration_client, test_package, unique_test_id
):
"""Test ref_count adjusts when a tag is updated to point to different artifact."""
project_name, package_name = test_package
# Upload two different artifacts
content1 = f"artifact one {unique_test_id}".encode()
content2 = f"artifact two {unique_test_id}".encode()
hash1 = compute_sha256(content1)
hash2 = compute_sha256(content2)
# Upload first artifact with tag "latest"
upload_test_file(
integration_client, project_name, package_name, content1, tag="latest"
)
# Verify first artifact has ref_count 1
response = integration_client.get(f"/api/v1/artifact/{hash1}")
assert response.json()["ref_count"] == 1
# Upload second artifact with different tag
upload_test_file(
integration_client, project_name, package_name, content2, tag="stable"
)
# Now update "latest" tag to point to second artifact
upload_test_file(
integration_client, project_name, package_name, content2, tag="latest"
)
# Verify first artifact ref_count decreased to 0
response = integration_client.get(f"/api/v1/artifact/{hash1}")
assert response.json()["ref_count"] == 0
# Verify second artifact ref_count increased to 2
response = integration_client.get(f"/api/v1/artifact/{hash2}")
assert response.json()["ref_count"] == 2
@pytest.mark.integration
def test_ref_count_unchanged_when_tag_same_artifact(
self, integration_client, test_package, unique_test_id
):
"""Test ref_count doesn't change when tag is 'updated' to same artifact."""
project_name, package_name = test_package
content = f"same artifact {unique_test_id}".encode()
expected_hash = compute_sha256(content)
# Upload with tag
upload_test_file(
integration_client, project_name, package_name, content, tag="same-v1"
)
# Verify ref_count is 1
response = integration_client.get(f"/api/v1/artifact/{expected_hash}")
assert response.json()["ref_count"] == 1
# Upload same content with same tag (no-op)
upload_test_file(
integration_client, project_name, package_name, content, tag="same-v1"
)
# Verify ref_count is still 1
response = integration_client.get(f"/api/v1/artifact/{expected_hash}")
assert response.json()["ref_count"] == 1
@pytest.mark.integration
def test_tag_via_post_endpoint_increments_ref_count(
self, integration_client, test_package, unique_test_id
):
"""Test creating tag via POST /tags endpoint increments ref_count."""
project_name, package_name = test_package
content = f"tag endpoint test {unique_test_id}".encode()
expected_hash = compute_sha256(content)
# Upload artifact without tag
result = upload_test_file(
integration_client, project_name, package_name, content, filename="test.bin"
)
artifact_id = result["artifact_id"]
# Verify ref_count is 0 (no tags yet)
response = integration_client.get(f"/api/v1/artifact/{expected_hash}")
assert response.json()["ref_count"] == 0
# Create tag via POST endpoint
tag_response = integration_client.post(
f"/api/v1/project/{project_name}/{package_name}/tags",
json={"name": "post-v1", "artifact_id": artifact_id},
)
assert tag_response.status_code == 200
# Verify ref_count is now 1
response = integration_client.get(f"/api/v1/artifact/{expected_hash}")
assert response.json()["ref_count"] == 1
# Create another tag via POST endpoint
tag_response = integration_client.post(
f"/api/v1/project/{project_name}/{package_name}/tags",
json={"name": "post-latest", "artifact_id": artifact_id},
)
assert tag_response.status_code == 200
# Verify ref_count is now 2
response = integration_client.get(f"/api/v1/artifact/{expected_hash}")
assert response.json()["ref_count"] == 2

View File

@@ -0,0 +1,316 @@
"""
Integration tests for Teams API endpoints.
"""
import pytest
@pytest.mark.integration
class TestTeamsCRUD:
"""Tests for team creation, listing, updating, and deletion."""
def test_create_team(self, integration_client, unique_test_id):
"""Test creating a new team."""
team_name = f"Test Team {unique_test_id}"
team_slug = f"test-team-{unique_test_id}"
response = integration_client.post(
"/api/v1/teams",
json={
"name": team_name,
"slug": team_slug,
"description": "A test team",
},
)
assert response.status_code == 201, f"Failed to create team: {response.text}"
data = response.json()
assert data["name"] == team_name
assert data["slug"] == team_slug
assert data["description"] == "A test team"
assert data["user_role"] == "owner"
assert data["member_count"] == 1
assert data["project_count"] == 0
# Cleanup
integration_client.delete(f"/api/v1/teams/{team_slug}")
def test_create_team_duplicate_slug(self, integration_client, unique_test_id):
"""Test that duplicate team slugs are rejected."""
team_slug = f"dup-team-{unique_test_id}"
# Create first team
response = integration_client.post(
"/api/v1/teams",
json={"name": "First Team", "slug": team_slug},
)
assert response.status_code == 201
# Try to create second team with same slug
response = integration_client.post(
"/api/v1/teams",
json={"name": "Second Team", "slug": team_slug},
)
assert response.status_code == 400
assert "already exists" in response.json()["detail"].lower()
# Cleanup
integration_client.delete(f"/api/v1/teams/{team_slug}")
def test_create_team_invalid_slug(self, integration_client):
"""Test that invalid team slugs are rejected."""
invalid_slugs = [
"UPPERCASE",
"with spaces",
"-starts-with-hyphen",
"ends-with-hyphen-",
"has--double--hyphen",
]
for invalid_slug in invalid_slugs:
response = integration_client.post(
"/api/v1/teams",
json={"name": "Test", "slug": invalid_slug},
)
assert response.status_code == 422, f"Slug '{invalid_slug}' should be invalid"
def test_list_teams(self, integration_client, unique_test_id):
"""Test listing teams the user belongs to."""
# Create a team
team_slug = f"list-team-{unique_test_id}"
integration_client.post(
"/api/v1/teams",
json={"name": "List Test Team", "slug": team_slug},
)
# List teams
response = integration_client.get("/api/v1/teams")
assert response.status_code == 200
data = response.json()
assert "items" in data
assert "pagination" in data
# Find our team
team = next((t for t in data["items"] if t["slug"] == team_slug), None)
assert team is not None
assert team["name"] == "List Test Team"
# Cleanup
integration_client.delete(f"/api/v1/teams/{team_slug}")
def test_get_team(self, integration_client, unique_test_id):
"""Test getting team details."""
team_slug = f"get-team-{unique_test_id}"
integration_client.post(
"/api/v1/teams",
json={"name": "Get Test Team", "slug": team_slug, "description": "Test"},
)
response = integration_client.get(f"/api/v1/teams/{team_slug}")
assert response.status_code == 200
data = response.json()
assert data["slug"] == team_slug
assert data["name"] == "Get Test Team"
assert data["user_role"] == "owner"
# Cleanup
integration_client.delete(f"/api/v1/teams/{team_slug}")
def test_get_nonexistent_team(self, integration_client):
"""Test getting a team that doesn't exist."""
response = integration_client.get("/api/v1/teams/nonexistent-team-12345")
assert response.status_code == 404
def test_update_team(self, integration_client, unique_test_id):
"""Test updating team details."""
team_slug = f"update-team-{unique_test_id}"
integration_client.post(
"/api/v1/teams",
json={"name": "Original Name", "slug": team_slug},
)
response = integration_client.put(
f"/api/v1/teams/{team_slug}",
json={"name": "Updated Name", "description": "New description"},
)
assert response.status_code == 200
data = response.json()
assert data["name"] == "Updated Name"
assert data["description"] == "New description"
assert data["slug"] == team_slug # Slug should not change
# Cleanup
integration_client.delete(f"/api/v1/teams/{team_slug}")
def test_delete_team(self, integration_client, unique_test_id):
"""Test deleting a team."""
team_slug = f"delete-team-{unique_test_id}"
integration_client.post(
"/api/v1/teams",
json={"name": "Delete Test Team", "slug": team_slug},
)
response = integration_client.delete(f"/api/v1/teams/{team_slug}")
assert response.status_code == 204
# Verify team is gone
response = integration_client.get(f"/api/v1/teams/{team_slug}")
assert response.status_code == 404
@pytest.mark.integration
class TestTeamMembers:
"""Tests for team membership management."""
@pytest.fixture
def test_team(self, integration_client, unique_test_id):
"""Create a test team for member tests."""
team_slug = f"member-team-{unique_test_id}"
response = integration_client.post(
"/api/v1/teams",
json={"name": "Member Test Team", "slug": team_slug},
)
assert response.status_code == 201
yield team_slug
# Cleanup
try:
integration_client.delete(f"/api/v1/teams/{team_slug}")
except Exception:
pass
def test_list_members(self, integration_client, test_team):
"""Test listing team members."""
response = integration_client.get(f"/api/v1/teams/{test_team}/members")
assert response.status_code == 200
members = response.json()
assert len(members) == 1
assert members[0]["role"] == "owner"
def test_owner_is_first_member(self, integration_client, test_team):
"""Test that the team creator is automatically the owner."""
response = integration_client.get(f"/api/v1/teams/{test_team}/members")
members = response.json()
assert len(members) >= 1
owner = next((m for m in members if m["role"] == "owner"), None)
assert owner is not None
@pytest.mark.integration
class TestTeamProjects:
"""Tests for team project management."""
@pytest.fixture
def test_team(self, integration_client, unique_test_id):
"""Create a test team for project tests."""
team_slug = f"proj-team-{unique_test_id}"
response = integration_client.post(
"/api/v1/teams",
json={"name": "Project Test Team", "slug": team_slug},
)
assert response.status_code == 201
data = response.json()
yield {"slug": team_slug, "id": data["id"]}
# Cleanup
try:
integration_client.delete(f"/api/v1/teams/{team_slug}")
except Exception:
pass
def test_list_team_projects_empty(self, integration_client, test_team):
"""Test listing projects in an empty team."""
response = integration_client.get(f"/api/v1/teams/{test_team['slug']}/projects")
assert response.status_code == 200
data = response.json()
assert data["items"] == []
assert data["pagination"]["total"] == 0
def test_create_project_in_team(self, integration_client, test_team, unique_test_id):
"""Test creating a project within a team."""
project_name = f"team-project-{unique_test_id}"
response = integration_client.post(
"/api/v1/projects",
json={
"name": project_name,
"description": "A team project",
"team_id": test_team["id"],
},
)
assert response.status_code == 200, f"Failed to create project: {response.text}"
data = response.json()
assert data["team_id"] == test_team["id"]
assert data["team_slug"] == test_team["slug"]
# Verify project appears in team projects list
response = integration_client.get(f"/api/v1/teams/{test_team['slug']}/projects")
assert response.status_code == 200
projects = response.json()["items"]
assert any(p["name"] == project_name for p in projects)
# Cleanup
integration_client.delete(f"/api/v1/projects/{project_name}")
def test_project_team_info_in_response(self, integration_client, test_team, unique_test_id):
"""Test that project responses include team info."""
project_name = f"team-info-project-{unique_test_id}"
# Create project in team
integration_client.post(
"/api/v1/projects",
json={"name": project_name, "team_id": test_team["id"]},
)
# Get project and verify team info
response = integration_client.get(f"/api/v1/projects/{project_name}")
assert response.status_code == 200
data = response.json()
assert data["team_id"] == test_team["id"]
assert data["team_slug"] == test_team["slug"]
assert data["team_name"] == "Project Test Team"
# Cleanup
integration_client.delete(f"/api/v1/projects/{project_name}")
@pytest.mark.integration
class TestTeamAuthorization:
"""Tests for team-based authorization."""
def test_cannot_delete_team_with_projects(self, integration_client, unique_test_id):
"""Test that teams with projects cannot be deleted."""
team_slug = f"nodelete-team-{unique_test_id}"
project_name = f"nodelete-project-{unique_test_id}"
# Create team
response = integration_client.post(
"/api/v1/teams",
json={"name": "No Delete Team", "slug": team_slug},
)
team_id = response.json()["id"]
# Create project in team
integration_client.post(
"/api/v1/projects",
json={"name": project_name, "team_id": team_id},
)
# Try to delete team - should fail
response = integration_client.delete(f"/api/v1/teams/{team_slug}")
assert response.status_code == 400
assert "project" in response.json()["detail"].lower()
# Cleanup - delete project first, then team
integration_client.delete(f"/api/v1/projects/{project_name}")
integration_client.delete(f"/api/v1/teams/{team_slug}")

View File

@@ -10,6 +10,7 @@ Tests cover:
- S3 storage verification
"""
import os
import pytest
import io
import threading
@@ -25,6 +26,19 @@ from tests.factories import (
class TestUploadBasics:
"""Tests for basic upload functionality."""
@pytest.mark.integration
def test_upload_returns_200(self, integration_client, test_package):
"""Test upload with valid file returns 200."""
project, package = test_package
content = b"valid file upload test"
files = {"file": ("test.bin", io.BytesIO(content), "application/octet-stream")}
response = integration_client.post(
f"/api/v1/project/{project}/{package}/upload",
files=files,
)
assert response.status_code == 200
@pytest.mark.integration
def test_upload_returns_artifact_id(self, integration_client, test_package):
"""Test upload returns the artifact ID (SHA256 hash)."""
@@ -33,7 +47,7 @@ class TestUploadBasics:
expected_hash = compute_sha256(content)
result = upload_test_file(
integration_client, project_name, package_name, content, tag="v1"
integration_client, project_name, package_name, content, version="v1"
)
assert result["artifact_id"] == expected_hash
@@ -101,6 +115,79 @@ class TestUploadBasics:
assert "created_at" in result
assert result["created_at"] is not None
@pytest.mark.integration
def test_upload_without_version_succeeds(self, integration_client, test_package):
"""Test upload without version succeeds (no version created)."""
project, package = test_package
content = b"upload without version test"
expected_hash = compute_sha256(content)
files = {"file": ("no_version.bin", io.BytesIO(content), "application/octet-stream")}
response = integration_client.post(
f"/api/v1/project/{project}/{package}/upload",
files=files,
# No version parameter
)
assert response.status_code == 200
result = response.json()
assert result["artifact_id"] == expected_hash
# Version should be None when not specified
assert result.get("version") is None
@pytest.mark.integration
def test_upload_creates_artifact_in_database(self, integration_client, test_package):
"""Test upload creates artifact record in database."""
project, package = test_package
content = b"database artifact test"
expected_hash = compute_sha256(content)
upload_test_file(integration_client, project, package, content)
# Verify artifact exists via API
response = integration_client.get(f"/api/v1/artifact/{expected_hash}")
assert response.status_code == 200
artifact = response.json()
assert artifact["id"] == expected_hash
assert artifact["size"] == len(content)
@pytest.mark.integration
@pytest.mark.requires_direct_s3
def test_upload_creates_object_in_s3(self, integration_client, test_package):
"""Test upload creates object in S3 storage."""
project, package = test_package
content = b"s3 object creation test"
expected_hash = compute_sha256(content)
upload_test_file(integration_client, project, package, content)
# Verify S3 object exists
assert s3_object_exists(expected_hash), "S3 object should exist after upload"
@pytest.mark.integration
def test_upload_with_version_creates_version_record(self, integration_client, test_package):
"""Test upload with version creates version record."""
project, package = test_package
content = b"version creation test"
expected_hash = compute_sha256(content)
version_name = "1.0.0"
result = upload_test_file(
integration_client, project, package, content, version=version_name
)
# Verify version was created
assert result.get("version") == version_name
assert result["artifact_id"] == expected_hash
# Verify version exists in versions list
versions_response = integration_client.get(
f"/api/v1/project/{project}/{package}/versions"
)
assert versions_response.status_code == 200
versions = versions_response.json()
version_names = [v["version"] for v in versions.get("items", [])]
assert version_name in version_names
class TestDuplicateUploads:
"""Tests for duplicate upload deduplication behavior."""
@@ -116,36 +203,44 @@ class TestDuplicateUploads:
# First upload
result1 = upload_test_file(
integration_client, project, package, content, tag="first"
integration_client, project, package, content, version="first"
)
assert result1["artifact_id"] == expected_hash
# Second upload
result2 = upload_test_file(
integration_client, project, package, content, tag="second"
integration_client, project, package, content, version="second"
)
assert result2["artifact_id"] == expected_hash
assert result1["artifact_id"] == result2["artifact_id"]
@pytest.mark.integration
def test_same_file_twice_increments_ref_count(
def test_same_file_twice_returns_existing_version(
self, integration_client, test_package
):
"""Test uploading same file twice increments ref_count to 2."""
"""Test uploading same file twice in same package returns existing version.
Same artifact can only have one version per package. Uploading the same content
with a different version name returns the existing version, not a new one.
ref_count stays at 1 because there's still only one PackageVersion reference.
"""
project, package = test_package
content = b"content for ref count increment test"
# First upload
result1 = upload_test_file(
integration_client, project, package, content, tag="v1"
integration_client, project, package, content, version="v1"
)
assert result1["ref_count"] == 1
# Second upload
# Second upload with different version name returns existing version
result2 = upload_test_file(
integration_client, project, package, content, tag="v2"
integration_client, project, package, content, version="v2"
)
assert result2["ref_count"] == 2
# Same artifact, same package = same version returned, ref_count stays 1
assert result2["ref_count"] == 1
assert result2["deduplicated"] is True
assert result1["version"] == result2["version"] # Both return "v1"
@pytest.mark.integration
def test_same_file_different_packages_shares_artifact(
@@ -170,12 +265,12 @@ class TestDuplicateUploads:
)
# Upload to first package
result1 = upload_test_file(integration_client, project, pkg1, content, tag="v1")
result1 = upload_test_file(integration_client, project, pkg1, content, version="v1")
assert result1["artifact_id"] == expected_hash
assert result1["deduplicated"] is False
# Upload to second package
result2 = upload_test_file(integration_client, project, pkg2, content, tag="v1")
result2 = upload_test_file(integration_client, project, pkg2, content, version="v1")
assert result2["artifact_id"] == expected_hash
assert result2["deduplicated"] is True
@@ -195,7 +290,7 @@ class TestDuplicateUploads:
package,
content,
filename="file1.bin",
tag="v1",
version="v1",
)
assert result1["artifact_id"] == expected_hash
@@ -206,7 +301,7 @@ class TestDuplicateUploads:
package,
content,
filename="file2.bin",
tag="v2",
version="v2",
)
assert result2["artifact_id"] == expected_hash
assert result2["deduplicated"] is True
@@ -216,17 +311,17 @@ class TestDownload:
"""Tests for download functionality."""
@pytest.mark.integration
def test_download_by_tag(self, integration_client, test_package):
"""Test downloading artifact by tag name."""
def test_download_by_version(self, integration_client, test_package):
"""Test downloading artifact by version."""
project, package = test_package
original_content = b"download by tag test"
original_content = b"download by version test"
upload_test_file(
integration_client, project, package, original_content, tag="download-tag"
integration_client, project, package, original_content, version="1.0.0"
)
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/download-tag",
f"/api/v1/project/{project}/{package}/+/1.0.0",
params={"mode": "proxy"},
)
assert response.status_code == 200
@@ -249,12 +344,56 @@ class TestDownload:
assert response.content == original_content
@pytest.mark.integration
def test_download_nonexistent_tag(self, integration_client, test_package):
"""Test downloading nonexistent tag returns 404."""
def test_download_by_version_prefix(self, integration_client, test_package):
"""Test downloading artifact using version: prefix."""
project, package = test_package
original_content = b"download by version prefix test"
upload_test_file(
integration_client, project, package, original_content, version="2.0.0"
)
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/version:2.0.0",
params={"mode": "proxy"},
)
assert response.status_code == 200
assert response.content == original_content
@pytest.mark.integration
def test_download_nonexistent_version(self, integration_client, test_package):
"""Test downloading nonexistent version returns 404."""
project, package = test_package
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/nonexistent-tag"
f"/api/v1/project/{project}/{package}/+/nonexistent-version"
)
assert response.status_code == 404
@pytest.mark.integration
def test_download_nonexistent_artifact(self, integration_client, test_package):
"""Test downloading nonexistent artifact ID returns 404."""
project, package = test_package
fake_hash = "0" * 64
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/artifact:{fake_hash}"
)
assert response.status_code == 404
@pytest.mark.integration
def test_download_from_nonexistent_project(self, integration_client, unique_test_id):
"""Test downloading from nonexistent project returns 404."""
response = integration_client.get(
f"/api/v1/project/nonexistent-project-{unique_test_id}/somepackage/+/sometag"
)
assert response.status_code == 404
@pytest.mark.integration
def test_download_from_nonexistent_package(self, integration_client, test_project, unique_test_id):
"""Test downloading from nonexistent package returns 404."""
response = integration_client.get(
f"/api/v1/project/{test_project}/nonexistent-package-{unique_test_id}/+/sometag"
)
assert response.status_code == 404
@@ -265,7 +404,7 @@ class TestDownload:
original_content = b"exact content verification test data 12345"
upload_test_file(
integration_client, project, package, original_content, tag="verify"
integration_client, project, package, original_content, version="verify"
)
response = integration_client.get(
@@ -275,21 +414,140 @@ class TestDownload:
assert response.content == original_content
class TestDownloadHeaders:
"""Tests for download response headers."""
@pytest.mark.integration
def test_download_content_type_header(self, integration_client, test_package):
"""Test download returns correct Content-Type header."""
project, package = test_package
content = b"content type header test"
upload_test_file(
integration_client, project, package, content,
filename="test.txt", version="content-type-test"
)
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/content-type-test",
params={"mode": "proxy"},
)
assert response.status_code == 200
# Content-Type should be set (either text/plain or application/octet-stream)
assert "content-type" in response.headers
@pytest.mark.integration
def test_download_content_length_header(self, integration_client, test_package):
"""Test download returns correct Content-Length header."""
project, package = test_package
content = b"content length header test - exactly 41 bytes!"
expected_length = len(content)
upload_test_file(
integration_client, project, package, content, version="content-length-test"
)
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/content-length-test",
params={"mode": "proxy"},
)
assert response.status_code == 200
assert "content-length" in response.headers
assert int(response.headers["content-length"]) == expected_length
@pytest.mark.integration
def test_download_content_disposition_header(self, integration_client, test_package):
"""Test download returns correct Content-Disposition header."""
project, package = test_package
content = b"content disposition test"
filename = "my-test-file.bin"
upload_test_file(
integration_client, project, package, content,
filename=filename, version="disposition-test"
)
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/disposition-test",
params={"mode": "proxy"},
)
assert response.status_code == 200
assert "content-disposition" in response.headers
disposition = response.headers["content-disposition"]
assert "attachment" in disposition
assert filename in disposition
@pytest.mark.integration
def test_download_checksum_headers(self, integration_client, test_package):
"""Test download returns checksum headers."""
project, package = test_package
content = b"checksum header test content"
expected_hash = compute_sha256(content)
upload_test_file(
integration_client, project, package, content, version="checksum-headers"
)
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/checksum-headers",
params={"mode": "proxy"},
)
assert response.status_code == 200
# Check for checksum headers
assert "x-checksum-sha256" in response.headers
assert response.headers["x-checksum-sha256"] == expected_hash
@pytest.mark.integration
def test_download_etag_header(self, integration_client, test_package):
"""Test download returns ETag header (artifact ID)."""
project, package = test_package
content = b"etag header test"
expected_hash = compute_sha256(content)
upload_test_file(
integration_client, project, package, content, version="etag-test"
)
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/etag-test",
params={"mode": "proxy"},
)
assert response.status_code == 200
assert "etag" in response.headers
# ETag should contain the artifact ID (hash)
etag = response.headers["etag"].strip('"')
assert etag == expected_hash
class TestConcurrentUploads:
"""Tests for concurrent upload handling."""
@pytest.mark.integration
def test_concurrent_uploads_same_file(self, integration_client, test_package):
"""Test concurrent uploads of same file handle deduplication correctly."""
project, package = test_package
def test_concurrent_uploads_same_file(self, integration_client, test_project, unique_test_id):
"""Test concurrent uploads of same file to different packages handle deduplication correctly.
Same artifact can only have one version per package, so we create multiple packages
to test that concurrent uploads to different packages correctly increment ref_count.
"""
content = b"content for concurrent upload test"
expected_hash = compute_sha256(content)
num_concurrent = 5
# Create packages for each concurrent upload
packages = []
for i in range(num_concurrent):
pkg_name = f"concurrent-pkg-{unique_test_id}-{i}"
response = integration_client.post(
f"/api/v1/project/{test_project}/packages",
json={"name": pkg_name},
)
assert response.status_code == 200
packages.append(pkg_name)
# Create an API key for worker threads
api_key_response = integration_client.post(
"/api/v1/auth/keys",
json={"name": "concurrent-test-key"},
json={"name": f"concurrent-test-key-{unique_test_id}"},
)
assert api_key_response.status_code == 200, f"Failed to create API key: {api_key_response.text}"
api_key = api_key_response.json()["key"]
@@ -297,23 +555,23 @@ class TestConcurrentUploads:
results = []
errors = []
def upload_worker(tag_suffix):
def upload_worker(idx):
try:
from httpx import Client
base_url = "http://localhost:8080"
base_url = os.environ.get("ORCHARD_TEST_URL", "http://localhost:8080")
with Client(base_url=base_url, timeout=30.0) as client:
files = {
"file": (
f"concurrent-{tag_suffix}.bin",
f"concurrent-{idx}.bin",
io.BytesIO(content),
"application/octet-stream",
)
}
response = client.post(
f"/api/v1/project/{project}/{package}/upload",
f"/api/v1/project/{test_project}/{packages[idx]}/upload",
files=files,
data={"tag": f"concurrent-{tag_suffix}"},
data={"version": "1.0.0"},
headers={"Authorization": f"Bearer {api_key}"},
)
if response.status_code == 200:
@@ -336,7 +594,7 @@ class TestConcurrentUploads:
assert len(artifact_ids) == 1
assert expected_hash in artifact_ids
# Verify final ref_count
# Verify final ref_count equals number of packages
response = integration_client.get(f"/api/v1/artifact/{expected_hash}")
assert response.status_code == 200
assert response.json()["ref_count"] == num_concurrent
@@ -365,7 +623,7 @@ class TestFileSizeValidation:
content = b"X"
result = upload_test_file(
integration_client, project, package, content, tag="tiny"
integration_client, project, package, content, version="tiny"
)
assert result["artifact_id"] is not None
@@ -381,7 +639,7 @@ class TestFileSizeValidation:
expected_size = len(content)
result = upload_test_file(
integration_client, project, package, content, tag="size-test"
integration_client, project, package, content, version="size-test"
)
assert result["size"] == expected_size
@@ -397,6 +655,7 @@ class TestUploadFailureCleanup:
"""Tests for cleanup when uploads fail."""
@pytest.mark.integration
@pytest.mark.requires_direct_s3
def test_upload_failure_invalid_project_no_orphaned_s3(
self, integration_client, unique_test_id
):
@@ -408,7 +667,7 @@ class TestUploadFailureCleanup:
response = integration_client.post(
f"/api/v1/project/nonexistent-project-{unique_test_id}/nonexistent-pkg/upload",
files=files,
data={"tag": "test"},
data={"version": "test"},
)
assert response.status_code == 404
@@ -419,6 +678,7 @@ class TestUploadFailureCleanup:
)
@pytest.mark.integration
@pytest.mark.requires_direct_s3
def test_upload_failure_invalid_package_no_orphaned_s3(
self, integration_client, test_project, unique_test_id
):
@@ -430,7 +690,7 @@ class TestUploadFailureCleanup:
response = integration_client.post(
f"/api/v1/project/{test_project}/nonexistent-package-{unique_test_id}/upload",
files=files,
data={"tag": "test"},
data={"version": "test"},
)
assert response.status_code == 404
@@ -451,7 +711,7 @@ class TestUploadFailureCleanup:
response = integration_client.post(
f"/api/v1/project/{test_project}/nonexistent-package-{unique_test_id}/upload",
files=files,
data={"tag": "test"},
data={"version": "test"},
)
assert response.status_code == 404
@@ -466,6 +726,7 @@ class TestS3StorageVerification:
"""Tests to verify S3 storage behavior."""
@pytest.mark.integration
@pytest.mark.requires_direct_s3
def test_s3_single_object_after_duplicates(
self, integration_client, test_package, unique_test_id
):
@@ -476,7 +737,7 @@ class TestS3StorageVerification:
# Upload same content multiple times
for tag in ["s3test1", "s3test2", "s3test3"]:
upload_test_file(integration_client, project, package, content, tag=tag)
upload_test_file(integration_client, project, package, content, version=tag)
# Verify only one S3 object exists
s3_objects = list_s3_objects_by_hash(expected_hash)
@@ -492,16 +753,26 @@ class TestS3StorageVerification:
@pytest.mark.integration
def test_artifact_table_single_row_after_duplicates(
self, integration_client, test_package
self, integration_client, test_project, unique_test_id
):
"""Test artifact table contains only one row after duplicate uploads."""
project, package = test_package
"""Test artifact table contains only one row after duplicate uploads to different packages.
Same artifact can only have one version per package, so we create multiple packages
to test deduplication across packages.
"""
content = b"content for single row test"
expected_hash = compute_sha256(content)
# Upload same content multiple times
for tag in ["v1", "v2", "v3"]:
upload_test_file(integration_client, project, package, content, tag=tag)
# Create 3 packages and upload same content to each
for i in range(3):
pkg_name = f"single-row-pkg-{unique_test_id}-{i}"
integration_client.post(
f"/api/v1/project/{test_project}/packages",
json={"name": pkg_name},
)
upload_test_file(
integration_client, test_project, pkg_name, content, version="1.0.0"
)
# Query artifact
response = integration_client.get(f"/api/v1/artifact/{expected_hash}")
@@ -521,6 +792,7 @@ class TestSecurityPathTraversal:
"""
@pytest.mark.integration
@pytest.mark.requires_direct_s3
def test_path_traversal_in_filename_stored_safely(
self, integration_client, test_package
):
@@ -539,7 +811,7 @@ class TestSecurityPathTraversal:
response = integration_client.post(
f"/api/v1/project/{project}/{package}/upload",
files=files,
data={"tag": "traversal-test"},
data={"version": "traversal-test"},
)
assert response.status_code == 200
result = response.json()
@@ -557,48 +829,16 @@ class TestSecurityPathTraversal:
assert response.status_code in [400, 404, 422]
@pytest.mark.integration
def test_path_traversal_in_tag_name(self, integration_client, test_package):
"""Test tag names with path traversal are handled safely."""
def test_path_traversal_in_version_name(self, integration_client, test_package):
"""Test version names with path traversal are handled safely."""
project, package = test_package
content = b"tag traversal test"
content = b"version traversal test"
files = {"file": ("test.bin", io.BytesIO(content), "application/octet-stream")}
response = integration_client.post(
f"/api/v1/project/{project}/{package}/upload",
files=files,
data={"tag": "../../../etc/passwd"},
)
assert response.status_code in [200, 400, 422]
@pytest.mark.integration
def test_download_path_traversal_in_ref(self, integration_client, test_package):
"""Test download ref with path traversal is rejected."""
project, package = test_package
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/../../../etc/passwd"
)
assert response.status_code in [400, 404, 422]
@pytest.mark.integration
def test_path_traversal_in_package_name(self, integration_client, test_project):
"""Test package names with path traversal sequences are rejected."""
response = integration_client.get(
f"/api/v1/project/{test_project}/packages/../../../etc/passwd"
)
assert response.status_code in [400, 404, 422]
@pytest.mark.integration
def test_path_traversal_in_tag_name(self, integration_client, test_package):
"""Test tag names with path traversal are rejected or handled safely."""
project, package = test_package
content = b"tag traversal test"
files = {"file": ("test.bin", io.BytesIO(content), "application/octet-stream")}
response = integration_client.post(
f"/api/v1/project/{project}/{package}/upload",
files=files,
data={"tag": "../../../etc/passwd"},
data={"version": "../../../etc/passwd"},
)
assert response.status_code in [200, 400, 422]
@@ -623,7 +863,7 @@ class TestSecurityMalformedRequests:
response = integration_client.post(
f"/api/v1/project/{project}/{package}/upload",
data={"tag": "no-file"},
data={"version": "no-file"},
)
assert response.status_code == 422

View File

@@ -0,0 +1,305 @@
"""
Integration tests for package version API endpoints.
Tests cover:
- Version creation via upload
- Version auto-detection from filename
- Version listing and retrieval
- Download by version prefix
- Version deletion
"""
import pytest
import io
from tests.factories import (
compute_sha256,
upload_test_file,
)
class TestVersionCreation:
"""Tests for creating versions via upload."""
@pytest.mark.integration
def test_upload_with_explicit_version(self, integration_client, test_package):
"""Test upload with explicit version parameter creates version record."""
project, package = test_package
content = b"version creation test"
expected_hash = compute_sha256(content)
files = {"file": ("app.tar.gz", io.BytesIO(content), "application/octet-stream")}
response = integration_client.post(
f"/api/v1/project/{project}/{package}/upload",
files=files,
data={"version": "1.0.0"},
)
assert response.status_code == 200
result = response.json()
assert result["artifact_id"] == expected_hash
assert result.get("version") == "1.0.0"
assert result.get("version_source") == "explicit"
@pytest.mark.integration
def test_duplicate_version_same_content_succeeds(self, integration_client, test_package):
"""Test uploading same version with same content succeeds (deduplication)."""
project, package = test_package
content = b"version dedup test"
# First upload with version
files1 = {"file": ("app1.tar.gz", io.BytesIO(content), "application/octet-stream")}
response1 = integration_client.post(
f"/api/v1/project/{project}/{package}/upload",
files=files1,
data={"version": "3.0.0"},
)
assert response1.status_code == 200
# Second upload with same version and same content succeeds
files2 = {"file": ("app2.tar.gz", io.BytesIO(content), "application/octet-stream")}
response2 = integration_client.post(
f"/api/v1/project/{project}/{package}/upload",
files=files2,
data={"version": "3.0.0"},
)
# This succeeds because it's the same artifact (deduplication)
assert response2.status_code == 200
class TestVersionAutoDetection:
"""Tests for automatic version detection from filename."""
@pytest.mark.integration
def test_version_detected_from_filename_tarball(self, integration_client, test_package):
"""Test version is auto-detected from tarball filename or metadata."""
project, package = test_package
content = b"auto detect version tarball"
files = {"file": ("myapp-1.2.3.tar.gz", io.BytesIO(content), "application/octet-stream")}
response = integration_client.post(
f"/api/v1/project/{project}/{package}/upload",
files=files,
)
assert response.status_code == 200
result = response.json()
assert result.get("version") == "1.2.3"
# Version source can be 'filename' or 'metadata' depending on detection order
assert result.get("version_source") in ["filename", "metadata"]
@pytest.mark.integration
def test_version_detected_from_filename_zip(self, integration_client, test_package):
"""Test version is auto-detected from zip filename."""
project, package = test_package
content = b"auto detect version zip"
files = {"file": ("package-2.0.0.zip", io.BytesIO(content), "application/octet-stream")}
response = integration_client.post(
f"/api/v1/project/{project}/{package}/upload",
files=files,
)
assert response.status_code == 200
result = response.json()
assert result.get("version") == "2.0.0"
assert result.get("version_source") == "filename"
@pytest.mark.integration
def test_explicit_version_overrides_filename(self, integration_client, test_package):
"""Test explicit version parameter overrides filename detection."""
project, package = test_package
content = b"explicit override test"
files = {"file": ("myapp-1.0.0.tar.gz", io.BytesIO(content), "application/octet-stream")}
response = integration_client.post(
f"/api/v1/project/{project}/{package}/upload",
files=files,
data={"version": "9.9.9"},
)
assert response.status_code == 200
result = response.json()
assert result.get("version") == "9.9.9"
assert result.get("version_source") == "explicit"
@pytest.mark.integration
def test_no_version_detected_from_plain_filename(self, integration_client, test_package):
"""Test no version is created for filenames without version pattern."""
project, package = test_package
content = b"no version in filename"
files = {"file": ("plain-file.bin", io.BytesIO(content), "application/octet-stream")}
response = integration_client.post(
f"/api/v1/project/{project}/{package}/upload",
files=files,
)
assert response.status_code == 200
result = response.json()
# Version should be None or not present
assert result.get("version") is None
class TestVersionListing:
"""Tests for listing and retrieving versions."""
@pytest.mark.integration
def test_list_versions(self, integration_client, test_package):
"""Test listing all versions for a package."""
project, package = test_package
# Create multiple versions
for ver in ["1.0.0", "1.1.0", "2.0.0"]:
content = f"version {ver} content".encode()
files = {"file": (f"app-{ver}.tar.gz", io.BytesIO(content), "application/octet-stream")}
response = integration_client.post(
f"/api/v1/project/{project}/{package}/upload",
files=files,
data={"version": ver},
)
assert response.status_code == 200
# List versions
response = integration_client.get(
f"/api/v1/project/{project}/{package}/versions"
)
assert response.status_code == 200
data = response.json()
versions = [v["version"] for v in data.get("items", data)]
assert "1.0.0" in versions
assert "1.1.0" in versions
assert "2.0.0" in versions
@pytest.mark.integration
def test_get_specific_version(self, integration_client, test_package):
"""Test getting details for a specific version."""
project, package = test_package
content = b"specific version test"
expected_hash = compute_sha256(content)
# Create version
files = {"file": ("app-4.0.0.tar.gz", io.BytesIO(content), "application/octet-stream")}
integration_client.post(
f"/api/v1/project/{project}/{package}/upload",
files=files,
data={"version": "4.0.0"},
)
# Get version details
response = integration_client.get(
f"/api/v1/project/{project}/{package}/versions/4.0.0"
)
assert response.status_code == 200
data = response.json()
assert data["version"] == "4.0.0"
assert data["artifact_id"] == expected_hash
@pytest.mark.integration
def test_get_nonexistent_version_returns_404(self, integration_client, test_package):
"""Test getting nonexistent version returns 404."""
project, package = test_package
response = integration_client.get(
f"/api/v1/project/{project}/{package}/versions/99.99.99"
)
assert response.status_code == 404
class TestDownloadByVersion:
"""Tests for downloading artifacts by version."""
@pytest.mark.integration
def test_download_by_version_prefix(self, integration_client, test_package):
"""Test downloading artifact using version: prefix."""
project, package = test_package
content = b"download by version test"
expected_hash = compute_sha256(content)
# Upload with version
files = {"file": ("app.tar.gz", io.BytesIO(content), "application/octet-stream")}
integration_client.post(
f"/api/v1/project/{project}/{package}/upload",
files=files,
data={"version": "5.0.0"},
)
# Download by version prefix
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/version:5.0.0",
params={"mode": "proxy"},
)
assert response.status_code == 200
assert response.content == content
@pytest.mark.integration
def test_download_nonexistent_version_returns_404(self, integration_client, test_package):
"""Test downloading nonexistent version returns 404."""
project, package = test_package
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/version:99.0.0"
)
assert response.status_code == 404
@pytest.mark.integration
def test_version_resolution_with_prefix(self, integration_client, test_package):
"""Test that version: prefix explicitly resolves to version."""
project, package = test_package
version_content = b"this is the version content"
# Create a version 6.0.0
files1 = {"file": ("app-v.tar.gz", io.BytesIO(version_content), "application/octet-stream")}
integration_client.post(
f"/api/v1/project/{project}/{package}/upload",
files=files1,
data={"version": "6.0.0"},
)
# Download with version: prefix should get version content
response = integration_client.get(
f"/api/v1/project/{project}/{package}/+/version:6.0.0",
params={"mode": "proxy"},
)
assert response.status_code == 200
assert response.content == version_content
class TestVersionDeletion:
"""Tests for deleting versions."""
@pytest.mark.integration
def test_delete_version(self, integration_client, test_package):
"""Test deleting a version."""
project, package = test_package
content = b"delete version test"
# Create version
files = {"file": ("app.tar.gz", io.BytesIO(content), "application/octet-stream")}
integration_client.post(
f"/api/v1/project/{project}/{package}/upload",
files=files,
data={"version": "7.0.0"},
)
# Verify version exists
response = integration_client.get(
f"/api/v1/project/{project}/{package}/versions/7.0.0"
)
assert response.status_code == 200
# Delete version - returns 204 No Content on success
delete_response = integration_client.delete(
f"/api/v1/project/{project}/{package}/versions/7.0.0"
)
assert delete_response.status_code == 204
# Verify version no longer exists
response2 = integration_client.get(
f"/api/v1/project/{project}/{package}/versions/7.0.0"
)
assert response2.status_code == 404
@pytest.mark.integration
def test_delete_nonexistent_version_returns_404(self, integration_client, test_package):
"""Test deleting nonexistent version returns 404."""
project, package = test_package
response = integration_client.delete(
f"/api/v1/project/{project}/{package}/versions/99.0.0"
)
assert response.status_code == 404

View File

@@ -0,0 +1,322 @@
"""
Integration tests for version API endpoints.
Tests cover:
- Version creation via upload
- Version auto-detection from filename
- Version listing with pagination
- Version deletion
- Download by version ref
- ref_count behavior with version operations
"""
import pytest
from tests.factories import upload_test_file
class TestVersionCreation:
"""Tests for version creation during upload."""
@pytest.mark.integration
def test_upload_with_explicit_version(self, integration_client, test_package):
"""Test creating a version via explicit version parameter."""
project_name, package_name = test_package
result = upload_test_file(
integration_client,
project_name,
package_name,
b"version create test",
version="1.0.0",
)
assert result["version"] == "1.0.0"
assert result["version_source"] == "explicit"
assert result["artifact_id"]
@pytest.mark.integration
def test_upload_with_version_auto_detect_from_tarball(
self, integration_client, test_package
):
"""Test version auto-detection from tarball filename pattern."""
project_name, package_name = test_package
result = upload_test_file(
integration_client,
project_name,
package_name,
b"auto version test",
filename="myapp-2.1.0.tar.gz",
)
assert result["version"] == "2.1.0"
# Tarball metadata extractor parses version from filename
assert result["version_source"] == "metadata"
@pytest.mark.integration
def test_upload_with_version_auto_detect_v_prefix(
self, integration_client, test_package
):
"""Test version auto-detection strips 'v' prefix from tarball filename."""
project_name, package_name = test_package
result = upload_test_file(
integration_client,
project_name,
package_name,
b"v prefix test",
filename="package-v3.0.0.tar.gz",
)
assert result["version"] == "3.0.0"
# Tarball metadata extractor parses version from filename
assert result["version_source"] == "metadata"
@pytest.mark.integration
def test_upload_duplicate_version_warning(self, integration_client, test_package):
"""Test that duplicate version during upload returns response without error."""
project_name, package_name = test_package
# Upload with version 1.0.0
upload_test_file(
integration_client,
project_name,
package_name,
b"first upload",
version="1.0.0",
)
# Upload different content with same version - should succeed but no new version
result = upload_test_file(
integration_client,
project_name,
package_name,
b"second upload different content",
version="1.0.0",
)
# Upload succeeds but version may not be set (duplicate)
assert result["artifact_id"]
class TestVersionCRUD:
"""Tests for version list, get, delete operations."""
@pytest.mark.integration
def test_list_versions(self, integration_client, test_package):
"""Test listing versions for a package."""
project_name, package_name = test_package
# Create some versions
upload_test_file(
integration_client,
project_name,
package_name,
b"v1 content",
version="1.0.0",
)
upload_test_file(
integration_client,
project_name,
package_name,
b"v2 content",
version="2.0.0",
)
response = integration_client.get(
f"/api/v1/project/{project_name}/{package_name}/versions"
)
assert response.status_code == 200
data = response.json()
assert "items" in data
assert "pagination" in data
versions = [v["version"] for v in data["items"]]
assert "1.0.0" in versions
assert "2.0.0" in versions
@pytest.mark.integration
def test_list_versions_with_artifact_info(self, integration_client, test_package):
"""Test that version list includes artifact metadata."""
project_name, package_name = test_package
upload_test_file(
integration_client,
project_name,
package_name,
b"version with info",
version="1.0.0",
)
response = integration_client.get(
f"/api/v1/project/{project_name}/{package_name}/versions"
)
assert response.status_code == 200
data = response.json()
assert len(data["items"]) >= 1
version_item = next(
(v for v in data["items"] if v["version"] == "1.0.0"), None
)
assert version_item is not None
assert "size" in version_item
assert "artifact_id" in version_item
@pytest.mark.integration
def test_get_version(self, integration_client, test_package):
"""Test getting a specific version."""
project_name, package_name = test_package
upload_result = upload_test_file(
integration_client,
project_name,
package_name,
b"get version test",
version="3.0.0",
)
response = integration_client.get(
f"/api/v1/project/{project_name}/{package_name}/versions/3.0.0"
)
assert response.status_code == 200
data = response.json()
assert data["version"] == "3.0.0"
assert data["artifact_id"] == upload_result["artifact_id"]
assert data["version_source"] == "explicit"
@pytest.mark.integration
def test_get_version_not_found(self, integration_client, test_package):
"""Test getting a non-existent version returns 404."""
project_name, package_name = test_package
response = integration_client.get(
f"/api/v1/project/{project_name}/{package_name}/versions/99.99.99"
)
assert response.status_code == 404
@pytest.mark.integration
def test_delete_version(self, integration_client, test_package):
"""Test deleting a version."""
project_name, package_name = test_package
upload_test_file(
integration_client,
project_name,
package_name,
b"delete version test",
version="4.0.0",
)
# Delete version
response = integration_client.delete(
f"/api/v1/project/{project_name}/{package_name}/versions/4.0.0"
)
assert response.status_code == 204
# Verify deleted
response = integration_client.get(
f"/api/v1/project/{project_name}/{package_name}/versions/4.0.0"
)
assert response.status_code == 404
class TestVersionDownload:
"""Tests for downloading artifacts by version reference."""
@pytest.mark.integration
def test_download_by_version_prefix(self, integration_client, test_package):
"""Test downloading an artifact using version: prefix."""
project_name, package_name = test_package
content = b"download by version test"
upload_test_file(
integration_client,
project_name,
package_name,
content,
version="5.0.0",
)
response = integration_client.get(
f"/api/v1/project/{project_name}/{package_name}/+/version:5.0.0",
follow_redirects=False,
)
# Should either redirect or return content
assert response.status_code in [200, 302, 307]
@pytest.mark.integration
def test_download_by_implicit_version(self, integration_client, test_package):
"""Test downloading an artifact using version number directly (no prefix)."""
project_name, package_name = test_package
content = b"implicit version download test"
upload_test_file(
integration_client,
project_name,
package_name,
content,
version="6.0.0",
)
response = integration_client.get(
f"/api/v1/project/{project_name}/{package_name}/+/6.0.0",
follow_redirects=False,
)
# Should resolve version
assert response.status_code in [200, 302, 307]
class TestVersionPagination:
"""Tests for version listing pagination and sorting."""
@pytest.mark.integration
def test_versions_pagination(self, integration_client, test_package):
"""Test version listing respects pagination."""
project_name, package_name = test_package
response = integration_client.get(
f"/api/v1/project/{project_name}/{package_name}/versions?limit=5"
)
assert response.status_code == 200
data = response.json()
assert "pagination" in data
assert data["pagination"]["limit"] == 5
@pytest.mark.integration
def test_versions_sorting(self, integration_client, test_package):
"""Test version listing can be sorted."""
project_name, package_name = test_package
# Create versions with different timestamps
upload_test_file(
integration_client,
project_name,
package_name,
b"sort test 1",
version="1.0.0",
)
upload_test_file(
integration_client,
project_name,
package_name,
b"sort test 2",
version="2.0.0",
)
# Test ascending sort
response = integration_client.get(
f"/api/v1/project/{project_name}/{package_name}/versions?sort=version&order=asc"
)
assert response.status_code == 200
data = response.json()
versions = [v["version"] for v in data["items"]]
# First version should be 1.0.0 when sorted ascending
if len(versions) >= 2:
assert versions.index("1.0.0") < versions.index("2.0.0")

File diff suppressed because it is too large Load Diff

View File

@@ -26,16 +26,16 @@ def upload_test_file(integration_client):
Factory fixture to upload a test file and return its artifact ID.
Usage:
artifact_id = upload_test_file(project, package, content, tag="v1.0")
artifact_id = upload_test_file(project, package, content, version="v1.0")
"""
def _upload(project_name: str, package_name: str, content: bytes, tag: str = None):
def _upload(project_name: str, package_name: str, content: bytes, version: str = None):
files = {
"file": ("test-file.bin", io.BytesIO(content), "application/octet-stream")
}
data = {}
if tag:
data["tag"] = tag
if version:
data["version"] = version
response = integration_client.post(
f"/api/v1/project/{project_name}/{package_name}/upload",
@@ -66,7 +66,7 @@ class TestDownloadChecksumHeaders:
# Upload file
artifact_id = upload_test_file(
project_name, package_name, content, tag="sha256-header-test"
project_name, package_name, content, version="sha256-header-test"
)
# Download with proxy mode
@@ -88,7 +88,7 @@ class TestDownloadChecksumHeaders:
content = b"Content for ETag header test"
artifact_id = upload_test_file(
project_name, package_name, content, tag="etag-test"
project_name, package_name, content, version="etag-test"
)
response = integration_client.get(
@@ -110,7 +110,7 @@ class TestDownloadChecksumHeaders:
content = b"Content for Digest header test"
sha256 = hashlib.sha256(content).hexdigest()
upload_test_file(project_name, package_name, content, tag="digest-test")
upload_test_file(project_name, package_name, content, version="digest-test")
response = integration_client.get(
f"/api/v1/project/{project_name}/{package_name}/+/digest-test",
@@ -137,7 +137,7 @@ class TestDownloadChecksumHeaders:
project_name, package_name = test_package
content = b"Content for X-Content-Length test"
upload_test_file(project_name, package_name, content, tag="content-length-test")
upload_test_file(project_name, package_name, content, version="content-length-test")
response = integration_client.get(
f"/api/v1/project/{project_name}/{package_name}/+/content-length-test",
@@ -156,7 +156,7 @@ class TestDownloadChecksumHeaders:
project_name, package_name = test_package
content = b"Content for X-Verified false test"
upload_test_file(project_name, package_name, content, tag="verified-false-test")
upload_test_file(project_name, package_name, content, version="verified-false-test")
response = integration_client.get(
f"/api/v1/project/{project_name}/{package_name}/+/verified-false-test",
@@ -184,7 +184,7 @@ class TestPreVerificationMode:
project_name, package_name = test_package
content = b"Content for pre-verification success test"
upload_test_file(project_name, package_name, content, tag="pre-verify-success")
upload_test_file(project_name, package_name, content, version="pre-verify-success")
response = integration_client.get(
f"/api/v1/project/{project_name}/{package_name}/+/pre-verify-success",
@@ -205,7 +205,7 @@ class TestPreVerificationMode:
# Use binary content to verify no corruption
content = bytes(range(256)) * 10 # 2560 bytes of all byte values
upload_test_file(project_name, package_name, content, tag="pre-verify-content")
upload_test_file(project_name, package_name, content, version="pre-verify-content")
response = integration_client.get(
f"/api/v1/project/{project_name}/{package_name}/+/pre-verify-content",
@@ -233,7 +233,7 @@ class TestStreamingVerificationMode:
content = b"Content for streaming verification success test"
upload_test_file(
project_name, package_name, content, tag="stream-verify-success"
project_name, package_name, content, version="stream-verify-success"
)
response = integration_client.get(
@@ -255,7 +255,7 @@ class TestStreamingVerificationMode:
# 100KB of content
content = b"x" * (100 * 1024)
upload_test_file(project_name, package_name, content, tag="stream-verify-large")
upload_test_file(project_name, package_name, content, version="stream-verify-large")
response = integration_client.get(
f"/api/v1/project/{project_name}/{package_name}/+/stream-verify-large",
@@ -283,7 +283,7 @@ class TestHeadRequestHeaders:
content = b"Content for HEAD SHA256 test"
artifact_id = upload_test_file(
project_name, package_name, content, tag="head-sha256-test"
project_name, package_name, content, version="head-sha256-test"
)
response = integration_client.head(
@@ -303,7 +303,7 @@ class TestHeadRequestHeaders:
content = b"Content for HEAD ETag test"
artifact_id = upload_test_file(
project_name, package_name, content, tag="head-etag-test"
project_name, package_name, content, version="head-etag-test"
)
response = integration_client.head(
@@ -322,7 +322,7 @@ class TestHeadRequestHeaders:
project_name, package_name = test_package
content = b"Content for HEAD Digest test"
upload_test_file(project_name, package_name, content, tag="head-digest-test")
upload_test_file(project_name, package_name, content, version="head-digest-test")
response = integration_client.head(
f"/api/v1/project/{project_name}/{package_name}/+/head-digest-test"
@@ -340,7 +340,7 @@ class TestHeadRequestHeaders:
project_name, package_name = test_package
content = b"Content for HEAD Content-Length test"
upload_test_file(project_name, package_name, content, tag="head-length-test")
upload_test_file(project_name, package_name, content, version="head-length-test")
response = integration_client.head(
f"/api/v1/project/{project_name}/{package_name}/+/head-length-test"
@@ -356,7 +356,7 @@ class TestHeadRequestHeaders:
project_name, package_name = test_package
content = b"Content for HEAD no-body test"
upload_test_file(project_name, package_name, content, tag="head-no-body-test")
upload_test_file(project_name, package_name, content, version="head-no-body-test")
response = integration_client.head(
f"/api/v1/project/{project_name}/{package_name}/+/head-no-body-test"
@@ -382,7 +382,7 @@ class TestRangeRequestHeaders:
project_name, package_name = test_package
content = b"Content for range request checksum header test"
upload_test_file(project_name, package_name, content, tag="range-checksum-test")
upload_test_file(project_name, package_name, content, version="range-checksum-test")
response = integration_client.get(
f"/api/v1/project/{project_name}/{package_name}/+/range-checksum-test",
@@ -412,7 +412,7 @@ class TestClientSideVerification:
project_name, package_name = test_package
content = b"Content for client-side verification test"
upload_test_file(project_name, package_name, content, tag="client-verify-test")
upload_test_file(project_name, package_name, content, version="client-verify-test")
response = integration_client.get(
f"/api/v1/project/{project_name}/{package_name}/+/client-verify-test",
@@ -438,7 +438,7 @@ class TestClientSideVerification:
project_name, package_name = test_package
content = b"Content for Digest header verification"
upload_test_file(project_name, package_name, content, tag="digest-verify-test")
upload_test_file(project_name, package_name, content, version="digest-verify-test")
response = integration_client.get(
f"/api/v1/project/{project_name}/{package_name}/+/digest-verify-test",

File diff suppressed because it is too large Load Diff

View File

@@ -0,0 +1,95 @@
"""Unit tests for authentication module."""
import pytest
from unittest.mock import patch, MagicMock
class TestCreateDefaultAdmin:
"""Tests for the create_default_admin function."""
def test_create_default_admin_with_env_password(self):
"""Test that ORCHARD_ADMIN_PASSWORD env var sets admin password."""
from app.auth import create_default_admin, verify_password
# Create mock settings with custom password
mock_settings = MagicMock()
mock_settings.admin_password = "my-custom-password-123"
# Mock database session
mock_db = MagicMock()
mock_db.query.return_value.count.return_value = 0 # No existing users
# Track the user that gets created
created_user = None
def capture_user(user):
nonlocal created_user
created_user = user
mock_db.add.side_effect = capture_user
with patch("app.auth.get_settings", return_value=mock_settings):
admin = create_default_admin(mock_db)
# Verify the user was created
assert mock_db.add.called
assert created_user is not None
assert created_user.username == "admin"
assert created_user.is_admin is True
# Password should NOT require change when set via env var
assert created_user.must_change_password is False
# Verify password was hashed correctly
assert verify_password("my-custom-password-123", created_user.password_hash)
def test_create_default_admin_with_default_password(self):
"""Test that default password 'changeme123' is used when env var not set."""
from app.auth import create_default_admin, verify_password
# Create mock settings with empty password (default)
mock_settings = MagicMock()
mock_settings.admin_password = ""
# Mock database session
mock_db = MagicMock()
mock_db.query.return_value.count.return_value = 0 # No existing users
# Track the user that gets created
created_user = None
def capture_user(user):
nonlocal created_user
created_user = user
mock_db.add.side_effect = capture_user
with patch("app.auth.get_settings", return_value=mock_settings):
admin = create_default_admin(mock_db)
# Verify the user was created
assert mock_db.add.called
assert created_user is not None
assert created_user.username == "admin"
assert created_user.is_admin is True
# Password SHOULD require change when using default
assert created_user.must_change_password is True
# Verify default password was used
assert verify_password("changeme123", created_user.password_hash)
def test_create_default_admin_skips_when_users_exist(self):
"""Test that no admin is created when users already exist."""
from app.auth import create_default_admin
# Create mock settings
mock_settings = MagicMock()
mock_settings.admin_password = "some-password"
# Mock database session with existing users
mock_db = MagicMock()
mock_db.query.return_value.count.return_value = 1 # Users exist
with patch("app.auth.get_settings", return_value=mock_settings):
result = create_default_admin(mock_db)
# Should return None and not create any user
assert result is None
assert not mock_db.add.called

View File

@@ -0,0 +1,374 @@
"""Tests for CacheService."""
import pytest
from unittest.mock import MagicMock, AsyncMock, patch
class TestCacheCategory:
"""Tests for cache category enum."""
@pytest.mark.unit
def test_immutable_categories_have_no_ttl(self):
"""Immutable categories should return None for TTL."""
from app.cache_service import CacheCategory, get_category_ttl
from app.config import Settings
settings = Settings()
assert get_category_ttl(CacheCategory.ARTIFACT_METADATA, settings) is None
assert get_category_ttl(CacheCategory.ARTIFACT_DEPENDENCIES, settings) is None
assert get_category_ttl(CacheCategory.DEPENDENCY_RESOLUTION, settings) is None
@pytest.mark.unit
def test_mutable_categories_have_ttl(self):
"""Mutable categories should return configured TTL."""
from app.cache_service import CacheCategory, get_category_ttl
from app.config import Settings
settings = Settings(
cache_ttl_index=300,
cache_ttl_upstream=3600,
)
assert get_category_ttl(CacheCategory.PACKAGE_INDEX, settings) == 300
assert get_category_ttl(CacheCategory.UPSTREAM_SOURCES, settings) == 3600
class TestCacheService:
"""Tests for Redis cache service."""
@pytest.mark.asyncio
@pytest.mark.unit
async def test_disabled_cache_returns_none(self):
"""When Redis disabled, get() should return None."""
from app.cache_service import CacheService, CacheCategory
from app.config import Settings
settings = Settings(redis_enabled=False)
cache = CacheService(settings)
await cache.startup()
result = await cache.get(CacheCategory.PACKAGE_INDEX, "test-key")
assert result is None
await cache.shutdown()
@pytest.mark.asyncio
@pytest.mark.unit
async def test_disabled_cache_set_is_noop(self):
"""When Redis disabled, set() should be a no-op."""
from app.cache_service import CacheService, CacheCategory
from app.config import Settings
settings = Settings(redis_enabled=False)
cache = CacheService(settings)
await cache.startup()
# Should not raise
await cache.set(CacheCategory.PACKAGE_INDEX, "test-key", b"test-value")
await cache.shutdown()
@pytest.mark.asyncio
@pytest.mark.unit
async def test_cache_key_namespacing(self):
"""Cache keys should be properly namespaced."""
from app.cache_service import CacheService, CacheCategory
key = CacheService._make_key(CacheCategory.PACKAGE_INDEX, "pypi", "numpy")
assert key == "orchard:index:pypi:numpy"
@pytest.mark.asyncio
@pytest.mark.unit
async def test_ping_returns_false_when_disabled(self):
"""ping() should return False when Redis is disabled."""
from app.cache_service import CacheService
from app.config import Settings
settings = Settings(redis_enabled=False)
cache = CacheService(settings)
await cache.startup()
result = await cache.ping()
assert result is False
await cache.shutdown()
@pytest.mark.asyncio
@pytest.mark.unit
async def test_enabled_property(self):
"""enabled property should reflect Redis state."""
from app.cache_service import CacheService
from app.config import Settings
settings = Settings(redis_enabled=False)
cache = CacheService(settings)
assert cache.enabled is False
@pytest.mark.asyncio
@pytest.mark.unit
async def test_delete_is_noop_when_disabled(self):
"""delete() should be a no-op when Redis is disabled."""
from app.cache_service import CacheService, CacheCategory
from app.config import Settings
settings = Settings(redis_enabled=False)
cache = CacheService(settings)
await cache.startup()
# Should not raise
await cache.delete(CacheCategory.PACKAGE_INDEX, "test-key")
await cache.shutdown()
@pytest.mark.asyncio
@pytest.mark.unit
async def test_invalidate_pattern_returns_zero_when_disabled(self):
"""invalidate_pattern() should return 0 when Redis is disabled."""
from app.cache_service import CacheService, CacheCategory
from app.config import Settings
settings = Settings(redis_enabled=False)
cache = CacheService(settings)
await cache.startup()
result = await cache.invalidate_pattern(CacheCategory.PACKAGE_INDEX)
assert result == 0
await cache.shutdown()
@pytest.mark.asyncio
@pytest.mark.unit
async def test_startup_already_started(self):
"""startup() should be idempotent."""
from app.cache_service import CacheService
from app.config import Settings
settings = Settings(redis_enabled=False)
cache = CacheService(settings)
await cache.startup()
await cache.startup() # Should not raise
assert cache._started is True
await cache.shutdown()
@pytest.mark.asyncio
@pytest.mark.unit
async def test_shutdown_not_started(self):
"""shutdown() should handle not-started state."""
from app.cache_service import CacheService
from app.config import Settings
settings = Settings(redis_enabled=False)
cache = CacheService(settings)
# Should not raise
await cache.shutdown()
@pytest.mark.asyncio
@pytest.mark.unit
async def test_make_key_with_default_protocol(self):
"""_make_key should work with default protocol."""
from app.cache_service import CacheService, CacheCategory
key = CacheService._make_key(CacheCategory.ARTIFACT_METADATA, "default", "abc123")
assert key == "orchard:artifact:default:abc123"
class TestCacheServiceWithMockedRedis:
"""Tests for CacheService with mocked Redis client."""
@pytest.mark.asyncio
@pytest.mark.unit
async def test_get_returns_cached_value(self):
"""get() should return cached value when available."""
from app.cache_service import CacheService, CacheCategory
from app.config import Settings
settings = Settings(redis_enabled=True)
cache = CacheService(settings)
# Mock the redis client
mock_redis = AsyncMock()
mock_redis.get.return_value = b"cached-data"
cache._redis = mock_redis
cache._enabled = True
cache._started = True
result = await cache.get(CacheCategory.PACKAGE_INDEX, "test-key", "pypi")
assert result == b"cached-data"
mock_redis.get.assert_called_once_with("orchard:index:pypi:test-key")
@pytest.mark.asyncio
@pytest.mark.unit
async def test_set_with_ttl(self):
"""set() should use setex for mutable categories."""
from app.cache_service import CacheService, CacheCategory
from app.config import Settings
settings = Settings(redis_enabled=True, cache_ttl_index=300)
cache = CacheService(settings)
mock_redis = AsyncMock()
cache._redis = mock_redis
cache._enabled = True
cache._started = True
await cache.set(CacheCategory.PACKAGE_INDEX, "test-key", b"test-value", "pypi")
mock_redis.setex.assert_called_once_with(
"orchard:index:pypi:test-key", 300, b"test-value"
)
@pytest.mark.asyncio
@pytest.mark.unit
async def test_set_without_ttl(self):
"""set() should use set (no expiry) for immutable categories."""
from app.cache_service import CacheService, CacheCategory
from app.config import Settings
settings = Settings(redis_enabled=True)
cache = CacheService(settings)
mock_redis = AsyncMock()
cache._redis = mock_redis
cache._enabled = True
cache._started = True
await cache.set(
CacheCategory.ARTIFACT_METADATA, "abc123", b"metadata", "pypi"
)
mock_redis.set.assert_called_once_with(
"orchard:artifact:pypi:abc123", b"metadata"
)
@pytest.mark.asyncio
@pytest.mark.unit
async def test_delete_calls_redis_delete(self):
"""delete() should call Redis delete."""
from app.cache_service import CacheService, CacheCategory
from app.config import Settings
settings = Settings(redis_enabled=True)
cache = CacheService(settings)
mock_redis = AsyncMock()
cache._redis = mock_redis
cache._enabled = True
cache._started = True
await cache.delete(CacheCategory.PACKAGE_INDEX, "test-key", "pypi")
mock_redis.delete.assert_called_once_with("orchard:index:pypi:test-key")
@pytest.mark.asyncio
@pytest.mark.unit
async def test_invalidate_pattern_deletes_matching_keys(self):
"""invalidate_pattern() should delete all matching keys."""
from app.cache_service import CacheService, CacheCategory
from app.config import Settings
settings = Settings(redis_enabled=True)
cache = CacheService(settings)
mock_redis = AsyncMock()
# Create an async generator for scan_iter
async def mock_scan_iter(match=None):
for key in [b"orchard:index:pypi:numpy", b"orchard:index:pypi:requests"]:
yield key
mock_redis.scan_iter = mock_scan_iter
mock_redis.delete.return_value = 2
cache._redis = mock_redis
cache._enabled = True
cache._started = True
result = await cache.invalidate_pattern(CacheCategory.PACKAGE_INDEX, "*", "pypi")
assert result == 2
mock_redis.delete.assert_called_once()
@pytest.mark.asyncio
@pytest.mark.unit
async def test_ping_returns_true_when_connected(self):
"""ping() should return True when Redis responds."""
from app.cache_service import CacheService
from app.config import Settings
settings = Settings(redis_enabled=True)
cache = CacheService(settings)
mock_redis = AsyncMock()
mock_redis.ping.return_value = True
cache._redis = mock_redis
cache._enabled = True
cache._started = True
result = await cache.ping()
assert result is True
@pytest.mark.asyncio
@pytest.mark.unit
async def test_get_handles_exception(self):
"""get() should return None and log warning on exception."""
from app.cache_service import CacheService, CacheCategory
from app.config import Settings
settings = Settings(redis_enabled=True)
cache = CacheService(settings)
mock_redis = AsyncMock()
mock_redis.get.side_effect = Exception("Connection lost")
cache._redis = mock_redis
cache._enabled = True
cache._started = True
result = await cache.get(CacheCategory.PACKAGE_INDEX, "test-key")
assert result is None
@pytest.mark.asyncio
@pytest.mark.unit
async def test_set_handles_exception(self):
"""set() should log warning on exception."""
from app.cache_service import CacheService, CacheCategory
from app.config import Settings
settings = Settings(redis_enabled=True, cache_ttl_index=300)
cache = CacheService(settings)
mock_redis = AsyncMock()
mock_redis.setex.side_effect = Exception("Connection lost")
cache._redis = mock_redis
cache._enabled = True
cache._started = True
# Should not raise
await cache.set(CacheCategory.PACKAGE_INDEX, "test-key", b"value")
@pytest.mark.asyncio
@pytest.mark.unit
async def test_ping_returns_false_on_exception(self):
"""ping() should return False on exception."""
from app.cache_service import CacheService
from app.config import Settings
settings = Settings(redis_enabled=True)
cache = CacheService(settings)
mock_redis = AsyncMock()
mock_redis.ping.side_effect = Exception("Connection lost")
cache._redis = mock_redis
cache._enabled = True
cache._started = True
result = await cache.ping()
assert result is False

View File

@@ -0,0 +1,167 @@
"""Tests for database utility functions."""
import pytest
from unittest.mock import MagicMock, patch
class TestArtifactRepository:
"""Tests for ArtifactRepository."""
def test_batch_dependency_values_formatting(self):
"""batch_upsert_dependencies should format values correctly."""
from app.db_utils import ArtifactRepository
deps = [
("_pypi", "numpy", ">=1.21.0"),
("_pypi", "requests", "*"),
("myproject", "mylib", "==1.0.0"),
]
values = ArtifactRepository._format_dependency_values("abc123", deps)
assert len(values) == 3
assert values[0] == {
"artifact_id": "abc123",
"dependency_project": "_pypi",
"dependency_package": "numpy",
"version_constraint": ">=1.21.0",
}
assert values[2]["dependency_project"] == "myproject"
def test_empty_dependencies_returns_empty_list(self):
"""Empty dependency list should return empty values."""
from app.db_utils import ArtifactRepository
values = ArtifactRepository._format_dependency_values("abc123", [])
assert values == []
def test_format_dependency_values_preserves_special_characters(self):
"""Version constraints with special characters should be preserved."""
from app.db_utils import ArtifactRepository
deps = [
("_pypi", "package-name", ">=1.0.0,<2.0.0"),
("_pypi", "another_pkg", "~=1.4.2"),
]
values = ArtifactRepository._format_dependency_values("hash123", deps)
assert values[0]["version_constraint"] == ">=1.0.0,<2.0.0"
assert values[1]["version_constraint"] == "~=1.4.2"
def test_batch_upsert_dependencies_returns_zero_for_empty(self):
"""batch_upsert_dependencies should return 0 for empty list without DB call."""
from app.db_utils import ArtifactRepository
mock_db = MagicMock()
repo = ArtifactRepository(mock_db)
result = repo.batch_upsert_dependencies("abc123", [])
assert result == 0
# Verify no DB operations were performed
mock_db.execute.assert_not_called()
def test_get_or_create_artifact_builds_correct_statement(self):
"""get_or_create_artifact should use ON CONFLICT DO UPDATE."""
from app.db_utils import ArtifactRepository
from app.models import Artifact
mock_db = MagicMock()
mock_result = MagicMock()
mock_artifact = MagicMock()
mock_artifact.ref_count = 1
mock_result.scalar_one.return_value = mock_artifact
mock_db.execute.return_value = mock_result
repo = ArtifactRepository(mock_db)
artifact, created = repo.get_or_create_artifact(
sha256="abc123def456",
size=1024,
filename="test.whl",
content_type="application/zip",
)
assert mock_db.execute.called
assert created is True
assert artifact == mock_artifact
def test_get_or_create_artifact_existing_not_created(self):
"""get_or_create_artifact should return created=False for existing artifact."""
from app.db_utils import ArtifactRepository
mock_db = MagicMock()
mock_result = MagicMock()
mock_artifact = MagicMock()
mock_artifact.ref_count = 5 # Existing artifact with ref_count > 1
mock_result.scalar_one.return_value = mock_artifact
mock_db.execute.return_value = mock_result
repo = ArtifactRepository(mock_db)
artifact, created = repo.get_or_create_artifact(
sha256="abc123def456",
size=1024,
filename="test.whl",
)
assert created is False
def test_get_cached_url_with_artifact_returns_tuple(self):
"""get_cached_url_with_artifact should return (CachedUrl, Artifact) tuple."""
from app.db_utils import ArtifactRepository
mock_db = MagicMock()
mock_cached_url = MagicMock()
mock_artifact = MagicMock()
mock_db.query.return_value.join.return_value.filter.return_value.first.return_value = (
mock_cached_url,
mock_artifact,
)
repo = ArtifactRepository(mock_db)
result = repo.get_cached_url_with_artifact("url_hash_123")
assert result == (mock_cached_url, mock_artifact)
def test_get_cached_url_with_artifact_returns_none_when_not_found(self):
"""get_cached_url_with_artifact should return None when URL not cached."""
from app.db_utils import ArtifactRepository
mock_db = MagicMock()
mock_db.query.return_value.join.return_value.filter.return_value.first.return_value = None
repo = ArtifactRepository(mock_db)
result = repo.get_cached_url_with_artifact("nonexistent_hash")
assert result is None
def test_get_artifact_dependencies_returns_list(self):
"""get_artifact_dependencies should return list of dependencies."""
from app.db_utils import ArtifactRepository
mock_db = MagicMock()
mock_dep1 = MagicMock()
mock_dep2 = MagicMock()
mock_db.query.return_value.filter.return_value.all.return_value = [
mock_dep1,
mock_dep2,
]
repo = ArtifactRepository(mock_db)
result = repo.get_artifact_dependencies("artifact_hash_123")
assert len(result) == 2
assert result[0] == mock_dep1
assert result[1] == mock_dep2
def test_get_artifact_dependencies_returns_empty_list(self):
"""get_artifact_dependencies should return empty list when no dependencies."""
from app.db_utils import ArtifactRepository
mock_db = MagicMock()
mock_db.query.return_value.filter.return_value.all.return_value = []
repo = ArtifactRepository(mock_db)
result = repo.get_artifact_dependencies("artifact_without_deps")
assert result == []

View File

@@ -0,0 +1,194 @@
"""Tests for HttpClientManager."""
import pytest
from unittest.mock import MagicMock, AsyncMock, patch
class TestHttpClientManager:
"""Tests for HTTP client pool management."""
@pytest.mark.unit
def test_manager_initializes_with_settings(self):
"""Manager should initialize with config settings."""
from app.http_client import HttpClientManager
from app.config import Settings
settings = Settings(
http_max_connections=50,
http_connect_timeout=15.0,
)
manager = HttpClientManager(settings)
assert manager.max_connections == 50
assert manager.connect_timeout == 15.0
assert manager._default_client is None # Not started yet
@pytest.mark.asyncio
@pytest.mark.unit
async def test_startup_creates_client(self):
"""Startup should create the default async client."""
from app.http_client import HttpClientManager
from app.config import Settings
settings = Settings()
manager = HttpClientManager(settings)
await manager.startup()
assert manager._default_client is not None
await manager.shutdown()
@pytest.mark.asyncio
@pytest.mark.unit
async def test_shutdown_closes_client(self):
"""Shutdown should close all clients gracefully."""
from app.http_client import HttpClientManager
from app.config import Settings
settings = Settings()
manager = HttpClientManager(settings)
await manager.startup()
client = manager._default_client
await manager.shutdown()
assert manager._default_client is None
assert client.is_closed
@pytest.mark.asyncio
@pytest.mark.unit
async def test_get_client_returns_default(self):
"""get_client() should return the default client."""
from app.http_client import HttpClientManager
from app.config import Settings
settings = Settings()
manager = HttpClientManager(settings)
await manager.startup()
client = manager.get_client()
assert client is manager._default_client
await manager.shutdown()
@pytest.mark.asyncio
@pytest.mark.unit
async def test_get_client_raises_if_not_started(self):
"""get_client() should raise RuntimeError if manager not started."""
from app.http_client import HttpClientManager
from app.config import Settings
settings = Settings()
manager = HttpClientManager(settings)
with pytest.raises(RuntimeError, match="not started"):
manager.get_client()
@pytest.mark.asyncio
@pytest.mark.unit
async def test_run_blocking_executes_in_thread_pool(self):
"""run_blocking should execute sync functions in thread pool."""
from app.http_client import HttpClientManager
from app.config import Settings
import threading
settings = Settings()
manager = HttpClientManager(settings)
await manager.startup()
main_thread = threading.current_thread()
execution_thread = None
def blocking_func():
nonlocal execution_thread
execution_thread = threading.current_thread()
return "result"
result = await manager.run_blocking(blocking_func)
assert result == "result"
assert execution_thread is not main_thread
await manager.shutdown()
@pytest.mark.asyncio
@pytest.mark.unit
async def test_run_blocking_raises_if_not_started(self):
"""run_blocking should raise RuntimeError if manager not started."""
from app.http_client import HttpClientManager
from app.config import Settings
settings = Settings()
manager = HttpClientManager(settings)
with pytest.raises(RuntimeError, match="not started"):
await manager.run_blocking(lambda: None)
@pytest.mark.asyncio
@pytest.mark.unit
async def test_startup_idempotent(self):
"""Calling startup multiple times should be safe."""
from app.http_client import HttpClientManager
from app.config import Settings
settings = Settings()
manager = HttpClientManager(settings)
await manager.startup()
client1 = manager._default_client
await manager.startup() # Should not create a new client
client2 = manager._default_client
assert client1 is client2 # Same client instance
await manager.shutdown()
@pytest.mark.asyncio
@pytest.mark.unit
async def test_shutdown_idempotent(self):
"""Calling shutdown multiple times should be safe."""
from app.http_client import HttpClientManager
from app.config import Settings
settings = Settings()
manager = HttpClientManager(settings)
await manager.startup()
await manager.shutdown()
await manager.shutdown() # Should not raise
assert manager._default_client is None
@pytest.mark.asyncio
@pytest.mark.unit
async def test_properties_return_configured_values(self):
"""Properties should return configured values."""
from app.http_client import HttpClientManager
from app.config import Settings
settings = Settings(
http_max_connections=75,
http_worker_threads=16,
)
manager = HttpClientManager(settings)
await manager.startup()
assert manager.pool_size == 75
assert manager.executor_max == 16
await manager.shutdown()
@pytest.mark.asyncio
@pytest.mark.unit
async def test_active_connections_when_not_started(self):
"""active_connections should return 0 when not started."""
from app.http_client import HttpClientManager
from app.config import Settings
settings = Settings()
manager = HttpClientManager(settings)
assert manager.active_connections == 0

View File

@@ -0,0 +1,243 @@
"""Unit tests for metadata extraction functionality."""
import io
import gzip
import tarfile
import zipfile
import pytest
from app.metadata import (
extract_metadata,
extract_deb_metadata,
extract_wheel_metadata,
extract_tarball_metadata,
extract_jar_metadata,
parse_deb_control,
)
class TestDebMetadata:
"""Tests for Debian package metadata extraction."""
def test_parse_deb_control_basic(self):
"""Test parsing a basic control file."""
control = """Package: my-package
Version: 1.2.3
Architecture: amd64
Maintainer: Test <test@example.com>
Description: A test package
"""
result = parse_deb_control(control)
assert result["package_name"] == "my-package"
assert result["version"] == "1.2.3"
assert result["architecture"] == "amd64"
assert result["format"] == "deb"
def test_parse_deb_control_with_epoch(self):
"""Test parsing version with epoch."""
control = """Package: another-pkg
Version: 2:1.0.0-1
"""
result = parse_deb_control(control)
assert result["version"] == "2:1.0.0-1"
assert result["package_name"] == "another-pkg"
assert result["format"] == "deb"
def test_extract_deb_metadata_invalid_magic(self):
"""Test that invalid ar magic returns empty dict."""
file = io.BytesIO(b"not an ar archive")
result = extract_deb_metadata(file)
assert result == {}
def test_extract_deb_metadata_valid_ar_no_control(self):
"""Test ar archive without control.tar returns empty."""
# Create minimal ar archive with just debian-binary
ar_data = b"!<arch>\n"
ar_data += b"debian-binary/ 0 0 0 100644 4 `\n"
ar_data += b"2.0\n"
file = io.BytesIO(ar_data)
result = extract_deb_metadata(file)
# Should return empty since no control.tar found
assert result == {} or "version" not in result
class TestWheelMetadata:
"""Tests for Python wheel metadata extraction."""
def _create_wheel_with_metadata(self, metadata_content: str) -> io.BytesIO:
"""Helper to create a wheel file with given METADATA content."""
buf = io.BytesIO()
with zipfile.ZipFile(buf, 'w') as zf:
zf.writestr('package-1.0.0.dist-info/METADATA', metadata_content)
buf.seek(0)
return buf
def test_extract_wheel_version(self):
"""Test extracting version from wheel METADATA."""
metadata = """Metadata-Version: 2.1
Name: my-package
Version: 2.3.4
Summary: A test package
"""
file = self._create_wheel_with_metadata(metadata)
result = extract_wheel_metadata(file)
assert result.get("version") == "2.3.4"
assert result.get("package_name") == "my-package"
assert result.get("format") == "wheel"
def test_extract_wheel_no_version(self):
"""Test wheel without version field."""
metadata = """Metadata-Version: 2.1
Name: no-version-pkg
"""
file = self._create_wheel_with_metadata(metadata)
result = extract_wheel_metadata(file)
assert "version" not in result
assert result.get("package_name") == "no-version-pkg"
assert result.get("format") == "wheel"
def test_extract_wheel_invalid_zip(self):
"""Test that invalid zip returns format-only dict."""
file = io.BytesIO(b"not a zip file")
result = extract_wheel_metadata(file)
assert result == {"format": "wheel"}
def test_extract_wheel_no_metadata_file(self):
"""Test wheel without METADATA file returns format-only dict."""
buf = io.BytesIO()
with zipfile.ZipFile(buf, 'w') as zf:
zf.writestr('some_file.py', 'print("hello")')
buf.seek(0)
result = extract_wheel_metadata(buf)
assert result == {"format": "wheel"}
class TestTarballMetadata:
"""Tests for tarball metadata extraction from filename."""
def test_extract_version_from_filename_standard(self):
"""Test standard package-version.tar.gz format."""
file = io.BytesIO(b"") # Content doesn't matter for filename extraction
result = extract_tarball_metadata(file, "mypackage-1.2.3.tar.gz")
assert result.get("version") == "1.2.3"
assert result.get("package_name") == "mypackage"
assert result.get("format") == "tarball"
def test_extract_version_with_v_prefix(self):
"""Test version with v prefix."""
file = io.BytesIO(b"")
result = extract_tarball_metadata(file, "package-v2.0.0.tar.gz")
assert result.get("version") == "2.0.0"
assert result.get("package_name") == "package"
assert result.get("format") == "tarball"
def test_extract_version_underscore_separator(self):
"""Test package_version format."""
file = io.BytesIO(b"")
result = extract_tarball_metadata(file, "my_package_3.1.4.tar.gz")
assert result.get("version") == "3.1.4"
assert result.get("package_name") == "my_package"
assert result.get("format") == "tarball"
def test_extract_version_complex(self):
"""Test complex version string."""
file = io.BytesIO(b"")
result = extract_tarball_metadata(file, "package-1.0.0-beta.1.tar.gz")
# The regex handles versions with suffix like -beta_1
assert result.get("format") == "tarball"
# May or may not extract version depending on regex match
if "version" in result:
assert result.get("package_name") == "package"
def test_extract_no_version_in_filename(self):
"""Test filename without version returns format-only dict."""
file = io.BytesIO(b"")
result = extract_tarball_metadata(file, "package.tar.gz")
# Should return format but no version
assert result.get("version") is None
assert result.get("format") == "tarball"
class TestJarMetadata:
"""Tests for JAR/Java metadata extraction."""
def _create_jar_with_manifest(self, manifest_content: str) -> io.BytesIO:
"""Helper to create a JAR file with given MANIFEST.MF content."""
buf = io.BytesIO()
with zipfile.ZipFile(buf, 'w') as zf:
zf.writestr('META-INF/MANIFEST.MF', manifest_content)
buf.seek(0)
return buf
def test_extract_jar_version_from_manifest(self):
"""Test extracting version from MANIFEST.MF."""
manifest = """Manifest-Version: 1.0
Implementation-Title: my-library
Implementation-Version: 4.5.6
"""
file = self._create_jar_with_manifest(manifest)
result = extract_jar_metadata(file)
assert result.get("version") == "4.5.6"
assert result.get("package_name") == "my-library"
assert result.get("format") == "jar"
def test_extract_jar_bundle_version(self):
"""Test extracting OSGi Bundle-Version."""
manifest = """Manifest-Version: 1.0
Bundle-Version: 2.1.0
Bundle-Name: Test Bundle
"""
file = self._create_jar_with_manifest(manifest)
result = extract_jar_metadata(file)
# Bundle-Version is stored in bundle_version, not version
assert result.get("bundle_version") == "2.1.0"
assert result.get("bundle_name") == "Test Bundle"
assert result.get("format") == "jar"
def test_extract_jar_invalid_zip(self):
"""Test that invalid JAR returns format-only dict."""
file = io.BytesIO(b"not a jar file")
result = extract_jar_metadata(file)
assert result == {"format": "jar"}
class TestExtractMetadataDispatch:
"""Tests for the main extract_metadata dispatcher function."""
def test_dispatch_to_wheel(self):
"""Test that .whl files use wheel extractor."""
buf = io.BytesIO()
with zipfile.ZipFile(buf, 'w') as zf:
zf.writestr('pkg-1.0.dist-info/METADATA', 'Version: 1.0.0\nName: pkg')
buf.seek(0)
result = extract_metadata(buf, "package-1.0.0-py3-none-any.whl")
assert result.get("version") == "1.0.0"
assert result.get("package_name") == "pkg"
assert result.get("format") == "wheel"
def test_dispatch_to_tarball(self):
"""Test that .tar.gz files use tarball extractor."""
file = io.BytesIO(b"")
result = extract_metadata(file, "mypackage-2.3.4.tar.gz")
assert result.get("version") == "2.3.4"
assert result.get("package_name") == "mypackage"
assert result.get("format") == "tarball"
def test_dispatch_unknown_extension(self):
"""Test that unknown extensions return empty dict."""
file = io.BytesIO(b"some content")
result = extract_metadata(file, "unknown.xyz")
assert result == {}
def test_file_position_reset_after_extraction(self):
"""Test that file position is reset to start after extraction."""
buf = io.BytesIO()
with zipfile.ZipFile(buf, 'w') as zf:
zf.writestr('pkg-1.0.dist-info/METADATA', 'Version: 1.0.0\nName: pkg')
buf.seek(0)
extract_metadata(buf, "package.whl")
# File should be back at position 0
assert buf.tell() == 0

View File

@@ -145,54 +145,6 @@ class TestPackageModel:
assert platform_col.default.arg == "any"
class TestTagModel:
"""Tests for the Tag model."""
@pytest.mark.unit
def test_tag_requires_package_id(self):
"""Test tag requires package_id."""
from app.models import Tag
tag = Tag(
name="v1.0.0",
package_id=uuid.uuid4(),
artifact_id="f" * 64,
created_by="test-user",
)
assert tag.package_id is not None
assert tag.artifact_id == "f" * 64
class TestTagHistoryModel:
"""Tests for the TagHistory model."""
@pytest.mark.unit
def test_tag_history_default_change_type(self):
"""Test tag history change_type column has default value of 'update'."""
from app.models import TagHistory
# Check the column definition has the right default
change_type_col = TagHistory.__table__.columns["change_type"]
assert change_type_col.default is not None
assert change_type_col.default.arg == "update"
@pytest.mark.unit
def test_tag_history_allows_null_old_artifact(self):
"""Test tag history allows null old_artifact_id (for create events)."""
from app.models import TagHistory
history = TagHistory(
tag_id=uuid.uuid4(),
old_artifact_id=None,
new_artifact_id="h" * 64,
change_type="create",
changed_by="test-user",
)
assert history.old_artifact_id is None
class TestUploadModel:
"""Tests for the Upload model."""

View File

@@ -0,0 +1,85 @@
"""Unit tests for PyPI proxy functionality."""
import pytest
from app.pypi_proxy import _parse_requires_dist
class TestParseRequiresDist:
"""Tests for _parse_requires_dist function."""
def test_simple_package(self):
"""Test parsing a simple package name."""
name, version = _parse_requires_dist("numpy")
assert name == "numpy"
assert version is None
def test_package_with_version(self):
"""Test parsing package with version constraint."""
name, version = _parse_requires_dist("numpy>=1.21.0")
assert name == "numpy"
assert version == ">=1.21.0"
def test_package_with_parenthesized_version(self):
"""Test parsing package with parenthesized version."""
name, version = _parse_requires_dist("requests (>=2.25.0)")
assert name == "requests"
assert version == ">=2.25.0"
def test_package_with_python_version_marker(self):
"""Test that python_version markers are preserved but marker stripped."""
name, version = _parse_requires_dist("typing-extensions; python_version < '3.8'")
assert name == "typing-extensions"
assert version is None
def test_filters_extra_dependencies(self):
"""Test that extra dependencies are filtered out."""
# Extra dependencies should return (None, None)
name, version = _parse_requires_dist("pytest; extra == 'test'")
assert name is None
assert version is None
name, version = _parse_requires_dist("sphinx; extra == 'docs'")
assert name is None
assert version is None
def test_filters_platform_specific_darwin(self):
"""Test that macOS-specific dependencies are filtered out."""
name, version = _parse_requires_dist("pyobjc; sys_platform == 'darwin'")
assert name is None
assert version is None
def test_filters_platform_specific_win32(self):
"""Test that Windows-specific dependencies are filtered out."""
name, version = _parse_requires_dist("pywin32; sys_platform == 'win32'")
assert name is None
assert version is None
def test_filters_platform_system_marker(self):
"""Test that platform_system markers are filtered out."""
name, version = _parse_requires_dist("jaraco-windows; platform_system == 'Windows'")
assert name is None
assert version is None
def test_normalizes_package_name(self):
"""Test that package names are normalized (PEP 503)."""
name, version = _parse_requires_dist("Typing_Extensions>=3.7.4")
assert name == "typing-extensions"
assert version == ">=3.7.4"
def test_complex_version_constraint(self):
"""Test parsing complex version constraints."""
name, version = _parse_requires_dist("gast!=0.5.0,!=0.5.1,!=0.5.2,>=0.2.1")
assert name == "gast"
assert version == "!=0.5.0,!=0.5.1,!=0.5.2,>=0.2.1"
def test_version_range(self):
"""Test parsing version range constraints."""
name, version = _parse_requires_dist("grpcio<2.0,>=1.24.3")
assert name == "grpcio"
assert version == "<2.0,>=1.24.3"
def test_tilde_version(self):
"""Test parsing tilde version constraints."""
name, version = _parse_requires_dist("tensorboard~=2.20.0")
assert name == "tensorboard"
assert version == "~=2.20.0"

View File

@@ -0,0 +1,65 @@
"""Unit tests for rate limiting configuration."""
import os
import pytest
class TestRateLimitConfiguration:
"""Tests for rate limit configuration."""
def test_default_login_rate_limit(self):
"""Test default login rate limit is 5/minute."""
# Import fresh to get default value
import importlib
import app.rate_limit as rate_limit_module
# Save original env value
original = os.environ.get("ORCHARD_LOGIN_RATE_LIMIT")
try:
# Clear env variable to test default
if "ORCHARD_LOGIN_RATE_LIMIT" in os.environ:
del os.environ["ORCHARD_LOGIN_RATE_LIMIT"]
# Reload module to pick up new env
importlib.reload(rate_limit_module)
assert rate_limit_module.LOGIN_RATE_LIMIT == "5/minute"
finally:
# Restore original env value
if original is not None:
os.environ["ORCHARD_LOGIN_RATE_LIMIT"] = original
importlib.reload(rate_limit_module)
def test_custom_login_rate_limit(self):
"""Test custom login rate limit from environment."""
import importlib
import app.rate_limit as rate_limit_module
# Save original env value
original = os.environ.get("ORCHARD_LOGIN_RATE_LIMIT")
try:
# Set custom rate limit
os.environ["ORCHARD_LOGIN_RATE_LIMIT"] = "10/minute"
# Reload module to pick up new env
importlib.reload(rate_limit_module)
assert rate_limit_module.LOGIN_RATE_LIMIT == "10/minute"
finally:
# Restore original env value
if original is not None:
os.environ["ORCHARD_LOGIN_RATE_LIMIT"] = original
else:
if "ORCHARD_LOGIN_RATE_LIMIT" in os.environ:
del os.environ["ORCHARD_LOGIN_RATE_LIMIT"]
importlib.reload(rate_limit_module)
def test_limiter_exists(self):
"""Test that limiter object is created."""
from app.rate_limit import limiter
assert limiter is not None
# Limiter should have a key_func set
assert limiter._key_func is not None

View File

@@ -0,0 +1,300 @@
"""Unit tests for registry client functionality."""
import pytest
from unittest.mock import AsyncMock, MagicMock, patch
import httpx
from packaging.specifiers import SpecifierSet
from app.registry_client import (
PyPIRegistryClient,
VersionInfo,
FetchResult,
get_registry_client,
)
class TestPyPIRegistryClient:
"""Tests for PyPI registry client."""
@pytest.fixture
def mock_http_client(self):
"""Create a mock async HTTP client."""
return AsyncMock(spec=httpx.AsyncClient)
@pytest.fixture
def client(self, mock_http_client):
"""Create a PyPI registry client with mocked HTTP."""
return PyPIRegistryClient(
http_client=mock_http_client,
upstream_sources=[],
pypi_api_url="https://pypi.org/pypi",
)
def test_source_type(self, client):
"""Test source_type returns 'pypi'."""
assert client.source_type == "pypi"
def test_normalize_package_name(self, client):
"""Test package name normalization per PEP 503."""
assert client._normalize_package_name("My_Package") == "my-package"
assert client._normalize_package_name("my.package") == "my-package"
assert client._normalize_package_name("my-package") == "my-package"
assert client._normalize_package_name("MY-PACKAGE") == "my-package"
assert client._normalize_package_name("my__package") == "my-package"
assert client._normalize_package_name("my..package") == "my-package"
@pytest.mark.asyncio
async def test_get_available_versions_success(self, client, mock_http_client):
"""Test fetching available versions from PyPI."""
mock_response = MagicMock()
mock_response.status_code = 200
mock_response.json.return_value = {
"releases": {
"1.0.0": [{"packagetype": "bdist_wheel"}],
"1.1.0": [{"packagetype": "bdist_wheel"}],
"2.0.0": [{"packagetype": "bdist_wheel"}],
}
}
mock_http_client.get.return_value = mock_response
versions = await client.get_available_versions("test-package")
assert "1.0.0" in versions
assert "1.1.0" in versions
assert "2.0.0" in versions
mock_http_client.get.assert_called_once()
@pytest.mark.asyncio
async def test_get_available_versions_empty(self, client, mock_http_client):
"""Test handling package with no releases."""
mock_response = MagicMock()
mock_response.status_code = 200
mock_response.json.return_value = {"releases": {}}
mock_http_client.get.return_value = mock_response
versions = await client.get_available_versions("empty-package")
assert versions == []
@pytest.mark.asyncio
async def test_get_available_versions_404(self, client, mock_http_client):
"""Test handling non-existent package."""
mock_response = MagicMock()
mock_response.status_code = 404
mock_http_client.get.return_value = mock_response
versions = await client.get_available_versions("nonexistent")
assert versions == []
@pytest.mark.asyncio
async def test_resolve_constraint_wildcard(self, client, mock_http_client):
"""Test resolving wildcard constraint returns latest."""
mock_response = MagicMock()
mock_response.status_code = 200
mock_response.json.return_value = {
"info": {"version": "2.0.0"},
"releases": {
"1.0.0": [
{
"packagetype": "bdist_wheel",
"url": "https://files.pythonhosted.org/test-1.0.0.whl",
"filename": "test-1.0.0.whl",
"digests": {"sha256": "abc123"},
"size": 1000,
}
],
"2.0.0": [
{
"packagetype": "bdist_wheel",
"url": "https://files.pythonhosted.org/test-2.0.0.whl",
"filename": "test-2.0.0.whl",
"digests": {"sha256": "def456"},
"size": 2000,
}
],
},
}
mock_http_client.get.return_value = mock_response
result = await client.resolve_constraint("test-package", "*")
assert result is not None
assert result.version == "2.0.0"
@pytest.mark.asyncio
async def test_resolve_constraint_specific_version(self, client, mock_http_client):
"""Test resolving specific version constraint."""
mock_response = MagicMock()
mock_response.status_code = 200
mock_response.json.return_value = {
"releases": {
"1.0.0": [
{
"packagetype": "bdist_wheel",
"url": "https://files.pythonhosted.org/test-1.0.0.whl",
"filename": "test-1.0.0.whl",
"digests": {"sha256": "abc123"},
"size": 1000,
}
],
"2.0.0": [
{
"packagetype": "bdist_wheel",
"url": "https://files.pythonhosted.org/test-2.0.0.whl",
"filename": "test-2.0.0.whl",
}
],
},
}
mock_http_client.get.return_value = mock_response
result = await client.resolve_constraint("test-package", ">=1.0.0,<2.0.0")
assert result is not None
assert result.version == "1.0.0"
@pytest.mark.asyncio
async def test_resolve_constraint_no_match(self, client, mock_http_client):
"""Test resolving constraint with no matching version."""
mock_response = MagicMock()
mock_response.status_code = 200
mock_response.json.return_value = {
"releases": {
"1.0.0": [
{
"packagetype": "bdist_wheel",
"url": "https://files.pythonhosted.org/test-1.0.0.whl",
"filename": "test-1.0.0.whl",
}
],
},
}
mock_http_client.get.return_value = mock_response
result = await client.resolve_constraint("test-package", ">=5.0.0")
assert result is None
@pytest.mark.asyncio
async def test_resolve_constraint_bare_version(self, client, mock_http_client):
"""Test resolving bare version string as exact match."""
mock_response = MagicMock()
mock_response.status_code = 200
mock_response.json.return_value = {
"info": {"version": "2.0.0"},
"releases": {
"1.0.0": [
{
"packagetype": "bdist_wheel",
"url": "https://files.pythonhosted.org/test-1.0.0.whl",
"filename": "test-1.0.0.whl",
"digests": {"sha256": "abc123"},
"size": 1000,
}
],
"2.0.0": [
{
"packagetype": "bdist_wheel",
"url": "https://files.pythonhosted.org/test-2.0.0.whl",
"filename": "test-2.0.0.whl",
"digests": {"sha256": "def456"},
"size": 2000,
}
],
},
}
mock_http_client.get.return_value = mock_response
# Bare version "1.0.0" should resolve to exactly 1.0.0, not latest
result = await client.resolve_constraint("test-package", "1.0.0")
assert result is not None
assert result.version == "1.0.0"
class TestVersionInfo:
"""Tests for VersionInfo dataclass."""
def test_create_version_info(self):
"""Test creating VersionInfo with all fields."""
info = VersionInfo(
version="1.0.0",
download_url="https://example.com/pkg-1.0.0.whl",
filename="pkg-1.0.0.whl",
sha256="abc123",
size=5000,
content_type="application/zip",
)
assert info.version == "1.0.0"
assert info.download_url == "https://example.com/pkg-1.0.0.whl"
assert info.filename == "pkg-1.0.0.whl"
assert info.sha256 == "abc123"
assert info.size == 5000
def test_create_version_info_minimal(self):
"""Test creating VersionInfo with only required fields."""
info = VersionInfo(
version="1.0.0",
download_url="https://example.com/pkg.whl",
filename="pkg.whl",
)
assert info.sha256 is None
assert info.size is None
class TestFetchResult:
"""Tests for FetchResult dataclass."""
def test_create_fetch_result(self):
"""Test creating FetchResult."""
result = FetchResult(
artifact_id="abc123def456",
size=10000,
version="2.0.0",
filename="pkg-2.0.0.whl",
already_cached=True,
)
assert result.artifact_id == "abc123def456"
assert result.size == 10000
assert result.version == "2.0.0"
assert result.already_cached is True
def test_fetch_result_default_not_cached(self):
"""Test FetchResult defaults to not cached."""
result = FetchResult(
artifact_id="xyz",
size=100,
version="1.0.0",
filename="pkg.whl",
)
assert result.already_cached is False
class TestGetRegistryClient:
"""Tests for registry client factory function."""
def test_get_pypi_client(self):
"""Test getting PyPI client."""
mock_client = MagicMock()
mock_sources = []
client = get_registry_client("pypi", mock_client, mock_sources)
assert isinstance(client, PyPIRegistryClient)
def test_get_unsupported_client(self):
"""Test getting unsupported registry type returns None."""
mock_client = MagicMock()
client = get_registry_client("npm", mock_client, [])
assert client is None
def test_get_unknown_client(self):
"""Test getting unknown registry type returns None."""
mock_client = MagicMock()
client = get_registry_client("unknown", mock_client, [])
assert client is None

View File

@@ -0,0 +1,213 @@
"""
Unit tests for TeamAuthorizationService.
"""
import pytest
from unittest.mock import MagicMock, patch
import uuid
class TestTeamRoleHierarchy:
"""Tests for team role hierarchy functions."""
def test_get_team_role_rank(self):
"""Test role ranking."""
from app.auth import get_team_role_rank
assert get_team_role_rank("member") == 0
assert get_team_role_rank("admin") == 1
assert get_team_role_rank("owner") == 2
assert get_team_role_rank("invalid") == -1
def test_has_sufficient_team_role(self):
"""Test role sufficiency checks."""
from app.auth import has_sufficient_team_role
# Same role should be sufficient
assert has_sufficient_team_role("member", "member") is True
assert has_sufficient_team_role("admin", "admin") is True
assert has_sufficient_team_role("owner", "owner") is True
# Higher role should be sufficient for lower requirements
assert has_sufficient_team_role("admin", "member") is True
assert has_sufficient_team_role("owner", "member") is True
assert has_sufficient_team_role("owner", "admin") is True
# Lower role should NOT be sufficient for higher requirements
assert has_sufficient_team_role("member", "admin") is False
assert has_sufficient_team_role("member", "owner") is False
assert has_sufficient_team_role("admin", "owner") is False
class TestTeamAuthorizationService:
"""Tests for TeamAuthorizationService class."""
@pytest.fixture
def mock_db(self):
"""Create a mock database session."""
return MagicMock()
@pytest.fixture
def mock_user(self):
"""Create a mock user."""
user = MagicMock()
user.id = uuid.uuid4()
user.username = "testuser"
user.is_admin = False
return user
@pytest.fixture
def mock_admin_user(self):
"""Create a mock admin user."""
user = MagicMock()
user.id = uuid.uuid4()
user.username = "adminuser"
user.is_admin = True
return user
def test_get_user_team_role_no_user(self, mock_db):
"""Test that None is returned for anonymous users."""
from app.auth import TeamAuthorizationService
service = TeamAuthorizationService(mock_db)
result = service.get_user_team_role("team-id", None)
assert result is None
def test_get_user_team_role_admin_user(self, mock_db, mock_admin_user):
"""Test that system admins who are not members get admin role."""
from app.auth import TeamAuthorizationService
# Mock no membership found
mock_db.query.return_value.filter.return_value.first.return_value = None
service = TeamAuthorizationService(mock_db)
result = service.get_user_team_role("team-id", mock_admin_user)
assert result == "admin"
def test_get_user_team_role_member(self, mock_db, mock_user):
"""Test getting role for a team member."""
from app.auth import TeamAuthorizationService
# Mock the membership query
mock_membership = MagicMock()
mock_membership.role = "member"
mock_db.query.return_value.filter.return_value.first.return_value = mock_membership
service = TeamAuthorizationService(mock_db)
result = service.get_user_team_role("team-id", mock_user)
assert result == "member"
def test_get_user_team_role_not_member(self, mock_db, mock_user):
"""Test getting role for a non-member."""
from app.auth import TeamAuthorizationService
# Mock no membership found
mock_db.query.return_value.filter.return_value.first.return_value = None
service = TeamAuthorizationService(mock_db)
result = service.get_user_team_role("team-id", mock_user)
assert result is None
def test_check_team_access_member(self, mock_db, mock_user):
"""Test access check for member requiring member role."""
from app.auth import TeamAuthorizationService
# Mock the membership query
mock_membership = MagicMock()
mock_membership.role = "member"
mock_db.query.return_value.filter.return_value.first.return_value = mock_membership
service = TeamAuthorizationService(mock_db)
# Member should have member access
assert service.check_team_access("team-id", mock_user, "member") is True
# Member should not have admin access
assert service.check_team_access("team-id", mock_user, "admin") is False
# Member should not have owner access
assert service.check_team_access("team-id", mock_user, "owner") is False
def test_check_team_access_admin(self, mock_db, mock_user):
"""Test access check for admin role."""
from app.auth import TeamAuthorizationService
# Mock admin membership
mock_membership = MagicMock()
mock_membership.role = "admin"
mock_db.query.return_value.filter.return_value.first.return_value = mock_membership
service = TeamAuthorizationService(mock_db)
assert service.check_team_access("team-id", mock_user, "member") is True
assert service.check_team_access("team-id", mock_user, "admin") is True
assert service.check_team_access("team-id", mock_user, "owner") is False
def test_check_team_access_owner(self, mock_db, mock_user):
"""Test access check for owner role."""
from app.auth import TeamAuthorizationService
# Mock owner membership
mock_membership = MagicMock()
mock_membership.role = "owner"
mock_db.query.return_value.filter.return_value.first.return_value = mock_membership
service = TeamAuthorizationService(mock_db)
assert service.check_team_access("team-id", mock_user, "member") is True
assert service.check_team_access("team-id", mock_user, "admin") is True
assert service.check_team_access("team-id", mock_user, "owner") is True
def test_can_create_project(self, mock_db, mock_user):
"""Test can_create_project requires admin role."""
from app.auth import TeamAuthorizationService
service = TeamAuthorizationService(mock_db)
# Member cannot create projects
mock_membership = MagicMock()
mock_membership.role = "member"
mock_db.query.return_value.filter.return_value.first.return_value = mock_membership
assert service.can_create_project("team-id", mock_user) is False
# Admin can create projects
mock_membership.role = "admin"
assert service.can_create_project("team-id", mock_user) is True
# Owner can create projects
mock_membership.role = "owner"
assert service.can_create_project("team-id", mock_user) is True
def test_can_manage_members(self, mock_db, mock_user):
"""Test can_manage_members requires admin role."""
from app.auth import TeamAuthorizationService
service = TeamAuthorizationService(mock_db)
# Member cannot manage members
mock_membership = MagicMock()
mock_membership.role = "member"
mock_db.query.return_value.filter.return_value.first.return_value = mock_membership
assert service.can_manage_members("team-id", mock_user) is False
# Admin can manage members
mock_membership.role = "admin"
assert service.can_manage_members("team-id", mock_user) is True
def test_can_delete_team(self, mock_db, mock_user):
"""Test can_delete_team requires owner role."""
from app.auth import TeamAuthorizationService
service = TeamAuthorizationService(mock_db)
# Member cannot delete team
mock_membership = MagicMock()
mock_membership.role = "member"
mock_db.query.return_value.filter.return_value.first.return_value = mock_membership
assert service.can_delete_team("team-id", mock_user) is False
# Admin cannot delete team
mock_membership.role = "admin"
assert service.can_delete_team("team-id", mock_user) is False
# Only owner can delete team
mock_membership.role = "owner"
assert service.can_delete_team("team-id", mock_user) is True

View File

@@ -26,6 +26,8 @@ services:
- ORCHARD_REDIS_PORT=6379
# Higher rate limit for local development/testing
- ORCHARD_LOGIN_RATE_LIMIT=1000/minute
# Admin password - set in .env file or environment (see .env.example)
- ORCHARD_ADMIN_PASSWORD=${ORCHARD_ADMIN_PASSWORD:-}
depends_on:
postgres:
condition: service_healthy

View File

@@ -0,0 +1,672 @@
# Epic: Upstream Artifact Caching for Hermetic Builds
## Overview
Orchard will act as a permanent, content-addressable cache for upstream artifacts (npm, PyPI, Maven, Docker, etc.). Once an artifact is cached, it is stored forever by SHA256 hash - enabling reproducible builds years later regardless of whether the upstream source still exists.
## Problem Statement
Build reproducibility is critical for enterprise environments:
- Packages get deleted, yanked, or modified upstream
- Registries go down or change URLs
- Version constraints resolve differently over time
- Air-gapped environments cannot access public internet
Teams need to guarantee that a build from 5 years ago produces the exact same output today.
## Solution
Orchard becomes "the cache that never forgets":
1. **Fetch once, store forever** - When a build needs `lodash@4.17.21`, Orchard fetches it from npm, stores it by SHA256 hash, and never deletes it
2. **Content-addressable** - Same hash = same bytes, guaranteed
3. **Format-agnostic** - Orchard doesn't need to understand npm/PyPI/Maven protocols; the client provides the URL, Orchard fetches and stores
4. **Air-gap support** - Disable public internet entirely, only allow configured private upstreams
## User Workflow
```
1. Build tool resolves dependencies npm install / pip install / mvn resolve
2. Generate lockfile with URLs package-lock.json / requirements.txt
3. Cache all URLs in Orchard orchard cache --file urls.txt
4. Pin by SHA256 hash lodash = "sha256:abc123..."
5. Future builds fetch by hash Always get exact same bytes
```
## Key Features
- **Multiple upstream sources** - Configure npm, PyPI, Maven Central, private Artifactory, etc.
- **Per-source authentication** - Basic auth, bearer tokens, API keys
- **System cache projects** - `_npm`, `_pypi`, `_maven` organize cached packages by format
- **Cross-referencing** - Link cached artifacts to user projects for visibility
- **URL tracking** - Know which URLs map to which hashes, audit provenance
- **Air-gap mode** - Global kill switch for all public internet access
- **Environment variable config** - 12-factor friendly for containerized deployments
## Architecture
```
┌─────────────────────────────────────────────────────────────────┐
│ Orchard Server │
├─────────────────────────────────────────────────────────────────┤
│ POST /api/v1/cache │
│ ├── Check if URL already cached (url_hash lookup) │
│ ├── Match URL to upstream source (get auth) │
│ ├── Fetch via UpstreamClient (stream + compute SHA256) │
│ ├── Store artifact in S3 (content-addressable) │
│ ├── Create tag in system project (_npm/lodash:4.17.21) │
│ ├── Optionally create tag in user project │
│ └── Record in cached_urls table (provenance) │
├─────────────────────────────────────────────────────────────────┤
│ Tables │
│ ├── upstream_sources (npm-public, pypi-public, artifactory) │
│ ├── cache_settings (allow_public_internet, etc.) │
│ ├── cached_urls (url → artifact_id mapping) │
│ └── projects.is_system (for _npm, _pypi, etc.) │
└─────────────────────────────────────────────────────────────────┘
```
## Issues Summary
| Issue | Title | Status | Dependencies |
|-------|-------|--------|--------------|
| #68 | Schema: Upstream Sources & Cache Tracking | ✅ Complete | None |
| #69 | HTTP Client: Generic URL Fetcher | Pending | None |
| #70 | Cache API Endpoint | Pending | #68, #69 |
| #71 | System Projects (Cache Namespaces) | Pending | #68, #70 |
| #72 | Upstream Sources Admin API | Pending | #68 |
| #73 | Global Cache Settings API | Pending | #68 |
| #74 | Environment Variable Overrides | Pending | #68, #72, #73 |
| #75 | Frontend: Upstream Sources Management | Pending | #72, #73 |
| #105 | Frontend: System Projects Integration | Pending | #71 |
| #77 | CLI: Cache Command | Pending | #70 |
## Implementation Phases
**Phase 1 - Core (MVP):**
- #68 Schema ✅
- #69 HTTP Client
- #70 Cache API
- #71 System Projects
**Phase 2 - Admin:**
- #72 Upstream Sources API
- #73 Cache Settings API
- #74 Environment Variables
**Phase 3 - Frontend:**
- #75 Upstream Sources UI
- #105 System Projects UI
**Phase 4 - CLI:**
- #77 Cache Command
---
# Issue #68: Schema - Upstream Sources & Cache Tracking
**Status: ✅ Complete**
## Description
Create database schema for flexible multi-source upstream configuration and URL-to-artifact tracking. This replaces the previous singleton proxy_config design with a more flexible model supporting multiple upstream sources, air-gap mode, and provenance tracking.
## Acceptance Criteria
- [x] `upstream_sources` table:
- id (UUID, primary key)
- name (VARCHAR(255), unique, e.g., "npm-public", "artifactory-private")
- source_type (VARCHAR(50), enum: npm, pypi, maven, docker, helm, nuget, deb, rpm, generic)
- url (VARCHAR(2048), base URL of upstream)
- enabled (BOOLEAN, default false)
- is_public (BOOLEAN, true if this is a public internet source)
- auth_type (VARCHAR(20), enum: none, basic, bearer, api_key)
- username (VARCHAR(255), nullable)
- password_encrypted (BYTEA, nullable, Fernet encrypted)
- headers_encrypted (BYTEA, nullable, for custom headers like API keys)
- priority (INTEGER, default 100, lower = checked first)
- created_at, updated_at timestamps
- [x] `cache_settings` table (singleton, id always 1):
- id (INTEGER, primary key, check id = 1)
- allow_public_internet (BOOLEAN, default true, air-gap kill switch)
- auto_create_system_projects (BOOLEAN, default true)
- created_at, updated_at timestamps
- [x] `cached_urls` table:
- id (UUID, primary key)
- url (VARCHAR(4096), original URL fetched)
- url_hash (VARCHAR(64), SHA256 of URL for fast lookup, indexed)
- artifact_id (VARCHAR(64), FK to artifacts)
- source_id (UUID, FK to upstream_sources, nullable for manual imports)
- fetched_at (TIMESTAMP WITH TIME ZONE)
- response_headers (JSONB, original upstream headers for provenance)
- created_at timestamp
- [x] Add `is_system` BOOLEAN column to projects table (default false)
- [x] Migration SQL file in migrations/
- [x] Runtime migration in database.py
- [x] SQLAlchemy models for all new tables
- [x] Pydantic schemas for API input/output (passwords write-only)
- [x] Encryption helpers for password/headers fields
- [x] Seed default upstream sources (disabled by default):
- npm-public: https://registry.npmjs.org
- pypi-public: https://pypi.org/simple
- maven-central: https://repo1.maven.org/maven2
- docker-hub: https://registry-1.docker.io
- [x] Unit tests for models and schemas
## Files Modified
- `migrations/010_upstream_caching.sql`
- `backend/app/database.py` (migrations 016-020)
- `backend/app/models.py` (UpstreamSource, CacheSettings, CachedUrl, Project.is_system)
- `backend/app/schemas.py` (all caching schemas)
- `backend/app/encryption.py` (renamed env var)
- `backend/app/config.py` (renamed setting)
- `backend/tests/test_upstream_caching.py` (37 tests)
- `frontend/src/components/Layout.tsx` (footer tagline)
- `CHANGELOG.md`
---
# Issue #69: HTTP Client - Generic URL Fetcher
**Status: Pending**
## Description
Create a reusable HTTP client for fetching artifacts from upstream sources. Supports multiple auth methods, streaming for large files, and computes SHA256 while downloading.
## Acceptance Criteria
- [ ] `UpstreamClient` class in `backend/app/upstream.py`
- [ ] `fetch(url)` method that:
- Streams response body (doesn't load large files into memory)
- Computes SHA256 hash while streaming
- Returns file content, hash, size, and response headers
- [ ] Auth support based on upstream source configuration:
- None (anonymous)
- Basic auth (username/password)
- Bearer token (Authorization: Bearer {token})
- API key (custom header name/value)
- [ ] URL-to-source matching:
- Match URL to configured upstream source by URL prefix
- Apply auth from matched source
- Respect source priority for multiple matches
- [ ] Configuration options:
- Timeout (connect and read, default 30s/300s)
- Max retries (default 3)
- Follow redirects (default true, max 5)
- Max file size (reject if Content-Length exceeds limit)
- [ ] Respect `allow_public_internet` setting:
- If false, reject URLs matching `is_public=true` sources
- If false, reject URLs not matching any configured source
- [ ] Capture response headers for provenance tracking
- [ ] Proper error handling:
- Connection errors (retry with backoff)
- HTTP errors (4xx, 5xx)
- Timeout errors
- SSL/TLS errors
- [ ] Logging for debugging (URL, source matched, status, timing)
- [ ] Unit tests with mocked HTTP responses
- [ ] Integration tests against httpbin.org or similar (optional, marked)
## Technical Notes
- Use `httpx` for async HTTP support (already in requirements)
- Stream to temp file to avoid memory issues with large artifacts
- Consider checksum verification if upstream provides it (e.g., npm provides shasum)
---
# Issue #70: Cache API Endpoint
**Status: Pending**
## Description
API endpoint to cache an artifact from an upstream URL. This is the core endpoint that fetches from upstream, stores in Orchard, and creates appropriate tags.
## Acceptance Criteria
- [ ] `POST /api/v1/cache` endpoint
- [ ] Request body:
```json
{
"url": "https://registry.npmjs.org/lodash/-/lodash-4.17.21.tgz",
"source_type": "npm",
"package_name": "lodash",
"tag": "4.17.21",
"user_project": "my-app",
"user_package": "npm-deps",
"user_tag": "lodash-4.17.21",
"expected_hash": "sha256:abc123..."
}
```
- `url` (required): URL to fetch
- `source_type` (required): Determines system project (_npm, _pypi, etc.)
- `package_name` (optional): Package name in system project, derived from URL if not provided
- `tag` (optional): Tag name in system project, derived from URL if not provided
- `user_project`, `user_package`, `user_tag` (optional): Cross-reference in user's project
- `expected_hash` (optional): Verify downloaded content matches
- [ ] Response:
```json
{
"artifact_id": "abc123...",
"sha256": "abc123...",
"size": 12345,
"content_type": "application/gzip",
"already_cached": false,
"source_url": "https://registry.npmjs.org/lodash/-/lodash-4.17.21.tgz",
"source_name": "npm-public",
"system_project": "_npm",
"system_package": "lodash",
"system_tag": "4.17.21",
"user_reference": "my-app/npm-deps:lodash-4.17.21"
}
```
- [ ] Behavior:
- Check if URL already cached (by url_hash in cached_urls)
- If cached: return existing artifact, optionally create user tag
- If not cached: fetch via UpstreamClient, store artifact, create tags
- Create/get system project if needed (e.g., `_npm`)
- Create package in system project (e.g., `_npm/lodash`)
- Create tag in system project (e.g., `_npm/lodash:4.17.21`)
- If user reference provided, create tag in user's project
- Record in cached_urls table with provenance
- [ ] Error handling:
- 400: Invalid request (bad URL format, missing required fields)
- 403: Air-gap mode enabled and URL is from public source
- 404: Upstream returned 404
- 409: Hash mismatch (if expected_hash provided)
- 502: Upstream fetch failed (connection error, timeout)
- 503: Upstream source disabled
- [ ] Authentication required (any authenticated user can cache)
- [ ] Audit logging for cache operations
- [ ] Integration tests covering success and error cases
## Technical Notes
- URL parsing for package_name/tag derivation is format-specific:
- npm: `/{package}/-/{package}-{version}.tgz` → package=lodash, tag=4.17.21
- pypi: `/packages/.../requests-2.28.0.tar.gz` → package=requests, tag=2.28.0
- maven: `/{group}/{artifact}/{version}/{artifact}-{version}.jar`
- Deduplication: if same SHA256 already exists, just create new tag pointing to it
---
# Issue #71: System Projects (Cache Namespaces)
**Status: Pending**
## Description
Implement auto-created system projects for organizing cached artifacts by format type. These are special projects that provide a browsable namespace for all cached upstream packages.
## Acceptance Criteria
- [ ] System project names: `_npm`, `_pypi`, `_maven`, `_docker`, `_helm`, `_nuget`, `_deb`, `_rpm`, `_generic`
- [ ] Auto-creation:
- Created automatically on first cache request for that format
- Created by cache endpoint, not at startup
- Uses system user as creator (`created_by = "system"`)
- [ ] System project properties:
- `is_system = true`
- `is_public = true` (readable by all authenticated users)
- `description` = "System cache for {format} packages"
- [ ] Restrictions:
- Cannot be deleted (return 403 with message)
- Cannot be renamed
- Cannot change `is_public` to false
- Only admins can modify description
- [ ] Helper function: `get_or_create_system_project(source_type)` in routes.py or new cache.py module
- [ ] Update project deletion endpoint to check `is_system` flag
- [ ] Update project update endpoint to enforce restrictions
- [ ] Query helper: list all system projects for UI dropdown
- [ ] Unit tests for restrictions
- [ ] Integration tests for auto-creation and restrictions
## Technical Notes
- System projects are identified by `is_system=true`, not just naming convention
- The `_` prefix is a convention for display purposes
- Packages within system projects follow upstream naming (e.g., `_npm/lodash`, `_npm/@types/node`)
---
# Issue #72: Upstream Sources Admin API
**Status: Pending**
## Description
CRUD API endpoints for managing upstream sources configuration. Admin-only access.
## Acceptance Criteria
- [ ] `GET /api/v1/admin/upstream-sources` - List all upstream sources
- Returns array of sources with id, name, source_type, url, enabled, is_public, auth_type, priority, has_credentials, created_at, updated_at
- Supports `?enabled=true/false` filter
- Supports `?source_type=npm,pypi` filter
- Passwords/tokens never returned
- [ ] `POST /api/v1/admin/upstream-sources` - Create upstream source
- Request: name, source_type, url, enabled, is_public, auth_type, username, password, headers, priority
- Validates unique name
- Validates URL format
- Encrypts password/headers before storage
- Returns created source (without secrets)
- [ ] `GET /api/v1/admin/upstream-sources/{id}` - Get source details
- Returns source with `has_credentials` boolean, not actual credentials
- [ ] `PUT /api/v1/admin/upstream-sources/{id}` - Update source
- Partial update supported
- If password provided, re-encrypt; if omitted, keep existing
- Special value `password: null` clears credentials
- [ ] `DELETE /api/v1/admin/upstream-sources/{id}` - Delete source
- Returns 400 if source has cached_urls referencing it (optional: cascade or reassign)
- [ ] `POST /api/v1/admin/upstream-sources/{id}/test` - Test connectivity
- Attempts HEAD request to source URL
- Returns success/failure with status code and timing
- Does not cache anything
- [ ] All endpoints require admin role
- [ ] Audit logging for all mutations
- [ ] Pydantic schemas: UpstreamSourceCreate, UpstreamSourceUpdate, UpstreamSourceResponse
- [ ] Integration tests for all endpoints
## Technical Notes
- Test endpoint should respect auth configuration to verify credentials work
- Consider adding `last_used_at` and `last_error` fields for observability (future enhancement)
---
# Issue #73: Global Cache Settings API
**Status: Pending**
## Description
API endpoints for managing global cache settings including air-gap mode.
## Acceptance Criteria
- [ ] `GET /api/v1/admin/cache-settings` - Get current settings
- Returns: allow_public_internet, auto_create_system_projects, created_at, updated_at
- [ ] `PUT /api/v1/admin/cache-settings` - Update settings
- Partial update supported
- Returns updated settings
- [ ] Settings fields:
- `allow_public_internet` (boolean): When false, blocks all requests to sources marked `is_public=true`
- `auto_create_system_projects` (boolean): When false, system projects must be created manually
- [ ] Admin-only access
- [ ] Audit logging for changes (especially air-gap mode changes)
- [ ] Pydantic schemas: CacheSettingsResponse, CacheSettingsUpdate
- [ ] Initialize singleton row on first access if not exists
- [ ] Integration tests
## Technical Notes
- Air-gap mode change should be logged prominently (security-relevant)
- Consider requiring confirmation header for disabling air-gap mode (similar to factory reset)
---
# Issue #74: Environment Variable Overrides
**Status: Pending**
## Description
Allow cache and upstream configuration via environment variables for containerized deployments. Environment variables override database settings following 12-factor app principles.
## Acceptance Criteria
- [ ] Global settings overrides:
- `ORCHARD_CACHE_ALLOW_PUBLIC_INTERNET=true/false`
- `ORCHARD_CACHE_AUTO_CREATE_SYSTEM_PROJECTS=true/false`
- `ORCHARD_CACHE_ENCRYPTION_KEY` (Fernet key for credential encryption)
- [ ] Upstream source definition via env vars:
- `ORCHARD_UPSTREAM__{NAME}__URL` (double underscore as separator)
- `ORCHARD_UPSTREAM__{NAME}__TYPE` (npm, pypi, maven, etc.)
- `ORCHARD_UPSTREAM__{NAME}__ENABLED` (true/false)
- `ORCHARD_UPSTREAM__{NAME}__IS_PUBLIC` (true/false)
- `ORCHARD_UPSTREAM__{NAME}__AUTH_TYPE` (none, basic, bearer, api_key)
- `ORCHARD_UPSTREAM__{NAME}__USERNAME`
- `ORCHARD_UPSTREAM__{NAME}__PASSWORD`
- `ORCHARD_UPSTREAM__{NAME}__PRIORITY`
- Example: `ORCHARD_UPSTREAM__NPM_PRIVATE__URL=https://npm.corp.com`
- [ ] Env var sources:
- Loaded at startup
- Merged with database sources
- Env var sources have `source = "env"` marker
- Cannot be modified via API (return 400)
- Cannot be deleted via API (return 400)
- [ ] Update Settings class in config.py
- [ ] Update get/list endpoints to include env-defined sources
- [ ] Document all env vars in CLAUDE.md
- [ ] Unit tests for env var parsing
- [ ] Integration tests with env vars set
## Technical Notes
- Double underscore (`__`) separator allows source names with single underscores
- Env-defined sources should appear in API responses but marked as read-only
- Consider startup validation that warns about invalid env var combinations
---
# Issue #75: Frontend - Upstream Sources Management
**Status: Pending**
## Description
Admin UI for managing upstream sources and cache settings.
## Acceptance Criteria
- [ ] New admin page: `/admin/cache` or `/admin/upstream-sources`
- [ ] Upstream sources section:
- Table listing all sources with: name, type, URL, enabled toggle, public badge, priority, actions
- Visual distinction for env-defined sources (locked icon, no edit/delete)
- Create button opens modal/form
- Edit button for DB-defined sources
- Delete with confirmation modal
- Test connection button with status indicator
- [ ] Create/edit form fields:
- Name (text, required)
- Source type (dropdown)
- URL (text, required)
- Priority (number)
- Is public (checkbox)
- Enabled (checkbox)
- Auth type (dropdown: none, basic, bearer, api_key)
- Conditional auth fields based on type:
- Basic: username, password
- Bearer: token
- API key: header name, header value
- Password fields masked, "unchanged" placeholder on edit
- [ ] Cache settings section:
- Air-gap mode toggle with warning
- Auto-create system projects toggle
- "Air-gap mode" shows prominent warning banner when enabled
- [ ] Link from main admin navigation
- [ ] Loading and error states
- [ ] Success/error toast notifications
## Technical Notes
- Use existing admin page patterns from user management
- Air-gap toggle should require confirmation (modal with warning text)
---
# Issue #105: Frontend - System Projects Integration
**Status: Pending**
## Description
Integrate system projects into the frontend UI with appropriate visual treatment and navigation.
## Acceptance Criteria
- [ ] Home page project dropdown:
- System projects shown in separate "Cached Packages" section
- Visual distinction (icon, different background, or badge)
- Format icon for each type (npm, pypi, maven, etc.)
- [ ] Project list/grid:
- System projects can be filtered: "Show system projects" toggle
- Or separate tab: "Projects" | "Package Cache"
- [ ] System project page:
- "System Cache" badge in header
- Description explains this is auto-managed cache
- Settings/delete buttons hidden or disabled
- Shows format type prominently
- [ ] Package page within system project:
- Shows "Cached from" with source URL (linked)
- Shows "First cached" timestamp
- Shows which upstream source provided it
- [ ] Artifact page:
- If artifact came from cache, show provenance:
- Original URL
- Upstream source name
- Fetch timestamp
- [ ] Search includes system projects (with filter option)
## Technical Notes
- Use React context or query params for system project filtering
- Consider dedicated route: `/cache/npm/lodash` as alias for `/_npm/lodash`
---
# Issue #77: CLI - Cache Command
**Status: Pending**
## Description
Add a new `orchard cache` command to the existing CLI for caching artifacts from upstream URLs. This integrates with the new cache API endpoint and can optionally update `orchard.ensure` with cached artifacts.
## Acceptance Criteria
- [ ] New command: `orchard cache <url>` in `orchard/commands/cache.py`
- [ ] Basic usage:
```bash
# Cache a URL, print artifact info
orchard cache https://registry.npmjs.org/lodash/-/lodash-4.17.21.tgz
# Output:
# Caching https://registry.npmjs.org/lodash/-/lodash-4.17.21.tgz...
# Source type: npm
# Package: lodash
# Version: 4.17.21
#
# Successfully cached artifact
# Artifact ID: abc123...
# Size: 1.2 MB
# System project: _npm
# System package: lodash
# System tag: 4.17.21
```
- [ ] Options:
| Option | Description |
|--------|-------------|
| `--type, -t TYPE` | Source type: npm, pypi, maven, docker, helm, generic (auto-detected from URL if not provided) |
| `--package, -p NAME` | Package name in system project (auto-derived from URL if not provided) |
| `--tag TAG` | Tag name in system project (auto-derived from URL if not provided) |
| `--project PROJECT` | Also create tag in this user project |
| `--user-package PKG` | Package name in user project (required if --project specified) |
| `--user-tag TAG` | Tag name in user project (default: same as system tag) |
| `--expected-hash HASH` | Verify downloaded content matches this SHA256 |
| `--add` | Add to orchard.ensure after caching |
| `--add-path PATH` | Extraction path for --add (default: `<package>/`) |
| `--file, -f FILE` | Path to orchard.ensure file |
| `--verbose, -v` | Show detailed output |
- [ ] URL type auto-detection:
- `registry.npmjs.org` → npm
- `pypi.org` or `files.pythonhosted.org` → pypi
- `repo1.maven.org` or contains `/maven2/` → maven
- `registry-1.docker.io` or `docker.io` → docker
- Otherwise → generic
- [ ] Package/version extraction from URL patterns:
- npm: `/{package}/-/{package}-{version}.tgz`
- pypi: `/packages/.../requests-{version}.tar.gz`
- maven: `/{group}/{artifact}/{version}/{artifact}-{version}.jar`
- [ ] Add `cache_artifact()` function to `orchard/api.py`
- [ ] Integration with `--add` flag:
- Parse existing orchard.ensure
- Add new dependency entry pointing to cached artifact
- Use artifact_id (SHA256) for hermetic pinning
- [ ] Batch mode: `orchard cache --file urls.txt`
- One URL per line
- Lines starting with `#` are comments
- Report success/failure for each
- [ ] Exit codes:
- 0: Success (or already cached)
- 1: Fetch failed
- 2: Hash mismatch
- 3: Air-gap mode blocked request
- [ ] Error handling consistent with existing CLI patterns
- [ ] Unit tests in `test/test_cache.py`
- [ ] Update README.md with cache command documentation
## Technical Notes
- Follow existing Click patterns from other commands
- Use `get_auth_headers()` from `orchard/auth.py`
- URL parsing can use `urllib.parse`
- Consider adding URL pattern registry for extensibility
- The `--add` flag should integrate with existing ensure file parsing in `orchard/ensure.py`
## Example Workflows
```bash
# Simple: cache a single URL
orchard cache https://registry.npmjs.org/lodash/-/lodash-4.17.21.tgz
# Cache and add to orchard.ensure for current project
orchard cache https://registry.npmjs.org/lodash/-/lodash-4.17.21.tgz \
--add --add-path libs/lodash/
# Cache with explicit metadata
orchard cache https://internal.corp/files/custom-lib.tar.gz \
--type generic \
--package custom-lib \
--tag v1.0.0
# Cache and cross-reference to user project
orchard cache https://registry.npmjs.org/lodash/-/lodash-4.17.21.tgz \
--project my-app \
--user-package npm-deps \
--user-tag lodash-4.17.21
# Batch cache from file
orchard cache --file deps-urls.txt
# Verify hash while caching
orchard cache https://example.com/file.tar.gz \
--expected-hash sha256:abc123...
```
---
## Out of Scope (Future Enhancements)
- Automatic transitive dependency resolution (client's responsibility)
- Lockfile parsing (`package-lock.json`, `requirements.txt`) - stretch goal for CLI
- Cache eviction policies (we cache forever by design)
- Mirroring/sync between Orchard instances
- Format-specific metadata extraction (npm package.json parsing, etc.)
## Success Criteria
- [ ] Can cache any URL and retrieve by SHA256 hash
- [ ] Cached artifacts persist indefinitely
- [ ] Air-gap mode blocks all public internet access
- [ ] Multiple upstream sources with different auth
- [ ] System projects organize cached packages by format
- [ ] CLI can cache URLs and update orchard.ensure
- [ ] Admin UI for upstream source management

View File

@@ -0,0 +1,294 @@
# Integrity Verification
Orchard uses content-addressable storage with SHA256 hashing to ensure artifact integrity. This document describes how integrity verification works and how to use it.
## How It Works
### Content-Addressable Storage
Orchard stores artifacts using their SHA256 hash as the unique identifier. This provides several benefits:
1. **Automatic deduplication**: Identical content is stored only once
2. **Built-in integrity**: The artifact ID *is* the content hash
3. **Tamper detection**: Any modification changes the hash, making corruption detectable
When you upload a file:
1. Orchard computes the SHA256 hash of the content
2. The hash becomes the artifact ID (64-character hex string)
3. The file is stored in S3 at `fruits/{hash[0:2]}/{hash[2:4]}/{hash}`
4. The hash and metadata are recorded in the database
### Hash Format
- Algorithm: SHA256
- Format: 64-character lowercase hexadecimal string
- Example: `dffd6021bb2bd5b0af676290809ec3a53191dd81c7f70a4b28688a362182986f`
## Client-Side Verification
### Before Upload
Compute the hash locally before uploading to verify the server received your content correctly:
```python
import hashlib
def compute_sha256(content: bytes) -> str:
return hashlib.sha256(content).hexdigest()
# Compute hash before upload
content = open("myfile.tar.gz", "rb").read()
local_hash = compute_sha256(content)
# Upload the file
response = requests.post(
f"{base_url}/api/v1/project/{project}/{package}/upload",
files={"file": ("myfile.tar.gz", content)},
)
result = response.json()
# Verify server computed the same hash
assert result["artifact_id"] == local_hash, "Hash mismatch!"
```
### Providing Expected Hash on Upload
You can provide the expected hash in the upload request. The server will reject the upload if the computed hash doesn't match:
```python
response = requests.post(
f"{base_url}/api/v1/project/{project}/{package}/upload",
files={"file": ("myfile.tar.gz", content)},
headers={"X-Checksum-SHA256": local_hash},
)
# Returns 422 if hash doesn't match
if response.status_code == 422:
print("Checksum mismatch - upload rejected")
```
### After Download
Verify downloaded content matches the expected hash using response headers:
```python
response = requests.get(
f"{base_url}/api/v1/project/{project}/{package}/+/{tag}",
params={"mode": "proxy"},
)
# Get expected hash from header
expected_hash = response.headers.get("X-Checksum-SHA256")
# Compute hash of downloaded content
actual_hash = compute_sha256(response.content)
# Verify
if actual_hash != expected_hash:
raise Exception(f"Integrity check failed! Expected {expected_hash}, got {actual_hash}")
```
### Response Headers for Verification
Download responses include multiple headers for verification:
| Header | Format | Description |
|--------|--------|-------------|
| `X-Checksum-SHA256` | Hex string | SHA256 hash (64 chars) |
| `ETag` | `"<hash>"` | SHA256 hash in quotes |
| `Digest` | `sha-256=<base64>` | RFC 3230 format (base64-encoded) |
| `Content-Length` | Integer | File size in bytes |
### Server-Side Verification on Download
Request server-side verification during download:
```bash
# Pre-verification: Server verifies before streaming (returns 500 if corrupt)
curl "${base_url}/api/v1/project/${project}/${package}/+/${tag}?mode=proxy&verify=true&verify_mode=pre"
# Stream verification: Server verifies while streaming (logs error if corrupt)
curl "${base_url}/api/v1/project/${project}/${package}/+/${tag}?mode=proxy&verify=true&verify_mode=stream"
```
The `X-Verified` header indicates whether server-side verification was performed:
- `X-Verified: true` - Content was verified by the server
## Server-Side Consistency Check
### Consistency Check Endpoint
Administrators can run a consistency check to verify all stored artifacts:
```bash
curl "${base_url}/api/v1/admin/consistency-check"
```
Response:
```json
{
"total_artifacts_checked": 1234,
"healthy": true,
"orphaned_s3_objects": 0,
"missing_s3_objects": 0,
"size_mismatches": 0,
"orphaned_s3_keys": [],
"missing_s3_keys": [],
"size_mismatch_artifacts": []
}
```
### What the Check Verifies
1. **Missing S3 objects**: Database records with no corresponding S3 object
2. **Orphaned S3 objects**: S3 objects with no database record
3. **Size mismatches**: S3 object size doesn't match database record
### Running Consistency Checks
**Manual check:**
```bash
# Check all artifacts
curl "${base_url}/api/v1/admin/consistency-check"
# Limit results (for large deployments)
curl "${base_url}/api/v1/admin/consistency-check?limit=100"
```
**Scheduled checks (recommended):**
Set up a cron job or Kubernetes CronJob to run periodic checks:
```yaml
# Kubernetes CronJob example
apiVersion: batch/v1
kind: CronJob
metadata:
name: orchard-consistency-check
spec:
schedule: "0 2 * * *" # Daily at 2 AM
jobTemplate:
spec:
template:
spec:
containers:
- name: check
image: curlimages/curl
command:
- /bin/sh
- -c
- |
response=$(curl -s "${ORCHARD_URL}/api/v1/admin/consistency-check")
healthy=$(echo "$response" | jq -r '.healthy')
if [ "$healthy" != "true" ]; then
echo "ALERT: Consistency check failed!"
echo "$response"
exit 1
fi
echo "Consistency check passed"
restartPolicy: OnFailure
```
## Recovery Procedures
### Corrupted Artifact (Size Mismatch)
If the consistency check reports size mismatches:
1. **Identify affected artifacts:**
```bash
curl "${base_url}/api/v1/admin/consistency-check" | jq '.size_mismatch_artifacts'
```
2. **Check if artifact can be re-uploaded:**
- If the original content is available, delete the corrupted artifact and re-upload
- The same content will produce the same artifact ID
3. **If original content is lost:**
- The artifact data is corrupted and cannot be recovered
- Delete the artifact record and notify affected users
- Consider restoring from backup if available
### Missing S3 Object
If database records exist but S3 objects are missing:
1. **Identify affected artifacts:**
```bash
curl "${base_url}/api/v1/admin/consistency-check" | jq '.missing_s3_keys'
```
2. **Check S3 bucket:**
- Verify the S3 bucket exists and is accessible
- Check S3 access logs for deletion events
- Check if objects were moved or lifecycle-deleted
3. **Recovery options:**
- Restore from S3 versioning (if enabled)
- Restore from backup
- Re-upload original content (if available)
- Delete orphaned database records
### Orphaned S3 Objects
If S3 objects exist without database records:
1. **Identify orphaned objects:**
```bash
curl "${base_url}/api/v1/admin/consistency-check" | jq '.orphaned_s3_keys'
```
2. **Investigate cause:**
- Upload interrupted before database commit?
- Database record deleted but S3 cleanup failed?
3. **Resolution:**
- If content is needed, create database record manually
- If content is not needed, delete the S3 object to reclaim storage
### Preventive Measures
1. **Enable S3 versioning** to recover from accidental deletions
2. **Regular backups** of both database and S3 bucket
3. **Scheduled consistency checks** to detect issues early
4. **Monitoring and alerting** on consistency check failures
5. **Audit logging** to track all artifact operations
## Verification in CI/CD
### Verifying Artifacts in Pipelines
```bash
#!/bin/bash
# Download and verify artifact in CI pipeline
ARTIFACT_URL="${ORCHARD_URL}/api/v1/project/${PROJECT}/${PACKAGE}/+/${TAG}"
# Download with verification headers
response=$(curl -s -D - "${ARTIFACT_URL}?mode=proxy" -o artifact.tar.gz)
expected_hash=$(echo "$response" | grep -i "X-Checksum-SHA256" | cut -d: -f2 | tr -d ' \r')
# Compute actual hash
actual_hash=$(sha256sum artifact.tar.gz | cut -d' ' -f1)
# Verify
if [ "$actual_hash" != "$expected_hash" ]; then
echo "ERROR: Integrity check failed!"
echo "Expected: $expected_hash"
echo "Actual: $actual_hash"
exit 1
fi
echo "Integrity verified: $actual_hash"
```
### Using Server-Side Verification
For critical deployments, use server-side pre-verification:
```bash
# Server verifies before streaming - returns 500 if corrupt
curl -f "${ARTIFACT_URL}?mode=proxy&verify=true&verify_mode=pre" -o artifact.tar.gz
```
This ensures the artifact is verified before any bytes are streamed to your pipeline.

View File

@@ -0,0 +1,228 @@
# PyPI Proxy Performance & Multi-Protocol Architecture Design
**Date:** 2026-02-04
**Status:** Approved
**Branch:** fix/pypi-proxy-timeout
## Overview
Comprehensive infrastructure overhaul to address latency, throughput, and resource consumption issues in the PyPI proxy, while establishing a foundation for npm, Maven, and other package protocols.
## Goals
1. **Reduce latency** - Eliminate per-request connection overhead, cache aggressively
2. **Increase throughput** - Handle hundreds of concurrent requests without degradation
3. **Lower resource usage** - Connection pooling, efficient DB queries, proper async I/O
4. **Enable multi-protocol** - Abstract base class ready for npm/Maven/etc.
5. **Maintain hermetic builds** - Immutable artifact content and metadata, mutable discovery data
## Architecture
```
┌─────────────────────────────────────────────────────────────────────┐
│ FastAPI Application │
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │
│ │ PyPI Proxy │ │ npm Proxy │ │ Maven Proxy │ │ (future) │ │
│ │ Router │ │ Router │ │ Router │ │ │ │
│ └──────┬──────┘ └──────┬──────┘ └──────┬──────┘ └─────────────┘ │
│ │ │ │ │
│ └────────────────┼────────────────┘ │
│ ▼ │
│ ┌───────────────────────┐ │
│ │ PackageProxyBase │ ← Abstract base class │
│ │ - check_cache() │ │
│ │ - fetch_upstream() │ │
│ │ - store_artifact() │ │
│ │ - serve_artifact() │ │
│ └───────────┬───────────┘ │
│ │ │
│ ┌────────────────┼────────────────┐ │
│ ▼ ▼ ▼ │
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │
│ │ HttpClient │ │ CacheService│ │ ThreadPool │ │
│ │ Manager │ │ (Redis) │ │ Executor │ │
│ └─────────────┘ └─────────────┘ └─────────────┘ │
│ │ │ │ │
└─────────┼────────────────┼────────────────┼──────────────────────────┘
▼ ▼ ▼
┌──────────┐ ┌──────────┐ ┌──────────────┐
│ Upstream │ │ Redis │ │ S3/MinIO │
│ Sources │ │ │ │ │
└──────────┘ └──────────┘ └──────────────┘
```
## Components
### 1. HttpClientManager
Manages httpx.AsyncClient pools with FastAPI lifespan integration.
**Features:**
- Default pool for general requests
- Per-upstream pools for sources needing specific config/auth
- Graceful shutdown drains in-flight requests
- Dedicated thread pool for blocking operations
**Configuration:**
```bash
ORCHARD_HTTP_MAX_CONNECTIONS=100 # Default pool size
ORCHARD_HTTP_KEEPALIVE_CONNECTIONS=20 # Keep-alive connections
ORCHARD_HTTP_CONNECT_TIMEOUT=30 # Connection timeout (seconds)
ORCHARD_HTTP_READ_TIMEOUT=60 # Read timeout (seconds)
ORCHARD_HTTP_WORKER_THREADS=32 # Thread pool size
```
**File:** `backend/app/http_client.py`
### 2. CacheService (Redis Layer)
Redis-backed caching with category-aware TTL and invalidation.
**Cache Categories:**
| Category | TTL | Invalidation | Purpose |
|----------|-----|--------------|---------|
| ARTIFACT_METADATA | Forever | Never (immutable) | Artifact info by SHA256 |
| ARTIFACT_DEPENDENCIES | Forever | Never (immutable) | Extracted deps by SHA256 |
| DEPENDENCY_RESOLUTION | Forever | Manual/refresh param | Resolution results |
| UPSTREAM_SOURCES | 1 hour | On DB change | Upstream config |
| PACKAGE_INDEX | 5 min | TTL only | PyPI/npm index pages |
| PACKAGE_VERSIONS | 5 min | TTL only | Version listings |
**Key format:** `orchard:{category}:{protocol}:{identifier}`
**Configuration:**
```bash
ORCHARD_REDIS_HOST=redis
ORCHARD_REDIS_PORT=6379
ORCHARD_REDIS_DB=0
ORCHARD_CACHE_TTL_INDEX=300 # Package index: 5 minutes
ORCHARD_CACHE_TTL_VERSIONS=300 # Version listings: 5 minutes
ORCHARD_CACHE_TTL_UPSTREAM=3600 # Upstream config: 1 hour
```
**File:** `backend/app/cache_service.py`
### 3. PackageProxyBase
Abstract base class defining the cache→fetch→store→serve flow.
**Abstract methods (protocol-specific):**
- `get_protocol_name()` - Return 'pypi', 'npm', 'maven'
- `get_system_project_name()` - Return '_pypi', '_npm'
- `rewrite_index_html()` - Rewrite upstream index to Orchard URLs
- `extract_metadata()` - Extract deps from package file
- `parse_package_url()` - Parse URL into package/version/filename
**Concrete methods (shared):**
- `serve_index()` - Serve package index with caching
- `serve_artifact()` - Full cache→fetch→store→serve flow
**File:** `backend/app/proxy_base.py`
### 4. ArtifactRepository (DB Optimization)
Optimized database operations eliminating N+1 queries.
**Key methods:**
- `get_or_create_artifact()` - Atomic upsert via ON CONFLICT
- `batch_upsert_dependencies()` - Single INSERT for all deps
- `get_cached_url_with_artifact()` - Joined query for cache lookup
**Query reduction:**
| Operation | Before | After |
|-----------|--------|-------|
| Cache hit check | 2 queries | 1 query (joined) |
| Store artifact | 3-4 queries | 1 query (upsert) |
| Store 50 deps | 50+ queries | 1 query (batch) |
**Configuration:**
```bash
ORCHARD_DATABASE_POOL_SIZE=20 # Base connections (up from 5)
ORCHARD_DATABASE_MAX_OVERFLOW=30 # Burst capacity (up from 10)
ORCHARD_DATABASE_POOL_TIMEOUT=30 # Wait timeout
ORCHARD_DATABASE_POOL_PRE_PING=false # Disable in prod for performance
```
**File:** `backend/app/db_utils.py`
### 5. Dependency Resolution Caching
Cache resolution results for ensure files and API queries.
**Cache key:** Hash of (artifact_id, max_depth, include_optional)
**Invalidation:** Manual only (immutable artifact deps mean cached resolutions stay valid)
**Refresh:** `?refresh=true` parameter forces fresh resolution
**File:** Updates to `backend/app/dependencies.py`
### 6. FastAPI Integration
Lifespan-managed infrastructure with dependency injection.
**Startup:**
1. Initialize HttpClientManager (connection pools)
2. Initialize CacheService (Redis connection)
3. Load upstream source configs
**Shutdown:**
1. Drain in-flight HTTP requests
2. Close Redis connections
3. Shutdown thread pool
**Health endpoint additions:**
- Database connection status
- Redis ping
- HTTP pool active/max connections
- Thread pool active/max workers
**File:** Updates to `backend/app/main.py`
## Files Summary
**New files:**
- `backend/app/http_client.py` - HttpClientManager
- `backend/app/cache_service.py` - CacheService
- `backend/app/proxy_base.py` - PackageProxyBase
- `backend/app/db_utils.py` - ArtifactRepository
**Modified files:**
- `backend/app/config.py` - New settings
- `backend/app/main.py` - Lifespan integration
- `backend/app/pypi_proxy.py` - Refactor to use base class
- `backend/app/dependencies.py` - Resolution caching
- `backend/app/routes.py` - Health endpoint, DI
## Hermetic Build Guarantees
**Immutable (cached forever):**
- Artifact content (by SHA256)
- Extracted dependencies for a specific artifact
- Dependency resolution results
**Mutable (TTL + event invalidation):**
- Package index listings
- Version discovery
- Upstream source configuration
Once an artifact is cached with SHA256 `abc123` and dependencies extracted, that data never changes.
## Performance Expectations
| Metric | Before | After |
|--------|--------|-------|
| HTTP connection setup | Per request (~100-500ms) | Pooled (~5ms) |
| Cache hit (index page) | N/A | ~5ms (Redis) |
| Store 50 dependencies | ~500ms (50 queries) | ~10ms (1 query) |
| Dependency resolution (cached) | N/A | ~5ms |
| Concurrent request capacity | ~15 (DB pool) | ~50 (configurable) |
## Testing Requirements
- Unit tests for each new component
- Integration tests for full proxy flow
- Load tests to verify pool sizing
- Cache hit/miss verification tests

File diff suppressed because it is too large Load Diff

File diff suppressed because it is too large Load Diff

View File

@@ -12,9 +12,12 @@
"test:coverage": "vitest run --coverage"
},
"dependencies": {
"@types/dagre": "^0.7.53",
"dagre": "^0.8.5",
"react": "^18.2.0",
"react-dom": "^18.2.0",
"react-router-dom": "6.28.0"
"react-router-dom": "6.28.0",
"reactflow": "^11.11.4"
},
"devDependencies": {
"@testing-library/jest-dom": "^6.4.2",
@@ -34,6 +37,15 @@
"ufo": "1.5.4",
"rollup": "4.52.4",
"caniuse-lite": "1.0.30001692",
"baseline-browser-mapping": "2.9.5"
"baseline-browser-mapping": "2.9.5",
"lodash": "4.17.21",
"electron-to-chromium": "1.5.72",
"@babel/core": "7.26.0",
"@babel/traverse": "7.26.4",
"@babel/types": "7.26.3",
"@babel/compat-data": "7.26.3",
"@babel/parser": "7.26.3",
"@babel/generator": "7.26.3",
"@babel/code-frame": "7.26.2"
}
}

View File

@@ -1,5 +1,6 @@
import { Routes, Route, Navigate, useLocation } from 'react-router-dom';
import { AuthProvider, useAuth } from './contexts/AuthContext';
import { TeamProvider } from './contexts/TeamContext';
import Layout from './components/Layout';
import Home from './pages/Home';
import ProjectPage from './pages/ProjectPage';
@@ -10,6 +11,12 @@ import ChangePasswordPage from './pages/ChangePasswordPage';
import APIKeysPage from './pages/APIKeysPage';
import AdminUsersPage from './pages/AdminUsersPage';
import AdminOIDCPage from './pages/AdminOIDCPage';
import AdminCachePage from './pages/AdminCachePage';
import ProjectSettingsPage from './pages/ProjectSettingsPage';
import TeamsPage from './pages/TeamsPage';
import TeamDashboardPage from './pages/TeamDashboardPage';
import TeamSettingsPage from './pages/TeamSettingsPage';
import TeamMembersPage from './pages/TeamMembersPage';
// Component that checks if user must change password
function RequirePasswordChange({ children }: { children: React.ReactNode }) {
@@ -44,7 +51,13 @@ function AppRoutes() {
<Route path="/settings/api-keys" element={<APIKeysPage />} />
<Route path="/admin/users" element={<AdminUsersPage />} />
<Route path="/admin/oidc" element={<AdminOIDCPage />} />
<Route path="/admin/cache" element={<AdminCachePage />} />
<Route path="/teams" element={<TeamsPage />} />
<Route path="/teams/:slug" element={<TeamDashboardPage />} />
<Route path="/teams/:slug/settings" element={<TeamSettingsPage />} />
<Route path="/teams/:slug/members" element={<TeamMembersPage />} />
<Route path="/project/:projectName" element={<ProjectPage />} />
<Route path="/project/:projectName/settings" element={<ProjectSettingsPage />} />
<Route path="/project/:projectName/:packageName" element={<PackagePage />} />
</Routes>
</Layout>
@@ -58,7 +71,9 @@ function AppRoutes() {
function App() {
return (
<AuthProvider>
<AppRoutes />
<TeamProvider>
<AppRoutes />
</TeamProvider>
</AuthProvider>
);
}

View File

@@ -1,14 +1,11 @@
import {
Project,
Package,
Tag,
TagDetail,
Artifact,
ArtifactDetail,
PackageArtifact,
UploadResponse,
PaginatedResponse,
ListParams,
TagListParams,
PackageListParams,
ArtifactListParams,
ProjectListParams,
@@ -32,6 +29,20 @@ import {
OIDCConfig,
OIDCConfigUpdate,
OIDCStatus,
PackageVersion,
ArtifactDependenciesResponse,
ReverseDependenciesResponse,
DependencyResolutionResponse,
TeamDetail,
TeamMember,
TeamCreate,
TeamUpdate,
TeamMemberCreate,
TeamMemberUpdate,
UpstreamSource,
UpstreamSourceCreate,
UpstreamSourceUpdate,
UpstreamSourceTestResult,
} from './types';
const API_BASE = '/api/v1';
@@ -64,7 +75,13 @@ export class ForbiddenError extends ApiError {
async function handleResponse<T>(response: Response): Promise<T> {
if (!response.ok) {
const error = await response.json().catch(() => ({ detail: 'Unknown error' }));
const message = error.detail || `HTTP ${response.status}`;
// Handle detail as string or object (backend may return structured errors)
let message: string;
if (typeof error.detail === 'object') {
message = JSON.stringify(error.detail);
} else {
message = error.detail || `HTTP ${response.status}`;
}
if (response.status === 401) {
throw new UnauthorizedError(message);
@@ -156,7 +173,7 @@ export async function listProjectsSimple(params: ListParams = {}): Promise<Proje
return data.items;
}
export async function createProject(data: { name: string; description?: string; is_public?: boolean }): Promise<Project> {
export async function createProject(data: { name: string; description?: string; is_public?: boolean; team_id?: string }): Promise<Project> {
const response = await fetch(`${API_BASE}/projects`, {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
@@ -170,6 +187,30 @@ export async function getProject(name: string): Promise<Project> {
return handleResponse<Project>(response);
}
export async function updateProject(
projectName: string,
data: { description?: string; is_public?: boolean }
): Promise<Project> {
const response = await fetch(`${API_BASE}/projects/${projectName}`, {
method: 'PUT',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify(data),
credentials: 'include',
});
return handleResponse<Project>(response);
}
export async function deleteProject(projectName: string): Promise<void> {
const response = await fetch(`${API_BASE}/projects/${projectName}`, {
method: 'DELETE',
credentials: 'include',
});
if (!response.ok) {
const error = await response.json().catch(() => ({ detail: 'Unknown error' }));
throw new Error(error.detail || `HTTP ${response.status}`);
}
}
// Package API
export async function listPackages(projectName: string, params: PackageListParams = {}): Promise<PaginatedResponse<Package>> {
const query = buildQueryString(params as Record<string, unknown>);
@@ -196,32 +237,6 @@ export async function createPackage(projectName: string, data: { name: string; d
return handleResponse<Package>(response);
}
// Tag API
export async function listTags(projectName: string, packageName: string, params: TagListParams = {}): Promise<PaginatedResponse<TagDetail>> {
const query = buildQueryString(params as Record<string, unknown>);
const response = await fetch(`${API_BASE}/project/${projectName}/${packageName}/tags${query}`);
return handleResponse<PaginatedResponse<TagDetail>>(response);
}
export async function listTagsSimple(projectName: string, packageName: string, params: TagListParams = {}): Promise<TagDetail[]> {
const data = await listTags(projectName, packageName, params);
return data.items;
}
export async function getTag(projectName: string, packageName: string, tagName: string): Promise<TagDetail> {
const response = await fetch(`${API_BASE}/project/${projectName}/${packageName}/tags/${tagName}`);
return handleResponse<TagDetail>(response);
}
export async function createTag(projectName: string, packageName: string, data: { name: string; artifact_id: string }): Promise<Tag> {
const response = await fetch(`${API_BASE}/project/${projectName}/${packageName}/tags`, {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify(data),
});
return handleResponse<Tag>(response);
}
// Artifact API
export async function getArtifact(artifactId: string): Promise<ArtifactDetail> {
const response = await fetch(`${API_BASE}/artifact/${artifactId}`);
@@ -232,18 +247,23 @@ export async function listPackageArtifacts(
projectName: string,
packageName: string,
params: ArtifactListParams = {}
): Promise<PaginatedResponse<Artifact & { tags: string[] }>> {
): Promise<PaginatedResponse<PackageArtifact>> {
const query = buildQueryString(params as Record<string, unknown>);
const response = await fetch(`${API_BASE}/project/${projectName}/${packageName}/artifacts${query}`);
return handleResponse<PaginatedResponse<Artifact & { tags: string[] }>>(response);
return handleResponse<PaginatedResponse<PackageArtifact>>(response);
}
// Upload
export async function uploadArtifact(projectName: string, packageName: string, file: File, tag?: string): Promise<UploadResponse> {
export async function uploadArtifact(
projectName: string,
packageName: string,
file: File,
version?: string
): Promise<UploadResponse> {
const formData = new FormData();
formData.append('file', file);
if (tag) {
formData.append('tag', tag);
if (version) {
formData.append('version', version);
}
const response = await fetch(`${API_BASE}/project/${projectName}/${packageName}/upload`, {
@@ -443,3 +463,260 @@ export function getOIDCLoginUrl(returnTo?: string): string {
const query = params.toString();
return `${API_BASE}/auth/oidc/login${query ? `?${query}` : ''}`;
}
// Version API
export async function listVersions(
projectName: string,
packageName: string,
params: ListParams = {}
): Promise<PaginatedResponse<PackageVersion>> {
const query = buildQueryString(params as Record<string, unknown>);
const response = await fetch(`${API_BASE}/project/${projectName}/${packageName}/versions${query}`);
return handleResponse<PaginatedResponse<PackageVersion>>(response);
}
export async function getVersion(
projectName: string,
packageName: string,
version: string
): Promise<PackageVersion> {
const response = await fetch(`${API_BASE}/project/${projectName}/${packageName}/versions/${version}`);
return handleResponse<PackageVersion>(response);
}
export async function deleteVersion(
projectName: string,
packageName: string,
version: string
): Promise<void> {
const response = await fetch(`${API_BASE}/project/${projectName}/${packageName}/versions/${version}`, {
method: 'DELETE',
credentials: 'include',
});
if (!response.ok) {
const error = await response.json().catch(() => ({ detail: 'Unknown error' }));
throw new Error(error.detail || `HTTP ${response.status}`);
}
}
// Dependency API
export async function getArtifactDependencies(artifactId: string): Promise<ArtifactDependenciesResponse> {
const response = await fetch(`${API_BASE}/artifact/${artifactId}/dependencies`);
return handleResponse<ArtifactDependenciesResponse>(response);
}
export async function getDependenciesByRef(
projectName: string,
packageName: string,
ref: string
): Promise<ArtifactDependenciesResponse> {
const response = await fetch(`${API_BASE}/project/${projectName}/${packageName}/+/${ref}/dependencies`);
return handleResponse<ArtifactDependenciesResponse>(response);
}
export async function getReverseDependencies(
projectName: string,
packageName: string,
params: { page?: number; limit?: number } = {}
): Promise<ReverseDependenciesResponse> {
const query = buildQueryString(params as Record<string, unknown>);
const response = await fetch(`${API_BASE}/project/${projectName}/${packageName}/reverse-dependencies${query}`);
return handleResponse<ReverseDependenciesResponse>(response);
}
export async function resolveDependencies(
projectName: string,
packageName: string,
ref: string
): Promise<DependencyResolutionResponse> {
const response = await fetch(`${API_BASE}/project/${projectName}/${packageName}/+/${ref}/resolve`);
return handleResponse<DependencyResolutionResponse>(response);
}
export async function getEnsureFile(
projectName: string,
packageName: string,
ref: string
): Promise<string> {
const response = await fetch(`${API_BASE}/project/${projectName}/${packageName}/+/${ref}/ensure`);
if (!response.ok) {
const error = await response.json().catch(() => ({ detail: 'Unknown error' }));
throw new ApiError(error.detail || `HTTP ${response.status}`, response.status);
}
return response.text();
}
// Team API
export async function listTeams(params: ListParams = {}): Promise<PaginatedResponse<TeamDetail>> {
const query = buildQueryString(params as Record<string, unknown>);
const response = await fetch(`${API_BASE}/teams${query}`, {
credentials: 'include',
});
return handleResponse<PaginatedResponse<TeamDetail>>(response);
}
export async function createTeam(data: TeamCreate): Promise<TeamDetail> {
const response = await fetch(`${API_BASE}/teams`, {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify(data),
credentials: 'include',
});
return handleResponse<TeamDetail>(response);
}
export async function getTeam(slug: string): Promise<TeamDetail> {
const response = await fetch(`${API_BASE}/teams/${slug}`, {
credentials: 'include',
});
return handleResponse<TeamDetail>(response);
}
export async function updateTeam(slug: string, data: TeamUpdate): Promise<TeamDetail> {
const response = await fetch(`${API_BASE}/teams/${slug}`, {
method: 'PUT',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify(data),
credentials: 'include',
});
return handleResponse<TeamDetail>(response);
}
export async function deleteTeam(slug: string): Promise<void> {
const response = await fetch(`${API_BASE}/teams/${slug}`, {
method: 'DELETE',
credentials: 'include',
});
if (!response.ok) {
const error = await response.json().catch(() => ({ detail: 'Unknown error' }));
throw new ApiError(error.detail || `HTTP ${response.status}`, response.status);
}
}
export async function listTeamMembers(slug: string): Promise<TeamMember[]> {
const response = await fetch(`${API_BASE}/teams/${slug}/members`, {
credentials: 'include',
});
return handleResponse<TeamMember[]>(response);
}
export async function addTeamMember(slug: string, data: TeamMemberCreate): Promise<TeamMember> {
const response = await fetch(`${API_BASE}/teams/${slug}/members`, {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify(data),
credentials: 'include',
});
return handleResponse<TeamMember>(response);
}
export async function updateTeamMember(
slug: string,
username: string,
data: TeamMemberUpdate
): Promise<TeamMember> {
const response = await fetch(`${API_BASE}/teams/${slug}/members/${username}`, {
method: 'PUT',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify(data),
credentials: 'include',
});
return handleResponse<TeamMember>(response);
}
export async function removeTeamMember(slug: string, username: string): Promise<void> {
const response = await fetch(`${API_BASE}/teams/${slug}/members/${username}`, {
method: 'DELETE',
credentials: 'include',
});
if (!response.ok) {
const error = await response.json().catch(() => ({ detail: 'Unknown error' }));
throw new ApiError(error.detail || `HTTP ${response.status}`, response.status);
}
}
export async function listTeamProjects(
slug: string,
params: ProjectListParams = {}
): Promise<PaginatedResponse<Project>> {
const query = buildQueryString(params as Record<string, unknown>);
const response = await fetch(`${API_BASE}/teams/${slug}/projects${query}`, {
credentials: 'include',
});
return handleResponse<PaginatedResponse<Project>>(response);
}
// User search (for autocomplete)
export interface UserSearchResult {
id: string;
username: string;
is_admin: boolean;
}
export async function searchUsers(query: string, limit: number = 10): Promise<UserSearchResult[]> {
const response = await fetch(`${API_BASE}/users/search?q=${encodeURIComponent(query)}&limit=${limit}`, {
credentials: 'include',
});
return handleResponse<UserSearchResult[]>(response);
}
// Upstream Sources Admin API
export interface UpstreamSourceListParams {
enabled?: boolean;
source_type?: string;
}
export async function listUpstreamSources(params: UpstreamSourceListParams = {}): Promise<UpstreamSource[]> {
const query = buildQueryString(params as Record<string, unknown>);
const response = await fetch(`${API_BASE}/admin/upstream-sources${query}`, {
credentials: 'include',
});
return handleResponse<UpstreamSource[]>(response);
}
export async function createUpstreamSource(data: UpstreamSourceCreate): Promise<UpstreamSource> {
const response = await fetch(`${API_BASE}/admin/upstream-sources`, {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify(data),
credentials: 'include',
});
return handleResponse<UpstreamSource>(response);
}
export async function getUpstreamSource(id: string): Promise<UpstreamSource> {
const response = await fetch(`${API_BASE}/admin/upstream-sources/${id}`, {
credentials: 'include',
});
return handleResponse<UpstreamSource>(response);
}
export async function updateUpstreamSource(id: string, data: UpstreamSourceUpdate): Promise<UpstreamSource> {
const response = await fetch(`${API_BASE}/admin/upstream-sources/${id}`, {
method: 'PUT',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify(data),
credentials: 'include',
});
return handleResponse<UpstreamSource>(response);
}
export async function deleteUpstreamSource(id: string): Promise<void> {
const response = await fetch(`${API_BASE}/admin/upstream-sources/${id}`, {
method: 'DELETE',
credentials: 'include',
});
if (!response.ok) {
const error = await response.json().catch(() => ({ detail: 'Unknown error' }));
throw new ApiError(error.detail || `HTTP ${response.status}`, response.status);
}
}
export async function testUpstreamSource(id: string): Promise<UpstreamSourceTestResult> {
const response = await fetch(`${API_BASE}/admin/upstream-sources/${id}/test`, {
method: 'POST',
credentials: 'include',
});
return handleResponse<UpstreamSourceTestResult>(response);
}

View File

@@ -114,3 +114,32 @@
font-size: 0.875rem;
color: var(--text-primary);
}
/* Access source styling */
.access-source {
display: inline-block;
padding: 0.2rem 0.4rem;
border-radius: 4px;
font-size: 0.75rem;
font-weight: 500;
}
.access-source--explicit {
background: var(--bg-tertiary);
color: var(--text-secondary);
}
.access-source--team {
background: var(--color-info-bg, #e3f2fd);
color: var(--color-info, #1976d2);
}
/* Team access row styling */
.team-access-row {
background: var(--bg-secondary, #fafafa);
}
.team-access-row td.actions .text-muted {
font-size: 0.8125rem;
font-style: italic;
}

View File

@@ -208,85 +208,104 @@ export function AccessManagement({ projectName }: AccessManagementProps) {
<tr>
<th>User</th>
<th>Access Level</th>
<th>Source</th>
<th>Granted</th>
<th>Expires</th>
<th>Actions</th>
</tr>
</thead>
<tbody>
{permissions.map((p) => (
<tr key={p.id}>
<td>{p.user_id}</td>
<td>
{editingUser === p.user_id ? (
<select
value={editLevel}
onChange={(e) => setEditLevel(e.target.value as AccessLevel)}
disabled={submitting}
>
<option value="read">Read</option>
<option value="write">Write</option>
<option value="admin">Admin</option>
</select>
) : (
<span className={`access-badge access-badge--${p.level}`}>
{p.level}
</span>
)}
</td>
<td>{new Date(p.created_at).toLocaleDateString()}</td>
<td>
{editingUser === p.user_id ? (
<input
type="date"
value={editExpiresAt}
onChange={(e) => setEditExpiresAt(e.target.value)}
disabled={submitting}
min={new Date().toISOString().split('T')[0]}
/>
) : (
formatExpiration(p.expires_at)
)}
</td>
<td className="actions">
{editingUser === p.user_id ? (
<>
<button
className="btn btn-sm btn-primary"
onClick={() => handleUpdate(p.user_id)}
{permissions.map((p) => {
const isTeamBased = p.source === 'team';
return (
<tr key={p.id} className={isTeamBased ? 'team-access-row' : ''}>
<td>{p.user_id}</td>
<td>
{editingUser === p.user_id && !isTeamBased ? (
<select
value={editLevel}
onChange={(e) => setEditLevel(e.target.value as AccessLevel)}
disabled={submitting}
>
Save
</button>
<button
className="btn btn-sm"
onClick={cancelEdit}
<option value="read">Read</option>
<option value="write">Write</option>
<option value="admin">Admin</option>
</select>
) : (
<span className={`access-badge access-badge--${p.level}`}>
{p.level}
</span>
)}
</td>
<td>
{isTeamBased ? (
<span className="access-source access-source--team" title={`Team role: ${p.team_role}`}>
Team: {p.team_slug}
</span>
) : (
<span className="access-source access-source--explicit">
Explicit
</span>
)}
</td>
<td>{new Date(p.created_at).toLocaleDateString()}</td>
<td>
{editingUser === p.user_id && !isTeamBased ? (
<input
type="date"
value={editExpiresAt}
onChange={(e) => setEditExpiresAt(e.target.value)}
disabled={submitting}
>
Cancel
</button>
</>
) : (
<>
<button
className="btn btn-sm"
onClick={() => startEdit(p)}
disabled={submitting}
>
Edit
</button>
<button
className="btn btn-sm btn-danger"
onClick={() => handleRevoke(p.user_id)}
disabled={submitting}
>
Revoke
</button>
</>
)}
</td>
</tr>
))}
min={new Date().toISOString().split('T')[0]}
/>
) : (
formatExpiration(p.expires_at)
)}
</td>
<td className="actions">
{isTeamBased ? (
<span className="text-muted" title="Manage access via team settings">
Via team
</span>
) : editingUser === p.user_id ? (
<>
<button
className="btn btn-sm btn-primary"
onClick={() => handleUpdate(p.user_id)}
disabled={submitting}
>
Save
</button>
<button
className="btn btn-sm"
onClick={cancelEdit}
disabled={submitting}
>
Cancel
</button>
</>
) : (
<>
<button
className="btn btn-sm"
onClick={() => startEdit(p)}
disabled={submitting}
>
Edit
</button>
<button
className="btn btn-sm btn-danger"
onClick={() => handleRevoke(p.user_id)}
disabled={submitting}
>
Revoke
</button>
</>
)}
</td>
</tr>
);
})}
</tbody>
</table>
)}

View File

@@ -0,0 +1,323 @@
/* Dependency Graph Modal */
.dependency-graph-modal {
position: fixed;
top: 0;
left: 0;
right: 0;
bottom: 0;
background: rgba(0, 0, 0, 0.8);
display: flex;
align-items: center;
justify-content: center;
z-index: 1000;
padding: 24px;
}
.dependency-graph-content {
background: var(--bg-secondary);
border: 1px solid var(--border-primary);
border-radius: var(--radius-lg);
width: 100%;
max-width: 1200px;
height: 80vh;
display: flex;
flex-direction: column;
overflow: hidden;
}
.dependency-graph-header {
display: flex;
align-items: center;
gap: 16px;
padding: 16px 20px;
border-bottom: 1px solid var(--border-primary);
background: var(--bg-tertiary);
}
.dependency-graph-header h2 {
margin: 0;
font-size: 1.125rem;
font-weight: 600;
color: var(--text-primary);
}
.dependency-graph-info {
display: flex;
align-items: center;
gap: 12px;
flex: 1;
font-size: 0.875rem;
color: var(--text-secondary);
}
.graph-stats {
color: var(--text-muted);
font-size: 0.8125rem;
}
.missing-count {
color: #f59e0b;
}
.close-btn {
background: transparent;
border: none;
color: var(--text-secondary);
cursor: pointer;
padding: 4px;
border-radius: var(--radius-sm);
display: flex;
align-items: center;
justify-content: center;
}
.close-btn:hover {
background: var(--bg-hover);
color: var(--text-primary);
}
.dependency-graph-container {
flex: 1;
overflow: hidden;
position: relative;
background: var(--bg-primary);
}
/* React Flow Customization */
.react-flow__background {
background-color: var(--bg-primary) !important;
}
.react-flow__controls {
background: var(--bg-tertiary);
border: 1px solid var(--border-primary);
border-radius: var(--radius-md);
box-shadow: 0 4px 12px rgba(0, 0, 0, 0.3);
}
.react-flow__controls-button {
background: var(--bg-tertiary);
border: none;
border-bottom: 1px solid var(--border-primary);
color: var(--text-secondary);
width: 28px;
height: 28px;
}
.react-flow__controls-button:hover {
background: var(--bg-hover);
color: var(--text-primary);
}
.react-flow__controls-button:last-child {
border-bottom: none;
}
.react-flow__controls-button svg {
fill: currentColor;
}
.react-flow__attribution {
background: transparent !important;
}
.react-flow__attribution a {
color: var(--text-muted) !important;
font-size: 10px;
}
/* Custom Flow Nodes */
.flow-node {
background: var(--bg-tertiary);
border: 2px solid var(--border-primary);
border-radius: var(--radius-md);
padding: 12px 16px;
min-width: 160px;
cursor: pointer;
transition: all var(--transition-fast);
text-align: center;
}
.flow-node:hover {
border-color: var(--accent-primary);
box-shadow: 0 4px 12px rgba(16, 185, 129, 0.2);
}
.flow-node--root {
background: linear-gradient(135deg, rgba(16, 185, 129, 0.15) 0%, rgba(5, 150, 105, 0.15) 100%);
border-color: var(--accent-primary);
}
.flow-node__name {
font-weight: 600;
color: var(--accent-primary);
font-family: 'JetBrains Mono', monospace;
font-size: 0.8125rem;
margin-bottom: 4px;
word-break: break-word;
}
.flow-node__details {
display: flex;
align-items: center;
justify-content: center;
gap: 8px;
font-size: 0.6875rem;
color: var(--text-muted);
}
.flow-node__version {
font-family: 'JetBrains Mono', monospace;
color: var(--text-secondary);
}
.flow-node__size {
color: var(--text-muted);
}
/* Flow Handles (connection points) */
.flow-handle {
width: 8px !important;
height: 8px !important;
background: var(--border-primary) !important;
border: 2px solid var(--bg-tertiary) !important;
}
.flow-node:hover .flow-handle {
background: var(--accent-primary) !important;
}
/* Loading, Error, Empty States */
.graph-loading,
.graph-error,
.graph-empty {
display: flex;
flex-direction: column;
align-items: center;
justify-content: center;
height: 100%;
gap: 16px;
color: var(--text-muted);
}
.graph-loading .spinner {
width: 32px;
height: 32px;
border: 3px solid var(--border-primary);
border-top-color: var(--accent-primary);
border-radius: 50%;
animation: spin 1s linear infinite;
}
@keyframes spin {
to { transform: rotate(360deg); }
}
.graph-error {
color: var(--error-color, #ef4444);
}
.graph-error svg {
opacity: 0.6;
}
.graph-error p {
max-width: 400px;
text-align: center;
line-height: 1.5;
}
.graph-warning {
display: flex;
align-items: center;
gap: 8px;
padding: 8px 16px;
background: rgba(245, 158, 11, 0.1);
border-top: 1px solid rgba(245, 158, 11, 0.3);
color: var(--warning-color, #f59e0b);
font-size: 0.875rem;
}
.graph-warning svg {
flex-shrink: 0;
}
/* Missing Dependencies */
.missing-dependencies {
border-top: 1px solid var(--border-primary);
padding: 16px 20px;
background: rgba(245, 158, 11, 0.05);
max-height: 200px;
overflow-y: auto;
}
.missing-dependencies h3 {
margin: 0 0 8px 0;
font-size: 0.875rem;
font-weight: 600;
color: #f59e0b;
}
.missing-hint {
margin: 0 0 12px 0;
font-size: 0.75rem;
color: var(--text-muted);
}
.missing-list {
list-style: none;
padding: 0;
margin: 0;
display: flex;
flex-wrap: wrap;
gap: 8px;
}
.missing-item {
display: inline-flex;
align-items: center;
gap: 4px;
background: var(--bg-tertiary);
border: 1px solid rgba(245, 158, 11, 0.3);
border-radius: var(--radius-sm);
padding: 4px 8px;
font-size: 0.75rem;
}
.missing-name {
font-family: 'JetBrains Mono', monospace;
color: var(--text-secondary);
}
.missing-constraint {
color: var(--text-muted);
font-family: 'JetBrains Mono', monospace;
}
.missing-required-by {
color: var(--text-muted);
font-size: 0.6875rem;
}
/* Responsive */
@media (max-width: 768px) {
.dependency-graph-modal {
padding: 0;
}
.dependency-graph-content {
height: 100vh;
border-radius: 0;
max-width: none;
}
.dependency-graph-header {
flex-wrap: wrap;
}
.dependency-graph-info {
flex-basis: 100%;
order: 3;
margin-top: 8px;
}
}

View File

@@ -0,0 +1,367 @@
import { useState, useEffect, useCallback, useMemo } from 'react';
import { useNavigate } from 'react-router-dom';
import ReactFlow, {
Node,
Edge,
Controls,
Background,
useNodesState,
useEdgesState,
MarkerType,
NodeProps,
Handle,
Position,
} from 'reactflow';
import dagre from 'dagre';
import 'reactflow/dist/style.css';
import { ResolvedArtifact, DependencyResolutionResponse, Dependency } from '../types';
import { resolveDependencies, getArtifactDependencies } from '../api';
import './DependencyGraph.css';
interface DependencyGraphProps {
projectName: string;
packageName: string;
tagName: string;
onClose: () => void;
}
interface NodeData {
label: string;
project: string;
package: string;
version: string | null;
size: number;
isRoot: boolean;
onNavigate: (project: string, pkg: string) => void;
}
function formatBytes(bytes: number): string {
if (bytes === 0) return '0 B';
const k = 1024;
const sizes = ['B', 'KB', 'MB', 'GB'];
const i = Math.floor(Math.log(bytes) / Math.log(k));
return parseFloat((bytes / Math.pow(k, i)).toFixed(1)) + ' ' + sizes[i];
}
// Custom node component
function DependencyNode({ data }: NodeProps<NodeData>) {
return (
<div
className={`flow-node ${data.isRoot ? 'flow-node--root' : ''}`}
onClick={() => data.onNavigate(data.project, data.package)}
>
<Handle type="target" position={Position.Top} className="flow-handle" />
<div className="flow-node__name">{data.package}</div>
<div className="flow-node__details">
{data.version && <span className="flow-node__version">{data.version}</span>}
<span className="flow-node__size">{formatBytes(data.size)}</span>
</div>
<Handle type="source" position={Position.Bottom} className="flow-handle" />
</div>
);
}
const nodeTypes = { dependency: DependencyNode };
// Dagre layout function
function getLayoutedElements(
nodes: Node<NodeData>[],
edges: Edge[],
direction: 'TB' | 'LR' = 'TB'
) {
const dagreGraph = new dagre.graphlib.Graph();
dagreGraph.setDefaultEdgeLabel(() => ({}));
const nodeWidth = 180;
const nodeHeight = 60;
dagreGraph.setGraph({ rankdir: direction, nodesep: 50, ranksep: 80 });
nodes.forEach((node) => {
dagreGraph.setNode(node.id, { width: nodeWidth, height: nodeHeight });
});
edges.forEach((edge) => {
dagreGraph.setEdge(edge.source, edge.target);
});
dagre.layout(dagreGraph);
const layoutedNodes = nodes.map((node) => {
const nodeWithPosition = dagreGraph.node(node.id);
return {
...node,
position: {
x: nodeWithPosition.x - nodeWidth / 2,
y: nodeWithPosition.y - nodeHeight / 2,
},
};
});
return { nodes: layoutedNodes, edges };
}
function DependencyGraph({ projectName, packageName, tagName, onClose }: DependencyGraphProps) {
const navigate = useNavigate();
const [loading, setLoading] = useState(true);
const [error, setError] = useState<string | null>(null);
const [warning, setWarning] = useState<string | null>(null);
const [resolution, setResolution] = useState<DependencyResolutionResponse | null>(null);
const [nodes, setNodes, onNodesChange] = useNodesState<NodeData>([]);
const [edges, setEdges, onEdgesChange] = useEdgesState([]);
const handleNavigate = useCallback((project: string, pkg: string) => {
navigate(`/project/${project}/${pkg}`);
onClose();
}, [navigate, onClose]);
// Build graph structure from resolution data
const buildFlowGraph = useCallback(async (
resolutionData: DependencyResolutionResponse,
onNavigate: (project: string, pkg: string) => void
) => {
const artifactMap = new Map<string, ResolvedArtifact>();
resolutionData.resolved.forEach(artifact => {
artifactMap.set(artifact.artifact_id, artifact);
});
// Fetch dependencies for each artifact
const depsMap = new Map<string, Dependency[]>();
const failedFetches: string[] = [];
for (const artifact of resolutionData.resolved) {
try {
const deps = await getArtifactDependencies(artifact.artifact_id);
depsMap.set(artifact.artifact_id, deps.dependencies);
} catch (err) {
console.warn(`Failed to fetch dependencies for ${artifact.package}:`, err);
failedFetches.push(artifact.package);
depsMap.set(artifact.artifact_id, []);
}
}
// Report warning if some fetches failed
if (failedFetches.length > 0) {
setWarning(`Could not load dependency details for: ${failedFetches.slice(0, 3).join(', ')}${failedFetches.length > 3 ? ` and ${failedFetches.length - 3} more` : ''}`);
}
// Find the root artifact
const rootArtifact = resolutionData.resolved.find(
a => a.project === resolutionData.requested.project &&
a.package === resolutionData.requested.package
);
if (!rootArtifact) {
return { nodes: [], edges: [] };
}
const flowNodes: Node<NodeData>[] = [];
const flowEdges: Edge[] = [];
const visited = new Set<string>();
const nodeIdMap = new Map<string, string>(); // artifact_id -> node id
// Build nodes and edges recursively
const processNode = (artifact: ResolvedArtifact, isRoot: boolean) => {
if (visited.has(artifact.artifact_id)) {
return nodeIdMap.get(artifact.artifact_id);
}
visited.add(artifact.artifact_id);
const nodeId = `node-${flowNodes.length}`;
nodeIdMap.set(artifact.artifact_id, nodeId);
flowNodes.push({
id: nodeId,
type: 'dependency',
position: { x: 0, y: 0 }, // Will be set by dagre
data: {
label: `${artifact.project}/${artifact.package}`,
project: artifact.project,
package: artifact.package,
version: artifact.version,
size: artifact.size,
isRoot,
onNavigate,
},
});
const deps = depsMap.get(artifact.artifact_id) || [];
for (const dep of deps) {
const childArtifact = resolutionData.resolved.find(
a => a.project === dep.project && a.package === dep.package
);
if (childArtifact) {
const childNodeId = processNode(childArtifact, false);
if (childNodeId) {
flowEdges.push({
id: `edge-${nodeId}-${childNodeId}`,
source: nodeId,
target: childNodeId,
markerEnd: {
type: MarkerType.ArrowClosed,
width: 15,
height: 15,
color: 'var(--accent-primary)',
},
style: {
stroke: 'var(--border-primary)',
strokeWidth: 2,
},
});
}
}
}
return nodeId;
};
processNode(rootArtifact, true);
// Apply dagre layout
return getLayoutedElements(flowNodes, flowEdges);
}, []);
useEffect(() => {
async function loadData() {
setLoading(true);
setError(null);
try {
const result = await resolveDependencies(projectName, packageName, tagName);
// If only the root package (no dependencies) and no missing deps, close the modal
const hasDeps = result.artifact_count > 1 || (result.missing && result.missing.length > 0);
if (!hasDeps) {
onClose();
return;
}
setResolution(result);
const { nodes: layoutedNodes, edges: layoutedEdges } = await buildFlowGraph(result, handleNavigate);
setNodes(layoutedNodes);
setEdges(layoutedEdges);
} catch (err) {
if (err instanceof Error) {
try {
const errorData = JSON.parse(err.message);
if (errorData.error === 'circular_dependency') {
setError(`Circular dependency detected: ${errorData.cycle?.join(' → ')}`);
} else if (errorData.error === 'dependency_conflict') {
setError(`Dependency conflict: ${errorData.message}`);
} else {
setError(err.message);
}
} catch {
setError(err.message);
}
} else {
setError('Failed to load dependency graph');
}
} finally {
setLoading(false);
}
}
loadData();
}, [projectName, packageName, tagName, buildFlowGraph, handleNavigate, onClose, setNodes, setEdges]);
const defaultViewport = useMemo(() => ({ x: 50, y: 50, zoom: 0.8 }), []);
return (
<div className="dependency-graph-modal" onClick={onClose}>
<div className="dependency-graph-content" onClick={e => e.stopPropagation()}>
<div className="dependency-graph-header">
<h2>Dependency Graph</h2>
<div className="dependency-graph-info">
<span>{projectName}/{packageName} @ {tagName}</span>
{resolution && (
<span className="graph-stats">
{resolution.artifact_count} cached
{resolution.missing && resolution.missing.length > 0 && (
<span className="missing-count"> {resolution.missing.length} not cached</span>
)}
{formatBytes(resolution.total_size)} total
</span>
)}
</div>
<button className="close-btn" onClick={onClose} title="Close">
<svg width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" strokeWidth="2">
<line x1="18" y1="6" x2="6" y2="18"></line>
<line x1="6" y1="6" x2="18" y2="18"></line>
</svg>
</button>
</div>
<div className="dependency-graph-container">
{loading ? (
<div className="graph-loading">
<div className="spinner"></div>
<span>Resolving dependencies...</span>
</div>
) : error ? (
<div className="graph-error">
<svg width="48" height="48" viewBox="0 0 24 24" fill="none" stroke="currentColor" strokeWidth="2">
<circle cx="12" cy="12" r="10"></circle>
<line x1="12" y1="8" x2="12" y2="12"></line>
<line x1="12" y1="16" x2="12.01" y2="16"></line>
</svg>
<p>{error}</p>
</div>
) : nodes.length > 0 ? (
<ReactFlow
nodes={nodes}
edges={edges}
onNodesChange={onNodesChange}
onEdgesChange={onEdgesChange}
nodeTypes={nodeTypes}
defaultViewport={defaultViewport}
fitView
fitViewOptions={{ padding: 0.2 }}
minZoom={0.1}
maxZoom={2}
attributionPosition="bottom-left"
>
<Controls />
<Background color="var(--border-primary)" gap={20} />
</ReactFlow>
) : (
<div className="graph-empty">No dependencies to display</div>
)}
</div>
{warning && (
<div className="graph-warning">
<svg width="16" height="16" viewBox="0 0 24 24" fill="none" stroke="currentColor" strokeWidth="2">
<path d="M10.29 3.86L1.82 18a2 2 0 0 0 1.71 3h16.94a2 2 0 0 0 1.71-3L13.71 3.86a2 2 0 0 0-3.42 0z"></path>
<line x1="12" y1="9" x2="12" y2="13"></line>
<line x1="12" y1="17" x2="12.01" y2="17"></line>
</svg>
<span>{warning}</span>
</div>
)}
{resolution && resolution.missing && resolution.missing.length > 0 && (
<div className="missing-dependencies">
<h3>Not Cached ({resolution.missing.length})</h3>
<p className="missing-hint">These dependencies are referenced but not yet cached on the server.</p>
<ul className="missing-list">
{resolution.missing.map((dep, i) => (
<li key={i} className="missing-item">
<span className="missing-name">{dep.project}/{dep.package}</span>
{dep.constraint && <span className="missing-constraint">@{dep.constraint}</span>}
{dep.required_by && <span className="missing-required-by"> {dep.required_by}</span>}
</li>
))}
</ul>
</div>
)}
</div>
</div>
);
}
export default DependencyGraph;

View File

@@ -290,20 +290,25 @@
color: var(--error-color, #dc3545);
}
/* Progress Bar */
.progress-bar {
/* Progress Bar - scoped to upload component */
.drag-drop-upload .progress-bar,
.upload-queue .progress-bar {
height: 8px;
background: var(--border-color, #ddd);
border-radius: 4px;
overflow: hidden;
width: 100%;
max-width: 100%;
}
.progress-bar--small {
.drag-drop-upload .progress-bar--small,
.upload-queue .progress-bar--small {
height: 4px;
margin-top: 0.25rem;
}
.progress-bar__fill {
.drag-drop-upload .progress-bar__fill,
.upload-queue .progress-bar__fill {
height: 100%;
background: var(--accent-color, #007bff);
border-radius: 4px;

View File

@@ -504,42 +504,4 @@ describe('DragDropUpload', () => {
});
});
});
describe('Tag Support', () => {
it('includes tag in upload request', async () => {
let capturedFormData: FormData | null = null;
class MockXHR {
status = 200;
responseText = JSON.stringify({ artifact_id: 'abc123', size: 100 });
timeout = 0;
upload = { addEventListener: vi.fn() };
addEventListener = vi.fn((event: string, handler: () => void) => {
if (event === 'load') setTimeout(handler, 10);
});
open = vi.fn();
send = vi.fn((data: FormData) => {
capturedFormData = data;
});
}
vi.stubGlobal('XMLHttpRequest', MockXHR);
render(<DragDropUpload {...defaultProps} tag="v1.0.0" />);
const input = document.querySelector('input[type="file"]') as HTMLInputElement;
const file = createMockFile('test.txt', 100, 'text/plain');
Object.defineProperty(input, 'files', {
value: Object.assign([file], { item: (i: number) => [file][i] }),
});
fireEvent.change(input);
await vi.advanceTimersByTimeAsync(100);
await waitFor(() => {
expect(capturedFormData?.get('tag')).toBe('v1.0.0');
});
});
});
});

View File

@@ -13,7 +13,6 @@ interface StoredUploadState {
completedParts: number[];
project: string;
package: string;
tag?: string;
createdAt: number;
}
@@ -87,7 +86,6 @@ export interface DragDropUploadProps {
maxFileSize?: number; // in bytes
maxConcurrentUploads?: number;
maxRetries?: number;
tag?: string;
className?: string;
disabled?: boolean;
disabledReason?: string;
@@ -230,7 +228,6 @@ export function DragDropUpload({
maxFileSize,
maxConcurrentUploads = 3,
maxRetries = 3,
tag,
className = '',
disabled = false,
disabledReason,
@@ -368,7 +365,6 @@ export function DragDropUpload({
expected_hash: fileHash,
filename: item.file.name,
size: item.file.size,
tag: tag || undefined,
}),
}
);
@@ -392,7 +388,6 @@ export function DragDropUpload({
completedParts: [],
project: projectName,
package: packageName,
tag: tag || undefined,
createdAt: Date.now(),
});
@@ -438,7 +433,6 @@ export function DragDropUpload({
completedParts,
project: projectName,
package: packageName,
tag: tag || undefined,
createdAt: Date.now(),
});
@@ -459,7 +453,7 @@ export function DragDropUpload({
{
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ tag: tag || undefined }),
body: JSON.stringify({}),
}
);
@@ -475,18 +469,15 @@ export function DragDropUpload({
size: completeData.size,
deduplicated: false,
};
}, [projectName, packageName, tag, isOnline]);
}, [projectName, packageName, isOnline]);
const uploadFileSimple = useCallback((item: UploadItem): Promise<UploadResult> => {
return new Promise((resolve, reject) => {
const xhr = new XMLHttpRequest();
xhrMapRef.current.set(item.id, xhr);
const formData = new FormData();
formData.append('file', item.file);
if (tag) {
formData.append('tag', tag);
}
let lastLoaded = 0;
let lastTime = Date.now();
@@ -549,13 +540,13 @@ export function DragDropUpload({
xhr.timeout = 300000;
xhr.send(formData);
setUploadQueue(prev => prev.map(u =>
u.id === item.id
setUploadQueue(prev => prev.map(u =>
u.id === item.id
? { ...u, status: 'uploading' as UploadStatus, startTime: Date.now() }
: u
));
});
}, [projectName, packageName, tag]);
}, [projectName, packageName]);
const uploadFile = useCallback((item: UploadItem): Promise<UploadResult> => {
if (item.file.size >= CHUNKED_UPLOAD_THRESHOLD) {

View File

@@ -233,7 +233,7 @@ export function GlobalSearch() {
const flatIndex = results.projects.length + results.packages.length + index;
return (
<button
key={artifact.tag_id}
key={artifact.artifact_id}
className={`global-search__result ${selectedIndex === flatIndex ? 'selected' : ''}`}
onClick={() => navigateToResult({ type: 'artifact', item: artifact })}
onMouseEnter={() => setSelectedIndex(flatIndex)}
@@ -243,7 +243,7 @@ export function GlobalSearch() {
<line x1="7" y1="7" x2="7.01" y2="7" />
</svg>
<div className="global-search__result-content">
<span className="global-search__result-name">{artifact.tag_name}</span>
<span className="global-search__result-name">{artifact.version}</span>
<span className="global-search__result-path">
{artifact.project_name} / {artifact.package_name}
</span>

View File

@@ -272,7 +272,7 @@
.footer {
background: var(--bg-secondary);
border-top: 1px solid var(--border-primary);
padding: 24px 0;
padding: 12px 0;
}
.footer-content {
@@ -284,7 +284,11 @@
.footer-brand {
display: flex;
align-items: center;
gap: 12px;
gap: 8px;
}
.footer-icon {
color: var(--accent-primary);
}
.footer-logo {
@@ -292,6 +296,10 @@
color: var(--text-primary);
}
.footer-separator {
color: var(--text-muted);
}
.footer-tagline {
color: var(--text-secondary);
font-size: 0.875rem;

View File

@@ -2,6 +2,8 @@ import { ReactNode, useState, useRef, useEffect } from 'react';
import { Link, NavLink, useLocation, useNavigate } from 'react-router-dom';
import { useAuth } from '../contexts/AuthContext';
import { GlobalSearch } from './GlobalSearch';
import { listTeams } from '../api';
import { TeamDetail } from '../types';
import './Layout.css';
interface LayoutProps {
@@ -13,8 +15,22 @@ function Layout({ children }: LayoutProps) {
const navigate = useNavigate();
const { user, loading, logout } = useAuth();
const [showUserMenu, setShowUserMenu] = useState(false);
const [userTeams, setUserTeams] = useState<TeamDetail[]>([]);
const menuRef = useRef<HTMLDivElement>(null);
// Fetch user's teams
useEffect(() => {
if (user) {
listTeams({ limit: 10 }).then(data => {
setUserTeams(data.items);
}).catch(() => {
setUserTeams([]);
});
} else {
setUserTeams([]);
}
}, [user]);
// Close menu when clicking outside
useEffect(() => {
function handleClickOutside(event: MouseEvent) {
@@ -68,15 +84,6 @@ function Layout({ children }: LayoutProps) {
</svg>
Projects
</Link>
<Link to="/dashboard" className={location.pathname === '/dashboard' ? 'active' : ''}>
<svg width="16" height="16" viewBox="0 0 24 24" fill="none" stroke="currentColor" strokeWidth="2">
<rect x="3" y="3" width="7" height="7" rx="1"/>
<rect x="14" y="3" width="7" height="7" rx="1"/>
<rect x="3" y="14" width="7" height="7" rx="1"/>
<rect x="14" y="14" width="7" height="7" rx="1"/>
</svg>
Dashboard
</Link>
<a href="/docs" className="nav-link-muted">
<svg width="16" height="16" viewBox="0 0 24 24" fill="none" stroke="currentColor" strokeWidth="2">
<path d="M14 2H6a2 2 0 0 0-2 2v16a2 2 0 0 0 2 2h12a2 2 0 0 0 2-2V8z"/>
@@ -118,6 +125,35 @@ function Layout({ children }: LayoutProps) {
)}
</div>
<div className="user-menu-divider"></div>
<NavLink
to="/dashboard"
className="user-menu-item"
onClick={() => setShowUserMenu(false)}
>
<svg width="16" height="16" viewBox="0 0 24 24" fill="none" stroke="currentColor" strokeWidth="2">
<rect x="3" y="3" width="7" height="7" rx="1"/>
<rect x="14" y="3" width="7" height="7" rx="1"/>
<rect x="3" y="14" width="7" height="7" rx="1"/>
<rect x="14" y="14" width="7" height="7" rx="1"/>
</svg>
Dashboard
</NavLink>
{userTeams.length > 0 && (
<NavLink
to={userTeams.length === 1 ? `/teams/${userTeams[0].slug}` : '/teams'}
className="user-menu-item"
onClick={() => setShowUserMenu(false)}
>
<svg width="16" height="16" viewBox="0 0 24 24" fill="none" stroke="currentColor" strokeWidth="2">
<path d="M17 21v-2a4 4 0 0 0-4-4H5a4 4 0 0 0-4 4v2"/>
<circle cx="9" cy="7" r="4"/>
<path d="M23 21v-2a4 4 0 0 0-3-3.87"/>
<path d="M16 3.13a4 4 0 0 1 0 7.75"/>
</svg>
{userTeams.length === 1 ? 'Team' : 'Teams'}
</NavLink>
)}
<div className="user-menu-divider"></div>
<NavLink
to="/settings/api-keys"
className="user-menu-item"
@@ -153,6 +189,18 @@ function Layout({ children }: LayoutProps) {
</svg>
SSO Configuration
</NavLink>
<NavLink
to="/admin/cache"
className="user-menu-item"
onClick={() => setShowUserMenu(false)}
>
<svg width="16" height="16" viewBox="0 0 24 24" fill="none" stroke="currentColor" strokeWidth="2">
<path d="M21 16V8a2 2 0 0 0-1-1.73l-7-4a2 2 0 0 0-2 0l-7 4A2 2 0 0 0 3 8v8a2 2 0 0 0 1 1.73l7 4a2 2 0 0 0 2 0l7-4A2 2 0 0 0 21 16z"/>
<polyline points="3.27 6.96 12 12.01 20.73 6.96"/>
<line x1="12" y1="22.08" x2="12" y2="12"/>
</svg>
Cache Management
</NavLink>
</>
)}
<div className="user-menu-divider"></div>
@@ -188,12 +236,21 @@ function Layout({ children }: LayoutProps) {
<footer className="footer">
<div className="container footer-content">
<div className="footer-brand">
<svg className="footer-icon" width="18" height="18" viewBox="0 0 24 24" fill="none" xmlns="http://www.w3.org/2000/svg">
<path d="M6 14 Q6 8 3 8 Q6 4 6 4 Q6 4 9 8 Q6 8 6 14" fill="currentColor" opacity="0.6"/>
<rect x="5.25" y="13" width="1.5" height="4" fill="currentColor" opacity="0.6"/>
<path d="M12 12 Q12 5 8 5 Q12 1 12 1 Q12 1 16 5 Q12 5 12 12" fill="currentColor"/>
<rect x="11.25" y="11" width="1.5" height="5" fill="currentColor"/>
<path d="M18 14 Q18 8 15 8 Q18 4 18 4 Q18 4 21 8 Q18 8 18 14" fill="currentColor" opacity="0.6"/>
<rect x="17.25" y="13" width="1.5" height="4" fill="currentColor" opacity="0.6"/>
<ellipse cx="12" cy="19" rx="9" ry="1.5" fill="currentColor" opacity="0.3"/>
</svg>
<span className="footer-logo">Orchard</span>
<span className="footer-tagline">Content-Addressable Storage</span>
<span className="footer-separator">·</span>
<span className="footer-tagline">The cache that never forgets</span>
</div>
<div className="footer-links">
<a href="/docs">Documentation</a>
<a href="/api/v1">API</a>
</div>
</div>
</footer>

View File

@@ -0,0 +1,163 @@
.team-selector {
position: relative;
}
.team-selector-trigger {
display: flex;
align-items: center;
gap: 0.5rem;
padding: 0.375rem 0.75rem;
background: var(--bg-secondary);
border: 1px solid var(--border-primary);
border-radius: var(--radius-md);
color: var(--text-primary);
font-size: 0.875rem;
cursor: pointer;
transition: all 0.15s ease;
min-width: 160px;
}
.team-selector-trigger:hover:not(:disabled) {
background: var(--bg-tertiary);
border-color: var(--border-secondary);
}
.team-selector-trigger:disabled {
opacity: 0.6;
cursor: not-allowed;
}
.team-selector-name {
flex: 1;
text-align: left;
overflow: hidden;
text-overflow: ellipsis;
white-space: nowrap;
}
.team-selector-chevron {
transition: transform 0.15s ease;
flex-shrink: 0;
}
.team-selector-chevron.open {
transform: rotate(180deg);
}
.team-selector-dropdown {
position: absolute;
top: 100%;
left: 0;
right: 0;
min-width: 240px;
margin-top: 0.25rem;
background: var(--bg-secondary);
border: 1px solid var(--border-primary);
border-radius: var(--radius-md);
box-shadow: var(--shadow-lg);
z-index: 100;
overflow: hidden;
}
.team-selector-empty {
padding: 1rem;
text-align: center;
color: var(--text-muted);
}
.team-selector-empty p {
margin: 0 0 0.75rem;
font-size: 0.875rem;
}
.team-selector-create-link {
color: var(--accent-primary);
font-size: 0.875rem;
text-decoration: none;
}
.team-selector-create-link:hover {
text-decoration: underline;
}
.team-selector-list {
list-style: none;
margin: 0;
padding: 0.25rem 0;
max-height: 280px;
overflow-y: auto;
}
.team-selector-item {
display: flex;
align-items: center;
gap: 0.5rem;
width: 100%;
padding: 0.5rem 0.75rem;
background: none;
border: none;
color: var(--text-primary);
font-size: 0.875rem;
cursor: pointer;
text-align: left;
transition: background 0.1s ease;
}
.team-selector-item:hover {
background: var(--bg-hover);
}
.team-selector-item.selected {
background: rgba(16, 185, 129, 0.1);
}
.team-selector-item-info {
flex: 1;
min-width: 0;
}
.team-selector-item-name {
display: block;
font-weight: 500;
overflow: hidden;
text-overflow: ellipsis;
white-space: nowrap;
}
.team-selector-item-meta {
display: block;
font-size: 0.75rem;
color: var(--text-muted);
}
.team-selector-item-role {
font-size: 0.75rem;
text-transform: capitalize;
flex-shrink: 0;
}
.team-selector-footer {
display: flex;
justify-content: space-between;
padding: 0.5rem 0.75rem;
border-top: 1px solid var(--border-primary);
background: var(--bg-tertiary);
}
.team-selector-link {
font-size: 0.8125rem;
color: var(--text-muted);
text-decoration: none;
}
.team-selector-link:hover {
color: var(--text-primary);
}
.team-selector-link-primary {
color: var(--accent-primary);
}
.team-selector-link-primary:hover {
color: var(--accent-primary-hover);
}

View File

@@ -0,0 +1,141 @@
import { useState, useRef, useEffect } from 'react';
import { Link } from 'react-router-dom';
import { useTeam } from '../contexts/TeamContext';
import { useAuth } from '../contexts/AuthContext';
import { TeamDetail } from '../types';
import './TeamSelector.css';
export function TeamSelector() {
const { user } = useAuth();
const { teams, currentTeam, loading, setCurrentTeam } = useTeam();
const [isOpen, setIsOpen] = useState(false);
const dropdownRef = useRef<HTMLDivElement>(null);
// Close dropdown when clicking outside
useEffect(() => {
function handleClickOutside(event: MouseEvent) {
if (dropdownRef.current && !dropdownRef.current.contains(event.target as Node)) {
setIsOpen(false);
}
}
document.addEventListener('mousedown', handleClickOutside);
return () => document.removeEventListener('mousedown', handleClickOutside);
}, []);
// Don't show if not authenticated
if (!user) {
return null;
}
const handleTeamSelect = (team: TeamDetail) => {
setCurrentTeam(team);
setIsOpen(false);
};
const roleColors: Record<string, string> = {
owner: 'var(--color-success)',
admin: 'var(--color-primary)',
member: 'var(--color-text-muted)',
};
return (
<div className="team-selector" ref={dropdownRef}>
<button
className="team-selector-trigger"
onClick={() => setIsOpen(!isOpen)}
disabled={loading}
aria-expanded={isOpen}
aria-haspopup="listbox"
>
<svg width="16" height="16" viewBox="0 0 24 24" fill="none" stroke="currentColor" strokeWidth="2">
<path d="M17 21v-2a4 4 0 0 0-4-4H5a4 4 0 0 0-4 4v2"/>
<circle cx="9" cy="7" r="4"/>
<path d="M23 21v-2a4 4 0 0 0-3-3.87"/>
<path d="M16 3.13a4 4 0 0 1 0 7.75"/>
</svg>
<span className="team-selector-name">
{loading ? 'Loading...' : currentTeam?.name || 'Select Team'}
</span>
<svg
className={`team-selector-chevron ${isOpen ? 'open' : ''}`}
width="12"
height="12"
viewBox="0 0 24 24"
fill="none"
stroke="currentColor"
strokeWidth="2"
>
<polyline points="6 9 12 15 18 9"/>
</svg>
</button>
{isOpen && (
<div className="team-selector-dropdown" role="listbox">
{teams.length === 0 ? (
<div className="team-selector-empty">
<p>You're not a member of any teams yet.</p>
<Link
to="/teams/new"
className="team-selector-create-link"
onClick={() => setIsOpen(false)}
>
Create your first team
</Link>
</div>
) : (
<>
<ul className="team-selector-list">
{teams.map(team => (
<li key={team.id}>
<button
className={`team-selector-item ${currentTeam?.id === team.id ? 'selected' : ''}`}
onClick={() => handleTeamSelect(team)}
role="option"
aria-selected={currentTeam?.id === team.id}
>
<div className="team-selector-item-info">
<span className="team-selector-item-name">{team.name}</span>
<span className="team-selector-item-meta">
{team.project_count} project{team.project_count !== 1 ? 's' : ''}
</span>
</div>
{team.user_role && (
<span
className="team-selector-item-role"
style={{ color: roleColors[team.user_role] || roleColors.member }}
>
{team.user_role}
</span>
)}
{currentTeam?.id === team.id && (
<svg width="16" height="16" viewBox="0 0 24 24" fill="none" stroke="currentColor" strokeWidth="2">
<polyline points="20 6 9 17 4 12"/>
</svg>
)}
</button>
</li>
))}
</ul>
<div className="team-selector-footer">
<Link
to="/teams"
className="team-selector-link"
onClick={() => setIsOpen(false)}
>
View all teams
</Link>
<Link
to="/teams/new"
className="team-selector-link team-selector-link-primary"
onClick={() => setIsOpen(false)}
>
+ New Team
</Link>
</div>
</>
)}
</div>
)}
</div>
);
}

View File

@@ -0,0 +1,105 @@
.user-autocomplete {
position: relative;
width: 100%;
}
.user-autocomplete__input-wrapper {
position: relative;
}
.user-autocomplete__input {
width: 100%;
padding: 0.625rem 2.5rem 0.625rem 0.75rem;
border: 1px solid var(--border-primary);
border-radius: var(--radius-md);
background: var(--bg-tertiary);
color: var(--text-primary);
font-size: 0.875rem;
}
.user-autocomplete__input:focus {
outline: none;
border-color: var(--accent-primary);
box-shadow: 0 0 0 3px rgba(16, 185, 129, 0.2);
}
.user-autocomplete__spinner {
position: absolute;
right: 0.75rem;
top: 50%;
transform: translateY(-50%);
width: 16px;
height: 16px;
border: 2px solid var(--border-primary);
border-top-color: var(--accent-primary);
border-radius: 50%;
animation: spin 0.6s linear infinite;
}
@keyframes spin {
to { transform: translateY(-50%) rotate(360deg); }
}
.user-autocomplete__dropdown {
position: absolute;
top: 100%;
left: 0;
right: 0;
margin-top: 4px;
padding: 0.25rem;
background: var(--bg-secondary);
border: 1px solid var(--border-primary);
border-radius: var(--radius-md);
box-shadow: var(--shadow-lg);
z-index: 100;
max-height: 240px;
overflow-y: auto;
list-style: none;
}
.user-autocomplete__option {
display: flex;
align-items: center;
gap: 0.75rem;
padding: 0.5rem 0.75rem;
border-radius: var(--radius-sm);
cursor: pointer;
transition: background 0.1s;
}
.user-autocomplete__option:hover,
.user-autocomplete__option.selected {
background: var(--bg-hover);
}
.user-autocomplete__avatar {
width: 32px;
height: 32px;
border-radius: 50%;
background: var(--accent-primary);
color: white;
display: flex;
align-items: center;
justify-content: center;
font-weight: 600;
font-size: 0.875rem;
flex-shrink: 0;
}
.user-autocomplete__user-info {
display: flex;
flex-direction: column;
min-width: 0;
}
.user-autocomplete__username {
font-weight: 500;
color: var(--text-primary);
}
.user-autocomplete__admin-badge {
font-size: 0.6875rem;
color: var(--text-muted);
text-transform: uppercase;
letter-spacing: 0.025em;
}

View File

@@ -0,0 +1,171 @@
import { useState, useEffect, useRef, useCallback } from 'react';
import { searchUsers, UserSearchResult } from '../api';
import './UserAutocomplete.css';
interface UserAutocompleteProps {
value: string;
onChange: (username: string) => void;
placeholder?: string;
disabled?: boolean;
autoFocus?: boolean;
}
export function UserAutocomplete({
value,
onChange,
placeholder = 'Search users...',
disabled = false,
autoFocus = false,
}: UserAutocompleteProps) {
const [query, setQuery] = useState(value);
const [results, setResults] = useState<UserSearchResult[]>([]);
const [loading, setLoading] = useState(false);
const [isOpen, setIsOpen] = useState(false);
const [selectedIndex, setSelectedIndex] = useState(-1);
const containerRef = useRef<HTMLDivElement>(null);
const inputRef = useRef<HTMLInputElement>(null);
const debounceRef = useRef<ReturnType<typeof setTimeout>>();
// Search for users with debounce
const doSearch = useCallback(async (searchQuery: string) => {
if (searchQuery.length < 1) {
setResults([]);
setIsOpen(false);
return;
}
setLoading(true);
try {
const users = await searchUsers(searchQuery);
setResults(users);
setIsOpen(users.length > 0);
setSelectedIndex(-1);
} catch {
setResults([]);
setIsOpen(false);
} finally {
setLoading(false);
}
}, []);
// Handle input change with debounce
const handleInputChange = (e: React.ChangeEvent<HTMLInputElement>) => {
const newValue = e.target.value;
setQuery(newValue);
onChange(newValue); // Update parent immediately for form validation
// Debounce the search
if (debounceRef.current) {
clearTimeout(debounceRef.current);
}
debounceRef.current = setTimeout(() => {
doSearch(newValue);
}, 200);
};
// Handle selecting a user
const handleSelect = (user: UserSearchResult) => {
setQuery(user.username);
onChange(user.username);
setIsOpen(false);
setResults([]);
inputRef.current?.focus();
};
// Handle keyboard navigation
const handleKeyDown = (e: React.KeyboardEvent) => {
if (!isOpen) return;
switch (e.key) {
case 'ArrowDown':
e.preventDefault();
setSelectedIndex(prev => (prev < results.length - 1 ? prev + 1 : prev));
break;
case 'ArrowUp':
e.preventDefault();
setSelectedIndex(prev => (prev > 0 ? prev - 1 : -1));
break;
case 'Enter':
e.preventDefault();
if (selectedIndex >= 0 && results[selectedIndex]) {
handleSelect(results[selectedIndex]);
}
break;
case 'Escape':
setIsOpen(false);
break;
}
};
// Close dropdown when clicking outside
useEffect(() => {
const handleClickOutside = (e: MouseEvent) => {
if (containerRef.current && !containerRef.current.contains(e.target as Node)) {
setIsOpen(false);
}
};
document.addEventListener('mousedown', handleClickOutside);
return () => document.removeEventListener('mousedown', handleClickOutside);
}, []);
// Sync external value changes
useEffect(() => {
setQuery(value);
}, [value]);
// Cleanup debounce on unmount
useEffect(() => {
return () => {
if (debounceRef.current) {
clearTimeout(debounceRef.current);
}
};
}, []);
return (
<div className="user-autocomplete" ref={containerRef}>
<div className="user-autocomplete__input-wrapper">
<input
ref={inputRef}
type="text"
value={query}
onChange={handleInputChange}
onKeyDown={handleKeyDown}
onFocus={() => query.length >= 1 && results.length > 0 && setIsOpen(true)}
placeholder={placeholder}
disabled={disabled}
autoFocus={autoFocus}
autoComplete="off"
className="user-autocomplete__input"
/>
{loading && (
<div className="user-autocomplete__spinner" />
)}
</div>
{isOpen && results.length > 0 && (
<ul className="user-autocomplete__dropdown">
{results.map((user, index) => (
<li
key={user.id}
className={`user-autocomplete__option ${index === selectedIndex ? 'selected' : ''}`}
onClick={() => handleSelect(user)}
onMouseEnter={() => setSelectedIndex(index)}
>
<div className="user-autocomplete__avatar">
{user.username.charAt(0).toUpperCase()}
</div>
<div className="user-autocomplete__user-info">
<span className="user-autocomplete__username">{user.username}</span>
{user.is_admin && (
<span className="user-autocomplete__admin-badge">Admin</span>
)}
</div>
</li>
))}
</ul>
)}
</div>
);
}

View File

@@ -0,0 +1,110 @@
import { createContext, useContext, useState, useEffect, useCallback, ReactNode } from 'react';
import { TeamDetail } from '../types';
import { listTeams } from '../api';
import { useAuth } from './AuthContext';
const SELECTED_TEAM_KEY = 'orchard_selected_team';
interface TeamContextType {
teams: TeamDetail[];
currentTeam: TeamDetail | null;
loading: boolean;
error: string | null;
setCurrentTeam: (team: TeamDetail | null) => void;
refreshTeams: () => Promise<void>;
clearError: () => void;
}
const TeamContext = createContext<TeamContextType | undefined>(undefined);
interface TeamProviderProps {
children: ReactNode;
}
export function TeamProvider({ children }: TeamProviderProps) {
const { user } = useAuth();
const [teams, setTeams] = useState<TeamDetail[]>([]);
const [currentTeam, setCurrentTeamState] = useState<TeamDetail | null>(null);
const [loading, setLoading] = useState(false);
const [error, setError] = useState<string | null>(null);
const loadTeams = useCallback(async () => {
if (!user) {
setTeams([]);
setCurrentTeamState(null);
return;
}
setLoading(true);
setError(null);
try {
const response = await listTeams({ limit: 100 });
setTeams(response.items);
// Try to restore previously selected team
const savedSlug = localStorage.getItem(SELECTED_TEAM_KEY);
if (savedSlug) {
const savedTeam = response.items.find(t => t.slug === savedSlug);
if (savedTeam) {
setCurrentTeamState(savedTeam);
return;
}
}
// Auto-select first team if none selected
if (response.items.length > 0 && !currentTeam) {
setCurrentTeamState(response.items[0]);
localStorage.setItem(SELECTED_TEAM_KEY, response.items[0].slug);
}
} catch (err) {
const message = err instanceof Error ? err.message : 'Failed to load teams';
setError(message);
} finally {
setLoading(false);
}
}, [user, currentTeam]);
// Load teams when user changes
useEffect(() => {
loadTeams();
}, [user]); // eslint-disable-line react-hooks/exhaustive-deps
const setCurrentTeam = useCallback((team: TeamDetail | null) => {
setCurrentTeamState(team);
if (team) {
localStorage.setItem(SELECTED_TEAM_KEY, team.slug);
} else {
localStorage.removeItem(SELECTED_TEAM_KEY);
}
}, []);
const refreshTeams = useCallback(async () => {
await loadTeams();
}, [loadTeams]);
const clearError = useCallback(() => {
setError(null);
}, []);
return (
<TeamContext.Provider value={{
teams,
currentTeam,
loading,
error,
setCurrentTeam,
refreshTeams,
clearError,
}}>
{children}
</TeamContext.Provider>
);
}
export function useTeam() {
const context = useContext(TeamContext);
if (context === undefined) {
throw new Error('useTeam must be used within a TeamProvider');
}
return context;
}

View File

@@ -0,0 +1,377 @@
.admin-cache-page {
padding: 2rem;
max-width: 1400px;
margin: 0 auto;
}
.admin-cache-page h1 {
margin-bottom: 2rem;
color: var(--text-primary);
}
.admin-cache-page h2 {
margin-bottom: 1rem;
color: var(--text-primary);
font-size: 1.25rem;
}
/* Success/Error Messages */
.success-message {
padding: 0.75rem 1rem;
background-color: #d4edda;
border: 1px solid #c3e6cb;
border-radius: 4px;
color: #155724;
margin-bottom: 1rem;
}
.error-message {
padding: 0.75rem 1rem;
background-color: #f8d7da;
border: 1px solid #f5c6cb;
border-radius: 4px;
color: #721c24;
margin-bottom: 1rem;
}
/* Sources Section */
.sources-section {
background: var(--bg-secondary);
border: 1px solid var(--border-color);
border-radius: 8px;
padding: 1.5rem;
}
.section-header {
display: flex;
justify-content: space-between;
align-items: center;
margin-bottom: 1rem;
}
.section-header h2 {
margin: 0;
}
/* Sources Table */
.sources-table {
width: 100%;
border-collapse: collapse;
background: var(--bg-primary);
border-radius: 4px;
overflow: hidden;
}
.sources-table th,
.sources-table td {
padding: 0.75rem 1rem;
text-align: center;
border-bottom: 1px solid var(--border-color);
}
.sources-table th {
background: var(--bg-tertiary);
font-weight: 600;
color: var(--text-secondary);
font-size: 0.85rem;
text-transform: uppercase;
}
.sources-table tr:last-child td {
border-bottom: none;
}
.sources-table tr.disabled-row {
opacity: 0.6;
}
.source-name {
font-weight: 500;
color: var(--text-primary);
white-space: nowrap;
}
/* Name column should be left-aligned */
.sources-table td:first-child {
text-align: left;
}
.url-cell {
font-family: monospace;
font-size: 0.9rem;
max-width: 300px;
overflow: hidden;
text-overflow: ellipsis;
white-space: nowrap;
text-align: left;
}
/* Badges */
.env-badge,
.status-badge {
display: inline-block;
padding: 0.2rem 0.5rem;
border-radius: 4px;
font-size: 0.75rem;
font-weight: 500;
margin-left: 0.5rem;
}
.env-badge {
background-color: #fff3e0;
color: #e65100;
}
.status-badge.enabled {
background-color: #e8f5e9;
color: #2e7d32;
}
.status-badge.disabled {
background-color: #ffebee;
color: #c62828;
}
.coming-soon-badge {
color: #9e9e9e;
font-style: italic;
font-size: 0.85em;
}
/* Actions */
.actions-cell {
white-space: nowrap;
}
.actions-cell .btn {
margin-right: 0.5rem;
}
.actions-cell .btn:last-child {
margin-right: 0;
}
.test-cell {
text-align: center;
width: 2rem;
}
.test-dot {
font-size: 1rem;
cursor: default;
}
.test-dot.success {
color: #2e7d32;
}
.test-dot.failure {
color: #c62828;
cursor: pointer;
}
.test-dot.failure:hover {
color: #b71c1c;
}
.test-dot.testing {
color: #1976d2;
animation: pulse 1s infinite;
}
@keyframes pulse {
0%, 100% { opacity: 1; }
50% { opacity: 0.4; }
}
/* Error Modal */
.error-modal-content {
background: var(--bg-primary);
border-radius: 8px;
padding: 2rem;
width: 100%;
max-width: 500px;
}
.error-modal-content h3 {
margin-top: 0;
color: #c62828;
}
.error-modal-content .error-details {
background: var(--bg-tertiary);
padding: 1rem;
border-radius: 4px;
font-family: monospace;
font-size: 0.9rem;
word-break: break-word;
white-space: pre-wrap;
}
.error-modal-content .modal-actions {
display: flex;
justify-content: flex-end;
margin-top: 1.5rem;
}
/* Buttons */
.btn {
padding: 0.5rem 1rem;
border: 1px solid var(--border-color);
border-radius: 4px;
background: var(--bg-primary);
color: var(--text-primary);
cursor: pointer;
font-size: 0.875rem;
}
.btn:hover {
background: var(--bg-tertiary);
}
.btn:disabled {
opacity: 0.6;
cursor: not-allowed;
}
.btn-primary {
background-color: var(--color-primary);
border-color: var(--color-primary);
color: white;
}
.btn-primary:hover {
background-color: var(--color-primary-hover);
}
.btn-danger {
background-color: #dc3545;
border-color: #dc3545;
color: white;
}
.btn-danger:hover {
background-color: #c82333;
}
.btn-sm {
padding: 0.25rem 0.75rem;
font-size: 0.8rem;
}
.btn-secondary {
background-color: var(--bg-tertiary);
border-color: var(--border-color);
color: var(--text-primary);
font-weight: 500;
}
.btn-secondary:hover {
background-color: var(--bg-secondary);
border-color: var(--text-secondary);
}
.empty-message {
color: var(--text-secondary);
font-style: italic;
padding: 2rem;
text-align: center;
}
/* Modal */
.modal-overlay {
position: fixed;
top: 0;
left: 0;
right: 0;
bottom: 0;
background: rgba(0, 0, 0, 0.5);
display: flex;
align-items: center;
justify-content: center;
z-index: 1000;
}
.modal-content {
background: var(--bg-primary);
border-radius: 8px;
padding: 2rem;
width: 100%;
max-width: 600px;
max-height: 90vh;
overflow-y: auto;
}
.modal-content h2 {
margin-top: 0;
}
/* Form */
.form-group {
margin-bottom: 1rem;
}
.form-group label {
display: block;
margin-bottom: 0.5rem;
font-weight: 500;
color: var(--text-primary);
}
.form-group input,
.form-group select {
width: 100%;
padding: 0.5rem;
border: 1px solid var(--border-color);
border-radius: 4px;
background: var(--bg-primary);
color: var(--text-primary);
font-size: 1rem;
}
.form-group input:focus,
.form-group select:focus {
outline: none;
border-color: var(--color-primary);
}
.form-row {
display: flex;
gap: 1rem;
}
.form-row .form-group {
flex: 1;
}
.checkbox-group label {
display: flex;
align-items: center;
gap: 0.5rem;
cursor: pointer;
}
.checkbox-group input[type="checkbox"] {
width: auto;
}
.help-text {
display: block;
font-size: 0.8rem;
color: var(--text-secondary);
margin-top: 0.25rem;
}
.form-actions {
display: flex;
justify-content: space-between;
align-items: center;
margin-top: 1.5rem;
padding-top: 1rem;
border-top: 1px solid var(--border-color);
}
.form-actions-right {
display: flex;
gap: 0.5rem;
}

View File

@@ -0,0 +1,509 @@
import { useState, useEffect } from 'react';
import { useNavigate } from 'react-router-dom';
import { useAuth } from '../contexts/AuthContext';
import {
listUpstreamSources,
createUpstreamSource,
updateUpstreamSource,
deleteUpstreamSource,
testUpstreamSource,
} from '../api';
import { UpstreamSource, SourceType, AuthType } from '../types';
import './AdminCachePage.css';
const SOURCE_TYPES: SourceType[] = ['npm', 'pypi', 'maven', 'docker', 'helm', 'nuget', 'deb', 'rpm', 'generic'];
const SUPPORTED_SOURCE_TYPES: Set<SourceType> = new Set(['pypi', 'generic']);
const AUTH_TYPES: AuthType[] = ['none', 'basic', 'bearer', 'api_key'];
function AdminCachePage() {
const { user, loading: authLoading } = useAuth();
const navigate = useNavigate();
// Upstream sources state
const [sources, setSources] = useState<UpstreamSource[]>([]);
const [loadingSources, setLoadingSources] = useState(true);
const [sourcesError, setSourcesError] = useState<string | null>(null);
// Create/Edit form state
const [showForm, setShowForm] = useState(false);
const [editingSource, setEditingSource] = useState<UpstreamSource | null>(null);
const [formData, setFormData] = useState({
name: '',
source_type: 'generic' as SourceType,
url: '',
enabled: true,
auth_type: 'none' as AuthType,
username: '',
password: '',
priority: 100,
});
const [formError, setFormError] = useState<string | null>(null);
const [isSaving, setIsSaving] = useState(false);
// Test result state
const [testingId, setTestingId] = useState<string | null>(null);
const [testResults, setTestResults] = useState<Record<string, { success: boolean; message: string }>>({});
// Delete confirmation state
const [deletingId, setDeletingId] = useState<string | null>(null);
// Success message
const [successMessage, setSuccessMessage] = useState<string | null>(null);
// Error modal state
const [showErrorModal, setShowErrorModal] = useState(false);
const [selectedError, setSelectedError] = useState<{ sourceName: string; error: string } | null>(null);
useEffect(() => {
if (!authLoading && !user) {
navigate('/login', { state: { from: '/admin/cache' } });
}
}, [user, authLoading, navigate]);
useEffect(() => {
if (user && user.is_admin) {
loadSources();
}
}, [user]);
useEffect(() => {
if (successMessage) {
const timer = setTimeout(() => setSuccessMessage(null), 3000);
return () => clearTimeout(timer);
}
}, [successMessage]);
async function loadSources() {
setLoadingSources(true);
setSourcesError(null);
try {
const data = await listUpstreamSources();
setSources(data);
} catch (err) {
setSourcesError(err instanceof Error ? err.message : 'Failed to load sources');
} finally {
setLoadingSources(false);
}
}
function openCreateForm() {
setEditingSource(null);
setFormData({
name: '',
source_type: 'generic',
url: '',
enabled: true,
auth_type: 'none',
username: '',
password: '',
priority: 100,
});
setFormError(null);
setShowForm(true);
}
function openEditForm(source: UpstreamSource) {
setEditingSource(source);
setFormData({
name: source.name,
source_type: source.source_type,
url: source.url,
enabled: source.enabled,
auth_type: source.auth_type,
username: source.username || '',
password: '',
priority: source.priority,
});
setFormError(null);
setShowForm(true);
}
async function handleFormSubmit(e: React.FormEvent) {
e.preventDefault();
if (!formData.name.trim()) {
setFormError('Name is required');
return;
}
if (!formData.url.trim()) {
setFormError('URL is required');
return;
}
setIsSaving(true);
setFormError(null);
try {
let savedSourceId: string | null = null;
if (editingSource) {
// Update existing source
await updateUpstreamSource(editingSource.id, {
name: formData.name.trim(),
source_type: formData.source_type,
url: formData.url.trim(),
enabled: formData.enabled,
auth_type: formData.auth_type,
username: formData.username.trim() || undefined,
password: formData.password || undefined,
priority: formData.priority,
});
savedSourceId = editingSource.id;
setSuccessMessage('Source updated successfully');
} else {
// Create new source
const newSource = await createUpstreamSource({
name: formData.name.trim(),
source_type: formData.source_type,
url: formData.url.trim(),
enabled: formData.enabled,
auth_type: formData.auth_type,
username: formData.username.trim() || undefined,
password: formData.password || undefined,
priority: formData.priority,
});
savedSourceId = newSource.id;
setSuccessMessage('Source created successfully');
}
setShowForm(false);
await loadSources();
// Auto-test the source after save
if (savedSourceId) {
testSourceById(savedSourceId);
}
} catch (err) {
setFormError(err instanceof Error ? err.message : 'Failed to save source');
} finally {
setIsSaving(false);
}
}
async function handleDelete(source: UpstreamSource) {
if (!window.confirm(`Delete upstream source "${source.name}"? This cannot be undone.`)) {
return;
}
setDeletingId(source.id);
try {
await deleteUpstreamSource(source.id);
setSuccessMessage(`Source "${source.name}" deleted`);
await loadSources();
} catch (err) {
setSourcesError(err instanceof Error ? err.message : 'Failed to delete source');
} finally {
setDeletingId(null);
}
}
async function handleTest(source: UpstreamSource) {
testSourceById(source.id);
}
async function testSourceById(sourceId: string) {
setTestingId(sourceId);
setTestResults((prev) => ({ ...prev, [sourceId]: { success: true, message: 'Testing...' } }));
try {
const result = await testUpstreamSource(sourceId);
setTestResults((prev) => ({
...prev,
[sourceId]: {
success: result.success,
message: result.success
? `OK (${result.elapsed_ms}ms)`
: result.error || `HTTP ${result.status_code}`,
},
}));
} catch (err) {
setTestResults((prev) => ({
...prev,
[sourceId]: {
success: false,
message: err instanceof Error ? err.message : 'Test failed',
},
}));
} finally {
setTestingId(null);
}
}
function showError(sourceName: string, error: string) {
setSelectedError({ sourceName, error });
setShowErrorModal(true);
}
if (authLoading) {
return <div className="admin-cache-page">Loading...</div>;
}
if (!user?.is_admin) {
return (
<div className="admin-cache-page">
<div className="error-message">Access denied. Admin privileges required.</div>
</div>
);
}
return (
<div className="admin-cache-page">
<h1>Upstream Sources</h1>
{successMessage && <div className="success-message">{successMessage}</div>}
{/* Upstream Sources Section */}
<section className="sources-section">
<div className="section-header">
<button className="btn btn-primary" onClick={openCreateForm}>
Add Source
</button>
</div>
{loadingSources ? (
<p>Loading sources...</p>
) : sourcesError ? (
<div className="error-message">{sourcesError}</div>
) : sources.length === 0 ? (
<p className="empty-message">No upstream sources configured.</p>
) : (
<table className="sources-table">
<thead>
<tr>
<th>Name</th>
<th>Type</th>
<th>URL</th>
<th>Priority</th>
<th>Status</th>
<th>Test</th>
<th>Actions</th>
</tr>
</thead>
<tbody>
{sources.map((source) => (
<tr key={source.id} className={source.enabled ? '' : 'disabled-row'}>
<td>
<span className="source-name">{source.name}</span>
{source.source === 'env' && (
<span className="env-badge" title="Defined via environment variable">ENV</span>
)}
</td>
<td>
{source.source_type}
{!SUPPORTED_SOURCE_TYPES.has(source.source_type) && (
<span className="coming-soon-badge"> (coming soon)</span>
)}
</td>
<td className="url-cell" title={source.url}>{source.url}</td>
<td>{source.priority}</td>
<td>
<span className={`status-badge ${source.enabled ? 'enabled' : 'disabled'}`}>
{source.enabled ? 'Enabled' : 'Disabled'}
</span>
</td>
<td className="test-cell">
{testingId === source.id ? (
<span className="test-dot testing" title="Testing..."></span>
) : testResults[source.id] ? (
testResults[source.id].success ? (
<span className="test-dot success" title={testResults[source.id].message}></span>
) : (
<span
className="test-dot failure"
title="Click to see error"
onClick={() => showError(source.name, testResults[source.id].message)}
></span>
)
) : null}
</td>
<td className="actions-cell">
<button
className="btn btn-sm btn-secondary"
onClick={() => handleTest(source)}
disabled={testingId === source.id}
>
Test
</button>
{source.source !== 'env' && (
<button className="btn btn-sm btn-secondary" onClick={() => openEditForm(source)}>
Edit
</button>
)}
</td>
</tr>
))}
</tbody>
</table>
)}
</section>
{/* Create/Edit Modal */}
{showForm && (
<div className="modal-overlay" onClick={() => setShowForm(false)}>
<div className="modal-content" onClick={(e) => e.stopPropagation()}>
<h2>{editingSource ? 'Edit Upstream Source' : 'Add Upstream Source'}</h2>
<form onSubmit={handleFormSubmit}>
{formError && <div className="error-message">{formError}</div>}
<div className="form-group">
<label htmlFor="name">Name</label>
<input
type="text"
id="name"
value={formData.name}
onChange={(e) => setFormData({ ...formData, name: e.target.value })}
placeholder="e.g., npm-private"
required
/>
</div>
<div className="form-row">
<div className="form-group">
<label htmlFor="source_type">Type</label>
<select
id="source_type"
value={formData.source_type}
onChange={(e) => setFormData({ ...formData, source_type: e.target.value as SourceType })}
>
{SOURCE_TYPES.map((type) => (
<option key={type} value={type}>
{type}{!SUPPORTED_SOURCE_TYPES.has(type) ? ' (coming soon)' : ''}
</option>
))}
</select>
</div>
<div className="form-group">
<label htmlFor="priority">Priority</label>
<input
type="number"
id="priority"
value={formData.priority}
onChange={(e) => setFormData({ ...formData, priority: parseInt(e.target.value) || 100 })}
min="1"
/>
<span className="help-text">Lower = higher priority</span>
</div>
</div>
<div className="form-group">
<label htmlFor="url">URL</label>
<input
type="url"
id="url"
value={formData.url}
onChange={(e) => setFormData({ ...formData, url: e.target.value })}
placeholder="https://registry.example.com"
required
/>
</div>
<div className="form-row">
<div className="form-group checkbox-group">
<label>
<input
type="checkbox"
checked={formData.enabled}
onChange={(e) => setFormData({ ...formData, enabled: e.target.checked })}
/>
Enabled
</label>
</div>
</div>
<div className="form-group">
<label htmlFor="auth_type">Authentication</label>
<select
id="auth_type"
value={formData.auth_type}
onChange={(e) => setFormData({ ...formData, auth_type: e.target.value as AuthType })}
>
{AUTH_TYPES.map((type) => (
<option key={type} value={type}>
{type === 'none' ? 'None' : type === 'api_key' ? 'API Key' : type.charAt(0).toUpperCase() + type.slice(1)}
</option>
))}
</select>
</div>
{formData.auth_type !== 'none' && (
<div className="form-row">
{(formData.auth_type === 'basic' || formData.auth_type === 'api_key') && (
<div className="form-group">
<label htmlFor="username">{formData.auth_type === 'api_key' ? 'Header Name' : 'Username'}</label>
<input
type="text"
id="username"
value={formData.username}
onChange={(e) => setFormData({ ...formData, username: e.target.value })}
placeholder={formData.auth_type === 'api_key' ? 'X-API-Key' : 'username'}
/>
</div>
)}
<div className="form-group">
<label htmlFor="password">
{formData.auth_type === 'bearer'
? 'Token'
: formData.auth_type === 'api_key'
? 'API Key Value'
: 'Password'}
</label>
<input
type="password"
id="password"
value={formData.password}
onChange={(e) => setFormData({ ...formData, password: e.target.value })}
placeholder={editingSource ? '(unchanged)' : ''}
/>
{editingSource && (
<span className="help-text">Leave empty to keep existing {formData.auth_type === 'bearer' ? 'token' : 'credentials'}</span>
)}
</div>
</div>
)}
<div className="form-actions">
{editingSource && (
<button
type="button"
className="btn btn-danger"
onClick={() => {
handleDelete(editingSource);
setShowForm(false);
}}
disabled={deletingId === editingSource.id}
>
{deletingId === editingSource.id ? 'Deleting...' : 'Delete'}
</button>
)}
<div className="form-actions-right">
<button type="button" className="btn" onClick={() => setShowForm(false)}>
Cancel
</button>
<button type="submit" className="btn btn-primary" disabled={isSaving}>
{isSaving ? 'Saving...' : editingSource ? 'Update' : 'Create'}
</button>
</div>
</div>
</form>
</div>
</div>
)}
{/* Error Details Modal */}
{showErrorModal && selectedError && (
<div className="modal-overlay" onClick={() => setShowErrorModal(false)}>
<div className="error-modal-content" onClick={(e) => e.stopPropagation()}>
<h3>Connection Error: {selectedError.sourceName}</h3>
<div className="error-details">{selectedError.error}</div>
<div className="modal-actions">
<button className="btn" onClick={() => setShowErrorModal(false)}>
Close
</button>
</div>
</div>
</div>
)}
</div>
);
}
export default AdminCachePage;

View File

@@ -358,6 +358,12 @@
gap: 4px;
}
.page-header__actions {
display: flex;
align-items: center;
gap: 12px;
}
/* Package card styles */
.package-card__header {
display: flex;
@@ -487,3 +493,16 @@
gap: 6px;
flex-wrap: wrap;
}
/* Cell name styles */
.cell-name {
display: flex;
align-items: center;
gap: 8px;
}
/* System project badge */
.system-badge {
font-size: 0.7rem;
padding: 2px 6px;
}

View File

@@ -179,16 +179,18 @@ function Home() {
</form>
)}
<div className="list-controls">
<FilterDropdown
label="Visibility"
options={VISIBILITY_OPTIONS}
value={visibility}
onChange={handleVisibilityChange}
/>
</div>
{user && (
<div className="list-controls">
<FilterDropdown
label="Visibility"
options={VISIBILITY_OPTIONS}
value={visibility}
onChange={handleVisibilityChange}
/>
</div>
)}
{hasActiveFilters && (
{user && hasActiveFilters && (
<FilterChipGroup onClearAll={clearFilters}>
{visibility && (
<FilterChip
@@ -222,6 +224,9 @@ function Home() {
<span className="cell-name">
{!project.is_public && <LockIcon />}
{project.name}
{project.is_system && (
<Badge variant="warning" className="system-badge">Cache</Badge>
)}
</span>
),
},
@@ -244,7 +249,7 @@ function Home() {
key: 'created_by',
header: 'Owner',
className: 'cell-owner',
render: (project) => project.created_by,
render: (project) => project.team_name || project.created_by,
},
...(user
? [

View File

@@ -127,6 +127,12 @@ h2 {
font-size: 0.75rem;
}
/* Action buttons in table */
.action-buttons {
display: flex;
gap: 8px;
}
/* Download by Artifact ID Section */
.download-by-id-section {
margin-top: 32px;
@@ -179,56 +185,6 @@ h2 {
color: var(--warning-color, #f59e0b);
}
/* Usage Section */
.usage-section {
margin-top: 32px;
background: var(--bg-secondary);
}
.usage-section h3 {
margin-bottom: 12px;
color: var(--text-primary);
font-size: 1rem;
font-weight: 600;
}
.usage-section p {
color: var(--text-secondary);
margin-bottom: 12px;
font-size: 0.875rem;
}
.usage-section pre {
background: #0d0d0f;
border: 1px solid var(--border-primary);
padding: 16px 20px;
border-radius: var(--radius-md);
overflow-x: auto;
margin-bottom: 16px;
}
.usage-section code {
font-family: 'JetBrains Mono', 'Fira Code', 'Consolas', monospace;
font-size: 0.8125rem;
color: #e2e8f0;
}
/* Syntax highlighting for code blocks */
.usage-section pre {
position: relative;
}
.usage-section pre::before {
content: 'bash';
position: absolute;
top: 8px;
right: 12px;
font-size: 0.6875rem;
color: var(--text-muted);
text-transform: uppercase;
letter-spacing: 0.05em;
}
/* Copy button for code blocks (optional enhancement) */
.code-block {
position: relative;
@@ -324,6 +280,86 @@ tr:hover .copy-btn {
color: var(--text-muted);
}
/* Version badge */
.version-badge {
font-family: 'JetBrains Mono', 'Fira Code', 'Consolas', monospace;
font-size: 0.8125rem;
color: var(--accent-primary);
background: rgba(16, 185, 129, 0.1);
padding: 2px 8px;
border-radius: var(--radius-sm);
}
/* Create Tag Section */
.create-tag-section {
margin-top: 32px;
background: var(--bg-secondary);
}
.create-tag-section h3 {
margin-bottom: 4px;
color: var(--text-primary);
font-size: 1rem;
font-weight: 600;
}
.section-description {
color: var(--text-muted);
font-size: 0.875rem;
margin-bottom: 16px;
}
.create-tag-form .form-row {
display: flex;
gap: 12px;
align-items: flex-end;
flex-wrap: wrap;
}
.create-tag-form .form-group {
flex: 1;
min-width: 150px;
}
.create-tag-form .form-group--wide {
flex: 2;
min-width: 300px;
}
.create-tag-form .form-group label {
display: block;
margin-bottom: 6px;
font-size: 0.75rem;
font-weight: 500;
color: var(--text-secondary);
text-transform: uppercase;
letter-spacing: 0.05em;
}
.create-tag-form .form-group input {
width: 100%;
padding: 10px 14px;
background: var(--bg-tertiary);
border: 1px solid var(--border-primary);
border-radius: var(--radius-md);
color: var(--text-primary);
font-size: 0.875rem;
}
.create-tag-form .form-group input:focus {
outline: none;
border-color: var(--accent-primary);
}
.create-tag-form .form-group input:disabled {
opacity: 0.6;
cursor: not-allowed;
}
.create-tag-form button {
flex-shrink: 0;
}
/* Created cell */
.created-cell {
display: flex;
@@ -344,6 +380,345 @@ tr:hover .copy-btn {
white-space: nowrap;
}
/* Dependencies Section */
.dependencies-section {
margin-top: 32px;
background: var(--bg-secondary);
}
.dependencies-header {
display: flex;
align-items: center;
justify-content: space-between;
margin-bottom: 12px;
}
.dependencies-header h3 {
margin: 0;
color: var(--text-primary);
font-size: 1rem;
font-weight: 600;
}
.dependencies-controls {
display: flex;
align-items: center;
gap: 8px;
}
.dependencies-controls .btn {
display: inline-flex;
align-items: center;
}
.dependencies-tag-select {
margin-bottom: 16px;
}
.tag-selector {
padding: 8px 12px;
background: var(--bg-tertiary);
border: 1px solid var(--border-primary);
border-radius: var(--radius-md);
color: var(--text-primary);
font-size: 0.875rem;
cursor: pointer;
min-width: 200px;
}
.tag-selector:focus {
outline: none;
border-color: var(--accent-primary);
}
.deps-loading {
color: var(--text-muted);
font-size: 0.875rem;
padding: 16px 0;
}
.deps-error {
color: var(--error-color, #ef4444);
font-size: 0.875rem;
padding: 12px 16px;
background: rgba(239, 68, 68, 0.1);
border-radius: var(--radius-md);
}
.deps-empty {
color: var(--text-muted);
font-size: 0.875rem;
padding: 16px 0;
}
.deps-summary {
color: var(--text-secondary);
font-size: 0.875rem;
margin-bottom: 12px;
}
.deps-summary strong {
color: var(--accent-primary);
}
.deps-items {
list-style: none;
margin: 0;
padding: 0;
display: flex;
flex-direction: column;
gap: 8px;
}
.dep-item {
display: flex;
align-items: center;
gap: 12px;
padding: 12px 16px;
background: var(--bg-tertiary);
border-radius: var(--radius-md);
border: 1px solid var(--border-primary);
}
.dep-link {
color: var(--accent-primary);
font-weight: 500;
text-decoration: none;
font-family: 'JetBrains Mono', 'Fira Code', 'Consolas', monospace;
font-size: 0.875rem;
}
.dep-link:hover {
text-decoration: underline;
}
.dep-constraint {
color: var(--text-muted);
font-size: 0.8125rem;
font-family: 'JetBrains Mono', 'Fira Code', 'Consolas', monospace;
}
.dep-status {
margin-left: auto;
font-size: 0.875rem;
font-weight: 600;
}
.dep-status--ok {
color: var(--success-color, #10b981);
}
.dep-status--missing {
color: var(--warning-color, #f59e0b);
}
/* Tag name link in table */
.tag-name-link {
color: var(--accent-primary);
transition: opacity var(--transition-fast);
}
.tag-name-link:hover {
opacity: 0.8;
}
.tag-name-link.selected {
text-decoration: underline;
}
/* Used By (Reverse Dependencies) Section */
.used-by-section {
margin-top: 32px;
background: var(--bg-secondary);
}
.used-by-section h3 {
margin-bottom: 16px;
color: var(--text-primary);
font-size: 1rem;
font-weight: 600;
}
.reverse-dep-item {
display: flex;
align-items: center;
gap: 12px;
flex-wrap: wrap;
}
.dep-version {
color: var(--accent-primary);
font-family: 'JetBrains Mono', 'Fira Code', 'Consolas', monospace;
font-size: 0.8125rem;
background: rgba(16, 185, 129, 0.1);
padding: 2px 8px;
border-radius: var(--radius-sm);
}
.dep-requires {
color: var(--text-muted);
font-size: 0.8125rem;
font-family: 'JetBrains Mono', 'Fira Code', 'Consolas', monospace;
margin-left: auto;
}
.reverse-deps-pagination {
display: flex;
align-items: center;
justify-content: center;
gap: 16px;
margin-top: 16px;
padding-top: 16px;
border-top: 1px solid var(--border-primary);
}
.pagination-info {
color: var(--text-secondary);
font-size: 0.875rem;
}
/* Ensure File Modal */
.modal-overlay {
position: fixed;
top: 0;
left: 0;
right: 0;
bottom: 0;
background: rgba(0, 0, 0, 0.7);
display: flex;
align-items: center;
justify-content: center;
z-index: 1000;
padding: 20px;
}
/* Ensure file modal needs higher z-index when opened from deps modal */
.modal-overlay:has(.ensure-file-modal) {
z-index: 1100;
}
.ensure-file-modal {
background: var(--bg-secondary);
border: 1px solid var(--border-primary);
border-radius: var(--radius-lg);
max-width: 700px;
width: 100%;
max-height: 80vh;
display: flex;
flex-direction: column;
box-shadow: 0 20px 50px rgba(0, 0, 0, 0.5);
}
.ensure-file-header {
display: flex;
align-items: center;
justify-content: space-between;
padding: 16px 20px;
border-bottom: 1px solid var(--border-primary);
}
.ensure-file-header h3 {
margin: 0;
color: var(--text-primary);
font-size: 1rem;
font-weight: 600;
}
.ensure-file-actions {
display: flex;
align-items: center;
gap: 8px;
}
.ensure-file-actions .copy-btn {
opacity: 1;
width: 32px;
height: 32px;
}
.modal-close {
display: flex;
align-items: center;
justify-content: center;
width: 32px;
height: 32px;
padding: 0;
background: transparent;
border: none;
border-radius: var(--radius-sm);
color: var(--text-muted);
cursor: pointer;
transition: all var(--transition-fast);
}
.modal-close:hover {
background: var(--bg-hover);
color: var(--text-primary);
}
.ensure-file-content {
flex: 1;
overflow: auto;
padding: 20px;
}
.ensure-file-loading {
color: var(--text-muted);
text-align: center;
padding: 40px 20px;
}
.ensure-file-error {
color: var(--error-color, #ef4444);
padding: 16px;
background: rgba(239, 68, 68, 0.1);
border-radius: var(--radius-md);
}
.ensure-file-empty {
color: var(--text-muted);
text-align: center;
padding: 40px 20px;
font-style: italic;
}
.ensure-file-yaml {
margin: 0;
padding: 16px;
background: #0d0d0f;
border: 1px solid var(--border-primary);
border-radius: var(--radius-md);
overflow-x: auto;
}
.ensure-file-yaml code {
font-family: 'JetBrains Mono', 'Fira Code', 'Consolas', monospace;
font-size: 0.8125rem;
color: #e2e8f0;
white-space: pre;
}
.ensure-file-footer {
padding: 16px 20px;
border-top: 1px solid var(--border-primary);
background: var(--bg-tertiary);
border-radius: 0 0 var(--radius-lg) var(--radius-lg);
}
.ensure-file-hint {
margin: 0;
color: var(--text-muted);
font-size: 0.8125rem;
}
.ensure-file-hint code {
background: rgba(0, 0, 0, 0.2);
padding: 2px 6px;
border-radius: var(--radius-sm);
font-family: 'JetBrains Mono', 'Fira Code', 'Consolas', monospace;
color: var(--accent-primary);
}
/* Responsive adjustments */
@media (max-width: 768px) {
.upload-form {
@@ -359,4 +734,208 @@ tr:hover .copy-btn {
flex-wrap: wrap;
gap: 12px;
}
.dependencies-header {
flex-direction: column;
align-items: flex-start;
gap: 12px;
}
.tag-selector {
width: 100%;
}
.ensure-file-modal {
max-height: 90vh;
}
.action-menu-dropdown {
right: 0;
left: auto;
}
}
/* Header upload button */
.header-upload-btn {
margin-left: auto;
}
/* Tag/Version cell */
.tag-version-cell {
display: flex;
flex-direction: column;
gap: 4px;
}
.tag-version-cell .version-badge {
font-size: 0.75rem;
color: var(--text-muted);
}
/* Icon buttons */
.btn-icon {
display: flex;
align-items: center;
justify-content: center;
width: 32px;
height: 32px;
padding: 0;
background: transparent;
border: 1px solid transparent;
border-radius: var(--radius-sm);
color: var(--text-secondary);
cursor: pointer;
transition: all var(--transition-fast);
}
.btn-icon:hover {
background: var(--bg-hover);
color: var(--text-primary);
}
/* Action menu */
.action-buttons {
display: flex;
align-items: center;
gap: 4px;
}
.action-menu {
position: relative;
}
/* Action menu backdrop for click-outside */
.action-menu-backdrop {
position: fixed;
top: 0;
left: 0;
right: 0;
bottom: 0;
z-index: 999;
}
.action-menu-dropdown {
position: fixed;
z-index: 1000;
min-width: 180px;
padding: 4px 0;
background: var(--bg-secondary);
border: 1px solid var(--border-primary);
border-radius: var(--radius-md);
box-shadow: 0 4px 12px rgba(0, 0, 0, 0.15);
}
.action-menu-dropdown button {
display: block;
width: 100%;
padding: 8px 12px;
background: none;
border: none;
text-align: left;
font-size: 0.875rem;
color: var(--text-primary);
cursor: pointer;
transition: background var(--transition-fast);
}
.action-menu-dropdown button:hover {
background: var(--bg-hover);
}
/* Upload Modal */
.upload-modal,
.create-tag-modal {
background: var(--bg-secondary);
border-radius: var(--radius-lg);
width: 90%;
max-width: 500px;
max-height: 90vh;
overflow: hidden;
}
.modal-header {
display: flex;
align-items: center;
justify-content: space-between;
padding: 16px 20px;
border-bottom: 1px solid var(--border-primary);
}
.modal-header h3 {
margin: 0;
font-size: 1.125rem;
font-weight: 600;
}
.modal-body {
padding: 20px;
}
.modal-description {
margin-bottom: 16px;
color: var(--text-secondary);
font-size: 0.875rem;
}
.modal-actions {
display: flex;
justify-content: flex-end;
gap: 12px;
margin-top: 20px;
padding-top: 16px;
border-top: 1px solid var(--border-primary);
}
/* Dependencies Modal */
.deps-modal {
background: var(--bg-secondary);
border-radius: var(--radius-lg);
width: 90%;
max-width: 600px;
max-height: 80vh;
overflow: hidden;
display: flex;
flex-direction: column;
}
.deps-modal .modal-body {
overflow-y: auto;
flex: 1;
}
.deps-modal-controls {
display: flex;
gap: 8px;
margin-bottom: 16px;
}
/* Artifact ID Modal */
.artifact-id-modal {
background: var(--bg-secondary);
border-radius: var(--radius-lg);
width: 90%;
max-width: 500px;
}
.artifact-id-display {
display: flex;
align-items: center;
gap: 12px;
padding: 16px;
background: var(--bg-tertiary);
border-radius: var(--radius-md);
border: 1px solid var(--border-primary);
}
.artifact-id-display code {
font-family: 'JetBrains Mono', 'Fira Code', 'Consolas', monospace;
font-size: 0.8125rem;
color: var(--text-primary);
word-break: break-all;
flex: 1;
}
.artifact-id-display .copy-btn {
opacity: 1;
flex-shrink: 0;
}

View File

@@ -1,7 +1,7 @@
import { useState, useEffect, useCallback } from 'react';
import { useParams, useSearchParams, useNavigate, useLocation } from 'react-router-dom';
import { TagDetail, Package, PaginatedResponse, AccessLevel } from '../types';
import { listTags, getDownloadUrl, getPackage, getMyProjectAccess, UnauthorizedError, ForbiddenError } from '../api';
import { useParams, useSearchParams, useNavigate, useLocation, Link } from 'react-router-dom';
import { PackageArtifact, Package, PaginatedResponse, AccessLevel, Dependency, DependentInfo } from '../types';
import { listPackageArtifacts, getDownloadUrl, getPackage, getMyProjectAccess, getArtifactDependencies, getReverseDependencies, getEnsureFile, UnauthorizedError, ForbiddenError } from '../api';
import { Breadcrumb } from '../components/Breadcrumb';
import { Badge } from '../components/Badge';
import { SearchInput } from '../components/SearchInput';
@@ -10,6 +10,7 @@ import { DataTable } from '../components/DataTable';
import { Pagination } from '../components/Pagination';
import { DragDropUpload, UploadResult } from '../components/DragDropUpload';
import { useAuth } from '../contexts/AuthContext';
import DependencyGraph from '../components/DependencyGraph';
import './Home.css';
import './PackagePage.css';
@@ -56,23 +57,61 @@ function PackagePage() {
const { user } = useAuth();
const [pkg, setPkg] = useState<Package | null>(null);
const [tagsData, setTagsData] = useState<PaginatedResponse<TagDetail> | null>(null);
const [artifactsData, setArtifactsData] = useState<PaginatedResponse<PackageArtifact> | null>(null);
const [loading, setLoading] = useState(true);
const [error, setError] = useState<string | null>(null);
const [accessDenied, setAccessDenied] = useState(false);
const [uploadTag, setUploadTag] = useState('');
const [uploadSuccess, setUploadSuccess] = useState<string | null>(null);
const [artifactIdInput, setArtifactIdInput] = useState('');
const [accessLevel, setAccessLevel] = useState<AccessLevel | null>(null);
// UI state
const [showUploadModal, setShowUploadModal] = useState(false);
const [openMenuId, setOpenMenuId] = useState<string | null>(null);
const [menuPosition, setMenuPosition] = useState<{ top: number; left: number } | null>(null);
// Dependencies state
const [selectedArtifact, setSelectedArtifact] = useState<PackageArtifact | null>(null);
const [dependencies, setDependencies] = useState<Dependency[]>([]);
const [depsLoading, setDepsLoading] = useState(false);
const [depsError, setDepsError] = useState<string | null>(null);
// Reverse dependencies state
const [reverseDeps, setReverseDeps] = useState<DependentInfo[]>([]);
const [reverseDepsLoading, setReverseDepsLoading] = useState(false);
const [reverseDepsError, setReverseDepsError] = useState<string | null>(null);
const [reverseDepsPage, setReverseDepsPage] = useState(1);
const [reverseDepsTotal, setReverseDepsTotal] = useState(0);
const [reverseDepsHasMore, setReverseDepsHasMore] = useState(false);
// Dependency graph modal state
const [showGraph, setShowGraph] = useState(false);
// Dependencies modal state
const [showDepsModal, setShowDepsModal] = useState(false);
// Artifact ID modal state
const [showArtifactIdModal, setShowArtifactIdModal] = useState(false);
const [viewArtifactId, setViewArtifactId] = useState<string | null>(null);
// Ensure file modal state
const [showEnsureFile, setShowEnsureFile] = useState(false);
const [ensureFileContent, setEnsureFileContent] = useState<string | null>(null);
const [ensureFileLoading, setEnsureFileLoading] = useState(false);
const [ensureFileError, setEnsureFileError] = useState<string | null>(null);
const [ensureFileTagName, setEnsureFileTagName] = useState<string | null>(null);
// Derived permissions
const canWrite = accessLevel === 'write' || accessLevel === 'admin';
// Detect system projects (convention: name starts with "_")
const isSystemProject = projectName?.startsWith('_') ?? false;
// Get params from URL
// Valid sort fields for artifacts: created_at, size, original_name
const page = parseInt(searchParams.get('page') || '1', 10);
const search = searchParams.get('search') || '';
const sort = searchParams.get('sort') || 'name';
const order = (searchParams.get('order') || 'asc') as 'asc' | 'desc';
const sort = searchParams.get('sort') || 'created_at';
const order = (searchParams.get('order') || 'desc') as 'asc' | 'desc';
const updateParams = useCallback(
(updates: Record<string, string | undefined>) => {
@@ -95,13 +134,13 @@ function PackagePage() {
try {
setLoading(true);
setAccessDenied(false);
const [pkgData, tagsResult, accessResult] = await Promise.all([
const [pkgData, artifactsResult, accessResult] = await Promise.all([
getPackage(projectName, packageName),
listTags(projectName, packageName, { page, search, sort, order }),
listPackageArtifacts(projectName, packageName, { page, search, sort, order }),
getMyProjectAccess(projectName),
]);
setPkg(pkgData);
setTagsData(tagsResult);
setArtifactsData(artifactsResult);
setAccessLevel(accessResult.access_level);
setError(null);
} catch (err) {
@@ -125,6 +164,90 @@ function PackagePage() {
loadData();
}, [loadData]);
// Auto-select artifact when artifacts are loaded (prefer first artifact)
// Re-run when package changes to pick up new artifacts
useEffect(() => {
if (artifactsData?.items && artifactsData.items.length > 0) {
// Fall back to first artifact
setSelectedArtifact(artifactsData.items[0]);
setDependencies([]);
}
}, [artifactsData, projectName, packageName]);
// Fetch dependencies when selected tag changes
const fetchDependencies = useCallback(async (artifactId: string) => {
setDepsLoading(true);
setDepsError(null);
try {
const result = await getArtifactDependencies(artifactId);
setDependencies(result.dependencies);
} catch (err) {
setDepsError(err instanceof Error ? err.message : 'Failed to load dependencies');
setDependencies([]);
} finally {
setDepsLoading(false);
}
}, []);
useEffect(() => {
if (selectedArtifact) {
fetchDependencies(selectedArtifact.id);
}
}, [selectedArtifact, fetchDependencies]);
// Fetch reverse dependencies
const fetchReverseDeps = useCallback(async (pageNum: number = 1) => {
if (!projectName || !packageName) return;
setReverseDepsLoading(true);
setReverseDepsError(null);
try {
const result = await getReverseDependencies(projectName, packageName, { page: pageNum, limit: 10 });
setReverseDeps(result.dependents);
setReverseDepsTotal(result.pagination.total);
setReverseDepsHasMore(result.pagination.has_more);
setReverseDepsPage(pageNum);
} catch (err) {
setReverseDepsError(err instanceof Error ? err.message : 'Failed to load reverse dependencies');
setReverseDeps([]);
} finally {
setReverseDepsLoading(false);
}
}, [projectName, packageName]);
useEffect(() => {
if (projectName && packageName && !loading) {
fetchReverseDeps(1);
}
}, [projectName, packageName, loading, fetchReverseDeps]);
// Fetch ensure file for a specific version or artifact
const fetchEnsureFileForRef = useCallback(async (ref: string) => {
if (!projectName || !packageName) return;
setEnsureFileTagName(ref);
setEnsureFileLoading(true);
setEnsureFileError(null);
try {
const content = await getEnsureFile(projectName, packageName, ref);
setEnsureFileContent(content);
setShowEnsureFile(true);
} catch (err) {
setEnsureFileError(err instanceof Error ? err.message : 'Failed to load ensure file');
setShowEnsureFile(true);
} finally {
setEnsureFileLoading(false);
}
}, [projectName, packageName]);
// Fetch ensure file for selected artifact
const fetchEnsureFile = useCallback(async () => {
if (!selectedArtifact) return;
const version = getArtifactVersion(selectedArtifact);
const ref = version || `artifact:${selectedArtifact.id}`;
fetchEnsureFileForRef(ref);
}, [selectedArtifact, fetchEnsureFileForRef]);
// Keyboard navigation - go back with backspace
useEffect(() => {
const handleKeyDown = (e: KeyboardEvent) => {
@@ -139,11 +262,10 @@ function PackagePage() {
const handleUploadComplete = useCallback((results: UploadResult[]) => {
const count = results.length;
const message = count === 1
const message = count === 1
? `Uploaded successfully! Artifact ID: ${results[0].artifact_id}`
: `${count} files uploaded successfully!`;
setUploadSuccess(message);
setUploadTag('');
loadData();
// Auto-dismiss success message after 5 seconds
@@ -172,73 +294,225 @@ function PackagePage() {
};
const hasActiveFilters = search !== '';
const tags = tagsData?.items || [];
const pagination = tagsData?.pagination;
const artifacts = artifactsData?.items || [];
const pagination = artifactsData?.pagination;
const columns = [
{
key: 'name',
header: 'Tag',
sortable: true,
render: (t: TagDetail) => <strong>{t.name}</strong>,
},
{
key: 'artifact_id',
header: 'Artifact ID',
render: (t: TagDetail) => (
<div className="artifact-id-cell">
<code className="artifact-id">{t.artifact_id.substring(0, 12)}...</code>
<CopyButton text={t.artifact_id} />
</div>
),
},
{
key: 'artifact_size',
header: 'Size',
render: (t: TagDetail) => <span>{formatBytes(t.artifact_size)}</span>,
},
{
key: 'artifact_content_type',
header: 'Type',
render: (t: TagDetail) => (
<span className="content-type">{t.artifact_content_type || '-'}</span>
),
},
{
key: 'artifact_original_name',
header: 'Filename',
className: 'cell-truncate',
render: (t: TagDetail) => (
<span title={t.artifact_original_name || undefined}>{t.artifact_original_name || '-'}</span>
),
},
{
key: 'created_at',
header: 'Created',
sortable: true,
render: (t: TagDetail) => (
<div className="created-cell">
<span>{new Date(t.created_at).toLocaleString()}</span>
<span className="created-by">by {t.created_by}</span>
</div>
),
},
{
key: 'actions',
header: 'Actions',
render: (t: TagDetail) => (
<a
href={getDownloadUrl(projectName!, packageName!, t.name)}
className="btn btn-secondary btn-small"
download
const handleArtifactSelect = (artifact: PackageArtifact) => {
setSelectedArtifact(artifact);
};
const handleMenuOpen = (e: React.MouseEvent, artifactId: string) => {
e.stopPropagation();
if (openMenuId === artifactId) {
setOpenMenuId(null);
setMenuPosition(null);
} else {
const rect = e.currentTarget.getBoundingClientRect();
setMenuPosition({ top: rect.bottom + 4, left: rect.right - 180 });
setOpenMenuId(artifactId);
}
};
// Helper to get version from artifact - prefer direct version field, fallback to metadata
const getArtifactVersion = (a: PackageArtifact): string | null => {
return a.version || (a.format_metadata?.version as string) || null;
};
// Helper to get download ref - prefer version, fallback to artifact ID
const getDownloadRef = (a: PackageArtifact): string => {
const version = getArtifactVersion(a);
return version || `artifact:${a.id}`;
};
// System projects show Version first, regular projects show Tag first
const columns = isSystemProject
? [
// System project columns: Version first, then Filename
{
key: 'version',
header: 'Version',
// version is from format_metadata, not a sortable DB field
render: (a: PackageArtifact) => (
<strong
className={`tag-name-link ${selectedArtifact?.id === a.id ? 'selected' : ''}`}
onClick={() => handleArtifactSelect(a)}
style={{ cursor: 'pointer' }}
>
<span className="version-badge">{getArtifactVersion(a) || a.id.slice(0, 12)}</span>
</strong>
),
},
{
key: 'original_name',
header: 'Filename',
sortable: true,
className: 'cell-truncate',
render: (a: PackageArtifact) => (
<span title={a.original_name || a.id}>{a.original_name || a.id.slice(0, 12)}</span>
),
},
{
key: 'size',
header: 'Size',
sortable: true,
render: (a: PackageArtifact) => <span>{formatBytes(a.size)}</span>,
},
{
key: 'created_at',
header: 'Cached',
sortable: true,
render: (a: PackageArtifact) => (
<span>{new Date(a.created_at).toLocaleDateString()}</span>
),
},
{
key: 'actions',
header: '',
render: (a: PackageArtifact) => (
<div className="action-buttons">
<a
href={getDownloadUrl(projectName!, packageName!, getDownloadRef(a))}
className="btn btn-icon"
download
title="Download"
>
<svg width="16" height="16" viewBox="0 0 24 24" fill="none" stroke="currentColor" strokeWidth="2">
<path d="M21 15v4a2 2 0 0 1-2 2H5a2 2 0 0 1-2-2v-4" />
<polyline points="7 10 12 15 17 10" />
<line x1="12" y1="15" x2="12" y2="3" />
</svg>
</a>
<button
className="btn btn-icon"
onClick={(e) => handleMenuOpen(e, a.id)}
title="More actions"
>
<svg width="16" height="16" viewBox="0 0 24 24" fill="none" stroke="currentColor" strokeWidth="2">
<circle cx="12" cy="12" r="1" />
<circle cx="12" cy="5" r="1" />
<circle cx="12" cy="19" r="1" />
</svg>
</button>
</div>
),
},
]
: [
// Regular project columns: Version, Filename, Size, Created
// Valid sort fields: created_at, size, original_name
{
key: 'version',
header: 'Version',
// version is from format_metadata, not a sortable DB field
render: (a: PackageArtifact) => (
<strong
className={`tag-name-link ${selectedArtifact?.id === a.id ? 'selected' : ''}`}
onClick={() => handleArtifactSelect(a)}
style={{ cursor: 'pointer' }}
>
<span className="version-badge">{getArtifactVersion(a) || a.id.slice(0, 12)}</span>
</strong>
),
},
{
key: 'original_name',
header: 'Filename',
sortable: true,
className: 'cell-truncate',
render: (a: PackageArtifact) => (
<span title={a.original_name || undefined}>{a.original_name || '—'}</span>
),
},
{
key: 'size',
header: 'Size',
sortable: true,
render: (a: PackageArtifact) => <span>{formatBytes(a.size)}</span>,
},
{
key: 'created_at',
header: 'Created',
sortable: true,
render: (a: PackageArtifact) => (
<span title={`by ${a.created_by}`}>{new Date(a.created_at).toLocaleDateString()}</span>
),
},
{
key: 'actions',
header: '',
render: (a: PackageArtifact) => (
<div className="action-buttons">
<a
href={getDownloadUrl(projectName!, packageName!, getDownloadRef(a))}
className="btn btn-icon"
download
title="Download"
>
<svg width="16" height="16" viewBox="0 0 24 24" fill="none" stroke="currentColor" strokeWidth="2">
<path d="M21 15v4a2 2 0 0 1-2 2H5a2 2 0 0 1-2-2v-4" />
<polyline points="7 10 12 15 17 10" />
<line x1="12" y1="15" x2="12" y2="3" />
</svg>
</a>
<button
className="btn btn-icon"
onClick={(e) => handleMenuOpen(e, a.id)}
title="More actions"
>
<svg width="16" height="16" viewBox="0 0 24 24" fill="none" stroke="currentColor" strokeWidth="2">
<circle cx="12" cy="12" r="1" />
<circle cx="12" cy="5" r="1" />
<circle cx="12" cy="19" r="1" />
</svg>
</button>
</div>
),
},
];
// Find the artifact for the open menu
const openMenuArtifact = artifacts.find(a => a.id === openMenuId);
// Close menu when clicking outside
const handleClickOutside = () => {
if (openMenuId) {
setOpenMenuId(null);
setMenuPosition(null);
}
};
// Render dropdown menu as a portal-like element
const renderActionMenu = () => {
if (!openMenuId || !menuPosition || !openMenuArtifact) return null;
const a = openMenuArtifact;
return (
<div
className="action-menu-backdrop"
onClick={handleClickOutside}
>
<div
className="action-menu-dropdown"
style={{ top: menuPosition.top, left: menuPosition.left }}
onClick={(e) => e.stopPropagation()}
>
Download
</a>
),
},
];
<button onClick={() => { setViewArtifactId(a.id); setShowArtifactIdModal(true); setOpenMenuId(null); setMenuPosition(null); }}>
View Artifact ID
</button>
<button onClick={() => { navigator.clipboard.writeText(a.id); setOpenMenuId(null); setMenuPosition(null); }}>
Copy Artifact ID
</button>
<button onClick={() => { const version = getArtifactVersion(a); const ref = version || `artifact:${a.id}`; fetchEnsureFileForRef(ref); setOpenMenuId(null); setMenuPosition(null); }}>
View Ensure File
</button>
<button onClick={() => { handleArtifactSelect(a); setShowDepsModal(true); setOpenMenuId(null); setMenuPosition(null); }}>
View Dependencies
</button>
</div>
</div>
);
};
if (loading && !tagsData) {
if (loading && !artifactsData) {
return <div className="loading">Loading...</div>;
}
@@ -279,6 +553,19 @@ function PackagePage() {
<div className="page-header__title-row">
<h1>{packageName}</h1>
{pkg && <Badge variant="default">{pkg.format}</Badge>}
{user && canWrite && !isSystemProject && (
<button
className="btn btn-primary btn-small header-upload-btn"
onClick={() => setShowUploadModal(true)}
>
<svg width="14" height="14" viewBox="0 0 24 24" fill="none" stroke="currentColor" strokeWidth="2" style={{ marginRight: '6px' }}>
<path d="M21 15v4a2 2 0 0 1-2 2H5a2 2 0 0 1-2-2v-4" />
<polyline points="17 8 12 3 7 8" />
<line x1="12" y1="3" x2="12" y2="15" />
</svg>
Upload
</button>
)}
</div>
{pkg?.description && <p className="description">{pkg.description}</p>}
<div className="page-header__meta">
@@ -294,16 +581,11 @@ function PackagePage() {
</>
)}
</div>
{pkg && (pkg.tag_count !== undefined || pkg.artifact_count !== undefined) && (
{pkg && pkg.artifact_count !== undefined && (
<div className="package-header-stats">
{pkg.tag_count !== undefined && (
<span className="stat-item">
<strong>{pkg.tag_count}</strong> tags
</span>
)}
{pkg.artifact_count !== undefined && (
<span className="stat-item">
<strong>{pkg.artifact_count}</strong> artifacts
<strong>{pkg.artifact_count}</strong> {isSystemProject ? 'versions' : 'artifacts'}
</span>
)}
{pkg.total_size !== undefined && pkg.total_size > 0 && (
@@ -311,11 +593,6 @@ function PackagePage() {
<strong>{formatBytes(pkg.total_size)}</strong> total
</span>
)}
{pkg.latest_tag && (
<span className="stat-item">
Latest: <strong className="accent">{pkg.latest_tag}</strong>
</span>
)}
</div>
)}
</div>
@@ -324,51 +601,16 @@ function PackagePage() {
{error && <div className="error-message">{error}</div>}
{uploadSuccess && <div className="success-message">{uploadSuccess}</div>}
{user && (
<div className="upload-section card">
<h3>Upload Artifact</h3>
{canWrite ? (
<div className="upload-form">
<div className="form-group">
<label htmlFor="upload-tag">Tag (optional)</label>
<input
id="upload-tag"
type="text"
value={uploadTag}
onChange={(e) => setUploadTag(e.target.value)}
placeholder="v1.0.0, latest, stable..."
/>
</div>
<DragDropUpload
projectName={projectName!}
packageName={packageName!}
tag={uploadTag || undefined}
onUploadComplete={handleUploadComplete}
onUploadError={handleUploadError}
/>
</div>
) : (
<DragDropUpload
projectName={projectName!}
packageName={packageName!}
disabled={true}
disabledReason="You have read-only access to this project and cannot upload artifacts."
onUploadComplete={handleUploadComplete}
onUploadError={handleUploadError}
/>
)}
</div>
)}
<div className="section-header">
<h2>Tags / Versions</h2>
<h2>{isSystemProject ? 'Versions' : 'Artifacts'}</h2>
</div>
<div className="list-controls">
<SearchInput
value={search}
onChange={handleSearchChange}
placeholder="Filter tags..."
placeholder="Filter artifacts..."
className="list-controls__search"
/>
</div>
@@ -381,13 +623,13 @@ function PackagePage() {
<div className="data-table--responsive">
<DataTable
data={tags}
data={artifacts}
columns={columns}
keyExtractor={(t) => t.id}
keyExtractor={(a) => a.id}
emptyMessage={
hasActiveFilters
? 'No tags match your filters. Try adjusting your search.'
: 'No tags yet. Upload an artifact with a tag to create one!'
? 'No artifacts match your filters. Try adjusting your search.'
: 'No artifacts yet. Upload a file to get started!'
}
onSort={handleSortChange}
sortKey={sort}
@@ -405,45 +647,242 @@ function PackagePage() {
/>
)}
<div className="download-by-id-section card">
<h3>Download by Artifact ID</h3>
<div className="download-by-id-form">
<input
type="text"
value={artifactIdInput}
onChange={(e) => setArtifactIdInput(e.target.value.toLowerCase().replace(/[^a-f0-9]/g, '').slice(0, 64))}
placeholder="Enter SHA256 artifact ID (64 hex characters)"
className="artifact-id-input"
/>
<a
href={artifactIdInput.length === 64 ? getDownloadUrl(projectName!, packageName!, `artifact:${artifactIdInput}`) : '#'}
className={`btn btn-primary ${artifactIdInput.length !== 64 ? 'btn-disabled' : ''}`}
download
onClick={(e) => {
if (artifactIdInput.length !== 64) {
e.preventDefault();
}
}}
>
Download
</a>
{/* Used By (Reverse Dependencies) Section - only show if there are reverse deps or error */}
{(reverseDeps.length > 0 || reverseDepsError) && (
<div className="used-by-section card">
<h3>Used By</h3>
{reverseDepsError && (
<div className="error-message">{reverseDepsError}</div>
)}
<div className="reverse-deps-list">
<div className="deps-summary">
{reverseDepsTotal} {reverseDepsTotal === 1 ? 'package depends' : 'packages depend'} on this:
</div>
<ul className="deps-items">
{reverseDeps.map((dep) => (
<li key={dep.artifact_id} className="dep-item reverse-dep-item">
<Link
to={`/project/${dep.project}/${dep.package}${dep.version ? `?version=${dep.version}` : ''}`}
className="dep-link"
>
{dep.project}/{dep.package}
{dep.version && (
<span className="dep-version">v{dep.version}</span>
)}
</Link>
<span className="dep-requires">
requires @ {dep.constraint_value}
</span>
</li>
))}
</ul>
{(reverseDepsHasMore || reverseDepsPage > 1) && (
<div className="reverse-deps-pagination">
<button
className="btn btn-secondary btn-small"
onClick={() => fetchReverseDeps(reverseDepsPage - 1)}
disabled={reverseDepsPage <= 1 || reverseDepsLoading}
>
Previous
</button>
<span className="pagination-info">Page {reverseDepsPage}</span>
<button
className="btn btn-secondary btn-small"
onClick={() => fetchReverseDeps(reverseDepsPage + 1)}
disabled={!reverseDepsHasMore || reverseDepsLoading}
>
Next
</button>
</div>
)}
</div>
</div>
{artifactIdInput.length > 0 && artifactIdInput.length !== 64 && (
<p className="validation-hint">Artifact ID must be exactly 64 hex characters ({artifactIdInput.length}/64)</p>
)}
</div>
)}
<div className="usage-section card">
<h3>Usage</h3>
<p>Download artifacts using:</p>
<pre>
<code>curl -O {window.location.origin}/api/v1/project/{projectName}/{packageName}/+/latest</code>
</pre>
<p>Or with a specific tag:</p>
<pre>
<code>curl -O {window.location.origin}/api/v1/project/{projectName}/{packageName}/+/v1.0.0</code>
</pre>
</div>
{/* Dependency Graph Modal */}
{showGraph && selectedArtifact && (
<DependencyGraph
projectName={projectName!}
packageName={packageName!}
tagName={getArtifactVersion(selectedArtifact) || `artifact:${selectedArtifact.id}`}
onClose={() => setShowGraph(false)}
/>
)}
{/* Upload Modal */}
{showUploadModal && (
<div className="modal-overlay" onClick={() => setShowUploadModal(false)}>
<div className="upload-modal" onClick={(e) => e.stopPropagation()}>
<div className="modal-header">
<h3>Upload Artifact</h3>
<button
className="modal-close"
onClick={() => setShowUploadModal(false)}
title="Close"
>
<svg width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" strokeWidth="2">
<line x1="18" y1="6" x2="6" y2="18"></line>
<line x1="6" y1="6" x2="18" y2="18"></line>
</svg>
</button>
</div>
<div className="modal-body">
<DragDropUpload
projectName={projectName!}
packageName={packageName!}
onUploadComplete={(result) => {
handleUploadComplete(result);
setShowUploadModal(false);
}}
onUploadError={handleUploadError}
/>
</div>
</div>
</div>
)}
{/* Ensure File Modal */}
{showEnsureFile && (
<div className="modal-overlay" onClick={() => setShowEnsureFile(false)}>
<div className="ensure-file-modal" onClick={(e) => e.stopPropagation()}>
<div className="ensure-file-header">
<h3>orchard.ensure for {ensureFileTagName}</h3>
<div className="ensure-file-actions">
{ensureFileContent && (
<CopyButton text={ensureFileContent} />
)}
<button
className="modal-close"
onClick={() => setShowEnsureFile(false)}
title="Close"
>
<svg width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" strokeWidth="2">
<line x1="18" y1="6" x2="6" y2="18"></line>
<line x1="6" y1="6" x2="18" y2="18"></line>
</svg>
</button>
</div>
</div>
<div className="ensure-file-content">
{ensureFileLoading ? (
<div className="ensure-file-loading">Loading...</div>
) : ensureFileError ? (
<div className="ensure-file-error">{ensureFileError}</div>
) : ensureFileContent ? (
<pre className="ensure-file-yaml"><code>{ensureFileContent}</code></pre>
) : (
<div className="ensure-file-empty">No dependencies defined for this artifact.</div>
)}
</div>
<div className="ensure-file-footer">
<p className="ensure-file-hint">
Save this as <code>orchard.ensure</code> in your project root to declare dependencies.
</p>
</div>
</div>
</div>
)}
{/* Dependencies Modal */}
{showDepsModal && selectedArtifact && (
<div className="modal-overlay" onClick={() => setShowDepsModal(false)}>
<div className="deps-modal" onClick={(e) => e.stopPropagation()}>
<div className="modal-header">
<h3>Dependencies for {selectedArtifact.original_name || selectedArtifact.id.slice(0, 12)}</h3>
<button
className="modal-close"
onClick={() => setShowDepsModal(false)}
title="Close"
>
<svg width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" strokeWidth="2">
<line x1="18" y1="6" x2="6" y2="18"></line>
<line x1="6" y1="6" x2="18" y2="18"></line>
</svg>
</button>
</div>
<div className="modal-body">
<div className="deps-modal-controls">
<button
className="btn btn-secondary btn-small"
onClick={fetchEnsureFile}
disabled={ensureFileLoading}
>
View Ensure File
</button>
<button
className="btn btn-secondary btn-small"
onClick={() => { setShowDepsModal(false); setShowGraph(true); }}
>
View Graph
</button>
</div>
{depsLoading ? (
<div className="deps-loading">Loading dependencies...</div>
) : depsError ? (
<div className="deps-error">{depsError}</div>
) : dependencies.length === 0 ? (
<div className="deps-empty">No dependencies</div>
) : (
<div className="deps-list">
<div className="deps-summary">
{dependencies.length} {dependencies.length === 1 ? 'dependency' : 'dependencies'}:
</div>
<ul className="deps-items">
{dependencies.map((dep) => (
<li key={dep.id} className="dep-item">
<Link
to={`/project/${dep.project}/${dep.package}`}
className="dep-link"
onClick={() => setShowDepsModal(false)}
>
{dep.project}/{dep.package}
</Link>
<span className="dep-constraint">
@ {dep.version}
</span>
<span className="dep-status dep-status--ok" title="Package exists">
&#10003;
</span>
</li>
))}
</ul>
</div>
)}
</div>
</div>
</div>
)}
{/* Artifact ID Modal */}
{showArtifactIdModal && viewArtifactId && (
<div className="modal-overlay" onClick={() => setShowArtifactIdModal(false)}>
<div className="artifact-id-modal" onClick={(e) => e.stopPropagation()}>
<div className="modal-header">
<h3>Artifact ID</h3>
<button
className="modal-close"
onClick={() => setShowArtifactIdModal(false)}
title="Close"
>
<svg width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" strokeWidth="2">
<line x1="18" y1="6" x2="6" y2="18"></line>
<line x1="6" y1="6" x2="18" y2="18"></line>
</svg>
</button>
</div>
<div className="modal-body">
<p className="modal-description">SHA256 hash identifying this artifact:</p>
<div className="artifact-id-display">
<code>{viewArtifactId}</code>
<CopyButton text={viewArtifactId} />
</div>
</div>
</div>
</div>
)}
{/* Action Menu Dropdown */}
{renderActionMenu()}
</div>
);
}

View File

@@ -8,7 +8,6 @@ import { DataTable } from '../components/DataTable';
import { SearchInput } from '../components/SearchInput';
import { FilterChip, FilterChipGroup } from '../components/FilterChip';
import { Pagination } from '../components/Pagination';
import { AccessManagement } from '../components/AccessManagement';
import { useAuth } from '../contexts/AuthContext';
import './Home.css';
@@ -196,6 +195,9 @@ function ProjectPage() {
<Badge variant={project.is_public ? 'public' : 'private'}>
{project.is_public ? 'Public' : 'Private'}
</Badge>
{project.is_system && (
<Badge variant="warning">System Cache</Badge>
)}
{accessLevel && (
<Badge variant={accessLevel === 'admin' ? 'success' : accessLevel === 'write' ? 'info' : 'default'}>
{isOwner ? 'Owner' : accessLevel.charAt(0).toUpperCase() + accessLevel.slice(1)}
@@ -211,15 +213,30 @@ function ProjectPage() {
<span className="meta-item">by {project.created_by}</span>
</div>
</div>
{canWrite ? (
<button className="btn btn-primary" onClick={() => setShowForm(!showForm)}>
{showForm ? 'Cancel' : '+ New Package'}
</button>
) : user ? (
<span className="text-muted" title="You have read-only access to this project">
Read-only access
</span>
) : null}
<div className="page-header__actions">
{canAdmin && !project.team_id && !project.is_system && (
<button
className="btn btn-secondary"
onClick={() => navigate(`/project/${projectName}/settings`)}
title="Project Settings"
>
<svg width="16" height="16" viewBox="0 0 24 24" fill="none" stroke="currentColor" strokeWidth="2" strokeLinecap="round" strokeLinejoin="round">
<circle cx="12" cy="12" r="3" />
<path d="M19.4 15a1.65 1.65 0 0 0 .33 1.82l.06.06a2 2 0 0 1 0 2.83 2 2 0 0 1-2.83 0l-.06-.06a1.65 1.65 0 0 0-1.82-.33 1.65 1.65 0 0 0-1 1.51V21a2 2 0 0 1-2 2 2 2 0 0 1-2-2v-.09A1.65 1.65 0 0 0 9 19.4a1.65 1.65 0 0 0-1.82.33l-.06.06a2 2 0 0 1-2.83 0 2 2 0 0 1 0-2.83l.06-.06a1.65 1.65 0 0 0 .33-1.82 1.65 1.65 0 0 0-1.51-1H3a2 2 0 0 1-2-2 2 2 0 0 1 2-2h.09A1.65 1.65 0 0 0 4.6 9a1.65 1.65 0 0 0-.33-1.82l-.06-.06a2 2 0 0 1 0-2.83 2 2 0 0 1 2.83 0l.06.06a1.65 1.65 0 0 0 1.82.33H9a1.65 1.65 0 0 0 1-1.51V3a2 2 0 0 1 2-2 2 2 0 0 1 2 2v.09a1.65 1.65 0 0 0 1 1.51 1.65 1.65 0 0 0 1.82-.33l.06-.06a2 2 0 0 1 2.83 0 2 2 0 0 1 0 2.83l-.06.06a1.65 1.65 0 0 0-.33 1.82V9a1.65 1.65 0 0 0 1.51 1H21a2 2 0 0 1 2 2 2 2 0 0 1-2 2h-.09a1.65 1.65 0 0 0-1.51 1z" />
</svg>
Settings
</button>
)}
{canWrite && !project.is_system ? (
<button className="btn btn-primary" onClick={() => setShowForm(!showForm)}>
{showForm ? 'Cancel' : '+ New Package'}
</button>
) : user && !project.is_system ? (
<span className="text-muted" title="You have read-only access to this project">
Read-only access
</span>
) : null}
</div>
</div>
{error && <div className="error-message">{error}</div>}
@@ -277,18 +294,20 @@ function ProjectPage() {
placeholder="Filter packages..."
className="list-controls__search"
/>
<select
className="list-controls__select"
value={format}
onChange={(e) => handleFormatChange(e.target.value)}
>
<option value="">All formats</option>
{FORMAT_OPTIONS.map((f) => (
<option key={f} value={f}>
{f}
</option>
))}
</select>
{!project?.is_system && (
<select
className="list-controls__select"
value={format}
onChange={(e) => handleFormatChange(e.target.value)}
>
<option value="">All formats</option>
{FORMAT_OPTIONS.map((f) => (
<option key={f} value={f}>
{f}
</option>
))}
</select>
)}
</div>
{hasActiveFilters && (
@@ -324,19 +343,19 @@ function ProjectPage() {
className: 'cell-description',
render: (pkg) => pkg.description || '—',
},
{
...(!project?.is_system ? [{
key: 'format',
header: 'Format',
render: (pkg) => <Badge variant="default">{pkg.format}</Badge>,
},
{
key: 'tag_count',
header: 'Tags',
render: (pkg) => pkg.tag_count ?? '—',
},
render: (pkg: Package) => <Badge variant="default">{pkg.format}</Badge>,
}] : []),
...(!project?.is_system ? [{
key: 'version_count',
header: 'Versions',
render: (pkg: Package) => pkg.version_count ?? '—',
}] : []),
{
key: 'artifact_count',
header: 'Artifacts',
header: project?.is_system ? 'Versions' : 'Artifacts',
render: (pkg) => pkg.artifact_count ?? '—',
},
{
@@ -345,12 +364,12 @@ function ProjectPage() {
render: (pkg) =>
pkg.total_size !== undefined && pkg.total_size > 0 ? formatBytes(pkg.total_size) : '—',
},
{
key: 'latest_tag',
...(!project?.is_system ? [{
key: 'latest_version',
header: 'Latest',
render: (pkg) =>
pkg.latest_tag ? <strong style={{ color: 'var(--accent-primary)' }}>{pkg.latest_tag}</strong> : '—',
},
render: (pkg: Package) =>
pkg.latest_version ? <strong style={{ color: 'var(--accent-primary)' }}>{pkg.latest_version}</strong> : '—',
}] : []),
{
key: 'created_at',
header: 'Created',
@@ -371,10 +390,6 @@ function ProjectPage() {
onPageChange={handlePageChange}
/>
)}
{canAdmin && projectName && (
<AccessManagement projectName={projectName} />
)}
</div>
);
}

View File

@@ -0,0 +1,476 @@
.project-settings-page {
max-width: 900px;
margin: 0 auto;
}
.project-settings-header {
margin-bottom: 32px;
}
.project-settings-header h1 {
font-size: 1.75rem;
font-weight: 600;
color: var(--text-primary);
margin-bottom: 8px;
letter-spacing: -0.02em;
}
.project-settings-subtitle {
color: var(--text-tertiary);
font-size: 0.9375rem;
}
.project-settings-loading {
display: flex;
align-items: center;
justify-content: center;
gap: 12px;
padding: 64px 24px;
color: var(--text-tertiary);
font-size: 0.9375rem;
}
.project-settings-spinner {
width: 20px;
height: 20px;
border: 2px solid var(--border-secondary);
border-top-color: var(--accent-primary);
border-radius: 50%;
animation: project-settings-spin 0.6s linear infinite;
}
@keyframes project-settings-spin {
to {
transform: rotate(360deg);
}
}
.project-settings-error {
display: flex;
align-items: center;
gap: 10px;
background: var(--error-bg);
border: 1px solid rgba(239, 68, 68, 0.2);
color: var(--error);
padding: 12px 16px;
border-radius: var(--radius-md);
margin-bottom: 24px;
font-size: 0.875rem;
}
.project-settings-success {
display: flex;
align-items: center;
gap: 10px;
background: var(--success-bg);
border: 1px solid rgba(34, 197, 94, 0.2);
color: var(--success);
padding: 12px 16px;
border-radius: var(--radius-md);
margin-bottom: 24px;
font-size: 0.875rem;
animation: project-settings-fade-in 0.2s ease;
}
@keyframes project-settings-fade-in {
from {
opacity: 0;
transform: translateY(-8px);
}
to {
opacity: 1;
transform: translateY(0);
}
}
.project-settings-section {
background: var(--bg-secondary);
border: 1px solid var(--border-primary);
border-radius: var(--radius-lg);
padding: 24px;
margin-bottom: 24px;
}
.project-settings-section h2 {
font-size: 1.125rem;
font-weight: 600;
color: var(--text-primary);
margin-bottom: 20px;
padding-bottom: 16px;
border-bottom: 1px solid var(--border-primary);
}
.project-settings-form {
display: flex;
flex-direction: column;
gap: 16px;
}
.project-settings-form-group {
display: flex;
flex-direction: column;
gap: 6px;
}
.project-settings-form-group label {
font-size: 0.8125rem;
font-weight: 500;
color: var(--text-secondary);
}
.project-settings-form-group textarea,
.project-settings-form-group input[type="text"] {
padding: 12px 14px;
background: var(--bg-tertiary);
border: 1px solid var(--border-primary);
border-radius: var(--radius-md);
font-size: 0.875rem;
color: var(--text-primary);
transition: all var(--transition-fast);
font-family: inherit;
resize: vertical;
}
.project-settings-form-group textarea {
min-height: 100px;
}
.project-settings-form-group textarea::placeholder,
.project-settings-form-group input::placeholder {
color: var(--text-muted);
}
.project-settings-form-group textarea:hover:not(:disabled),
.project-settings-form-group input:hover:not(:disabled) {
border-color: var(--border-secondary);
background: var(--bg-elevated);
}
.project-settings-form-group textarea:focus,
.project-settings-form-group input:focus {
outline: none;
border-color: var(--accent-primary);
box-shadow: 0 0 0 3px rgba(16, 185, 129, 0.15);
background: var(--bg-elevated);
}
.project-settings-form-group textarea:disabled,
.project-settings-form-group input:disabled {
opacity: 0.6;
cursor: not-allowed;
}
.project-settings-checkbox-group {
flex-direction: row;
align-items: center;
}
.project-settings-checkbox-label {
display: flex;
align-items: center;
gap: 10px;
cursor: pointer;
font-size: 0.875rem;
font-weight: 400;
color: var(--text-secondary);
user-select: none;
}
.project-settings-checkbox-label input[type="checkbox"] {
position: absolute;
opacity: 0;
width: 0;
height: 0;
}
.project-settings-checkbox-custom {
width: 18px;
height: 18px;
background: var(--bg-tertiary);
border: 1px solid var(--border-secondary);
border-radius: var(--radius-sm);
transition: all var(--transition-fast);
position: relative;
flex-shrink: 0;
}
.project-settings-checkbox-label input[type="checkbox"]:checked + .project-settings-checkbox-custom {
background: var(--accent-primary);
border-color: var(--accent-primary);
}
.project-settings-checkbox-label input[type="checkbox"]:checked + .project-settings-checkbox-custom::after {
content: '';
position: absolute;
left: 5px;
top: 2px;
width: 5px;
height: 9px;
border: solid white;
border-width: 0 2px 2px 0;
transform: rotate(45deg);
}
.project-settings-checkbox-label input[type="checkbox"]:focus + .project-settings-checkbox-custom {
box-shadow: 0 0 0 3px rgba(16, 185, 129, 0.15);
}
.project-settings-checkbox-label:hover .project-settings-checkbox-custom {
border-color: var(--accent-primary);
}
.project-settings-form-actions {
display: flex;
justify-content: flex-end;
gap: 12px;
margin-top: 8px;
}
.project-settings-save-button {
display: flex;
align-items: center;
justify-content: center;
gap: 8px;
padding: 10px 18px;
background: var(--accent-gradient);
border: none;
border-radius: var(--radius-md);
font-size: 0.875rem;
font-weight: 500;
color: white;
cursor: pointer;
transition: all var(--transition-fast);
min-width: 120px;
}
.project-settings-save-button:hover:not(:disabled) {
transform: translateY(-1px);
box-shadow: var(--shadow-sm), 0 0 20px rgba(16, 185, 129, 0.2);
}
.project-settings-save-button:disabled {
opacity: 0.5;
cursor: not-allowed;
transform: none;
}
.project-settings-button-spinner {
width: 14px;
height: 14px;
border: 2px solid rgba(255, 255, 255, 0.3);
border-top-color: white;
border-radius: 50%;
animation: project-settings-spin 0.6s linear infinite;
}
/* Danger Zone */
.project-settings-danger-zone {
background: var(--bg-secondary);
border: 1px solid rgba(239, 68, 68, 0.3);
border-radius: var(--radius-lg);
padding: 24px;
margin-bottom: 24px;
}
.project-settings-danger-zone h2 {
font-size: 1.125rem;
font-weight: 600;
color: var(--error);
margin-bottom: 20px;
padding-bottom: 16px;
border-bottom: 1px solid rgba(239, 68, 68, 0.2);
}
.project-settings-danger-item {
display: flex;
justify-content: space-between;
align-items: flex-start;
gap: 24px;
}
.project-settings-danger-info h3 {
font-size: 0.9375rem;
font-weight: 600;
color: var(--text-primary);
margin-bottom: 4px;
}
.project-settings-danger-info p {
color: var(--text-tertiary);
font-size: 0.8125rem;
max-width: 400px;
}
.project-settings-delete-button {
padding: 10px 18px;
background: transparent;
border: 1px solid rgba(239, 68, 68, 0.3);
border-radius: var(--radius-md);
font-size: 0.875rem;
font-weight: 500;
color: var(--error);
cursor: pointer;
transition: all var(--transition-fast);
flex-shrink: 0;
}
.project-settings-delete-button:hover:not(:disabled) {
background: var(--error-bg);
border-color: rgba(239, 68, 68, 0.5);
}
.project-settings-delete-button:disabled {
opacity: 0.5;
cursor: not-allowed;
}
/* Delete Confirmation */
.project-settings-delete-confirm {
margin-top: 20px;
padding-top: 20px;
border-top: 1px solid rgba(239, 68, 68, 0.2);
animation: project-settings-fade-in 0.2s ease;
}
.project-settings-delete-confirm p {
color: var(--text-secondary);
font-size: 0.875rem;
margin-bottom: 12px;
}
.project-settings-delete-confirm strong {
color: var(--text-primary);
font-family: 'JetBrains Mono', 'Fira Code', 'SF Mono', Monaco, monospace;
background: var(--bg-tertiary);
padding: 2px 6px;
border-radius: var(--radius-sm);
}
.project-settings-delete-confirm-input {
width: 100%;
padding: 12px 14px;
background: var(--bg-tertiary);
border: 1px solid var(--border-primary);
border-radius: var(--radius-md);
font-size: 0.875rem;
color: var(--text-primary);
transition: all var(--transition-fast);
margin-bottom: 16px;
}
.project-settings-delete-confirm-input:focus {
outline: none;
border-color: var(--error);
box-shadow: 0 0 0 3px rgba(239, 68, 68, 0.15);
}
.project-settings-delete-confirm-input::placeholder {
color: var(--text-muted);
}
.project-settings-delete-confirm-actions {
display: flex;
gap: 12px;
}
.project-settings-confirm-delete-button {
display: flex;
align-items: center;
justify-content: center;
gap: 8px;
padding: 10px 18px;
background: var(--error);
border: none;
border-radius: var(--radius-md);
font-size: 0.875rem;
font-weight: 500;
color: white;
cursor: pointer;
transition: all var(--transition-fast);
min-width: 120px;
}
.project-settings-confirm-delete-button:hover:not(:disabled) {
opacity: 0.9;
}
.project-settings-confirm-delete-button:disabled {
opacity: 0.5;
cursor: not-allowed;
}
.project-settings-cancel-button {
padding: 10px 18px;
background: transparent;
border: 1px solid var(--border-primary);
border-radius: var(--radius-md);
font-size: 0.875rem;
font-weight: 500;
color: var(--text-secondary);
cursor: pointer;
transition: all var(--transition-fast);
}
.project-settings-cancel-button:hover:not(:disabled) {
background: var(--bg-hover);
border-color: var(--border-secondary);
color: var(--text-primary);
}
.project-settings-cancel-button:disabled {
opacity: 0.5;
cursor: not-allowed;
}
.project-settings-delete-spinner {
width: 14px;
height: 14px;
border: 2px solid rgba(255, 255, 255, 0.3);
border-top-color: white;
border-radius: 50%;
animation: project-settings-spin 0.6s linear infinite;
}
/* Access denied */
.project-settings-access-denied {
display: flex;
flex-direction: column;
align-items: center;
justify-content: center;
padding: 80px 24px;
text-align: center;
}
.project-settings-access-denied h2 {
font-size: 1.5rem;
font-weight: 600;
color: var(--text-primary);
margin-bottom: 12px;
}
.project-settings-access-denied p {
color: var(--text-tertiary);
font-size: 0.9375rem;
max-width: 400px;
}
/* Responsive */
@media (max-width: 768px) {
.project-settings-danger-item {
flex-direction: column;
gap: 16px;
}
.project-settings-delete-button {
align-self: flex-start;
}
.project-settings-delete-confirm-actions {
flex-direction: column;
}
.project-settings-confirm-delete-button,
.project-settings-cancel-button {
width: 100%;
}
}

View File

@@ -0,0 +1,304 @@
import { useState, useEffect, useCallback } from 'react';
import { useParams, useNavigate } from 'react-router-dom';
import { Project } from '../types';
import {
getProject,
updateProject,
deleteProject,
getMyProjectAccess,
UnauthorizedError,
ForbiddenError,
} from '../api';
import { Breadcrumb } from '../components/Breadcrumb';
import { useAuth } from '../contexts/AuthContext';
import './ProjectSettingsPage.css';
function ProjectSettingsPage() {
const { projectName } = useParams<{ projectName: string }>();
const navigate = useNavigate();
const { user } = useAuth();
const [project, setProject] = useState<Project | null>(null);
const [loading, setLoading] = useState(true);
const [error, setError] = useState<string | null>(null);
const [success, setSuccess] = useState<string | null>(null);
const [accessDenied, setAccessDenied] = useState(false);
const [canAdmin, setCanAdmin] = useState(false);
// General settings form state
const [description, setDescription] = useState('');
const [isPublic, setIsPublic] = useState(false);
const [saving, setSaving] = useState(false);
// Delete confirmation state
const [showDeleteConfirm, setShowDeleteConfirm] = useState(false);
const [deleteConfirmText, setDeleteConfirmText] = useState('');
const [deleting, setDeleting] = useState(false);
const loadData = useCallback(async () => {
if (!projectName) return;
try {
setLoading(true);
setAccessDenied(false);
const [projectData, accessResult] = await Promise.all([
getProject(projectName),
getMyProjectAccess(projectName),
]);
setProject(projectData);
setDescription(projectData.description || '');
setIsPublic(projectData.is_public);
const hasAdminAccess = accessResult.access_level === 'admin';
setCanAdmin(hasAdminAccess);
if (!hasAdminAccess) {
setAccessDenied(true);
}
setError(null);
} catch (err) {
if (err instanceof UnauthorizedError) {
navigate('/login', { state: { from: `/project/${projectName}/settings` } });
return;
}
if (err instanceof ForbiddenError) {
setAccessDenied(true);
setLoading(false);
return;
}
setError(err instanceof Error ? err.message : 'Failed to load project');
} finally {
setLoading(false);
}
}, [projectName, navigate]);
useEffect(() => {
loadData();
}, [loadData]);
const handleSaveSettings = async (e: React.FormEvent) => {
e.preventDefault();
if (!projectName) return;
try {
setSaving(true);
setError(null);
const updatedProject = await updateProject(projectName, {
description: description || undefined,
is_public: isPublic,
});
setProject(updatedProject);
setSuccess('Settings saved successfully');
setTimeout(() => setSuccess(null), 3000);
} catch (err) {
setError(err instanceof Error ? err.message : 'Failed to save settings');
} finally {
setSaving(false);
}
};
const handleDeleteProject = async () => {
if (!projectName || deleteConfirmText !== projectName) return;
try {
setDeleting(true);
setError(null);
await deleteProject(projectName);
navigate('/');
} catch (err) {
setError(err instanceof Error ? err.message : 'Failed to delete project');
setDeleting(false);
}
};
const handleCancelDelete = () => {
setShowDeleteConfirm(false);
setDeleteConfirmText('');
};
if (loading) {
return (
<div className="project-settings-page">
<Breadcrumb
items={[
{ label: 'Projects', href: '/' },
{ label: projectName || '', href: `/project/${projectName}` },
{ label: 'Settings' },
]}
/>
<div className="project-settings-loading">
<div className="project-settings-spinner" />
<span>Loading...</span>
</div>
</div>
);
}
if (accessDenied || !canAdmin) {
return (
<div className="project-settings-page">
<Breadcrumb
items={[
{ label: 'Projects', href: '/' },
{ label: projectName || '', href: `/project/${projectName}` },
{ label: 'Settings' },
]}
/>
<div className="project-settings-access-denied">
<h2>Access Denied</h2>
<p>You must be a project admin to access settings.</p>
{!user && (
<p style={{ marginTop: '16px' }}>
<a href="/login" className="btn btn-primary">
Sign in
</a>
</p>
)}
</div>
</div>
);
}
if (!project) {
return (
<div className="project-settings-page">
<Breadcrumb
items={[
{ label: 'Projects', href: '/' },
{ label: projectName || '' },
]}
/>
<div className="project-settings-error">Project not found</div>
</div>
);
}
return (
<div className="project-settings-page">
<Breadcrumb
items={[
{ label: 'Projects', href: '/' },
{ label: project.name, href: `/project/${project.name}` },
{ label: 'Settings' },
]}
/>
<div className="project-settings-header">
<h1>Project Settings</h1>
<p className="project-settings-subtitle">Manage settings for {project.name}</p>
</div>
{error && <div className="project-settings-error">{error}</div>}
{success && <div className="project-settings-success">{success}</div>}
{/* General Settings Section */}
<div className="project-settings-section">
<h2>General</h2>
<form className="project-settings-form" onSubmit={handleSaveSettings}>
<div className="project-settings-form-group">
<label htmlFor="description">Description</label>
<textarea
id="description"
value={description}
onChange={(e) => setDescription(e.target.value)}
placeholder="Describe your project..."
disabled={saving}
/>
</div>
<div className="project-settings-form-group project-settings-checkbox-group">
<label className="project-settings-checkbox-label">
<input
type="checkbox"
checked={isPublic}
onChange={(e) => setIsPublic(e.target.checked)}
disabled={saving}
/>
<span className="project-settings-checkbox-custom" />
<span>Public project (visible to everyone)</span>
</label>
</div>
<div className="project-settings-form-actions">
<button type="submit" className="project-settings-save-button" disabled={saving}>
{saving ? (
<>
<span className="project-settings-button-spinner" />
Saving...
</>
) : (
'Save Changes'
)}
</button>
</div>
</form>
</div>
{/* Danger Zone Section */}
<div className="project-settings-danger-zone">
<h2>Danger Zone</h2>
<div className="project-settings-danger-item">
<div className="project-settings-danger-info">
<h3>Delete this project</h3>
<p>
Once you delete a project, there is no going back. This will permanently delete the
project, all packages, artifacts, and tags.
</p>
</div>
{!showDeleteConfirm && (
<button
className="project-settings-delete-button"
onClick={() => setShowDeleteConfirm(true)}
disabled={deleting}
>
Delete Project
</button>
)}
</div>
{showDeleteConfirm && (
<div className="project-settings-delete-confirm">
<p>
Type <strong>{project.name}</strong> to confirm deletion:
</p>
<input
type="text"
className="project-settings-delete-confirm-input"
value={deleteConfirmText}
onChange={(e) => setDeleteConfirmText(e.target.value)}
placeholder={project.name}
disabled={deleting}
autoFocus
/>
<div className="project-settings-delete-confirm-actions">
<button
className="project-settings-confirm-delete-button"
onClick={handleDeleteProject}
disabled={deleting || deleteConfirmText !== project.name}
>
{deleting ? (
<>
<span className="project-settings-delete-spinner" />
Deleting...
</>
) : (
'Yes, delete this project'
)}
</button>
<button
className="project-settings-cancel-button"
onClick={handleCancelDelete}
disabled={deleting}
>
Cancel
</button>
</div>
</div>
)}
</div>
</div>
);
}
export default ProjectSettingsPage;

View File

@@ -0,0 +1,270 @@
.team-dashboard {
padding: 1.5rem 0;
}
.team-header {
display: flex;
justify-content: space-between;
align-items: flex-start;
gap: 1.5rem;
margin-bottom: 2rem;
}
.team-header-left {
flex: 1;
}
.team-header-title {
display: flex;
align-items: center;
gap: 0.75rem;
margin-bottom: 0.5rem;
}
.team-header h1 {
margin: 0;
font-size: 1.5rem;
font-weight: 600;
}
.team-slug {
font-size: 0.875rem;
color: var(--text-muted);
}
.team-description {
margin: 0 0 0.5rem;
color: var(--text-secondary);
font-size: 0.9375rem;
max-width: 600px;
}
.team-header-actions {
display: flex;
gap: 0.5rem;
flex-shrink: 0;
}
.team-section {
margin-top: 2rem;
}
.section-header {
display: flex;
justify-content: space-between;
align-items: center;
margin-bottom: 1rem;
}
.section-header h2 {
margin: 0;
font-size: 1.25rem;
}
/* Table utility classes */
.text-muted {
color: var(--text-muted);
}
.btn-ghost {
background: transparent;
color: var(--text-muted);
border: none;
padding: 0.375rem;
cursor: pointer;
border-radius: var(--radius-sm);
}
.btn-ghost:hover {
background: var(--bg-tertiary);
color: var(--text-primary);
}
.section-footer {
margin-top: 1rem;
text-align: center;
}
.view-all-link {
font-size: 0.875rem;
color: var(--accent-primary);
text-decoration: none;
}
.view-all-link:hover {
text-decoration: underline;
}
/* States */
.loading-state,
.error-state {
text-align: center;
padding: 4rem 2rem;
}
.error-state h2 {
margin: 0 0 0.5rem;
}
.error-state p {
margin: 0 0 1.5rem;
color: var(--text-muted);
}
.empty-state {
text-align: center;
padding: 2rem;
background: var(--bg-secondary);
border: 1px dashed var(--border-primary);
border-radius: var(--radius-md);
color: var(--text-muted);
}
.empty-state p {
margin: 0;
}
.empty-hint {
margin-top: 0.5rem !important;
font-size: 0.875rem;
}
/* Buttons */
.btn {
display: inline-flex;
align-items: center;
gap: 0.5rem;
padding: 0.5rem 1rem;
border: none;
border-radius: var(--radius-md);
font-size: 0.875rem;
font-weight: 500;
cursor: pointer;
text-decoration: none;
transition: all 0.15s ease;
}
.btn-sm {
padding: 0.375rem 0.75rem;
font-size: 0.8125rem;
}
.btn-primary {
background: var(--accent-primary);
color: white;
}
.btn-primary:hover {
background: var(--accent-primary-hover);
}
.btn-secondary {
background: var(--bg-tertiary);
color: var(--text-primary);
border: 1px solid var(--border-primary);
}
.btn-secondary:hover {
background: var(--bg-hover);
}
/* Modal */
.modal-overlay {
position: fixed;
inset: 0;
background: rgba(0, 0, 0, 0.7);
display: flex;
align-items: center;
justify-content: center;
z-index: 1000;
padding: 1rem;
}
.modal-content {
background: var(--bg-secondary);
border: 1px solid var(--border-primary);
border-radius: var(--radius-lg);
padding: 1.5rem;
width: 100%;
max-width: 480px;
max-height: 90vh;
box-shadow: var(--shadow-lg);
overflow-y: auto;
}
.modal-content h2 {
margin: 0 0 1.5rem;
font-size: 1.25rem;
color: var(--text-primary);
}
/* Form */
.form-group {
margin-bottom: 1rem;
}
.form-group label {
display: block;
margin-bottom: 0.5rem;
font-weight: 500;
font-size: 0.875rem;
color: var(--text-primary);
}
.form-group input[type="text"],
.form-group textarea {
width: 100%;
padding: 0.625rem 0.75rem;
border: 1px solid var(--border-primary);
border-radius: var(--radius-md);
background: var(--bg-tertiary);
color: var(--text-primary);
font-size: 0.875rem;
}
.form-group input:focus,
.form-group textarea:focus {
outline: none;
border-color: var(--accent-primary);
box-shadow: 0 0 0 3px rgba(16, 185, 129, 0.2);
}
.form-group textarea {
resize: vertical;
min-height: 80px;
}
.checkbox-group label {
display: flex;
align-items: center;
gap: 0.5rem;
cursor: pointer;
}
.checkbox-group input[type="checkbox"] {
width: 1rem;
height: 1rem;
}
.form-hint {
display: block;
font-size: 0.8125rem;
color: var(--text-muted);
margin-top: 0.375rem;
}
.form-actions {
display: flex;
justify-content: flex-end;
gap: 0.75rem;
margin-top: 1.5rem;
}
.btn:disabled {
opacity: 0.6;
cursor: not-allowed;
}
.empty-state .btn {
margin-top: 1rem;
}

Some files were not shown because too many files have changed in this diff Show More