60 Commits

Author SHA1 Message Date
Mondo Diaz
72952d84a1 Skip self-dependencies in dependency resolver
PyPI packages can have self-referential dependencies for extras
(e.g., pytest[testing] depends on pytest). These were incorrectly
detected as circular dependencies. Now we skip them.
2026-02-02 19:45:34 -06:00
Mondo Diaz
e6d42d91cd Fix [object Object] error when API returns structured error detail
The backend returns detail as an object for some errors (circular dependency,
conflicts, etc.). The API client now JSON.stringifies object details so they
can be properly parsed by error handlers like DependencyGraph.
2026-02-02 18:33:55 -06:00
Mondo Diaz
b3ae3b03eb Show missing dependencies in dependency graph instead of failing
When dependencies are not cached on the server (common since we removed
proactive caching), the dependency graph now:
- Continues resolving what it can find
- Shows missing dependencies in a separate section with amber styling
- Displays the constraint and which package required them
- Updates the header stats to show "X cached • Y not cached"

This provides a better user experience than showing an error when
some dependencies haven't been downloaded yet.
2026-02-02 16:29:37 -06:00
Mondo Diaz
ba0a658611 Fix dependency graph error for invalid version constraints
When a dependency has an invalid version constraint like '>=' (without
a version number), the resolver now treats it as a wildcard and returns
the latest available version instead of failing with 'Dependency not found'.

This handles malformed metadata that may have been stored from PyPI packages.
2026-02-02 16:26:18 -06:00
Mondo Diaz
081cc6df83 Remove proactive PyPI dependency caching feature
The background task queue for proactively caching package dependencies was
causing server instability and unnecessary growth. The PyPI proxy now only
caches packages on-demand when users request them.

Removed:
- PyPI cache worker (background task queue and worker pool)
- PyPICacheTask model and related database schema
- Cache management API endpoints (/pypi/cache/*)
- Background Jobs admin dashboard
- Dependency extraction and queueing logic

Kept:
- On-demand package caching (still works when users request packages)
- Async httpx for non-blocking downloads (prevents health check failures)
- URL-based cache lookups for deduplication
2026-02-02 16:17:33 -06:00
Mondo Diaz
cf7bdccb3a Center text in jobs table columns 2026-02-02 15:30:46 -06:00
Mondo Diaz
1329d380a4 Convert PyPI proxy from sync to async httpx to prevent event loop blocking
The pypi_download_file, pypi_simple_index, and pypi_package_versions endpoints
were using synchronous httpx.Client inside async functions. When upstream PyPI
servers respond slowly, this blocked the entire FastAPI event loop, preventing
health checks from responding. Kubernetes would then kill the pod after the
liveness probe timed out.

Changes:
- httpx.Client → httpx.AsyncClient
- client.get() → await client.get()
- response.iter_bytes() → response.aiter_bytes()

This ensures the event loop remains responsive during slow upstream downloads,
allowing health checks to succeed even when downloads take 20+ seconds.
2026-02-02 15:26:24 -06:00
Mondo Diaz
361210a2bc Add cancel job button and improve jobs table UI
- Remove "All Jobs" title
- Move Status column to front of table
- Add Cancel button for in-progress jobs
- Add cancel endpoint: POST /pypi/cache/cancel/{package_name}
- Add btn-danger CSS styling
2026-02-02 15:18:59 -06:00
Mondo Diaz
415ad9a29a Stream downloads to temp file to reduce memory usage
- Download packages in 64KB chunks to temp file instead of loading into memory
- Upload to S3 from temp file (streaming)
- Clean up temp file after processing
- Reduces memory footprint from 2x file size to 1x file size
2026-02-02 15:10:25 -06:00
Mondo Diaz
1667c5a416 Increase memory to 1Gi and reduce workers to 1 for stability 2026-02-02 15:08:00 -06:00
Mondo Diaz
1021e2b942 Add PyPI cache config and bump memory in values-prod.yaml 2026-02-02 14:38:47 -06:00
Mondo Diaz
d0e91658d7 Add PyPI cache config and bump memory in values-stage.yaml 2026-02-02 14:38:21 -06:00
Mondo Diaz
7b89f41704 Add PyPI cache config and bump memory in values-dev.yaml 2026-02-02 14:37:55 -06:00
Mondo Diaz
ba43110123 Add PyPI cache worker config and increase memory limit
- Add orchard.pypiCache config section to helm values
- Set default workers to 2 (reduced from 5 to limit memory)
- Bump pod memory from 512Mi to 768Mi (request=limit)
- Add ORCHARD_PYPI_CACHE_* env vars to deployment template
2026-02-02 14:37:27 -06:00
Mondo Diaz
92edef92e6 Redesign jobs dashboard with unified table and progress bar
- Add overall progress bar showing completed/active/failed counts
- Unify all job types into single table with Type column
- Simplify status to Working/Pending/Failed badges
- Remove NPM "Coming Soon" section
- Add get_recent_activity() function for future activity feed
- Fix dark mode CSS using CSS variables
2026-02-02 14:34:48 -06:00
Mondo Diaz
47b137f4eb Improve Active Workers table and recover stale tasks
Backend:
- Add _recover_stale_tasks() to reset tasks stuck in 'in_progress'
  from previous crashes (tasks >5 min old get reset to pending)
- Called automatically on startup

Frontend:
- Fix dark mode colors using CSS variables instead of hardcoded values
- Add elapsed time column showing how long task has been running
- Add spinning indicator next to package name
- Add status badge (Running/Stale?)
- Highlight stale tasks (>5 min) in amber
- Auto-updates every 5 seconds with existing refresh
2026-02-02 14:29:17 -06:00
Mondo Diaz
1138309aaa Add Active Workers table to Background Jobs dashboard
Shows currently processing cache tasks in a dynamic table with:
- Package name and version constraint being cached
- Recursion depth and attempt number
- Start timestamp
- Pulsing indicator to show live activity

Backend changes:
- Add get_active_tasks() function to pypi_cache_worker.py
- Add GET /pypi/cache/active endpoint to pypi_proxy.py

Frontend changes:
- Add PyPICacheActiveTask type
- Add getPyPICacheActiveTasks() API function
- Add Active Workers section with animated table
- Auto-refreshes every 5 seconds with existing data
2026-02-02 13:50:45 -06:00
Mondo Diaz
3bdeade7ca Fix nested dependency depth tracking in PyPI cache worker
When the cache worker downloaded a package through the proxy, dependencies
were always queued with depth=0 instead of depth+1. This meant depth limits
weren't properly enforced for nested dependencies.

Changes:
- Add cache-depth query parameter to pypi_download_file endpoint
- Worker now passes its current depth when fetching packages
- Dependencies are queued at cache_depth+1 instead of hardcoded 0
- Add tests for depth tracking behavior
2026-02-02 13:47:22 -06:00
Mondo Diaz
8edb45879f Fix jobs dashboard showing misleading completion message
The dashboard was showing "All jobs completed successfully" whenever
there were no failed tasks, even if there were pending or in-progress
jobs. Now shows:
- "All jobs completed" only when pending=0 and in_progress=0
- "Jobs are processing. No failures yet." when jobs are in queue
2026-02-02 11:56:01 -06:00
Mondo Diaz
97b39d000b Add security fixes and code cleanup for PyPI cache
- Add require_admin authentication to cache management endpoints
- Add limit validation (1-500) on failed tasks query
- Add thread lock for worker pool thread safety
- Fix exception handling with separate recovery DB session
- Remove obsolete design doc
2026-02-02 11:37:25 -06:00
Mondo Diaz
ba708332a5 Add Background Jobs dashboard for admin users
New admin page at /admin/jobs showing:
- PyPI cache job status (pending, in-progress, completed, failed)
- Failed task list with error details
- Retry individual packages or retry all failed
- Auto-refresh every 5 seconds (toggleable)
- Placeholder for future NPM cache jobs

Accessible from admin dropdown menu as "Background Jobs".
2026-02-02 11:26:55 -06:00
Mondo Diaz
d274f3f375 Add robust PyPI dependency caching with task queue
Replace unbounded thread spawning with managed worker pool:
- New pypi_cache_tasks table tracks caching jobs
- Thread pool with 5 workers (configurable via ORCHARD_PYPI_CACHE_WORKERS)
- Automatic retries with exponential backoff (30s, 60s, then fail)
- Deduplication to prevent duplicate caching attempts

New API endpoints for visibility and control:
- GET /pypi/cache/status - queue health summary
- GET /pypi/cache/failed - list failed tasks with errors
- POST /pypi/cache/retry/{package} - retry single package
- POST /pypi/cache/retry-all - retry all failed packages

This fixes silent failures in background dependency caching where
packages would fail to cache without any tracking or retry mechanism.
2026-02-02 11:16:02 -06:00
Mondo Diaz
490b05438d Add design doc for PyPI cache robustness improvements 2026-02-02 11:06:51 -06:00
Mondo Diaz
3c2ab70ef0 Fix proactive dependency caching HTTPS redirect issue
When background threads fetch from our own proxy using the request's
base_url, it returns http:// but ingress requires https://. The 308
redirect was dropping trailing slashes, causing requests to hit the
frontend catch-all route instead of /pypi/simple/.

Force HTTPS explicitly in the background caching function to avoid
the redirect entirely.
2026-01-30 18:59:31 -06:00
Mondo Diaz
109a593f83 Add debug logging for proactive caching regex failures 2026-01-30 18:43:09 -06:00
Mondo Diaz
1d727b3f8c Fix proactive caching regex to match both hyphens and underscores
PEP 503 normalizes package names to use hyphens, but wheel filenames
may use underscores (e.g., typing_extensions-4.0.0-py3-none-any.whl).

Convert the search pattern to match either separator.
2026-01-30 18:25:30 -06:00
Mondo Diaz
47aa0afe91 Fix proactive caching failing on HTTP->HTTPS redirects
The background dependency caching was getting 308 redirects because
request.base_url returns http:// but the ingress redirects to https://.

Enable follow_redirects=True in httpx client to handle this.
2026-01-30 18:11:08 -06:00
Mondo Diaz
f992fc540e Add proactive dependency caching for PyPI packages
When a PyPI package is cached, its dependencies are now automatically
fetched in background threads. This ensures the entire dependency tree
is cached even if pip already has some packages installed locally.

Features:
- Background threads fetch each dependency without blocking the response
- Uses our own proxy endpoint to cache, which recursively caches transitive deps
- Max depth of 10 to prevent infinite loops
- Daemon threads so they don't block process shutdown
2026-01-30 17:45:30 -06:00
Mondo Diaz
044a6c1d27 Fix duplicate dependency constraint causing 500 errors
- Deduplicate dependencies by package name before inserting
- Some packages (like anyio) list the same dep (trio) multiple times with
  different version constraints for different extras
- The unique constraint on (artifact_id, project, package) rejected these
- Also removed debug logging from dependencies.py
2026-01-30 17:43:49 -06:00
Mondo Diaz
62c77dc16d Add detailed debug logging to _resolve_dependency_to_artifact 2026-01-30 17:29:19 -06:00
Mondo Diaz
7c05360eed Add debug logging to resolve_dependencies 2026-01-30 17:21:04 -06:00
Mondo Diaz
76878279e9 Add backfill script for PyPI package dependencies
Script extracts Requires-Dist metadata from cached PyPI packages
and stores them in artifact_dependencies table.

Usage:
  docker exec <container> python -m backend.scripts.backfill_pypi_dependencies
  docker exec <container> python -m backend.scripts.backfill_pypi_dependencies --dry-run
2026-01-30 15:38:45 -06:00
Mondo Diaz
e1b01abf9b Add PEP 440 version constraint matching for dependency resolution
- Parse version constraints like >=1.9, <2.0 using packaging library
- Find the latest version that satisfies the constraint
- Support wildcard (*) to get latest version
- Fall back to exact version and tag matching
2026-01-30 15:34:19 -06:00
Mondo Diaz
d07936b666 Fix ensure file modal z-index when opened from deps modal 2026-01-30 15:32:06 -06:00
Mondo Diaz
47b3eb439d Extract and store dependencies from PyPI packages
- Add functions to parse Requires-Dist metadata from wheel and sdist files
- Store extracted dependencies in artifact_dependencies table
- Fix streaming response for cached artifacts (proper tuple unpacking)
- Fix version uniqueness check to use version string instead of artifact_id
- Skip creating versions for .metadata files
2026-01-30 15:14:52 -06:00
Mondo Diaz
c5f75e4fd6 Add is_system to all ProjectResponse constructions in routes 2026-01-30 13:34:26 -06:00
Mondo Diaz
ff31379649 Fix: ensure existing _pypi project gets is_system=true 2026-01-30 13:33:31 -06:00
Mondo Diaz
424b1e5770 Add is_system field to ProjectResponse schema 2026-01-30 13:11:11 -06:00
Mondo Diaz
7b5b0c78d8 Hide Tags and Latest columns for system projects in package table 2026-01-30 12:55:28 -06:00
Mondo Diaz
924826f07a Improve system project UX and make dependencies a modal
- Hide tag count stat for system projects (show "versions" instead of "artifacts")
- Hide "Latest" tag stat for system projects
- Change "Create/Update Tag" to only show for non-system projects
- Add "View Artifact ID" menu option with modal showing the SHA256 hash
- Move dependencies section to a modal (opened via "View Dependencies" menu)
- Add deps-modal and artifact-id-modal CSS styles
2026-01-30 12:36:40 -06:00
Mondo Diaz
fe6c6c52d2 Fix PyPI proxy UX and package stats calculation
- Fix artifact_count and total_size calculation to use Tags instead of
  Uploads, so PyPI cached packages show their stats correctly
- Fix PackagePage dropdown menu positioning (use fixed position with backdrop)
- Add system project detection for projects starting with "_"
- Show Version as primary column for system projects, hide Tag column
- Hide upload button for system projects (they're cache-only)
- Rename section header to "Versions" for system projects
- Fix test_projects_sort_by_name to exclude system projects from sort comparison
2026-01-30 12:16:05 -06:00
Mondo Diaz
701e11ce83 Hide format filter and column for system projects
System projects like _pypi only contain packages of one format,
so the format filter dropdown and column are redundant.
2026-01-30 11:55:09 -06:00
Mondo Diaz
ff9e02606e Hide Settings and New Package buttons for system projects
System projects should be system-controlled only. Users should not
be able to create packages or change settings on system cache projects.
2026-01-30 11:54:02 -06:00
Mondo Diaz
f3afdd3bbf Improve PyPI proxy and Package page UX
PyPI proxy improvements:
- Set package format to "pypi" instead of "generic"
- Extract version from filename and create PackageVersion record
- Support .whl, .tar.gz, and .zip filename formats

Package page UX overhaul:
- Move upload to header button with modal
- Simplify table: combine Tag/Version, remove Type and Artifact ID columns
- Add row action menu (⋯) with: Copy ID, Ensure File, Create Tag, Dependencies
- Remove cluttered "Download by Artifact ID" and "Create/Update Tag" sections
- Add modals for upload and create tag actions
- Cleaner, more scannable table layout
2026-01-30 11:52:37 -06:00
Mondo Diaz
4b73196664 Show team name instead of individual user in Owner column
Projects owned by teams now display the team name in the Owner column
for better organizational continuity when team members change.
Falls back to created_by if no team is assigned.
2026-01-30 11:25:01 -06:00
Mondo Diaz
7ef66745f1 Add "(coming soon)" label for unsupported upstream source types
Only pypi and generic are currently supported. Other types now show
"(coming soon)" in both the dropdown and the sources table.
2026-01-30 11:03:44 -06:00
Mondo Diaz
2dc7fe5a7b Fix PyPI proxy: use correct storage method and make project public
- Use storage.get_stream(s3_key) instead of non-existent get_artifact_stream()
- Make _pypi project public (is_public=True) so cached packages are visible
2026-01-30 10:59:50 -06:00
Mondo Diaz
534e4b964f Fix Project and Tag model fields in PyPI proxy
Use correct model fields:
- Project: is_public, is_system, created_by (not visibility)
- Tag: add required created_by field
2026-01-30 10:29:25 -06:00
Mondo Diaz
757e43fc34 Fix Artifact model field names in PyPI proxy
Use correct Artifact model fields:
- original_name instead of filename
- Add required created_by and s3_key fields
- Include checksum fields from storage result
2026-01-30 09:58:15 -06:00
Mondo Diaz
d78092de55 Fix PyPI proxy to use correct storage.store() method
The code was calling storage.store_artifact() which doesn't exist.
Changed to use storage.store() which handles content-addressable
storage with automatic deduplication.
2026-01-30 09:41:34 -06:00
Mondo Diaz
0fa991f536 Allow full path in PyPI upstream source URL
Users can now configure the full path including /simple in their
upstream source URL (e.g., https://example.com/api/pypi/repo/simple)
instead of having the code append /simple/ automatically.

This matches pip's --index-url format, making configuration more
intuitive and copy/paste friendly.
2026-01-30 09:24:05 -06:00
Mondo Diaz
00fb2729e4 Fix test_rewrite_relative_links assertion to expect correct URL
The test was checking for the wrong URL pattern. When urljoin resolves
../../packages/ab/cd/... relative to /api/pypi/pypi-remote/simple/requests/,
it correctly produces /api/pypi/pypi-remote/packages/ab/cd/... (not
/api/pypi/packages/...).
2026-01-30 08:51:30 -06:00
Mondo Diaz
8ae4d7a685 Improve PyPI proxy test assertions for all status codes
Tests now verify the correct response for each scenario:
- 200: HTML content-type
- 404: "not found" error message
- 503: "No PyPI upstream sources configured" error message
2026-01-29 19:35:20 -06:00
Mondo Diaz
4b887d1aad Fix PyPI proxy tests to work with or without upstream sources
- Tests now accept 200/404/503 responses since upstream sources may or
  may not be configured in the test environment
- Added upstream_base_url parameter to _rewrite_package_links test
- Added test for relative URL resolution (Artifactory-style URLs)
2026-01-29 19:34:33 -06:00
Mondo Diaz
4dc54ace8a Fix HTTPS scheme detection behind reverse proxy
When behind a reverse proxy that terminates SSL, the server sees HTTP
requests internally. Added _get_base_url() helper that respects the
X-Forwarded-Proto header to generate correct external HTTPS URLs.

This fixes links in the PyPI simple index showing http:// instead of
https:// when accessed via HTTPS through a load balancer.
2026-01-29 18:02:21 -06:00
Mondo Diaz
64bfd3902f Fix relative URL handling in PyPI proxy
Artifactory and other registries may return relative URLs in their
Simple API responses (e.g., ../../packages/...). The proxy now resolves
these to absolute URLs using urljoin() before encoding them in the
upstream parameter.

This fixes package downloads failing when the upstream registry uses
relative URLs in its package index.
2026-01-29 18:01:19 -06:00
Mondo Diaz
bdfed77cb1 Remove dead code from pypi_proxy.py
- Remove unused imports (UpstreamClient, UpstreamClientConfig,
  UpstreamHTTPError, UpstreamConnectionError, UpstreamTimeoutError)
- Simplify matched_source selection logic, removing dead conditional
  that always evaluated to True due to 'or True'
2026-01-29 16:42:53 -06:00
Mondo Diaz
140f6c926a Fix httpx.Timeout configuration in PyPI proxy
httpx.Timeout requires either a default value or all four parameters.
Changed to httpx.Timeout(default, connect=X) format.
2026-01-29 16:40:06 -06:00
Mondo Diaz
dcd405679a Merge branch 'feature/transparent-proxy' into 'main'
Add transparent PyPI proxy and improve upstream sources UI

Closes #108

See merge request esv/bsf/bsf-integration/orchard/orchard-mvp!56
2026-01-29 16:12:57 -06:00
Mondo Diaz
97498b2f86 Add transparent PyPI proxy and improve upstream sources UI 2026-01-29 16:12:57 -06:00
26 changed files with 1569 additions and 446 deletions

View File

@@ -64,6 +64,11 @@ class Settings(BaseSettings):
# Global cache settings override (None = use DB value, True/False = override DB)
cache_auto_create_system_projects: Optional[bool] = None # Override auto_create_system_projects
# PyPI Cache Worker settings
pypi_cache_workers: int = 5 # Number of concurrent cache workers
pypi_cache_max_depth: int = 10 # Maximum recursion depth for dependency caching
pypi_cache_max_attempts: int = 3 # Maximum retry attempts for failed cache tasks
# JWT Authentication settings (optional, for external identity providers)
jwt_enabled: bool = False # Enable JWT token validation
jwt_secret: str = "" # Secret key for HS256, or leave empty for RS256 with JWKS
@@ -88,6 +93,24 @@ class Settings(BaseSettings):
def is_production(self) -> bool:
return self.env.lower() == "production"
@property
def PORT(self) -> int:
"""Alias for server_port for compatibility."""
return self.server_port
# Uppercase aliases for PyPI cache settings (for backward compatibility)
@property
def PYPI_CACHE_WORKERS(self) -> int:
return self.pypi_cache_workers
@property
def PYPI_CACHE_MAX_DEPTH(self) -> int:
return self.pypi_cache_max_depth
@property
def PYPI_CACHE_MAX_ATTEMPTS(self) -> int:
return self.pypi_cache_max_attempts
class Config:
env_prefix = "ORCHARD_"
case_sensitive = False

View File

@@ -10,11 +10,20 @@ Handles:
- Conflict detection
"""
import re
import yaml
from typing import List, Dict, Any, Optional, Set, Tuple
from sqlalchemy.orm import Session
from sqlalchemy import and_
# Import packaging for PEP 440 version matching
try:
from packaging.specifiers import SpecifierSet, InvalidSpecifier
from packaging.version import Version, InvalidVersion
HAS_PACKAGING = True
except ImportError:
HAS_PACKAGING = False
from .models import (
Project,
Package,
@@ -33,6 +42,7 @@ from .schemas import (
ResolvedArtifact,
DependencyResolutionResponse,
DependencyConflict,
MissingDependency,
PaginationMeta,
)
@@ -304,6 +314,95 @@ def get_reverse_dependencies(
)
def _is_version_constraint(version_str: str) -> bool:
"""Check if a version string contains constraint operators."""
if not version_str:
return False
# Check for common constraint operators
return any(op in version_str for op in ['>=', '<=', '!=', '~=', '>', '<', '==', '*'])
def _resolve_version_constraint(
db: Session,
package: Package,
constraint: str,
) -> Optional[Tuple[str, str, int]]:
"""
Resolve a version constraint (e.g., '>=1.9') to a specific version.
Uses PEP 440 version matching to find the best matching version.
Args:
db: Database session
package: Package to search versions in
constraint: Version constraint string (e.g., '>=1.9', '<2.0,>=1.5')
Returns:
Tuple of (artifact_id, resolved_version, size) or None if not found
"""
if not HAS_PACKAGING:
# Fallback: if packaging not available, can't do constraint matching
return None
# Handle wildcard - return latest version
if constraint == '*':
# Get the latest version by created_at
latest = db.query(PackageVersion).filter(
PackageVersion.package_id == package.id,
).order_by(PackageVersion.created_at.desc()).first()
if latest:
artifact = db.query(Artifact).filter(Artifact.id == latest.artifact_id).first()
if artifact:
return (artifact.id, latest.version, artifact.size)
return None
try:
specifier = SpecifierSet(constraint)
except InvalidSpecifier:
# Invalid constraint (e.g., ">=" without version) - treat as wildcard
# This can happen with malformed metadata from PyPI packages
latest = db.query(PackageVersion).filter(
PackageVersion.package_id == package.id,
).order_by(PackageVersion.created_at.desc()).first()
if latest:
artifact = db.query(Artifact).filter(Artifact.id == latest.artifact_id).first()
if artifact:
return (artifact.id, latest.version, artifact.size)
return None
# Get all versions for this package
all_versions = db.query(PackageVersion).filter(
PackageVersion.package_id == package.id,
).all()
if not all_versions:
return None
# Find matching versions
matching = []
for pv in all_versions:
try:
v = Version(pv.version)
if v in specifier:
matching.append((pv, v))
except InvalidVersion:
# Skip invalid versions
continue
if not matching:
return None
# Sort by version (descending) and return the latest matching
matching.sort(key=lambda x: x[1], reverse=True)
best_match = matching[0][0]
artifact = db.query(Artifact).filter(Artifact.id == best_match.artifact_id).first()
if artifact:
return (artifact.id, best_match.version, artifact.size)
return None
def _resolve_dependency_to_artifact(
db: Session,
project_name: str,
@@ -314,11 +413,17 @@ def _resolve_dependency_to_artifact(
"""
Resolve a dependency constraint to an artifact ID.
Supports:
- Exact version matching (e.g., '1.2.3')
- Version constraints (e.g., '>=1.9', '<2.0,>=1.5')
- Tag matching
- Wildcard ('*' for any version)
Args:
db: Database session
project_name: Project name
package_name: Package name
version: Version constraint (exact)
version: Version or version constraint
tag: Tag constraint
Returns:
@@ -337,17 +442,23 @@ def _resolve_dependency_to_artifact(
return None
if version:
# Look up by version
pkg_version = db.query(PackageVersion).filter(
PackageVersion.package_id == package.id,
PackageVersion.version == version,
).first()
if pkg_version:
artifact = db.query(Artifact).filter(
Artifact.id == pkg_version.artifact_id
# Check if this is a version constraint (>=, <, etc.) or exact version
if _is_version_constraint(version):
result = _resolve_version_constraint(db, package, version)
if result:
return result
else:
# Look up by exact version
pkg_version = db.query(PackageVersion).filter(
PackageVersion.package_id == package.id,
PackageVersion.version == version,
).first()
if artifact:
return (artifact.id, version, artifact.size)
if pkg_version:
artifact = db.query(Artifact).filter(
Artifact.id == pkg_version.artifact_id
).first()
if artifact:
return (artifact.id, version, artifact.size)
# Also check if there's a tag with this exact name
tag_record = db.query(Tag).filter(
@@ -580,6 +691,8 @@ def resolve_dependencies(
# Track resolved artifacts and their versions
resolved_artifacts: Dict[str, ResolvedArtifact] = {}
# Track missing dependencies (not cached on server)
missing_dependencies: List[MissingDependency] = []
# Track version requirements for conflict detection
version_requirements: Dict[str, List[Dict[str, Any]]] = {} # pkg_key -> [(version, required_by)]
# Track visiting/visited for cycle detection
@@ -654,6 +767,10 @@ def resolve_dependencies(
# Resolve each dependency first (depth-first)
for dep in deps:
# Skip self-dependencies (can happen with PyPI extras like pytest[testing])
if dep.dependency_project == proj_name and dep.dependency_package == pkg_name:
continue
resolved_dep = _resolve_dependency_to_artifact(
db,
dep.dependency_project,
@@ -663,12 +780,15 @@ def resolve_dependencies(
)
if not resolved_dep:
# Dependency not cached on server - track as missing but continue
constraint = dep.version_constraint or dep.tag_constraint
raise DependencyNotFoundError(
dep.dependency_project,
dep.dependency_package,
constraint,
)
missing_dependencies.append(MissingDependency(
project=dep.dependency_project,
package=dep.dependency_package,
constraint=constraint,
required_by=pkg_key,
))
continue
dep_artifact_id, dep_version, dep_size = resolved_dep
_resolve_recursive(
@@ -718,6 +838,7 @@ def resolve_dependencies(
"ref": ref,
},
resolved=resolved_list,
missing=missing_dependencies,
total_size=total_size,
artifact_count=len(resolved_list),
)

View File

@@ -50,7 +50,6 @@ async def lifespan(app: FastAPI):
logger.info(f"Running in {settings.env} mode - skipping seed data")
yield
# Shutdown: cleanup if needed
app = FastAPI(

View File

@@ -7,7 +7,9 @@ Artifacts are cached on first access through configured upstream sources.
import hashlib
import logging
import os
import re
import tempfile
from typing import Optional
from urllib.parse import urljoin, urlparse, quote, unquote
@@ -17,26 +19,50 @@ from fastapi.responses import StreamingResponse, HTMLResponse
from sqlalchemy.orm import Session
from .database import get_db
from .models import UpstreamSource, CachedUrl, Artifact, Project, Package, Tag
from .models import UpstreamSource, CachedUrl, Artifact, Project, Package, Tag, PackageVersion
from .storage import S3Storage, get_storage
from .upstream import (
UpstreamClient,
UpstreamClientConfig,
UpstreamHTTPError,
UpstreamConnectionError,
UpstreamTimeoutError,
)
from .config import get_env_upstream_sources
logger = logging.getLogger(__name__)
router = APIRouter(prefix="/pypi", tags=["pypi-proxy"])
# Timeout configuration for proxy requests
PROXY_CONNECT_TIMEOUT = 30.0
PROXY_READ_TIMEOUT = 60.0
def _extract_pypi_version(filename: str) -> Optional[str]:
"""Extract version from PyPI filename.
Handles formats like:
- cowsay-6.1-py3-none-any.whl
- cowsay-1.0.tar.gz
- some_package-1.2.3.post1-cp39-cp39-linux_x86_64.whl
"""
# Remove extension
if filename.endswith('.whl'):
# Wheel: name-version-pytag-abitag-platform.whl
parts = filename[:-4].split('-')
if len(parts) >= 2:
return parts[1]
elif filename.endswith('.tar.gz'):
# Source: name-version.tar.gz
base = filename[:-7]
# Find the last hyphen that precedes a version-like string
match = re.match(r'^(.+)-(\d+.*)$', base)
if match:
return match.group(2)
elif filename.endswith('.zip'):
# Egg/zip: name-version.zip
base = filename[:-4]
match = re.match(r'^(.+)-(\d+.*)$', base)
if match:
return match.group(2)
return None
def _get_pypi_upstream_sources(db: Session) -> list[UpstreamSource]:
"""Get all enabled upstream sources configured for PyPI."""
# Get database sources
@@ -88,7 +114,27 @@ def _get_basic_auth(source) -> Optional[tuple[str, str]]:
return None
def _rewrite_package_links(html: str, base_url: str, package_name: str) -> str:
def _get_base_url(request: Request) -> str:
"""
Get the external base URL, respecting X-Forwarded-Proto header.
When behind a reverse proxy that terminates SSL, the request.base_url
will show http:// even though the external URL is https://. This function
checks the X-Forwarded-Proto header to determine the correct scheme.
"""
base_url = str(request.base_url).rstrip('/')
# Check for X-Forwarded-Proto header (set by reverse proxies)
forwarded_proto = request.headers.get('x-forwarded-proto')
if forwarded_proto:
# Replace the scheme with the forwarded protocol
parsed = urlparse(base_url)
base_url = f"{forwarded_proto}://{parsed.netloc}{parsed.path}"
return base_url
def _rewrite_package_links(html: str, base_url: str, package_name: str, upstream_base_url: str) -> str:
"""
Rewrite download links in a PyPI simple page to go through our proxy.
@@ -96,6 +142,7 @@ def _rewrite_package_links(html: str, base_url: str, package_name: str) -> str:
html: The HTML content from upstream
base_url: Our server's base URL
package_name: The package name for the URL path
upstream_base_url: The upstream URL used to fetch this page (for resolving relative URLs)
Returns:
HTML with rewritten download links
@@ -103,19 +150,31 @@ def _rewrite_package_links(html: str, base_url: str, package_name: str) -> str:
# Pattern to match href attributes in anchor tags
# PyPI simple pages have links like:
# <a href="https://files.pythonhosted.org/packages/.../file.tar.gz#sha256=...">file.tar.gz</a>
# Or relative URLs from Artifactory like:
# <a href="../../packages/packages/62/35/.../requests-0.10.0.tar.gz#sha256=...">
def replace_href(match):
original_url = match.group(1)
# Resolve relative URLs to absolute using the upstream base URL
if not original_url.startswith(('http://', 'https://')):
# Split off fragment before resolving
url_without_fragment = original_url.split('#')[0]
fragment_part = original_url[len(url_without_fragment):]
absolute_url = urljoin(upstream_base_url, url_without_fragment) + fragment_part
else:
absolute_url = original_url
# Extract the filename from the URL
parsed = urlparse(original_url)
parsed = urlparse(absolute_url)
path_parts = parsed.path.split('/')
filename = path_parts[-1] if path_parts else ''
# Keep the hash fragment if present
fragment = f"#{parsed.fragment}" if parsed.fragment else ""
# Encode the original URL for safe transmission
encoded_url = quote(original_url.split('#')[0], safe='')
# Encode the absolute URL (without fragment) for safe transmission
encoded_url = quote(absolute_url.split('#')[0], safe='')
# Build new URL pointing to our proxy
new_url = f"{base_url}/pypi/simple/{package_name}/{filename}?upstream={encoded_url}{fragment}"
@@ -154,12 +213,13 @@ async def pypi_simple_index(
headers.update(_build_auth_headers(source))
auth = _get_basic_auth(source)
simple_url = source.url.rstrip('/') + '/simple/'
# Use URL as-is - users should provide full path including /simple
simple_url = source.url.rstrip('/') + '/'
timeout = httpx.Timeout(PROXY_READ_TIMEOUT, connect=PROXY_CONNECT_TIMEOUT)
with httpx.Client(timeout=timeout, follow_redirects=False) as client:
response = client.get(
async with httpx.AsyncClient(timeout=timeout, follow_redirects=False) as client:
response = await client.get(
simple_url,
headers=headers,
auth=auth,
@@ -170,7 +230,7 @@ async def pypi_simple_index(
redirect_url = response.headers.get('location')
if redirect_url:
# Follow the redirect once
response = client.get(
response = await client.get(
redirect_url,
headers=headers,
auth=auth,
@@ -183,7 +243,7 @@ async def pypi_simple_index(
content = response.text
# Rewrite package links to go through our proxy
base_url = str(request.base_url).rstrip('/')
base_url = _get_base_url(request)
content = re.sub(
r'href="([^"]+)/"',
lambda m: f'href="{base_url}/pypi/simple/{m.group(1)}/"',
@@ -229,7 +289,7 @@ async def pypi_package_versions(
detail="No PyPI upstream sources configured"
)
base_url = str(request.base_url).rstrip('/')
base_url = _get_base_url(request)
# Normalize package name (PEP 503)
normalized_name = re.sub(r'[-_.]+', '-', package_name).lower()
@@ -242,12 +302,14 @@ async def pypi_package_versions(
headers.update(_build_auth_headers(source))
auth = _get_basic_auth(source)
package_url = source.url.rstrip('/') + f'/simple/{normalized_name}/'
# Use URL as-is - users should provide full path including /simple
package_url = source.url.rstrip('/') + f'/{normalized_name}/'
final_url = package_url # Track final URL after redirects
timeout = httpx.Timeout(PROXY_READ_TIMEOUT, connect=PROXY_CONNECT_TIMEOUT)
with httpx.Client(timeout=timeout, follow_redirects=False) as client:
response = client.get(
async with httpx.AsyncClient(timeout=timeout, follow_redirects=False) as client:
response = await client.get(
package_url,
headers=headers,
auth=auth,
@@ -262,9 +324,11 @@ async def pypi_package_versions(
# Make redirect URL absolute if needed
if not redirect_url.startswith('http'):
redirect_url = urljoin(package_url, redirect_url)
redirect_url = urljoin(final_url, redirect_url)
response = client.get(
final_url = redirect_url # Update final URL
response = await client.get(
redirect_url,
headers=headers,
auth=auth,
@@ -276,7 +340,8 @@ async def pypi_package_versions(
content = response.text
# Rewrite download links to go through our proxy
content = _rewrite_package_links(content, base_url, normalized_name)
# Pass final_url so relative URLs can be resolved correctly
content = _rewrite_package_links(content, base_url, normalized_name, final_url)
return HTMLResponse(content=content)
@@ -341,14 +406,22 @@ async def pypi_download_file(
# Stream from S3
try:
content_stream = storage.get_artifact_stream(artifact.id)
stream, content_length, _ = storage.get_stream(artifact.s3_key)
def stream_content():
"""Generator that yields chunks from the S3 stream."""
try:
for chunk in stream.iter_chunks():
yield chunk
finally:
stream.close()
return StreamingResponse(
content_stream,
stream_content(),
media_type=artifact.content_type or "application/octet-stream",
headers={
"Content-Disposition": f'attachment; filename="{filename}"',
"Content-Length": str(artifact.size),
"Content-Length": str(content_length),
"X-Checksum-SHA256": artifact.id,
"X-Cache": "HIT",
}
@@ -360,18 +433,10 @@ async def pypi_download_file(
# Not cached - fetch from upstream
sources = _get_pypi_upstream_sources(db)
# Find a source that matches the upstream URL
matched_source = None
for source in sources:
source_url = getattr(source, 'url', '')
# Check if the upstream URL could come from this source
# (This is a loose check - the URL might be from files.pythonhosted.org)
if urlparse(upstream_url).netloc in source_url or True: # Allow any source for now
matched_source = source
break
if not matched_source and sources:
matched_source = sources[0] # Use first source for auth if available
# Use the first available source for authentication headers
# Note: The upstream URL may point to files.pythonhosted.org or other CDNs,
# not the configured source URL directly, so we can't strictly validate the host
matched_source = sources[0] if sources else None
try:
headers = {"User-Agent": "Orchard-PyPI-Proxy/1.0"}
@@ -384,8 +449,8 @@ async def pypi_download_file(
# Fetch the file
logger.info(f"PyPI proxy: fetching {filename} from {upstream_url}")
with httpx.Client(timeout=timeout, follow_redirects=False) as client:
response = client.get(
async with httpx.AsyncClient(timeout=timeout, follow_redirects=False) as client:
response = await client.get(
upstream_url,
headers=headers,
auth=auth,
@@ -410,7 +475,7 @@ async def pypi_download_file(
redirect_headers.update(headers)
redirect_auth = auth
response = client.get(
response = await client.get(
redirect_url,
headers=redirect_headers,
auth=redirect_auth,
@@ -424,22 +489,33 @@ async def pypi_download_file(
detail=f"Upstream returned {response.status_code}"
)
content = response.content
content_type = response.headers.get('content-type', 'application/octet-stream')
# Compute hash
sha256 = hashlib.sha256(content).hexdigest()
size = len(content)
# Stream to temp file to avoid loading large packages into memory
# This keeps memory usage constant regardless of package size
# Using async iteration to avoid blocking the event loop
tmp_path = None
try:
with tempfile.NamedTemporaryFile(delete=False, suffix=f"_{filename}") as tmp_file:
tmp_path = tmp_file.name
async for chunk in response.aiter_bytes(chunk_size=65536): # 64KB chunks
tmp_file.write(chunk)
logger.info(f"PyPI proxy: downloaded {filename}, {size} bytes, sha256={sha256[:12]}")
# Store in S3 from temp file (computes hash and deduplicates automatically)
with open(tmp_path, 'rb') as f:
result = storage.store(f)
sha256 = result.sha256
size = result.size
# Store in S3
from io import BytesIO
artifact = storage.store_artifact(
file_obj=BytesIO(content),
filename=filename,
content_type=content_type,
)
# Read content for response
with open(tmp_path, 'rb') as f:
content = f.read()
logger.info(f"PyPI proxy: downloaded {filename}, {size} bytes, sha256={sha256[:12]}")
finally:
# Clean up temp file
if tmp_path and os.path.exists(tmp_path):
os.unlink(tmp_path)
# Check if artifact already exists
existing = db.query(Artifact).filter(Artifact.id == sha256).first()
@@ -451,10 +527,15 @@ async def pypi_download_file(
# Create artifact record
new_artifact = Artifact(
id=sha256,
filename=filename,
original_name=filename,
content_type=content_type,
size=size,
ref_count=1,
created_by="pypi-proxy",
s3_key=result.s3_key,
checksum_md5=result.md5,
checksum_sha1=result.sha1,
s3_etag=result.s3_etag,
)
db.add(new_artifact)
db.flush()
@@ -465,10 +546,16 @@ async def pypi_download_file(
system_project = Project(
name="_pypi",
description="System project for cached PyPI packages",
visibility="private",
is_public=True,
is_system=True,
created_by="pypi-proxy",
)
db.add(system_project)
db.flush()
elif not system_project.is_system:
# Ensure existing project is marked as system
system_project.is_system = True
db.flush()
# Normalize package name
normalized_name = re.sub(r'[-_.]+', '-', package_name).lower()
@@ -482,6 +569,7 @@ async def pypi_download_file(
project_id=system_project.id,
name=normalized_name,
description=f"PyPI package: {normalized_name}",
format="pypi",
)
db.add(package)
db.flush()
@@ -496,9 +584,29 @@ async def pypi_download_file(
package_id=package.id,
name=filename,
artifact_id=sha256,
created_by="pypi-proxy",
)
db.add(tag)
# Extract and create version
# Only create version for actual package files, not .metadata files
version = _extract_pypi_version(filename)
if version and not filename.endswith('.metadata'):
# Check by version string (the unique constraint is on package_id + version)
existing_version = db.query(PackageVersion).filter(
PackageVersion.package_id == package.id,
PackageVersion.version == version,
).first()
if not existing_version:
pkg_version = PackageVersion(
package_id=package.id,
artifact_id=sha256,
version=version,
version_source="filename",
created_by="pypi-proxy",
)
db.add(pkg_version)
# Cache the URL mapping
existing_cached = db.query(CachedUrl).filter(CachedUrl.url_hash == url_hash).first()
if not existing_cached:

View File

@@ -1680,6 +1680,7 @@ def create_project(
name=db_project.name,
description=db_project.description,
is_public=db_project.is_public,
is_system=db_project.is_system,
created_at=db_project.created_at,
updated_at=db_project.updated_at,
created_by=db_project.created_by,
@@ -1704,6 +1705,7 @@ def get_project(
name=project.name,
description=project.description,
is_public=project.is_public,
is_system=project.is_system,
created_at=project.created_at,
updated_at=project.updated_at,
created_by=project.created_by,
@@ -2704,6 +2706,7 @@ def list_team_projects(
name=p.name,
description=p.description,
is_public=p.is_public,
is_system=p.is_system,
created_at=p.created_at,
updated_at=p.updated_at,
created_by=p.created_by,
@@ -2827,14 +2830,15 @@ def list_packages(
db.query(func.count(Tag.id)).filter(Tag.package_id == pkg.id).scalar() or 0
)
# Get unique artifact count and total size via uploads
# Get unique artifact count and total size via tags
# (PyPI proxy creates tags without uploads, so query from tags)
artifact_stats = (
db.query(
func.count(func.distinct(Upload.artifact_id)),
func.count(func.distinct(Tag.artifact_id)),
func.coalesce(func.sum(Artifact.size), 0),
)
.join(Artifact, Upload.artifact_id == Artifact.id)
.filter(Upload.package_id == pkg.id)
.join(Artifact, Tag.artifact_id == Artifact.id)
.filter(Tag.package_id == pkg.id)
.first()
)
artifact_count = artifact_stats[0] if artifact_stats else 0
@@ -2930,14 +2934,15 @@ def get_package(
db.query(func.count(Tag.id)).filter(Tag.package_id == pkg.id).scalar() or 0
)
# Get unique artifact count and total size via uploads
# Get unique artifact count and total size via tags
# (PyPI proxy creates tags without uploads, so query from tags)
artifact_stats = (
db.query(
func.count(func.distinct(Upload.artifact_id)),
func.count(func.distinct(Tag.artifact_id)),
func.coalesce(func.sum(Artifact.size), 0),
)
.join(Artifact, Upload.artifact_id == Artifact.id)
.filter(Upload.package_id == pkg.id)
.join(Artifact, Tag.artifact_id == Artifact.id)
.filter(Tag.package_id == pkg.id)
.first()
)
artifact_count = artifact_stats[0] if artifact_stats else 0
@@ -6280,14 +6285,14 @@ def get_package_stats(
db.query(func.count(Tag.id)).filter(Tag.package_id == package.id).scalar() or 0
)
# Artifact stats via uploads
# Artifact stats via tags (tags exist for both user uploads and PyPI proxy)
artifact_stats = (
db.query(
func.count(func.distinct(Upload.artifact_id)),
func.count(func.distinct(Tag.artifact_id)),
func.coalesce(func.sum(Artifact.size), 0),
)
.join(Artifact, Upload.artifact_id == Artifact.id)
.filter(Upload.package_id == package.id)
.join(Artifact, Tag.artifact_id == Artifact.id)
.filter(Tag.package_id == package.id)
.first()
)
artifact_count = artifact_stats[0] if artifact_stats else 0

View File

@@ -33,6 +33,7 @@ class ProjectResponse(BaseModel):
name: str
description: Optional[str]
is_public: bool
is_system: bool = False
created_at: datetime
updated_at: datetime
created_by: str
@@ -1032,10 +1033,19 @@ class ResolvedArtifact(BaseModel):
download_url: str
class MissingDependency(BaseModel):
"""A dependency that could not be resolved (not cached on server)"""
project: str
package: str
constraint: Optional[str] = None
required_by: Optional[str] = None
class DependencyResolutionResponse(BaseModel):
"""Response from dependency resolution endpoint"""
requested: Dict[str, str] # project, package, ref
resolved: List[ResolvedArtifact]
missing: List[MissingDependency] = []
total_size: int
artifact_count: int

View File

@@ -0,0 +1 @@
# Scripts package

View File

@@ -0,0 +1,262 @@
#!/usr/bin/env python3
"""
Backfill script to extract dependencies from cached PyPI packages.
This script scans all artifacts in the _pypi project and extracts
Requires-Dist metadata from wheel and sdist files that don't already
have dependencies recorded.
Usage:
# From within the container:
python -m scripts.backfill_pypi_dependencies
# Or with docker exec:
docker exec orchard_orchard-server_1 python -m scripts.backfill_pypi_dependencies
# Dry run (preview only):
docker exec orchard_orchard-server_1 python -m scripts.backfill_pypi_dependencies --dry-run
"""
import argparse
import logging
import re
import sys
import tarfile
import zipfile
from io import BytesIO
from typing import List, Optional, Tuple
# Add parent directory to path for imports
sys.path.insert(0, "/app")
from sqlalchemy import create_engine
from sqlalchemy.orm import sessionmaker
from backend.app.config import get_settings
from backend.app.models import (
Artifact,
ArtifactDependency,
Package,
Project,
Tag,
)
from backend.app.storage import get_storage
logging.basicConfig(
level=logging.INFO,
format="%(asctime)s - %(levelname)s - %(message)s",
)
logger = logging.getLogger(__name__)
def parse_requires_dist(requires_dist: str) -> Tuple[Optional[str], Optional[str]]:
"""Parse a Requires-Dist line into (package_name, version_constraint)."""
# Remove any environment markers (after semicolon)
if ";" in requires_dist:
requires_dist = requires_dist.split(";")[0].strip()
# Match patterns like "package (>=1.0)" or "package>=1.0" or "package"
match = re.match(
r"^([a-zA-Z0-9][-a-zA-Z0-9._]*)\s*(?:\(([^)]+)\)|([<>=!~][^\s;]+))?",
requires_dist.strip(),
)
if not match:
return None, None
package_name = match.group(1)
version_constraint = match.group(2) or match.group(3)
# Normalize package name (PEP 503)
normalized_name = re.sub(r"[-_.]+", "-", package_name).lower()
if version_constraint:
version_constraint = version_constraint.strip()
return normalized_name, version_constraint
def extract_requires_from_metadata(metadata_content: str) -> List[Tuple[str, Optional[str]]]:
"""Extract all Requires-Dist entries from METADATA/PKG-INFO content."""
dependencies = []
for line in metadata_content.split("\n"):
if line.startswith("Requires-Dist:"):
value = line[len("Requires-Dist:"):].strip()
pkg_name, version = parse_requires_dist(value)
if pkg_name:
dependencies.append((pkg_name, version))
return dependencies
def extract_metadata_from_wheel(content: bytes) -> Optional[str]:
"""Extract METADATA file content from a wheel (zip) file."""
try:
with zipfile.ZipFile(BytesIO(content)) as zf:
for name in zf.namelist():
if name.endswith(".dist-info/METADATA"):
return zf.read(name).decode("utf-8", errors="replace")
except Exception as e:
logger.warning(f"Failed to extract metadata from wheel: {e}")
return None
def extract_metadata_from_sdist(content: bytes) -> Optional[str]:
"""Extract PKG-INFO file content from a source distribution (.tar.gz)."""
try:
with tarfile.open(fileobj=BytesIO(content), mode="r:gz") as tf:
for member in tf.getmembers():
if member.name.endswith("/PKG-INFO") and member.name.count("/") == 1:
f = tf.extractfile(member)
if f:
return f.read().decode("utf-8", errors="replace")
except Exception as e:
logger.warning(f"Failed to extract metadata from sdist: {e}")
return None
def extract_dependencies(content: bytes, filename: str) -> List[Tuple[str, Optional[str]]]:
"""Extract dependencies from a PyPI package file."""
metadata = None
if filename.endswith(".whl"):
metadata = extract_metadata_from_wheel(content)
elif filename.endswith(".tar.gz"):
metadata = extract_metadata_from_sdist(content)
if metadata:
return extract_requires_from_metadata(metadata)
return []
def backfill_dependencies(dry_run: bool = False):
"""Main backfill function."""
settings = get_settings()
# Create database connection
engine = create_engine(settings.database_url)
Session = sessionmaker(bind=engine)
db = Session()
# Create storage client
storage = get_storage()
try:
# Find the _pypi project
pypi_project = db.query(Project).filter(Project.name == "_pypi").first()
if not pypi_project:
logger.info("No _pypi project found. Nothing to backfill.")
return
# Get all packages in _pypi
packages = db.query(Package).filter(Package.project_id == pypi_project.id).all()
logger.info(f"Found {len(packages)} packages in _pypi project")
total_artifacts = 0
artifacts_with_deps = 0
artifacts_processed = 0
dependencies_added = 0
for package in packages:
# Get all tags (each tag points to an artifact)
tags = db.query(Tag).filter(Tag.package_id == package.id).all()
for tag in tags:
total_artifacts += 1
filename = tag.name
# Skip non-package files (like .metadata files)
if not (filename.endswith(".whl") or filename.endswith(".tar.gz")):
continue
# Check if this artifact already has dependencies
existing_deps = db.query(ArtifactDependency).filter(
ArtifactDependency.artifact_id == tag.artifact_id
).count()
if existing_deps > 0:
artifacts_with_deps += 1
continue
# Get the artifact
artifact = db.query(Artifact).filter(Artifact.id == tag.artifact_id).first()
if not artifact:
logger.warning(f"Artifact {tag.artifact_id} not found for tag {filename}")
continue
logger.info(f"Processing {package.name}/{filename}...")
if dry_run:
logger.info(f" [DRY RUN] Would extract dependencies from {filename}")
artifacts_processed += 1
continue
# Download the artifact from S3
try:
content = storage.get(artifact.s3_key)
except Exception as e:
logger.error(f" Failed to download {filename}: {e}")
continue
# Extract dependencies
deps = extract_dependencies(content, filename)
if deps:
logger.info(f" Found {len(deps)} dependencies")
for dep_name, dep_version in deps:
# Check if already exists (race condition protection)
existing = db.query(ArtifactDependency).filter(
ArtifactDependency.artifact_id == tag.artifact_id,
ArtifactDependency.dependency_project == "_pypi",
ArtifactDependency.dependency_package == dep_name,
).first()
if not existing:
dep = ArtifactDependency(
artifact_id=tag.artifact_id,
dependency_project="_pypi",
dependency_package=dep_name,
version_constraint=dep_version if dep_version else "*",
)
db.add(dep)
dependencies_added += 1
logger.info(f" + {dep_name} {dep_version or '*'}")
db.commit()
else:
logger.info(f" No dependencies found")
artifacts_processed += 1
logger.info("")
logger.info("=" * 50)
logger.info("Backfill complete!")
logger.info(f" Total artifacts: {total_artifacts}")
logger.info(f" Already had deps: {artifacts_with_deps}")
logger.info(f" Processed: {artifacts_processed}")
logger.info(f" Dependencies added: {dependencies_added}")
if dry_run:
logger.info(" (DRY RUN - no changes made)")
finally:
db.close()
def main():
parser = argparse.ArgumentParser(
description="Backfill dependencies for cached PyPI packages"
)
parser.add_argument(
"--dry-run",
action="store_true",
help="Preview what would be done without making changes",
)
args = parser.parse_args()
backfill_dependencies(dry_run=args.dry_run)
if __name__ == "__main__":
main()

View File

@@ -128,7 +128,9 @@ class TestProjectListingFilters:
assert response.status_code == 200
data = response.json()
names = [p["name"] for p in data["items"]]
# Filter out system projects (names starting with "_") as they may have
# collation-specific sort behavior and aren't part of the test data
names = [p["name"] for p in data["items"] if not p["name"].startswith("_")]
assert names == sorted(names)

View File

@@ -17,21 +17,31 @@ class TestPyPIProxyEndpoints:
"""
@pytest.mark.integration
def test_pypi_simple_index_no_sources(self):
"""Test that /pypi/simple/ returns 503 when no sources configured."""
def test_pypi_simple_index(self):
"""Test that /pypi/simple/ returns HTML response."""
with httpx.Client(base_url=get_base_url(), timeout=30.0) as client:
response = client.get("/pypi/simple/")
# Should return 503 when no PyPI upstream sources are configured
assert response.status_code == 503
assert "No PyPI upstream sources configured" in response.json()["detail"]
# Returns 200 if sources configured, 503 if not
assert response.status_code in (200, 503)
if response.status_code == 200:
assert "text/html" in response.headers.get("content-type", "")
else:
assert "No PyPI upstream sources configured" in response.json()["detail"]
@pytest.mark.integration
def test_pypi_package_no_sources(self):
"""Test that /pypi/simple/{package}/ returns 503 when no sources configured."""
def test_pypi_package_endpoint(self):
"""Test that /pypi/simple/{package}/ returns appropriate response."""
with httpx.Client(base_url=get_base_url(), timeout=30.0) as client:
response = client.get("/pypi/simple/requests/")
assert response.status_code == 503
assert "No PyPI upstream sources configured" in response.json()["detail"]
# Returns 200 if sources configured and package found,
# 404 if package not found, 503 if no sources
assert response.status_code in (200, 404, 503)
if response.status_code == 200:
assert "text/html" in response.headers.get("content-type", "")
elif response.status_code == 404:
assert "not found" in response.json()["detail"].lower()
else: # 503
assert "No PyPI upstream sources configured" in response.json()["detail"]
@pytest.mark.integration
def test_pypi_download_missing_upstream_param(self):
@@ -58,7 +68,13 @@ class TestPyPILinkRewriting:
</html>
'''
result = _rewrite_package_links(html, "http://localhost:8080", "requests")
# upstream_base_url is used to resolve relative URLs (not needed here since URLs are absolute)
result = _rewrite_package_links(
html,
"http://localhost:8080",
"requests",
"https://pypi.org/simple/requests/"
)
# Links should be rewritten to go through our proxy
assert "/pypi/simple/requests/requests-2.31.0.tar.gz?upstream=" in result
@@ -69,25 +85,53 @@ class TestPyPILinkRewriting:
assert "#sha256=abc123" in result
assert "#sha256=def456" in result
def test_rewrite_relative_links(self):
"""Test that relative URLs are resolved to absolute URLs."""
from app.pypi_proxy import _rewrite_package_links
# Artifactory-style relative URLs
html = '''
<html>
<body>
<a href="../../packages/ab/cd/requests-2.31.0.tar.gz#sha256=abc123">requests-2.31.0.tar.gz</a>
</body>
</html>
'''
result = _rewrite_package_links(
html,
"https://orchard.example.com",
"requests",
"https://artifactory.example.com/api/pypi/pypi-remote/simple/requests/"
)
# The relative URL should be resolved to absolute
# ../../packages/ab/cd/... from /api/pypi/pypi-remote/simple/requests/ resolves to /api/pypi/pypi-remote/packages/ab/cd/...
assert "upstream=https%3A%2F%2Fartifactory.example.com%2Fapi%2Fpypi%2Fpypi-remote%2Fpackages" in result
# Hash fragment should be preserved
assert "#sha256=abc123" in result
class TestPyPIPackageNormalization:
"""Tests for PyPI package name normalization."""
@pytest.mark.integration
def test_package_name_normalized(self):
"""Test that package names are normalized per PEP 503."""
# These should all be treated the same:
# requests, Requests, requests_, requests-
# The endpoint normalizes to lowercase with hyphens
"""Test that package names are normalized per PEP 503.
Different capitalizations/separators should all be valid paths.
The endpoint normalizes to lowercase with hyphens before lookup.
"""
with httpx.Client(base_url=get_base_url(), timeout=30.0) as client:
# Without upstream sources, we get 503, but the normalization
# happens before the source lookup
response = client.get("/pypi/simple/Requests/")
assert response.status_code == 503 # No sources, but path was valid
# Test various name formats - all should be valid endpoint paths
for package_name in ["Requests", "some_package", "some-package"]:
response = client.get(f"/pypi/simple/{package_name}/")
# 200 = found, 404 = not found, 503 = no sources configured
assert response.status_code in (200, 404, 503), \
f"Unexpected status {response.status_code} for {package_name}"
response = client.get("/pypi/simple/some_package/")
assert response.status_code == 503
response = client.get("/pypi/simple/some-package/")
assert response.status_code == 503
# Verify response is appropriate for the status code
if response.status_code == 200:
assert "text/html" in response.headers.get("content-type", "")
elif response.status_code == 503:
assert "No PyPI upstream sources configured" in response.json()["detail"]

View File

@@ -78,7 +78,13 @@ export class ForbiddenError extends ApiError {
async function handleResponse<T>(response: Response): Promise<T> {
if (!response.ok) {
const error = await response.json().catch(() => ({ detail: 'Unknown error' }));
const message = error.detail || `HTTP ${response.status}`;
// Handle detail as string or object (backend may return structured errors)
let message: string;
if (typeof error.detail === 'object') {
message = JSON.stringify(error.detail);
} else {
message = error.detail || `HTTP ${response.status}`;
}
if (response.status === 401) {
throw new UnauthorizedError(message);
@@ -746,3 +752,4 @@ export async function testUpstreamSource(id: string): Promise<UpstreamSourceTest
});
return handleResponse<UpstreamSourceTestResult>(response);
}

View File

@@ -55,6 +55,10 @@
font-size: 0.8125rem;
}
.missing-count {
color: #f59e0b;
}
.close-btn {
background: transparent;
border: none;
@@ -314,6 +318,63 @@
font-size: 0.75rem;
}
/* Missing Dependencies */
.missing-dependencies {
border-top: 1px solid var(--border-primary);
padding: 16px 20px;
background: rgba(245, 158, 11, 0.05);
max-height: 200px;
overflow-y: auto;
}
.missing-dependencies h3 {
margin: 0 0 8px 0;
font-size: 0.875rem;
font-weight: 600;
color: #f59e0b;
}
.missing-hint {
margin: 0 0 12px 0;
font-size: 0.75rem;
color: var(--text-muted);
}
.missing-list {
list-style: none;
padding: 0;
margin: 0;
display: flex;
flex-wrap: wrap;
gap: 8px;
}
.missing-item {
display: inline-flex;
align-items: center;
gap: 4px;
background: var(--bg-tertiary);
border: 1px solid rgba(245, 158, 11, 0.3);
border-radius: var(--radius-sm);
padding: 4px 8px;
font-size: 0.75rem;
}
.missing-name {
font-family: 'JetBrains Mono', monospace;
color: var(--text-secondary);
}
.missing-constraint {
color: var(--text-muted);
font-family: 'JetBrains Mono', monospace;
}
.missing-required-by {
color: var(--text-muted);
font-size: 0.6875rem;
}
/* Responsive */
@media (max-width: 768px) {
.dependency-graph-modal {

View File

@@ -244,7 +244,11 @@ function DependencyGraph({ projectName, packageName, tagName, onClose }: Depende
<span>{projectName}/{packageName} @ {tagName}</span>
{resolution && (
<span className="graph-stats">
{resolution.artifact_count} packages {formatBytes(resolution.total_size)} total
{resolution.artifact_count} cached
{resolution.missing && resolution.missing.length > 0 && (
<span className="missing-count"> {resolution.missing.length} not cached</span>
)}
{formatBytes(resolution.total_size)} total
</span>
)}
</div>
@@ -307,6 +311,22 @@ function DependencyGraph({ projectName, packageName, tagName, onClose }: Depende
)}
</div>
{resolution && resolution.missing && resolution.missing.length > 0 && (
<div className="missing-dependencies">
<h3>Not Cached ({resolution.missing.length})</h3>
<p className="missing-hint">These dependencies are referenced but not yet cached on the server.</p>
<ul className="missing-list">
{resolution.missing.map((dep, i) => (
<li key={i} className="missing-item">
<span className="missing-name">{dep.project}/{dep.package}</span>
{dep.constraint && <span className="missing-constraint">@{dep.constraint}</span>}
{dep.required_by && <span className="missing-required-by"> {dep.required_by}</span>}
</li>
))}
</ul>
</div>
)}
{hoveredNode && (
<div className="graph-tooltip">
<strong>{hoveredNode.project}/{hoveredNode.package}</strong>

View File

@@ -132,6 +132,12 @@
color: #c62828;
}
.coming-soon-badge {
color: #9e9e9e;
font-style: italic;
font-size: 0.85em;
}
/* Actions */
.actions-cell {
white-space: nowrap;

View File

@@ -12,6 +12,7 @@ import { UpstreamSource, SourceType, AuthType } from '../types';
import './AdminCachePage.css';
const SOURCE_TYPES: SourceType[] = ['npm', 'pypi', 'maven', 'docker', 'helm', 'nuget', 'deb', 'rpm', 'generic'];
const SUPPORTED_SOURCE_TYPES: Set<SourceType> = new Set(['pypi', 'generic']);
const AUTH_TYPES: AuthType[] = ['none', 'basic', 'bearer', 'api_key'];
function AdminCachePage() {
@@ -285,7 +286,12 @@ function AdminCachePage() {
<span className="env-badge" title="Defined via environment variable">ENV</span>
)}
</td>
<td>{source.source_type}</td>
<td>
{source.source_type}
{!SUPPORTED_SOURCE_TYPES.has(source.source_type) && (
<span className="coming-soon-badge"> (coming soon)</span>
)}
</td>
<td className="url-cell" title={source.url}>{source.url}</td>
<td>{source.priority}</td>
<td>
@@ -359,7 +365,7 @@ function AdminCachePage() {
>
{SOURCE_TYPES.map((type) => (
<option key={type} value={type}>
{type}
{type}{!SUPPORTED_SOURCE_TYPES.has(type) ? ' (coming soon)' : ''}
</option>
))}
</select>

View File

@@ -249,7 +249,7 @@ function Home() {
key: 'created_by',
header: 'Owner',
className: 'cell-owner',
render: (project) => project.created_by,
render: (project) => project.team_name || project.created_by,
},
...(user
? [

View File

@@ -642,6 +642,11 @@ tr:hover .copy-btn {
padding: 20px;
}
/* Ensure file modal needs higher z-index when opened from deps modal */
.modal-overlay:has(.ensure-file-modal) {
z-index: 1100;
}
.ensure-file-modal {
background: var(--bg-secondary);
border: 1px solid var(--border-primary);
@@ -793,4 +798,194 @@ tr:hover .copy-btn {
.ensure-file-modal {
max-height: 90vh;
}
.action-menu-dropdown {
right: 0;
left: auto;
}
}
/* Header upload button */
.header-upload-btn {
margin-left: auto;
}
/* Tag/Version cell */
.tag-version-cell {
display: flex;
flex-direction: column;
gap: 4px;
}
.tag-version-cell .version-badge {
font-size: 0.75rem;
color: var(--text-muted);
}
/* Icon buttons */
.btn-icon {
display: flex;
align-items: center;
justify-content: center;
width: 32px;
height: 32px;
padding: 0;
background: transparent;
border: 1px solid transparent;
border-radius: var(--radius-sm);
color: var(--text-secondary);
cursor: pointer;
transition: all var(--transition-fast);
}
.btn-icon:hover {
background: var(--bg-hover);
color: var(--text-primary);
}
/* Action menu */
.action-buttons {
display: flex;
align-items: center;
gap: 4px;
}
.action-menu {
position: relative;
}
/* Action menu backdrop for click-outside */
.action-menu-backdrop {
position: fixed;
top: 0;
left: 0;
right: 0;
bottom: 0;
z-index: 999;
}
.action-menu-dropdown {
position: fixed;
z-index: 1000;
min-width: 180px;
padding: 4px 0;
background: var(--bg-secondary);
border: 1px solid var(--border-primary);
border-radius: var(--radius-md);
box-shadow: 0 4px 12px rgba(0, 0, 0, 0.15);
}
.action-menu-dropdown button {
display: block;
width: 100%;
padding: 8px 12px;
background: none;
border: none;
text-align: left;
font-size: 0.875rem;
color: var(--text-primary);
cursor: pointer;
transition: background var(--transition-fast);
}
.action-menu-dropdown button:hover {
background: var(--bg-hover);
}
/* Upload Modal */
.upload-modal,
.create-tag-modal {
background: var(--bg-secondary);
border-radius: var(--radius-lg);
width: 90%;
max-width: 500px;
max-height: 90vh;
overflow: hidden;
}
.modal-header {
display: flex;
align-items: center;
justify-content: space-between;
padding: 16px 20px;
border-bottom: 1px solid var(--border-primary);
}
.modal-header h3 {
margin: 0;
font-size: 1.125rem;
font-weight: 600;
}
.modal-body {
padding: 20px;
}
.modal-description {
margin-bottom: 16px;
color: var(--text-secondary);
font-size: 0.875rem;
}
.modal-actions {
display: flex;
justify-content: flex-end;
gap: 12px;
margin-top: 20px;
padding-top: 16px;
border-top: 1px solid var(--border-primary);
}
/* Dependencies Modal */
.deps-modal {
background: var(--bg-secondary);
border-radius: var(--radius-lg);
width: 90%;
max-width: 600px;
max-height: 80vh;
overflow: hidden;
display: flex;
flex-direction: column;
}
.deps-modal .modal-body {
overflow-y: auto;
flex: 1;
}
.deps-modal-controls {
display: flex;
gap: 8px;
margin-bottom: 16px;
}
/* Artifact ID Modal */
.artifact-id-modal {
background: var(--bg-secondary);
border-radius: var(--radius-lg);
width: 90%;
max-width: 500px;
}
.artifact-id-display {
display: flex;
align-items: center;
gap: 12px;
padding: 16px;
background: var(--bg-tertiary);
border-radius: var(--radius-md);
border: 1px solid var(--border-primary);
}
.artifact-id-display code {
font-family: 'JetBrains Mono', 'Fira Code', 'Consolas', monospace;
font-size: 0.8125rem;
color: var(--text-primary);
word-break: break-all;
flex: 1;
}
.artifact-id-display .copy-btn {
opacity: 1;
flex-shrink: 0;
}

View File

@@ -63,12 +63,17 @@ function PackagePage() {
const [accessDenied, setAccessDenied] = useState(false);
const [uploadTag, setUploadTag] = useState('');
const [uploadSuccess, setUploadSuccess] = useState<string | null>(null);
const [artifactIdInput, setArtifactIdInput] = useState('');
const [accessLevel, setAccessLevel] = useState<AccessLevel | null>(null);
const [createTagName, setCreateTagName] = useState('');
const [createTagArtifactId, setCreateTagArtifactId] = useState('');
const [createTagLoading, setCreateTagLoading] = useState(false);
// UI state
const [showUploadModal, setShowUploadModal] = useState(false);
const [showCreateTagModal, setShowCreateTagModal] = useState(false);
const [openMenuId, setOpenMenuId] = useState<string | null>(null);
const [menuPosition, setMenuPosition] = useState<{ top: number; left: number } | null>(null);
// Dependencies state
const [selectedTag, setSelectedTag] = useState<TagDetail | null>(null);
const [dependencies, setDependencies] = useState<Dependency[]>([]);
@@ -86,6 +91,13 @@ function PackagePage() {
// Dependency graph modal state
const [showGraph, setShowGraph] = useState(false);
// Dependencies modal state
const [showDepsModal, setShowDepsModal] = useState(false);
// Artifact ID modal state
const [showArtifactIdModal, setShowArtifactIdModal] = useState(false);
const [viewArtifactId, setViewArtifactId] = useState<string | null>(null);
// Ensure file modal state
const [showEnsureFile, setShowEnsureFile] = useState(false);
const [ensureFileContent, setEnsureFileContent] = useState<string | null>(null);
@@ -96,6 +108,9 @@ function PackagePage() {
// Derived permissions
const canWrite = accessLevel === 'write' || accessLevel === 'admin';
// Detect system projects (convention: name starts with "_")
const isSystemProject = projectName?.startsWith('_') ?? false;
// Get params from URL
const page = parseInt(searchParams.get('page') || '1', 10);
const search = searchParams.get('search') || '';
@@ -323,92 +338,212 @@ function PackagePage() {
setSelectedTag(tag);
};
const columns = [
{
key: 'name',
header: 'Tag',
sortable: true,
render: (t: TagDetail) => (
<strong
className={`tag-name-link ${selectedTag?.id === t.id ? 'selected' : ''}`}
onClick={() => handleTagSelect(t)}
style={{ cursor: 'pointer' }}
const handleMenuOpen = (e: React.MouseEvent, tagId: string) => {
e.stopPropagation();
if (openMenuId === tagId) {
setOpenMenuId(null);
setMenuPosition(null);
} else {
const rect = e.currentTarget.getBoundingClientRect();
setMenuPosition({ top: rect.bottom + 4, left: rect.right - 180 });
setOpenMenuId(tagId);
}
};
// System projects show Version first, regular projects show Tag first
const columns = isSystemProject
? [
// System project columns: Version first, then Filename
{
key: 'version',
header: 'Version',
sortable: true,
render: (t: TagDetail) => (
<strong
className={`tag-name-link ${selectedTag?.id === t.id ? 'selected' : ''}`}
onClick={() => handleTagSelect(t)}
style={{ cursor: 'pointer' }}
>
<span className="version-badge">{t.version || t.name}</span>
</strong>
),
},
{
key: 'artifact_original_name',
header: 'Filename',
className: 'cell-truncate',
render: (t: TagDetail) => (
<span title={t.artifact_original_name || t.name}>{t.artifact_original_name || t.name}</span>
),
},
{
key: 'artifact_size',
header: 'Size',
render: (t: TagDetail) => <span>{formatBytes(t.artifact_size)}</span>,
},
{
key: 'created_at',
header: 'Cached',
sortable: true,
render: (t: TagDetail) => (
<span>{new Date(t.created_at).toLocaleDateString()}</span>
),
},
{
key: 'actions',
header: '',
render: (t: TagDetail) => (
<div className="action-buttons">
<a
href={getDownloadUrl(projectName!, packageName!, t.name)}
className="btn btn-icon"
download
title="Download"
>
<svg width="16" height="16" viewBox="0 0 24 24" fill="none" stroke="currentColor" strokeWidth="2">
<path d="M21 15v4a2 2 0 0 1-2 2H5a2 2 0 0 1-2-2v-4" />
<polyline points="7 10 12 15 17 10" />
<line x1="12" y1="15" x2="12" y2="3" />
</svg>
</a>
<button
className="btn btn-icon"
onClick={(e) => handleMenuOpen(e, t.id)}
title="More actions"
>
<svg width="16" height="16" viewBox="0 0 24 24" fill="none" stroke="currentColor" strokeWidth="2">
<circle cx="12" cy="12" r="1" />
<circle cx="12" cy="5" r="1" />
<circle cx="12" cy="19" r="1" />
</svg>
</button>
</div>
),
},
]
: [
// Regular project columns: Tag, Version, Filename
{
key: 'name',
header: 'Tag',
sortable: true,
render: (t: TagDetail) => (
<strong
className={`tag-name-link ${selectedTag?.id === t.id ? 'selected' : ''}`}
onClick={() => handleTagSelect(t)}
style={{ cursor: 'pointer' }}
>
{t.name}
</strong>
),
},
{
key: 'version',
header: 'Version',
render: (t: TagDetail) => (
<span className="version-badge">{t.version || '—'}</span>
),
},
{
key: 'artifact_original_name',
header: 'Filename',
className: 'cell-truncate',
render: (t: TagDetail) => (
<span title={t.artifact_original_name || undefined}>{t.artifact_original_name || '—'}</span>
),
},
{
key: 'artifact_size',
header: 'Size',
render: (t: TagDetail) => <span>{formatBytes(t.artifact_size)}</span>,
},
{
key: 'created_at',
header: 'Created',
sortable: true,
render: (t: TagDetail) => (
<span title={`by ${t.created_by}`}>{new Date(t.created_at).toLocaleDateString()}</span>
),
},
{
key: 'actions',
header: '',
render: (t: TagDetail) => (
<div className="action-buttons">
<a
href={getDownloadUrl(projectName!, packageName!, t.name)}
className="btn btn-icon"
download
title="Download"
>
<svg width="16" height="16" viewBox="0 0 24 24" fill="none" stroke="currentColor" strokeWidth="2">
<path d="M21 15v4a2 2 0 0 1-2 2H5a2 2 0 0 1-2-2v-4" />
<polyline points="7 10 12 15 17 10" />
<line x1="12" y1="15" x2="12" y2="3" />
</svg>
</a>
<button
className="btn btn-icon"
onClick={(e) => handleMenuOpen(e, t.id)}
title="More actions"
>
<svg width="16" height="16" viewBox="0 0 24 24" fill="none" stroke="currentColor" strokeWidth="2">
<circle cx="12" cy="12" r="1" />
<circle cx="12" cy="5" r="1" />
<circle cx="12" cy="19" r="1" />
</svg>
</button>
</div>
),
},
];
// Find the tag for the open menu
const openMenuTag = tags.find(t => t.id === openMenuId);
// Close menu when clicking outside
const handleClickOutside = () => {
if (openMenuId) {
setOpenMenuId(null);
setMenuPosition(null);
}
};
// Render dropdown menu as a portal-like element
const renderActionMenu = () => {
if (!openMenuId || !menuPosition || !openMenuTag) return null;
const t = openMenuTag;
return (
<div
className="action-menu-backdrop"
onClick={handleClickOutside}
>
<div
className="action-menu-dropdown"
style={{ top: menuPosition.top, left: menuPosition.left }}
onClick={(e) => e.stopPropagation()}
>
{t.name}
</strong>
),
},
{
key: 'version',
header: 'Version',
render: (t: TagDetail) => (
<span className="version-badge">{t.version || '-'}</span>
),
},
{
key: 'artifact_id',
header: 'Artifact ID',
render: (t: TagDetail) => (
<div className="artifact-id-cell">
<code className="artifact-id">{t.artifact_id.substring(0, 12)}...</code>
<CopyButton text={t.artifact_id} />
</div>
),
},
{
key: 'artifact_size',
header: 'Size',
render: (t: TagDetail) => <span>{formatBytes(t.artifact_size)}</span>,
},
{
key: 'artifact_content_type',
header: 'Type',
render: (t: TagDetail) => (
<span className="content-type">{t.artifact_content_type || '-'}</span>
),
},
{
key: 'artifact_original_name',
header: 'Filename',
className: 'cell-truncate',
render: (t: TagDetail) => (
<span title={t.artifact_original_name || undefined}>{t.artifact_original_name || '-'}</span>
),
},
{
key: 'created_at',
header: 'Created',
sortable: true,
render: (t: TagDetail) => (
<div className="created-cell">
<span>{new Date(t.created_at).toLocaleString()}</span>
<span className="created-by">by {t.created_by}</span>
</div>
),
},
{
key: 'actions',
header: 'Actions',
render: (t: TagDetail) => (
<div className="action-buttons">
<button
className="btn btn-secondary btn-small"
onClick={() => fetchEnsureFileForTag(t.name)}
title="View orchard.ensure file"
>
Ensure
<button onClick={() => { setViewArtifactId(t.artifact_id); setShowArtifactIdModal(true); setOpenMenuId(null); setMenuPosition(null); }}>
View Artifact ID
</button>
<button onClick={() => { navigator.clipboard.writeText(t.artifact_id); setOpenMenuId(null); setMenuPosition(null); }}>
Copy Artifact ID
</button>
<button onClick={() => { fetchEnsureFileForTag(t.name); setOpenMenuId(null); setMenuPosition(null); }}>
View Ensure File
</button>
{canWrite && !isSystemProject && (
<button onClick={() => { setCreateTagArtifactId(t.artifact_id); setShowCreateTagModal(true); setOpenMenuId(null); setMenuPosition(null); }}>
Create/Update Tag
</button>
)}
<button onClick={() => { handleTagSelect(t); setShowDepsModal(true); setOpenMenuId(null); setMenuPosition(null); }}>
View Dependencies
</button>
<a
href={getDownloadUrl(projectName!, packageName!, t.name)}
className="btn btn-secondary btn-small"
download
>
Download
</a>
</div>
),
},
];
</div>
);
};
if (loading && !tagsData) {
return <div className="loading">Loading...</div>;
@@ -451,6 +586,19 @@ function PackagePage() {
<div className="page-header__title-row">
<h1>{packageName}</h1>
{pkg && <Badge variant="default">{pkg.format}</Badge>}
{user && canWrite && !isSystemProject && (
<button
className="btn btn-primary btn-small header-upload-btn"
onClick={() => setShowUploadModal(true)}
>
<svg width="14" height="14" viewBox="0 0 24 24" fill="none" stroke="currentColor" strokeWidth="2" style={{ marginRight: '6px' }}>
<path d="M21 15v4a2 2 0 0 1-2 2H5a2 2 0 0 1-2-2v-4" />
<polyline points="17 8 12 3 7 8" />
<line x1="12" y1="3" x2="12" y2="15" />
</svg>
Upload
</button>
)}
</div>
{pkg?.description && <p className="description">{pkg.description}</p>}
<div className="page-header__meta">
@@ -468,14 +616,14 @@ function PackagePage() {
</div>
{pkg && (pkg.tag_count !== undefined || pkg.artifact_count !== undefined) && (
<div className="package-header-stats">
{pkg.tag_count !== undefined && (
{!isSystemProject && pkg.tag_count !== undefined && (
<span className="stat-item">
<strong>{pkg.tag_count}</strong> tags
</span>
)}
{pkg.artifact_count !== undefined && (
<span className="stat-item">
<strong>{pkg.artifact_count}</strong> artifacts
<strong>{pkg.artifact_count}</strong> {isSystemProject ? 'versions' : 'artifacts'}
</span>
)}
{pkg.total_size !== undefined && pkg.total_size > 0 && (
@@ -483,7 +631,7 @@ function PackagePage() {
<strong>{formatBytes(pkg.total_size)}</strong> total
</span>
)}
{pkg.latest_tag && (
{!isSystemProject && pkg.latest_tag && (
<span className="stat-item">
Latest: <strong className="accent">{pkg.latest_tag}</strong>
</span>
@@ -496,44 +644,9 @@ function PackagePage() {
{error && <div className="error-message">{error}</div>}
{uploadSuccess && <div className="success-message">{uploadSuccess}</div>}
{user && (
<div className="upload-section card">
<h3>Upload Artifact</h3>
{canWrite ? (
<div className="upload-form">
<div className="form-group">
<label htmlFor="upload-tag">Tag (optional)</label>
<input
id="upload-tag"
type="text"
value={uploadTag}
onChange={(e) => setUploadTag(e.target.value)}
placeholder="v1.0.0, latest, stable..."
/>
</div>
<DragDropUpload
projectName={projectName!}
packageName={packageName!}
tag={uploadTag || undefined}
onUploadComplete={handleUploadComplete}
onUploadError={handleUploadError}
/>
</div>
) : (
<DragDropUpload
projectName={projectName!}
packageName={packageName!}
disabled={true}
disabledReason="You have read-only access to this project and cannot upload artifacts."
onUploadComplete={handleUploadComplete}
onUploadError={handleUploadError}
/>
)}
</div>
)}
<div className="section-header">
<h2>Tags / Versions</h2>
<h2>{isSystemProject ? 'Versions' : 'Tags / Versions'}</h2>
</div>
<div className="list-controls">
@@ -577,110 +690,6 @@ function PackagePage() {
/>
)}
{/* Dependencies Section */}
{tags.length > 0 && (
<div className="dependencies-section card">
<div className="dependencies-header">
<h3>Dependencies</h3>
<div className="dependencies-controls">
{selectedTag && (
<>
<button
className="btn btn-secondary btn-small"
onClick={fetchEnsureFile}
disabled={ensureFileLoading}
title="View orchard.ensure file"
>
<svg width="14" height="14" viewBox="0 0 24 24" fill="none" stroke="currentColor" strokeWidth="2" style={{ marginRight: '6px' }}>
<path d="M14 2H6a2 2 0 0 0-2 2v16a2 2 0 0 0 2 2h12a2 2 0 0 0 2-2V8z"></path>
<polyline points="14 2 14 8 20 8"></polyline>
<line x1="16" y1="13" x2="8" y2="13"></line>
<line x1="16" y1="17" x2="8" y2="17"></line>
<polyline points="10 9 9 9 8 9"></polyline>
</svg>
{ensureFileLoading ? 'Loading...' : 'View Ensure File'}
</button>
<button
className="btn btn-secondary btn-small"
onClick={() => setShowGraph(true)}
title="View full dependency tree"
>
<svg width="14" height="14" viewBox="0 0 24 24" fill="none" stroke="currentColor" strokeWidth="2" style={{ marginRight: '6px' }}>
<circle cx="12" cy="12" r="3"></circle>
<circle cx="4" cy="4" r="2"></circle>
<circle cx="20" cy="4" r="2"></circle>
<circle cx="4" cy="20" r="2"></circle>
<circle cx="20" cy="20" r="2"></circle>
<line x1="9.5" y1="9.5" x2="5.5" y2="5.5"></line>
<line x1="14.5" y1="9.5" x2="18.5" y2="5.5"></line>
<line x1="9.5" y1="14.5" x2="5.5" y2="18.5"></line>
<line x1="14.5" y1="14.5" x2="18.5" y2="18.5"></line>
</svg>
View Graph
</button>
</>
)}
</div>
</div>
<div className="dependencies-tag-select">
{selectedTag && (
<select
className="tag-selector"
value={selectedTag.id}
onChange={(e) => {
const tag = tags.find(t => t.id === e.target.value);
if (tag) setSelectedTag(tag);
}}
>
{tags.map(t => (
<option key={t.id} value={t.id}>
{t.name}{t.version ? ` (${t.version})` : ''}
</option>
))}
</select>
)}
</div>
{depsLoading ? (
<div className="deps-loading">Loading dependencies...</div>
) : depsError ? (
<div className="deps-error">{depsError}</div>
) : dependencies.length === 0 ? (
<div className="deps-empty">
{selectedTag ? (
<span><strong>{selectedTag.name}</strong> has no dependencies</span>
) : (
<span>No dependencies</span>
)}
</div>
) : (
<div className="deps-list">
<div className="deps-summary">
<strong>{selectedTag?.name}</strong> has {dependencies.length} {dependencies.length === 1 ? 'dependency' : 'dependencies'}:
</div>
<ul className="deps-items">
{dependencies.map((dep) => (
<li key={dep.id} className="dep-item">
<Link
to={`/project/${dep.project}/${dep.package}`}
className="dep-link"
>
{dep.project}/{dep.package}
</Link>
<span className="dep-constraint">
@ {dep.version || dep.tag}
</span>
<span className="dep-status dep-status--ok" title="Package exists">
&#10003;
</span>
</li>
))}
</ul>
</div>
)}
</div>
)}
{/* Used By (Reverse Dependencies) Section */}
<div className="used-by-section card">
<h3>Used By</h3>
@@ -737,78 +746,6 @@ function PackagePage() {
)}
</div>
<div className="download-by-id-section card">
<h3>Download by Artifact ID</h3>
<div className="download-by-id-form">
<input
type="text"
value={artifactIdInput}
onChange={(e) => setArtifactIdInput(e.target.value.toLowerCase().replace(/[^a-f0-9]/g, '').slice(0, 64))}
placeholder="Enter SHA256 artifact ID (64 hex characters)"
className="artifact-id-input"
/>
<a
href={artifactIdInput.length === 64 ? getDownloadUrl(projectName!, packageName!, `artifact:${artifactIdInput}`) : '#'}
className={`btn btn-primary ${artifactIdInput.length !== 64 ? 'btn-disabled' : ''}`}
download
onClick={(e) => {
if (artifactIdInput.length !== 64) {
e.preventDefault();
}
}}
>
Download
</a>
</div>
{artifactIdInput.length > 0 && artifactIdInput.length !== 64 && (
<p className="validation-hint">Artifact ID must be exactly 64 hex characters ({artifactIdInput.length}/64)</p>
)}
</div>
{user && canWrite && (
<div className="create-tag-section card">
<h3>Create / Update Tag</h3>
<p className="section-description">Point a tag at any existing artifact by its ID</p>
<form onSubmit={handleCreateTag} className="create-tag-form">
<div className="form-row">
<div className="form-group">
<label htmlFor="create-tag-name">Tag Name</label>
<input
id="create-tag-name"
type="text"
value={createTagName}
onChange={(e) => setCreateTagName(e.target.value)}
placeholder="latest, stable, v1.0.0..."
disabled={createTagLoading}
/>
</div>
<div className="form-group form-group--wide">
<label htmlFor="create-tag-artifact">Artifact ID</label>
<input
id="create-tag-artifact"
type="text"
value={createTagArtifactId}
onChange={(e) => setCreateTagArtifactId(e.target.value.toLowerCase().replace(/[^a-f0-9]/g, '').slice(0, 64))}
placeholder="SHA256 hash (64 hex characters)"
className="artifact-id-input"
disabled={createTagLoading}
/>
</div>
<button
type="submit"
className="btn btn-primary"
disabled={createTagLoading || !createTagName.trim() || createTagArtifactId.length !== 64}
>
{createTagLoading ? 'Creating...' : 'Create Tag'}
</button>
</div>
{createTagArtifactId.length > 0 && createTagArtifactId.length !== 64 && (
<p className="validation-hint">Artifact ID must be exactly 64 hex characters ({createTagArtifactId.length}/64)</p>
)}
</form>
</div>
)}
<div className="usage-section card">
<h3>Usage</h3>
<p>Download artifacts using:</p>
@@ -831,6 +768,118 @@ function PackagePage() {
/>
)}
{/* Upload Modal */}
{showUploadModal && (
<div className="modal-overlay" onClick={() => setShowUploadModal(false)}>
<div className="upload-modal" onClick={(e) => e.stopPropagation()}>
<div className="modal-header">
<h3>Upload Artifact</h3>
<button
className="modal-close"
onClick={() => setShowUploadModal(false)}
title="Close"
>
<svg width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" strokeWidth="2">
<line x1="18" y1="6" x2="6" y2="18"></line>
<line x1="6" y1="6" x2="18" y2="18"></line>
</svg>
</button>
</div>
<div className="modal-body">
<div className="form-group">
<label htmlFor="upload-tag">Tag (optional)</label>
<input
id="upload-tag"
type="text"
value={uploadTag}
onChange={(e) => setUploadTag(e.target.value)}
placeholder="v1.0.0, latest, stable..."
/>
</div>
<DragDropUpload
projectName={projectName!}
packageName={packageName!}
tag={uploadTag || undefined}
onUploadComplete={(result) => {
handleUploadComplete(result);
setShowUploadModal(false);
setUploadTag('');
}}
onUploadError={handleUploadError}
/>
</div>
</div>
</div>
)}
{/* Create/Update Tag Modal */}
{showCreateTagModal && (
<div className="modal-overlay" onClick={() => setShowCreateTagModal(false)}>
<div className="create-tag-modal" onClick={(e) => e.stopPropagation()}>
<div className="modal-header">
<h3>Create / Update Tag</h3>
<button
className="modal-close"
onClick={() => { setShowCreateTagModal(false); setCreateTagName(''); setCreateTagArtifactId(''); }}
title="Close"
>
<svg width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" strokeWidth="2">
<line x1="18" y1="6" x2="6" y2="18"></line>
<line x1="6" y1="6" x2="18" y2="18"></line>
</svg>
</button>
</div>
<div className="modal-body">
<p className="modal-description">Point a tag at an artifact by its ID</p>
<form onSubmit={(e) => { handleCreateTag(e); setShowCreateTagModal(false); }}>
<div className="form-group">
<label htmlFor="modal-tag-name">Tag Name</label>
<input
id="modal-tag-name"
type="text"
value={createTagName}
onChange={(e) => setCreateTagName(e.target.value)}
placeholder="latest, stable, v1.0.0..."
disabled={createTagLoading}
/>
</div>
<div className="form-group">
<label htmlFor="modal-artifact-id">Artifact ID</label>
<input
id="modal-artifact-id"
type="text"
value={createTagArtifactId}
onChange={(e) => setCreateTagArtifactId(e.target.value.toLowerCase().replace(/[^a-f0-9]/g, '').slice(0, 64))}
placeholder="SHA256 hash (64 hex characters)"
className="artifact-id-input"
disabled={createTagLoading}
/>
{createTagArtifactId.length > 0 && createTagArtifactId.length !== 64 && (
<p className="validation-hint">{createTagArtifactId.length}/64 characters</p>
)}
</div>
<div className="modal-actions">
<button
type="button"
className="btn btn-secondary"
onClick={() => { setShowCreateTagModal(false); setCreateTagName(''); setCreateTagArtifactId(''); }}
>
Cancel
</button>
<button
type="submit"
className="btn btn-primary"
disabled={createTagLoading || !createTagName.trim() || createTagArtifactId.length !== 64}
>
{createTagLoading ? 'Creating...' : 'Create Tag'}
</button>
</div>
</form>
</div>
</div>
</div>
)}
{/* Ensure File Modal */}
{showEnsureFile && (
<div className="modal-overlay" onClick={() => setShowEnsureFile(false)}>
@@ -872,6 +921,107 @@ function PackagePage() {
</div>
</div>
)}
{/* Dependencies Modal */}
{showDepsModal && selectedTag && (
<div className="modal-overlay" onClick={() => setShowDepsModal(false)}>
<div className="deps-modal" onClick={(e) => e.stopPropagation()}>
<div className="modal-header">
<h3>Dependencies for {selectedTag.version || selectedTag.name}</h3>
<button
className="modal-close"
onClick={() => setShowDepsModal(false)}
title="Close"
>
<svg width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" strokeWidth="2">
<line x1="18" y1="6" x2="6" y2="18"></line>
<line x1="6" y1="6" x2="18" y2="18"></line>
</svg>
</button>
</div>
<div className="modal-body">
<div className="deps-modal-controls">
<button
className="btn btn-secondary btn-small"
onClick={fetchEnsureFile}
disabled={ensureFileLoading}
>
View Ensure File
</button>
<button
className="btn btn-secondary btn-small"
onClick={() => { setShowDepsModal(false); setShowGraph(true); }}
>
View Graph
</button>
</div>
{depsLoading ? (
<div className="deps-loading">Loading dependencies...</div>
) : depsError ? (
<div className="deps-error">{depsError}</div>
) : dependencies.length === 0 ? (
<div className="deps-empty">No dependencies</div>
) : (
<div className="deps-list">
<div className="deps-summary">
{dependencies.length} {dependencies.length === 1 ? 'dependency' : 'dependencies'}:
</div>
<ul className="deps-items">
{dependencies.map((dep) => (
<li key={dep.id} className="dep-item">
<Link
to={`/project/${dep.project}/${dep.package}`}
className="dep-link"
onClick={() => setShowDepsModal(false)}
>
{dep.project}/{dep.package}
</Link>
<span className="dep-constraint">
@ {dep.version || dep.tag}
</span>
<span className="dep-status dep-status--ok" title="Package exists">
&#10003;
</span>
</li>
))}
</ul>
</div>
)}
</div>
</div>
</div>
)}
{/* Artifact ID Modal */}
{showArtifactIdModal && viewArtifactId && (
<div className="modal-overlay" onClick={() => setShowArtifactIdModal(false)}>
<div className="artifact-id-modal" onClick={(e) => e.stopPropagation()}>
<div className="modal-header">
<h3>Artifact ID</h3>
<button
className="modal-close"
onClick={() => setShowArtifactIdModal(false)}
title="Close"
>
<svg width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" strokeWidth="2">
<line x1="18" y1="6" x2="6" y2="18"></line>
<line x1="6" y1="6" x2="18" y2="18"></line>
</svg>
</button>
</div>
<div className="modal-body">
<p className="modal-description">SHA256 hash identifying this artifact:</p>
<div className="artifact-id-display">
<code>{viewArtifactId}</code>
<CopyButton text={viewArtifactId} />
</div>
</div>
</div>
</div>
)}
{/* Action Menu Dropdown */}
{renderActionMenu()}
</div>
);
}

View File

@@ -214,7 +214,7 @@ function ProjectPage() {
</div>
</div>
<div className="page-header__actions">
{canAdmin && !project.team_id && (
{canAdmin && !project.team_id && !project.is_system && (
<button
className="btn btn-secondary"
onClick={() => navigate(`/project/${projectName}/settings`)}
@@ -227,11 +227,11 @@ function ProjectPage() {
Settings
</button>
)}
{canWrite ? (
{canWrite && !project.is_system ? (
<button className="btn btn-primary" onClick={() => setShowForm(!showForm)}>
{showForm ? 'Cancel' : '+ New Package'}
</button>
) : user ? (
) : user && !project.is_system ? (
<span className="text-muted" title="You have read-only access to this project">
Read-only access
</span>
@@ -294,18 +294,20 @@ function ProjectPage() {
placeholder="Filter packages..."
className="list-controls__search"
/>
<select
className="list-controls__select"
value={format}
onChange={(e) => handleFormatChange(e.target.value)}
>
<option value="">All formats</option>
{FORMAT_OPTIONS.map((f) => (
<option key={f} value={f}>
{f}
</option>
))}
</select>
{!project?.is_system && (
<select
className="list-controls__select"
value={format}
onChange={(e) => handleFormatChange(e.target.value)}
>
<option value="">All formats</option>
{FORMAT_OPTIONS.map((f) => (
<option key={f} value={f}>
{f}
</option>
))}
</select>
)}
</div>
{hasActiveFilters && (
@@ -341,19 +343,19 @@ function ProjectPage() {
className: 'cell-description',
render: (pkg) => pkg.description || '—',
},
{
...(!project?.is_system ? [{
key: 'format',
header: 'Format',
render: (pkg) => <Badge variant="default">{pkg.format}</Badge>,
},
{
render: (pkg: Package) => <Badge variant="default">{pkg.format}</Badge>,
}] : []),
...(!project?.is_system ? [{
key: 'tag_count',
header: 'Tags',
render: (pkg) => pkg.tag_count ?? '—',
},
render: (pkg: Package) => pkg.tag_count ?? '—',
}] : []),
{
key: 'artifact_count',
header: 'Artifacts',
header: project?.is_system ? 'Versions' : 'Artifacts',
render: (pkg) => pkg.artifact_count ?? '—',
},
{
@@ -362,12 +364,12 @@ function ProjectPage() {
render: (pkg) =>
pkg.total_size !== undefined && pkg.total_size > 0 ? formatBytes(pkg.total_size) : '—',
},
{
...(!project?.is_system ? [{
key: 'latest_tag',
header: 'Latest',
render: (pkg) =>
render: (pkg: Package) =>
pkg.latest_tag ? <strong style={{ color: 'var(--accent-primary)' }}>{pkg.latest_tag}</strong> : '—',
},
}] : []),
{
key: 'created_at',
header: 'Created',

View File

@@ -433,6 +433,13 @@ export interface ResolvedArtifact {
download_url: string;
}
export interface MissingDependency {
project: string;
package: string;
constraint: string | null;
required_by: string | null;
}
export interface DependencyResolutionResponse {
requested: {
project: string;
@@ -440,6 +447,7 @@ export interface DependencyResolutionResponse {
ref: string;
};
resolved: ResolvedArtifact[];
missing: MissingDependency[];
total_size: number;
artifact_count: number;
}

View File

@@ -144,6 +144,20 @@ spec:
- name: ORCHARD_DATABASE_POOL_TIMEOUT
value: {{ .Values.orchard.database.poolTimeout | quote }}
{{- end }}
{{- if .Values.orchard.pypiCache }}
{{- if .Values.orchard.pypiCache.workers }}
- name: ORCHARD_PYPI_CACHE_WORKERS
value: {{ .Values.orchard.pypiCache.workers | quote }}
{{- end }}
{{- if .Values.orchard.pypiCache.maxDepth }}
- name: ORCHARD_PYPI_CACHE_MAX_DEPTH
value: {{ .Values.orchard.pypiCache.maxDepth | quote }}
{{- end }}
{{- if .Values.orchard.pypiCache.maxAttempts }}
- name: ORCHARD_PYPI_CACHE_MAX_ATTEMPTS
value: {{ .Values.orchard.pypiCache.maxAttempts | quote }}
{{- end }}
{{- end }}
{{- if .Values.orchard.auth }}
{{- if or .Values.orchard.auth.secretsManager .Values.orchard.auth.existingSecret .Values.orchard.auth.adminPassword }}
- name: ORCHARD_ADMIN_PASSWORD

View File

@@ -59,10 +59,10 @@ ingress:
resources:
limits:
cpu: 500m
memory: 512Mi
memory: 1Gi
requests:
cpu: 200m
memory: 512Mi
memory: 1Gi
livenessProbe:
httpGet:
@@ -124,6 +124,12 @@ orchard:
mode: "presigned"
presignedUrlExpiry: 3600
# PyPI Cache Worker settings (reduced workers to limit memory usage)
pypiCache:
workers: 1
maxDepth: 10
maxAttempts: 3
# Relaxed rate limits for dev/feature environments (allows integration tests to run)
rateLimit:
login: "1000/minute" # Default is 5/minute, relaxed for CI integration tests

View File

@@ -57,10 +57,10 @@ ingress:
resources:
limits:
cpu: 500m
memory: 512Mi
memory: 768Mi
requests:
cpu: 500m
memory: 512Mi
memory: 768Mi
livenessProbe:
httpGet:
@@ -121,6 +121,12 @@ orchard:
mode: "presigned"
presignedUrlExpiry: 3600
# PyPI Cache Worker settings (reduced workers to limit memory usage)
pypiCache:
workers: 2
maxDepth: 10
maxAttempts: 3
# PostgreSQL subchart - disabled in prod, using RDS
postgresql:
enabled: false

View File

@@ -56,10 +56,10 @@ ingress:
resources:
limits:
cpu: 500m
memory: 512Mi
memory: 768Mi
requests:
cpu: 500m
memory: 512Mi
memory: 768Mi
livenessProbe:
httpGet:
@@ -122,6 +122,12 @@ orchard:
mode: "presigned" # presigned, redirect, or proxy
presignedUrlExpiry: 3600 # Presigned URL expiry in seconds
# PyPI Cache Worker settings (reduced workers to limit memory usage)
pypiCache:
workers: 2
maxDepth: 10
maxAttempts: 3
# Relaxed rate limits for stage (allows CI integration tests to run)
rateLimit:
login: "1000/minute" # Default is 5/minute, relaxed for CI integration tests

View File

@@ -54,10 +54,10 @@ ingress:
resources:
limits:
cpu: 500m
memory: 512Mi
memory: 768Mi
requests:
cpu: 500m
memory: 512Mi
memory: 768Mi
livenessProbe:
httpGet:
@@ -120,6 +120,12 @@ orchard:
mode: "presigned" # presigned, redirect, or proxy
presignedUrlExpiry: 3600 # Presigned URL expiry in seconds
# PyPI Cache Worker settings
pypiCache:
workers: 2 # Number of concurrent cache workers (reduced to limit memory usage)
maxDepth: 10 # Maximum recursion depth for dependency caching
maxAttempts: 3 # Maximum retry attempts for failed cache tasks
# Authentication settings
auth:
# Option 1: Plain admin password (creates K8s secret)

View File

@@ -0,0 +1,55 @@
-- Migration: 011_pypi_cache_tasks
-- Description: Add table for tracking PyPI dependency caching tasks
-- Date: 2026-02-02
-- Table for tracking PyPI cache tasks with retry support
CREATE TABLE pypi_cache_tasks (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
-- What to cache
package_name VARCHAR(255) NOT NULL,
version_constraint VARCHAR(255),
-- Origin tracking
parent_task_id UUID REFERENCES pypi_cache_tasks(id) ON DELETE SET NULL,
depth INTEGER NOT NULL DEFAULT 0,
triggered_by_artifact VARCHAR(64) REFERENCES artifacts(id) ON DELETE SET NULL,
-- Status
status VARCHAR(20) NOT NULL DEFAULT 'pending',
attempts INTEGER NOT NULL DEFAULT 0,
max_attempts INTEGER NOT NULL DEFAULT 3,
-- Results
cached_artifact_id VARCHAR(64) REFERENCES artifacts(id) ON DELETE SET NULL,
error_message TEXT,
-- Timing
created_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT NOW(),
started_at TIMESTAMP WITH TIME ZONE,
completed_at TIMESTAMP WITH TIME ZONE,
next_retry_at TIMESTAMP WITH TIME ZONE,
-- Constraints
CONSTRAINT check_task_status CHECK (status IN ('pending', 'in_progress', 'completed', 'failed')),
CONSTRAINT check_depth_non_negative CHECK (depth >= 0),
CONSTRAINT check_attempts_non_negative CHECK (attempts >= 0)
);
-- Index for finding tasks ready to process (pending with retry time passed)
CREATE INDEX idx_pypi_cache_tasks_status_retry ON pypi_cache_tasks(status, next_retry_at);
-- Index for deduplication check (is this package already queued?)
CREATE INDEX idx_pypi_cache_tasks_package_status ON pypi_cache_tasks(package_name, status);
-- Index for tracing dependency chains
CREATE INDEX idx_pypi_cache_tasks_parent ON pypi_cache_tasks(parent_task_id);
-- Index for finding tasks by artifact that triggered them
CREATE INDEX idx_pypi_cache_tasks_triggered_by ON pypi_cache_tasks(triggered_by_artifact);
-- Index for finding tasks by cached artifact
CREATE INDEX idx_pypi_cache_tasks_cached_artifact ON pypi_cache_tasks(cached_artifact_id);
-- Index for sorting by depth and creation time (processing order)
CREATE INDEX idx_pypi_cache_tasks_depth_created ON pypi_cache_tasks(depth, created_at);