37 Commits

Author SHA1 Message Date
Mondo Diaz
f992fc540e Add proactive dependency caching for PyPI packages
When a PyPI package is cached, its dependencies are now automatically
fetched in background threads. This ensures the entire dependency tree
is cached even if pip already has some packages installed locally.

Features:
- Background threads fetch each dependency without blocking the response
- Uses our own proxy endpoint to cache, which recursively caches transitive deps
- Max depth of 10 to prevent infinite loops
- Daemon threads so they don't block process shutdown
2026-01-30 17:45:30 -06:00
Mondo Diaz
044a6c1d27 Fix duplicate dependency constraint causing 500 errors
- Deduplicate dependencies by package name before inserting
- Some packages (like anyio) list the same dep (trio) multiple times with
  different version constraints for different extras
- The unique constraint on (artifact_id, project, package) rejected these
- Also removed debug logging from dependencies.py
2026-01-30 17:43:49 -06:00
Mondo Diaz
62c77dc16d Add detailed debug logging to _resolve_dependency_to_artifact 2026-01-30 17:29:19 -06:00
Mondo Diaz
7c05360eed Add debug logging to resolve_dependencies 2026-01-30 17:21:04 -06:00
Mondo Diaz
76878279e9 Add backfill script for PyPI package dependencies
Script extracts Requires-Dist metadata from cached PyPI packages
and stores them in artifact_dependencies table.

Usage:
  docker exec <container> python -m backend.scripts.backfill_pypi_dependencies
  docker exec <container> python -m backend.scripts.backfill_pypi_dependencies --dry-run
2026-01-30 15:38:45 -06:00
Mondo Diaz
e1b01abf9b Add PEP 440 version constraint matching for dependency resolution
- Parse version constraints like >=1.9, <2.0 using packaging library
- Find the latest version that satisfies the constraint
- Support wildcard (*) to get latest version
- Fall back to exact version and tag matching
2026-01-30 15:34:19 -06:00
Mondo Diaz
d07936b666 Fix ensure file modal z-index when opened from deps modal 2026-01-30 15:32:06 -06:00
Mondo Diaz
47b3eb439d Extract and store dependencies from PyPI packages
- Add functions to parse Requires-Dist metadata from wheel and sdist files
- Store extracted dependencies in artifact_dependencies table
- Fix streaming response for cached artifacts (proper tuple unpacking)
- Fix version uniqueness check to use version string instead of artifact_id
- Skip creating versions for .metadata files
2026-01-30 15:14:52 -06:00
Mondo Diaz
c5f75e4fd6 Add is_system to all ProjectResponse constructions in routes 2026-01-30 13:34:26 -06:00
Mondo Diaz
ff31379649 Fix: ensure existing _pypi project gets is_system=true 2026-01-30 13:33:31 -06:00
Mondo Diaz
424b1e5770 Add is_system field to ProjectResponse schema 2026-01-30 13:11:11 -06:00
Mondo Diaz
7b5b0c78d8 Hide Tags and Latest columns for system projects in package table 2026-01-30 12:55:28 -06:00
Mondo Diaz
924826f07a Improve system project UX and make dependencies a modal
- Hide tag count stat for system projects (show "versions" instead of "artifacts")
- Hide "Latest" tag stat for system projects
- Change "Create/Update Tag" to only show for non-system projects
- Add "View Artifact ID" menu option with modal showing the SHA256 hash
- Move dependencies section to a modal (opened via "View Dependencies" menu)
- Add deps-modal and artifact-id-modal CSS styles
2026-01-30 12:36:40 -06:00
Mondo Diaz
fe6c6c52d2 Fix PyPI proxy UX and package stats calculation
- Fix artifact_count and total_size calculation to use Tags instead of
  Uploads, so PyPI cached packages show their stats correctly
- Fix PackagePage dropdown menu positioning (use fixed position with backdrop)
- Add system project detection for projects starting with "_"
- Show Version as primary column for system projects, hide Tag column
- Hide upload button for system projects (they're cache-only)
- Rename section header to "Versions" for system projects
- Fix test_projects_sort_by_name to exclude system projects from sort comparison
2026-01-30 12:16:05 -06:00
Mondo Diaz
701e11ce83 Hide format filter and column for system projects
System projects like _pypi only contain packages of one format,
so the format filter dropdown and column are redundant.
2026-01-30 11:55:09 -06:00
Mondo Diaz
ff9e02606e Hide Settings and New Package buttons for system projects
System projects should be system-controlled only. Users should not
be able to create packages or change settings on system cache projects.
2026-01-30 11:54:02 -06:00
Mondo Diaz
f3afdd3bbf Improve PyPI proxy and Package page UX
PyPI proxy improvements:
- Set package format to "pypi" instead of "generic"
- Extract version from filename and create PackageVersion record
- Support .whl, .tar.gz, and .zip filename formats

Package page UX overhaul:
- Move upload to header button with modal
- Simplify table: combine Tag/Version, remove Type and Artifact ID columns
- Add row action menu (⋯) with: Copy ID, Ensure File, Create Tag, Dependencies
- Remove cluttered "Download by Artifact ID" and "Create/Update Tag" sections
- Add modals for upload and create tag actions
- Cleaner, more scannable table layout
2026-01-30 11:52:37 -06:00
Mondo Diaz
4b73196664 Show team name instead of individual user in Owner column
Projects owned by teams now display the team name in the Owner column
for better organizational continuity when team members change.
Falls back to created_by if no team is assigned.
2026-01-30 11:25:01 -06:00
Mondo Diaz
7ef66745f1 Add "(coming soon)" label for unsupported upstream source types
Only pypi and generic are currently supported. Other types now show
"(coming soon)" in both the dropdown and the sources table.
2026-01-30 11:03:44 -06:00
Mondo Diaz
2dc7fe5a7b Fix PyPI proxy: use correct storage method and make project public
- Use storage.get_stream(s3_key) instead of non-existent get_artifact_stream()
- Make _pypi project public (is_public=True) so cached packages are visible
2026-01-30 10:59:50 -06:00
Mondo Diaz
534e4b964f Fix Project and Tag model fields in PyPI proxy
Use correct model fields:
- Project: is_public, is_system, created_by (not visibility)
- Tag: add required created_by field
2026-01-30 10:29:25 -06:00
Mondo Diaz
757e43fc34 Fix Artifact model field names in PyPI proxy
Use correct Artifact model fields:
- original_name instead of filename
- Add required created_by and s3_key fields
- Include checksum fields from storage result
2026-01-30 09:58:15 -06:00
Mondo Diaz
d78092de55 Fix PyPI proxy to use correct storage.store() method
The code was calling storage.store_artifact() which doesn't exist.
Changed to use storage.store() which handles content-addressable
storage with automatic deduplication.
2026-01-30 09:41:34 -06:00
Mondo Diaz
0fa991f536 Allow full path in PyPI upstream source URL
Users can now configure the full path including /simple in their
upstream source URL (e.g., https://example.com/api/pypi/repo/simple)
instead of having the code append /simple/ automatically.

This matches pip's --index-url format, making configuration more
intuitive and copy/paste friendly.
2026-01-30 09:24:05 -06:00
Mondo Diaz
00fb2729e4 Fix test_rewrite_relative_links assertion to expect correct URL
The test was checking for the wrong URL pattern. When urljoin resolves
../../packages/ab/cd/... relative to /api/pypi/pypi-remote/simple/requests/,
it correctly produces /api/pypi/pypi-remote/packages/ab/cd/... (not
/api/pypi/packages/...).
2026-01-30 08:51:30 -06:00
Mondo Diaz
8ae4d7a685 Improve PyPI proxy test assertions for all status codes
Tests now verify the correct response for each scenario:
- 200: HTML content-type
- 404: "not found" error message
- 503: "No PyPI upstream sources configured" error message
2026-01-29 19:35:20 -06:00
Mondo Diaz
4b887d1aad Fix PyPI proxy tests to work with or without upstream sources
- Tests now accept 200/404/503 responses since upstream sources may or
  may not be configured in the test environment
- Added upstream_base_url parameter to _rewrite_package_links test
- Added test for relative URL resolution (Artifactory-style URLs)
2026-01-29 19:34:33 -06:00
Mondo Diaz
4dc54ace8a Fix HTTPS scheme detection behind reverse proxy
When behind a reverse proxy that terminates SSL, the server sees HTTP
requests internally. Added _get_base_url() helper that respects the
X-Forwarded-Proto header to generate correct external HTTPS URLs.

This fixes links in the PyPI simple index showing http:// instead of
https:// when accessed via HTTPS through a load balancer.
2026-01-29 18:02:21 -06:00
Mondo Diaz
64bfd3902f Fix relative URL handling in PyPI proxy
Artifactory and other registries may return relative URLs in their
Simple API responses (e.g., ../../packages/...). The proxy now resolves
these to absolute URLs using urljoin() before encoding them in the
upstream parameter.

This fixes package downloads failing when the upstream registry uses
relative URLs in its package index.
2026-01-29 18:01:19 -06:00
Mondo Diaz
bdfed77cb1 Remove dead code from pypi_proxy.py
- Remove unused imports (UpstreamClient, UpstreamClientConfig,
  UpstreamHTTPError, UpstreamConnectionError, UpstreamTimeoutError)
- Simplify matched_source selection logic, removing dead conditional
  that always evaluated to True due to 'or True'
2026-01-29 16:42:53 -06:00
Mondo Diaz
140f6c926a Fix httpx.Timeout configuration in PyPI proxy
httpx.Timeout requires either a default value or all four parameters.
Changed to httpx.Timeout(default, connect=X) format.
2026-01-29 16:40:06 -06:00
Mondo Diaz
dcd405679a Merge branch 'feature/transparent-proxy' into 'main'
Add transparent PyPI proxy and improve upstream sources UI

Closes #108

See merge request esv/bsf/bsf-integration/orchard/orchard-mvp!56
2026-01-29 16:12:57 -06:00
Mondo Diaz
97498b2f86 Add transparent PyPI proxy and improve upstream sources UI 2026-01-29 16:12:57 -06:00
Mondo Diaz
e8cf2462b7 Merge branch 'fix/upstream-caching-bugs-2' into 'main'
Simplify cache management UI and improve test status display (#107)

See merge request esv/bsf/bsf-integration/orchard/orchard-mvp!55
2026-01-29 14:25:19 -06:00
Mondo Diaz
038ad4ed1b Simplify cache management UI and improve test status display (#107) 2026-01-29 14:25:19 -06:00
Mondo Diaz
858b45d434 Merge branch 'fix/purge-seed-data-user-id' into 'main'
Fix purge_seed_data type mismatch for access_permissions.user_id (#107)

See merge request esv/bsf/bsf-integration/orchard/orchard-mvp!54
2026-01-29 13:48:21 -06:00
Mondo Diaz
95470b2bf6 Fix purge_seed_data type mismatch for access_permissions.user_id (#107) 2026-01-29 13:48:21 -06:00
19 changed files with 2431 additions and 604 deletions

View File

@@ -6,15 +6,32 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
## [Unreleased] ## [Unreleased]
### Added
- Added transparent PyPI proxy implementing PEP 503 Simple API (#108)
- `GET /pypi/simple/` - package index (proxied from upstream)
- `GET /pypi/simple/{package}/` - version list with rewritten download links
- `GET /pypi/simple/{package}/{filename}` - download with automatic caching
- Allows `pip install --index-url https://orchard.../pypi/simple/ <package>`
- Artifacts cached on first access through configured upstream sources
- Added `POST /api/v1/cache/resolve` endpoint to cache packages by coordinates instead of URL (#108)
### Changed
- Upstream sources table text is now centered under column headers (#108)
- ENV badge now appears inline with source name instead of separate column (#108)
- Test and Edit buttons now have more prominent button styling (#108)
- Reduced footer padding for cleaner layout (#108)
### Fixed ### Fixed
- Fixed purge_seed_data crash when deleting access permissions - was comparing UUID to VARCHAR column (#107) - Fixed purge_seed_data crash when deleting access permissions - was comparing UUID to VARCHAR column (#107)
### Changed ### Changed
- Upstream source connectivity test no longer follows redirects, fixing "Exceeded maximum allowed redirects" error with Artifactory proxies (#107) - Upstream source connectivity test no longer follows redirects, fixing "Exceeded maximum allowed redirects" error with Artifactory proxies (#107)
- Upstream sources table now has dedicated "Test" column with OK/Error status badges (#107)
- Test runs automatically after saving a new or updated upstream source (#107) - Test runs automatically after saving a new or updated upstream source (#107)
- Error states in upstream sources table are now clickable to show full error details in a modal (#107) - Test status now shows as colored dots (green=success, red=error) instead of text badges (#107)
- Clicking red dot shows error details in a modal (#107)
- Source name column no longer wraps text for better table layout (#107) - Source name column no longer wraps text for better table layout (#107)
- Renamed "Cache Management" page to "Upstream Sources" (#107)
- Moved Delete button from table row to edit modal for cleaner table layout (#107)
### Removed ### Removed
- Removed `is_public` field from upstream sources - all sources are now treated as internal/private (#107) - Removed `is_public` field from upstream sources - all sources are now treated as internal/private (#107)
@@ -22,6 +39,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- Removed seeding of public registry URLs (npm-public, pypi-public, maven-central, docker-hub) (#107) - Removed seeding of public registry URLs (npm-public, pypi-public, maven-central, docker-hub) (#107)
- Removed "Public" badge and checkbox from upstream sources UI (#107) - Removed "Public" badge and checkbox from upstream sources UI (#107)
- Removed "Allow Public Internet" toggle from cache settings UI (#107) - Removed "Allow Public Internet" toggle from cache settings UI (#107)
- Removed "Global Settings" section from cache management UI - auto-create system projects is always enabled (#107)
- Removed unused CacheSettings frontend types and API functions (#107)
### Added ### Added
- Added `ORCHARD_PURGE_SEED_DATA` environment variable support to stage helm values to remove seed data from long-running deployments (#107) - Added `ORCHARD_PURGE_SEED_DATA` environment variable support to stage helm values to remove seed data from long-running deployments (#107)

View File

@@ -10,11 +10,20 @@ Handles:
- Conflict detection - Conflict detection
""" """
import re
import yaml import yaml
from typing import List, Dict, Any, Optional, Set, Tuple from typing import List, Dict, Any, Optional, Set, Tuple
from sqlalchemy.orm import Session from sqlalchemy.orm import Session
from sqlalchemy import and_ from sqlalchemy import and_
# Import packaging for PEP 440 version matching
try:
from packaging.specifiers import SpecifierSet, InvalidSpecifier
from packaging.version import Version, InvalidVersion
HAS_PACKAGING = True
except ImportError:
HAS_PACKAGING = False
from .models import ( from .models import (
Project, Project,
Package, Package,
@@ -304,6 +313,87 @@ def get_reverse_dependencies(
) )
def _is_version_constraint(version_str: str) -> bool:
"""Check if a version string contains constraint operators."""
if not version_str:
return False
# Check for common constraint operators
return any(op in version_str for op in ['>=', '<=', '!=', '~=', '>', '<', '==', '*'])
def _resolve_version_constraint(
db: Session,
package: Package,
constraint: str,
) -> Optional[Tuple[str, str, int]]:
"""
Resolve a version constraint (e.g., '>=1.9') to a specific version.
Uses PEP 440 version matching to find the best matching version.
Args:
db: Database session
package: Package to search versions in
constraint: Version constraint string (e.g., '>=1.9', '<2.0,>=1.5')
Returns:
Tuple of (artifact_id, resolved_version, size) or None if not found
"""
if not HAS_PACKAGING:
# Fallback: if packaging not available, can't do constraint matching
return None
# Handle wildcard - return latest version
if constraint == '*':
# Get the latest version by created_at
latest = db.query(PackageVersion).filter(
PackageVersion.package_id == package.id,
).order_by(PackageVersion.created_at.desc()).first()
if latest:
artifact = db.query(Artifact).filter(Artifact.id == latest.artifact_id).first()
if artifact:
return (artifact.id, latest.version, artifact.size)
return None
try:
specifier = SpecifierSet(constraint)
except InvalidSpecifier:
# Invalid constraint, try as exact version
return None
# Get all versions for this package
all_versions = db.query(PackageVersion).filter(
PackageVersion.package_id == package.id,
).all()
if not all_versions:
return None
# Find matching versions
matching = []
for pv in all_versions:
try:
v = Version(pv.version)
if v in specifier:
matching.append((pv, v))
except InvalidVersion:
# Skip invalid versions
continue
if not matching:
return None
# Sort by version (descending) and return the latest matching
matching.sort(key=lambda x: x[1], reverse=True)
best_match = matching[0][0]
artifact = db.query(Artifact).filter(Artifact.id == best_match.artifact_id).first()
if artifact:
return (artifact.id, best_match.version, artifact.size)
return None
def _resolve_dependency_to_artifact( def _resolve_dependency_to_artifact(
db: Session, db: Session,
project_name: str, project_name: str,
@@ -314,11 +404,17 @@ def _resolve_dependency_to_artifact(
""" """
Resolve a dependency constraint to an artifact ID. Resolve a dependency constraint to an artifact ID.
Supports:
- Exact version matching (e.g., '1.2.3')
- Version constraints (e.g., '>=1.9', '<2.0,>=1.5')
- Tag matching
- Wildcard ('*' for any version)
Args: Args:
db: Database session db: Database session
project_name: Project name project_name: Project name
package_name: Package name package_name: Package name
version: Version constraint (exact) version: Version or version constraint
tag: Tag constraint tag: Tag constraint
Returns: Returns:
@@ -337,17 +433,23 @@ def _resolve_dependency_to_artifact(
return None return None
if version: if version:
# Look up by version # Check if this is a version constraint (>=, <, etc.) or exact version
pkg_version = db.query(PackageVersion).filter( if _is_version_constraint(version):
PackageVersion.package_id == package.id, result = _resolve_version_constraint(db, package, version)
PackageVersion.version == version, if result:
).first() return result
if pkg_version: else:
artifact = db.query(Artifact).filter( # Look up by exact version
Artifact.id == pkg_version.artifact_id pkg_version = db.query(PackageVersion).filter(
PackageVersion.package_id == package.id,
PackageVersion.version == version,
).first() ).first()
if artifact: if pkg_version:
return (artifact.id, version, artifact.size) artifact = db.query(Artifact).filter(
Artifact.id == pkg_version.artifact_id
).first()
if artifact:
return (artifact.id, version, artifact.size)
# Also check if there's a tag with this exact name # Also check if there's a tag with this exact name
tag_record = db.query(Tag).filter( tag_record = db.query(Tag).filter(

View File

@@ -11,6 +11,7 @@ from slowapi.errors import RateLimitExceeded
from .config import get_settings from .config import get_settings
from .database import init_db, SessionLocal from .database import init_db, SessionLocal
from .routes import router from .routes import router
from .pypi_proxy import router as pypi_router
from .seed import seed_database from .seed import seed_database
from .auth import create_default_admin from .auth import create_default_admin
from .rate_limit import limiter from .rate_limit import limiter
@@ -65,6 +66,7 @@ app.add_exception_handler(RateLimitExceeded, _rate_limit_exceeded_handler)
# Include API routes # Include API routes
app.include_router(router) app.include_router(router)
app.include_router(pypi_router)
# Serve static files (React build) if the directory exists # Serve static files (React build) if the directory exists
static_dir = os.path.join(os.path.dirname(__file__), "..", "..", "frontend", "dist") static_dir = os.path.join(os.path.dirname(__file__), "..", "..", "frontend", "dist")

872
backend/app/pypi_proxy.py Normal file
View File

@@ -0,0 +1,872 @@
"""
Transparent PyPI proxy implementing PEP 503 (Simple API).
Provides endpoints that allow pip to use Orchard as a PyPI index URL.
Artifacts are cached on first access through configured upstream sources.
"""
import hashlib
import logging
import re
import tarfile
import threading
import zipfile
from io import BytesIO
from typing import Optional, List, Tuple
from urllib.parse import urljoin, urlparse, quote, unquote
import httpx
from fastapi import APIRouter, BackgroundTasks, Depends, HTTPException, Request, Response
from fastapi.responses import StreamingResponse, HTMLResponse
from sqlalchemy.orm import Session
from .database import get_db
from .models import UpstreamSource, CachedUrl, Artifact, Project, Package, Tag, PackageVersion, ArtifactDependency
from .storage import S3Storage, get_storage
from .config import get_env_upstream_sources
logger = logging.getLogger(__name__)
router = APIRouter(prefix="/pypi", tags=["pypi-proxy"])
def _parse_requires_dist(requires_dist: str) -> Tuple[str, Optional[str]]:
"""Parse a Requires-Dist line into (package_name, version_constraint).
Examples:
"requests (>=2.25.0)" -> ("requests", ">=2.25.0")
"typing-extensions; python_version < '3.8'" -> ("typing-extensions", None)
"numpy>=1.21.0" -> ("numpy", ">=1.21.0")
"certifi" -> ("certifi", None)
Returns:
Tuple of (normalized_package_name, version_constraint or None)
"""
# Remove any environment markers (after semicolon)
if ';' in requires_dist:
requires_dist = requires_dist.split(';')[0].strip()
# Match patterns like "package (>=1.0)" or "package>=1.0" or "package"
# Pattern breakdown: package name, optional whitespace, optional version in parens or directly
match = re.match(
r'^([a-zA-Z0-9][-a-zA-Z0-9._]*)\s*(?:\(([^)]+)\)|([<>=!~][^\s;]+))?',
requires_dist.strip()
)
if not match:
return None, None
package_name = match.group(1)
# Version can be in parentheses (group 2) or directly after name (group 3)
version_constraint = match.group(2) or match.group(3)
# Normalize package name (PEP 503)
normalized_name = re.sub(r'[-_.]+', '-', package_name).lower()
# Clean up version constraint
if version_constraint:
version_constraint = version_constraint.strip()
return normalized_name, version_constraint
def _extract_requires_from_metadata(metadata_content: str) -> List[Tuple[str, Optional[str]]]:
"""Extract all Requires-Dist entries from METADATA/PKG-INFO content.
Args:
metadata_content: The content of a METADATA or PKG-INFO file
Returns:
List of (package_name, version_constraint) tuples
"""
dependencies = []
for line in metadata_content.split('\n'):
if line.startswith('Requires-Dist:'):
# Extract the value after "Requires-Dist:"
value = line[len('Requires-Dist:'):].strip()
pkg_name, version = _parse_requires_dist(value)
if pkg_name:
dependencies.append((pkg_name, version))
return dependencies
def _extract_metadata_from_wheel(content: bytes) -> Optional[str]:
"""Extract METADATA file content from a wheel (zip) file.
Wheel files have structure: {package}-{version}.dist-info/METADATA
Args:
content: The wheel file content as bytes
Returns:
METADATA file content as string, or None if not found
"""
try:
with zipfile.ZipFile(BytesIO(content)) as zf:
# Find the .dist-info directory
for name in zf.namelist():
if name.endswith('.dist-info/METADATA'):
return zf.read(name).decode('utf-8', errors='replace')
except Exception as e:
logger.warning(f"Failed to extract metadata from wheel: {e}")
return None
def _extract_metadata_from_sdist(content: bytes, filename: str) -> Optional[str]:
"""Extract PKG-INFO file content from a source distribution (.tar.gz).
Source distributions have structure: {package}-{version}/PKG-INFO
Args:
content: The tarball content as bytes
filename: The original filename (used to determine package name)
Returns:
PKG-INFO file content as string, or None if not found
"""
try:
with tarfile.open(fileobj=BytesIO(content), mode='r:gz') as tf:
# Find PKG-INFO in the root directory of the archive
for member in tf.getmembers():
if member.name.endswith('/PKG-INFO') and member.name.count('/') == 1:
f = tf.extractfile(member)
if f:
return f.read().decode('utf-8', errors='replace')
except Exception as e:
logger.warning(f"Failed to extract metadata from sdist {filename}: {e}")
return None
def _extract_dependencies(content: bytes, filename: str) -> List[Tuple[str, Optional[str]]]:
"""Extract dependencies from a PyPI package file.
Supports wheel (.whl) and source distribution (.tar.gz) formats.
Args:
content: The package file content as bytes
filename: The original filename
Returns:
List of (package_name, version_constraint) tuples
"""
metadata = None
if filename.endswith('.whl'):
metadata = _extract_metadata_from_wheel(content)
elif filename.endswith('.tar.gz'):
metadata = _extract_metadata_from_sdist(content, filename)
if metadata:
return _extract_requires_from_metadata(metadata)
return []
# Timeout configuration for proxy requests
PROXY_CONNECT_TIMEOUT = 30.0
PROXY_READ_TIMEOUT = 60.0
def _extract_pypi_version(filename: str) -> Optional[str]:
"""Extract version from PyPI filename.
Handles formats like:
- cowsay-6.1-py3-none-any.whl
- cowsay-1.0.tar.gz
- some_package-1.2.3.post1-cp39-cp39-linux_x86_64.whl
"""
# Remove extension
if filename.endswith('.whl'):
# Wheel: name-version-pytag-abitag-platform.whl
parts = filename[:-4].split('-')
if len(parts) >= 2:
return parts[1]
elif filename.endswith('.tar.gz'):
# Source: name-version.tar.gz
base = filename[:-7]
# Find the last hyphen that precedes a version-like string
match = re.match(r'^(.+)-(\d+.*)$', base)
if match:
return match.group(2)
elif filename.endswith('.zip'):
# Egg/zip: name-version.zip
base = filename[:-4]
match = re.match(r'^(.+)-(\d+.*)$', base)
if match:
return match.group(2)
return None
def _get_pypi_upstream_sources(db: Session) -> list[UpstreamSource]:
"""Get all enabled upstream sources configured for PyPI."""
# Get database sources
db_sources = (
db.query(UpstreamSource)
.filter(
UpstreamSource.source_type == "pypi",
UpstreamSource.enabled == True,
)
.order_by(UpstreamSource.priority)
.all()
)
# Get env sources
env_sources = [
s for s in get_env_upstream_sources()
if s.source_type == "pypi" and s.enabled
]
# Combine and sort by priority
all_sources = list(db_sources) + list(env_sources)
return sorted(all_sources, key=lambda s: s.priority)
def _build_auth_headers(source) -> dict:
"""Build authentication headers for an upstream source."""
headers = {}
if hasattr(source, 'auth_type'):
if source.auth_type == "bearer":
password = source.get_password() if hasattr(source, 'get_password') else getattr(source, 'password', None)
if password:
headers["Authorization"] = f"Bearer {password}"
elif source.auth_type == "api_key":
custom_headers = source.get_headers() if hasattr(source, 'get_headers') else {}
if custom_headers:
headers.update(custom_headers)
return headers
def _get_basic_auth(source) -> Optional[tuple[str, str]]:
"""Get basic auth credentials if applicable."""
if hasattr(source, 'auth_type') and source.auth_type == "basic":
username = getattr(source, 'username', None)
if username:
password = source.get_password() if hasattr(source, 'get_password') else getattr(source, 'password', '')
return (username, password or '')
return None
def _get_base_url(request: Request) -> str:
"""
Get the external base URL, respecting X-Forwarded-Proto header.
When behind a reverse proxy that terminates SSL, the request.base_url
will show http:// even though the external URL is https://. This function
checks the X-Forwarded-Proto header to determine the correct scheme.
"""
base_url = str(request.base_url).rstrip('/')
# Check for X-Forwarded-Proto header (set by reverse proxies)
forwarded_proto = request.headers.get('x-forwarded-proto')
if forwarded_proto:
# Replace the scheme with the forwarded protocol
parsed = urlparse(base_url)
base_url = f"{forwarded_proto}://{parsed.netloc}{parsed.path}"
return base_url
def _rewrite_package_links(html: str, base_url: str, package_name: str, upstream_base_url: str) -> str:
"""
Rewrite download links in a PyPI simple page to go through our proxy.
Args:
html: The HTML content from upstream
base_url: Our server's base URL
package_name: The package name for the URL path
upstream_base_url: The upstream URL used to fetch this page (for resolving relative URLs)
Returns:
HTML with rewritten download links
"""
# Pattern to match href attributes in anchor tags
# PyPI simple pages have links like:
# <a href="https://files.pythonhosted.org/packages/.../file.tar.gz#sha256=...">file.tar.gz</a>
# Or relative URLs from Artifactory like:
# <a href="../../packages/packages/62/35/.../requests-0.10.0.tar.gz#sha256=...">
def replace_href(match):
original_url = match.group(1)
# Resolve relative URLs to absolute using the upstream base URL
if not original_url.startswith(('http://', 'https://')):
# Split off fragment before resolving
url_without_fragment = original_url.split('#')[0]
fragment_part = original_url[len(url_without_fragment):]
absolute_url = urljoin(upstream_base_url, url_without_fragment) + fragment_part
else:
absolute_url = original_url
# Extract the filename from the URL
parsed = urlparse(absolute_url)
path_parts = parsed.path.split('/')
filename = path_parts[-1] if path_parts else ''
# Keep the hash fragment if present
fragment = f"#{parsed.fragment}" if parsed.fragment else ""
# Encode the absolute URL (without fragment) for safe transmission
encoded_url = quote(absolute_url.split('#')[0], safe='')
# Build new URL pointing to our proxy
new_url = f"{base_url}/pypi/simple/{package_name}/{filename}?upstream={encoded_url}{fragment}"
return f'href="{new_url}"'
# Match href="..." patterns
rewritten = re.sub(r'href="([^"]+)"', replace_href, html)
return rewritten
@router.get("/simple/")
async def pypi_simple_index(
request: Request,
db: Session = Depends(get_db),
):
"""
PyPI Simple API index - lists all packages.
Proxies to the first available upstream PyPI source.
"""
sources = _get_pypi_upstream_sources(db)
if not sources:
raise HTTPException(
status_code=503,
detail="No PyPI upstream sources configured"
)
# Try each source in priority order
last_error = None
for source in sources:
try:
headers = {"User-Agent": "Orchard-PyPI-Proxy/1.0"}
headers.update(_build_auth_headers(source))
auth = _get_basic_auth(source)
# Use URL as-is - users should provide full path including /simple
simple_url = source.url.rstrip('/') + '/'
timeout = httpx.Timeout(PROXY_READ_TIMEOUT, connect=PROXY_CONNECT_TIMEOUT)
with httpx.Client(timeout=timeout, follow_redirects=False) as client:
response = client.get(
simple_url,
headers=headers,
auth=auth,
)
# Handle redirects manually to avoid loops
if response.status_code in (301, 302, 303, 307, 308):
redirect_url = response.headers.get('location')
if redirect_url:
# Follow the redirect once
response = client.get(
redirect_url,
headers=headers,
auth=auth,
follow_redirects=False,
)
if response.status_code == 200:
# Return the index as-is (links are to package pages, not files)
# We could rewrite these too, but for now just proxy
content = response.text
# Rewrite package links to go through our proxy
base_url = _get_base_url(request)
content = re.sub(
r'href="([^"]+)/"',
lambda m: f'href="{base_url}/pypi/simple/{m.group(1)}/"',
content
)
return HTMLResponse(content=content)
last_error = f"HTTP {response.status_code}"
except httpx.ConnectError as e:
last_error = f"Connection failed: {e}"
logger.warning(f"PyPI proxy: failed to connect to {source.url}: {e}")
except httpx.TimeoutException as e:
last_error = f"Timeout: {e}"
logger.warning(f"PyPI proxy: timeout connecting to {source.url}: {e}")
except Exception as e:
last_error = str(e)
logger.warning(f"PyPI proxy: error fetching from {source.url}: {e}")
raise HTTPException(
status_code=502,
detail=f"Failed to fetch package index from upstream: {last_error}"
)
@router.get("/simple/{package_name}/")
async def pypi_package_versions(
request: Request,
package_name: str,
db: Session = Depends(get_db),
):
"""
PyPI Simple API package page - lists all versions/files for a package.
Proxies to upstream and rewrites download links to go through our cache.
"""
sources = _get_pypi_upstream_sources(db)
if not sources:
raise HTTPException(
status_code=503,
detail="No PyPI upstream sources configured"
)
base_url = _get_base_url(request)
# Normalize package name (PEP 503)
normalized_name = re.sub(r'[-_.]+', '-', package_name).lower()
# Try each source in priority order
last_error = None
for source in sources:
try:
headers = {"User-Agent": "Orchard-PyPI-Proxy/1.0"}
headers.update(_build_auth_headers(source))
auth = _get_basic_auth(source)
# Use URL as-is - users should provide full path including /simple
package_url = source.url.rstrip('/') + f'/{normalized_name}/'
final_url = package_url # Track final URL after redirects
timeout = httpx.Timeout(PROXY_READ_TIMEOUT, connect=PROXY_CONNECT_TIMEOUT)
with httpx.Client(timeout=timeout, follow_redirects=False) as client:
response = client.get(
package_url,
headers=headers,
auth=auth,
)
# Handle redirects manually
redirect_count = 0
while response.status_code in (301, 302, 303, 307, 308) and redirect_count < 5:
redirect_url = response.headers.get('location')
if not redirect_url:
break
# Make redirect URL absolute if needed
if not redirect_url.startswith('http'):
redirect_url = urljoin(final_url, redirect_url)
final_url = redirect_url # Update final URL
response = client.get(
redirect_url,
headers=headers,
auth=auth,
follow_redirects=False,
)
redirect_count += 1
if response.status_code == 200:
content = response.text
# Rewrite download links to go through our proxy
# Pass final_url so relative URLs can be resolved correctly
content = _rewrite_package_links(content, base_url, normalized_name, final_url)
return HTMLResponse(content=content)
if response.status_code == 404:
# Package not found in this source, try next
last_error = f"Package not found in {source.name}"
continue
last_error = f"HTTP {response.status_code}"
except httpx.ConnectError as e:
last_error = f"Connection failed: {e}"
logger.warning(f"PyPI proxy: failed to connect to {source.url}: {e}")
except httpx.TimeoutException as e:
last_error = f"Timeout: {e}"
logger.warning(f"PyPI proxy: timeout connecting to {source.url}: {e}")
except Exception as e:
last_error = str(e)
logger.warning(f"PyPI proxy: error fetching {package_name} from {source.url}: {e}")
raise HTTPException(
status_code=404,
detail=f"Package '{package_name}' not found: {last_error}"
)
def _cache_dependency_background(
base_url: str,
dep_name: str,
dep_version: Optional[str],
depth: int = 0,
max_depth: int = 10,
):
"""
Background task to proactively cache a dependency.
Fetches the dependency from upstream via our own proxy, which will
recursively cache its dependencies as well.
"""
if depth >= max_depth:
logger.warning(f"PyPI proxy: max depth {max_depth} reached caching {dep_name}")
return
try:
# Normalize package name for URL (PEP 503)
normalized_name = re.sub(r'[-_.]+', '-', dep_name).lower()
# First, get the simple index page to find available versions
simple_url = f"{base_url}/pypi/simple/{normalized_name}/"
logger.info(f"PyPI proxy: proactively caching {dep_name} (depth={depth})")
with httpx.Client(timeout=30.0) as client:
response = client.get(simple_url)
if response.status_code != 200:
logger.warning(f"PyPI proxy: failed to get index for {dep_name}: {response.status_code}")
return
# Parse the HTML to find wheel files
html = response.text
# Look for wheel files (.whl) - prefer them over sdist
wheel_pattern = rf'href="([^"]*{normalized_name}[^"]*\.whl[^"]*)"'
matches = re.findall(wheel_pattern, html, re.IGNORECASE)
if not matches:
# Try sdist
sdist_pattern = rf'href="([^"]*{normalized_name}[^"]*\.tar\.gz[^"]*)"'
matches = re.findall(sdist_pattern, html, re.IGNORECASE)
if not matches:
logger.warning(f"PyPI proxy: no downloadable files found for {dep_name}")
return
# Get the last match (usually the latest version)
# The URL might be relative or absolute
download_url = matches[-1]
if download_url.startswith('/'):
download_url = f"{base_url}{download_url}"
elif not download_url.startswith('http'):
download_url = f"{base_url}/pypi/simple/{normalized_name}/{download_url}"
# Download the file through our proxy (this will cache it)
logger.info(f"PyPI proxy: downloading dependency {dep_name} from {download_url}")
response = client.get(download_url)
if response.status_code == 200:
logger.info(f"PyPI proxy: successfully cached {dep_name}")
else:
logger.warning(f"PyPI proxy: failed to cache {dep_name}: {response.status_code}")
except Exception as e:
logger.warning(f"PyPI proxy: error caching dependency {dep_name}: {e}")
def _start_background_dependency_caching(base_url: str, dependencies: List[Tuple[str, Optional[str]]]):
"""Start background threads to cache dependencies."""
for dep_name, dep_version in dependencies:
# Use a thread to avoid blocking
thread = threading.Thread(
target=_cache_dependency_background,
args=(base_url, dep_name, dep_version),
daemon=True,
)
thread.start()
@router.get("/simple/{package_name}/{filename}")
async def pypi_download_file(
request: Request,
package_name: str,
filename: str,
upstream: Optional[str] = None,
db: Session = Depends(get_db),
storage: S3Storage = Depends(get_storage),
):
"""
Download a package file, caching it in Orchard.
Args:
package_name: The package name
filename: The filename to download
upstream: URL-encoded upstream URL to fetch from
"""
if not upstream:
raise HTTPException(
status_code=400,
detail="Missing 'upstream' query parameter with source URL"
)
# Decode the upstream URL
upstream_url = unquote(upstream)
# Check if we already have this URL cached
url_hash = hashlib.sha256(upstream_url.encode()).hexdigest()
cached_url = db.query(CachedUrl).filter(CachedUrl.url_hash == url_hash).first()
if cached_url:
# Serve from cache
artifact = db.query(Artifact).filter(Artifact.id == cached_url.artifact_id).first()
if artifact:
logger.info(f"PyPI proxy: serving cached {filename} (artifact {artifact.id[:12]})")
# Stream from S3
try:
stream, content_length, _ = storage.get_stream(artifact.s3_key)
def stream_content():
"""Generator that yields chunks from the S3 stream."""
try:
for chunk in stream.iter_chunks():
yield chunk
finally:
stream.close()
return StreamingResponse(
stream_content(),
media_type=artifact.content_type or "application/octet-stream",
headers={
"Content-Disposition": f'attachment; filename="{filename}"',
"Content-Length": str(content_length),
"X-Checksum-SHA256": artifact.id,
"X-Cache": "HIT",
}
)
except Exception as e:
logger.error(f"PyPI proxy: error streaming cached artifact: {e}")
# Fall through to fetch from upstream
# Not cached - fetch from upstream
sources = _get_pypi_upstream_sources(db)
# Use the first available source for authentication headers
# Note: The upstream URL may point to files.pythonhosted.org or other CDNs,
# not the configured source URL directly, so we can't strictly validate the host
matched_source = sources[0] if sources else None
try:
headers = {"User-Agent": "Orchard-PyPI-Proxy/1.0"}
if matched_source:
headers.update(_build_auth_headers(matched_source))
auth = _get_basic_auth(matched_source) if matched_source else None
timeout = httpx.Timeout(300.0, connect=PROXY_CONNECT_TIMEOUT) # 5 minutes for large files
# Fetch the file
logger.info(f"PyPI proxy: fetching {filename} from {upstream_url}")
with httpx.Client(timeout=timeout, follow_redirects=False) as client:
response = client.get(
upstream_url,
headers=headers,
auth=auth,
)
# Handle redirects manually
redirect_count = 0
while response.status_code in (301, 302, 303, 307, 308) and redirect_count < 5:
redirect_url = response.headers.get('location')
if not redirect_url:
break
if not redirect_url.startswith('http'):
redirect_url = urljoin(upstream_url, redirect_url)
logger.info(f"PyPI proxy: following redirect to {redirect_url}")
# Don't send auth to different hosts
redirect_headers = {"User-Agent": "Orchard-PyPI-Proxy/1.0"}
redirect_auth = None
if urlparse(redirect_url).netloc == urlparse(upstream_url).netloc:
redirect_headers.update(headers)
redirect_auth = auth
response = client.get(
redirect_url,
headers=redirect_headers,
auth=redirect_auth,
follow_redirects=False,
)
redirect_count += 1
if response.status_code != 200:
raise HTTPException(
status_code=response.status_code,
detail=f"Upstream returned {response.status_code}"
)
content = response.content
content_type = response.headers.get('content-type', 'application/octet-stream')
# Store in S3 (computes hash and deduplicates automatically)
from io import BytesIO
result = storage.store(BytesIO(content))
sha256 = result.sha256
size = result.size
logger.info(f"PyPI proxy: downloaded {filename}, {size} bytes, sha256={sha256[:12]}")
# Check if artifact already exists
existing = db.query(Artifact).filter(Artifact.id == sha256).first()
if existing:
# Increment ref count
existing.ref_count += 1
db.flush()
else:
# Create artifact record
new_artifact = Artifact(
id=sha256,
original_name=filename,
content_type=content_type,
size=size,
ref_count=1,
created_by="pypi-proxy",
s3_key=result.s3_key,
checksum_md5=result.md5,
checksum_sha1=result.sha1,
s3_etag=result.s3_etag,
)
db.add(new_artifact)
db.flush()
# Create/get system project and package
system_project = db.query(Project).filter(Project.name == "_pypi").first()
if not system_project:
system_project = Project(
name="_pypi",
description="System project for cached PyPI packages",
is_public=True,
is_system=True,
created_by="pypi-proxy",
)
db.add(system_project)
db.flush()
elif not system_project.is_system:
# Ensure existing project is marked as system
system_project.is_system = True
db.flush()
# Normalize package name
normalized_name = re.sub(r'[-_.]+', '-', package_name).lower()
package = db.query(Package).filter(
Package.project_id == system_project.id,
Package.name == normalized_name,
).first()
if not package:
package = Package(
project_id=system_project.id,
name=normalized_name,
description=f"PyPI package: {normalized_name}",
format="pypi",
)
db.add(package)
db.flush()
# Create tag with filename
existing_tag = db.query(Tag).filter(
Tag.package_id == package.id,
Tag.name == filename,
).first()
if not existing_tag:
tag = Tag(
package_id=package.id,
name=filename,
artifact_id=sha256,
created_by="pypi-proxy",
)
db.add(tag)
# Extract and create version
# Only create version for actual package files, not .metadata files
version = _extract_pypi_version(filename)
if version and not filename.endswith('.metadata'):
# Check by version string (the unique constraint is on package_id + version)
existing_version = db.query(PackageVersion).filter(
PackageVersion.package_id == package.id,
PackageVersion.version == version,
).first()
if not existing_version:
pkg_version = PackageVersion(
package_id=package.id,
artifact_id=sha256,
version=version,
version_source="filename",
created_by="pypi-proxy",
)
db.add(pkg_version)
# Cache the URL mapping
existing_cached = db.query(CachedUrl).filter(CachedUrl.url_hash == url_hash).first()
if not existing_cached:
cached_url_record = CachedUrl(
url_hash=url_hash,
url=upstream_url,
artifact_id=sha256,
)
db.add(cached_url_record)
# Extract and store dependencies
dependencies = _extract_dependencies(content, filename)
unique_deps = []
if dependencies:
# Deduplicate dependencies by package name (keep first occurrence)
seen_packages = set()
for dep_name, dep_version in dependencies:
if dep_name not in seen_packages:
seen_packages.add(dep_name)
unique_deps.append((dep_name, dep_version))
logger.info(f"PyPI proxy: extracted {len(unique_deps)} dependencies from {filename} (deduped from {len(dependencies)})")
for dep_name, dep_version in unique_deps:
# Check if this dependency already exists for this artifact
existing_dep = db.query(ArtifactDependency).filter(
ArtifactDependency.artifact_id == sha256,
ArtifactDependency.dependency_project == "_pypi",
ArtifactDependency.dependency_package == dep_name,
).first()
if not existing_dep:
dep = ArtifactDependency(
artifact_id=sha256,
dependency_project="_pypi",
dependency_package=dep_name,
version_constraint=dep_version if dep_version else "*",
)
db.add(dep)
db.commit()
# Proactively cache dependencies in the background
if unique_deps:
base_url = str(request.base_url).rstrip("/")
_start_background_dependency_caching(base_url, unique_deps)
# Return the file
return Response(
content=content,
media_type=content_type,
headers={
"Content-Disposition": f'attachment; filename="{filename}"',
"Content-Length": str(size),
"X-Checksum-SHA256": sha256,
"X-Cache": "MISS",
}
)
except httpx.ConnectError as e:
raise HTTPException(status_code=502, detail=f"Connection failed: {e}")
except httpx.TimeoutException as e:
raise HTTPException(status_code=504, detail=f"Timeout: {e}")
except HTTPException:
raise
except Exception as e:
logger.exception(f"PyPI proxy: error downloading {filename}")
raise HTTPException(status_code=500, detail=str(e))

View File

@@ -1680,6 +1680,7 @@ def create_project(
name=db_project.name, name=db_project.name,
description=db_project.description, description=db_project.description,
is_public=db_project.is_public, is_public=db_project.is_public,
is_system=db_project.is_system,
created_at=db_project.created_at, created_at=db_project.created_at,
updated_at=db_project.updated_at, updated_at=db_project.updated_at,
created_by=db_project.created_by, created_by=db_project.created_by,
@@ -1704,6 +1705,7 @@ def get_project(
name=project.name, name=project.name,
description=project.description, description=project.description,
is_public=project.is_public, is_public=project.is_public,
is_system=project.is_system,
created_at=project.created_at, created_at=project.created_at,
updated_at=project.updated_at, updated_at=project.updated_at,
created_by=project.created_by, created_by=project.created_by,
@@ -2704,6 +2706,7 @@ def list_team_projects(
name=p.name, name=p.name,
description=p.description, description=p.description,
is_public=p.is_public, is_public=p.is_public,
is_system=p.is_system,
created_at=p.created_at, created_at=p.created_at,
updated_at=p.updated_at, updated_at=p.updated_at,
created_by=p.created_by, created_by=p.created_by,
@@ -2827,14 +2830,15 @@ def list_packages(
db.query(func.count(Tag.id)).filter(Tag.package_id == pkg.id).scalar() or 0 db.query(func.count(Tag.id)).filter(Tag.package_id == pkg.id).scalar() or 0
) )
# Get unique artifact count and total size via uploads # Get unique artifact count and total size via tags
# (PyPI proxy creates tags without uploads, so query from tags)
artifact_stats = ( artifact_stats = (
db.query( db.query(
func.count(func.distinct(Upload.artifact_id)), func.count(func.distinct(Tag.artifact_id)),
func.coalesce(func.sum(Artifact.size), 0), func.coalesce(func.sum(Artifact.size), 0),
) )
.join(Artifact, Upload.artifact_id == Artifact.id) .join(Artifact, Tag.artifact_id == Artifact.id)
.filter(Upload.package_id == pkg.id) .filter(Tag.package_id == pkg.id)
.first() .first()
) )
artifact_count = artifact_stats[0] if artifact_stats else 0 artifact_count = artifact_stats[0] if artifact_stats else 0
@@ -2930,14 +2934,15 @@ def get_package(
db.query(func.count(Tag.id)).filter(Tag.package_id == pkg.id).scalar() or 0 db.query(func.count(Tag.id)).filter(Tag.package_id == pkg.id).scalar() or 0
) )
# Get unique artifact count and total size via uploads # Get unique artifact count and total size via tags
# (PyPI proxy creates tags without uploads, so query from tags)
artifact_stats = ( artifact_stats = (
db.query( db.query(
func.count(func.distinct(Upload.artifact_id)), func.count(func.distinct(Tag.artifact_id)),
func.coalesce(func.sum(Artifact.size), 0), func.coalesce(func.sum(Artifact.size), 0),
) )
.join(Artifact, Upload.artifact_id == Artifact.id) .join(Artifact, Tag.artifact_id == Artifact.id)
.filter(Upload.package_id == pkg.id) .filter(Tag.package_id == pkg.id)
.first() .first()
) )
artifact_count = artifact_stats[0] if artifact_stats else 0 artifact_count = artifact_stats[0] if artifact_stats else 0
@@ -6280,14 +6285,14 @@ def get_package_stats(
db.query(func.count(Tag.id)).filter(Tag.package_id == package.id).scalar() or 0 db.query(func.count(Tag.id)).filter(Tag.package_id == package.id).scalar() or 0
) )
# Artifact stats via uploads # Artifact stats via tags (tags exist for both user uploads and PyPI proxy)
artifact_stats = ( artifact_stats = (
db.query( db.query(
func.count(func.distinct(Upload.artifact_id)), func.count(func.distinct(Tag.artifact_id)),
func.coalesce(func.sum(Artifact.size), 0), func.coalesce(func.sum(Artifact.size), 0),
) )
.join(Artifact, Upload.artifact_id == Artifact.id) .join(Artifact, Tag.artifact_id == Artifact.id)
.filter(Upload.package_id == package.id) .filter(Tag.package_id == package.id)
.first() .first()
) )
artifact_count = artifact_stats[0] if artifact_stats else 0 artifact_count = artifact_stats[0] if artifact_stats else 0
@@ -8305,6 +8310,200 @@ def _create_user_cache_reference(
return f"{user_project_name}/{user_package_name}" return f"{user_project_name}/{user_package_name}"
# --- Cache Resolve Endpoint ---
from .schemas import CacheResolveRequest
@router.post(
"/api/v1/cache/resolve",
response_model=CacheResponse,
tags=["cache"],
summary="Cache an artifact by package coordinates",
)
def cache_resolve(
request: Request,
resolve_request: CacheResolveRequest,
db: Session = Depends(get_db),
storage: S3Storage = Depends(get_storage),
current_user: User = Depends(get_current_user),
):
"""
Cache an artifact by package coordinates (no URL required).
The server finds the appropriate download URL based on source_type
and configured upstream sources. Currently supports PyPI packages.
**Request Body:**
- `source_type` (required): Type of source (pypi, npm, maven, etc.)
- `package` (required): Package name
- `version` (required): Package version
- `user_project` (optional): Also create reference in this user project
- `user_package` (optional): Package name in user project
- `user_tag` (optional): Tag name in user project
**Example (curl):**
```bash
curl -X POST "http://localhost:8080/api/v1/cache/resolve" \\
-H "Authorization: Bearer <api-key>" \\
-H "Content-Type: application/json" \\
-d '{
"source_type": "pypi",
"package": "requests",
"version": "2.31.0"
}'
```
"""
import re
import httpx
from urllib.parse import quote, unquote
if resolve_request.source_type != "pypi":
raise HTTPException(
status_code=501,
detail=f"Cache resolve for '{resolve_request.source_type}' not yet implemented. Currently only 'pypi' is supported."
)
# Get PyPI upstream sources
sources = (
db.query(UpstreamSource)
.filter(
UpstreamSource.source_type == "pypi",
UpstreamSource.enabled == True,
)
.order_by(UpstreamSource.priority)
.all()
)
# Also get env sources
env_sources = [
s for s in get_env_upstream_sources()
if s.source_type == "pypi" and s.enabled
]
all_sources = list(sources) + list(env_sources)
all_sources = sorted(all_sources, key=lambda s: s.priority)
if not all_sources:
raise HTTPException(
status_code=503,
detail="No PyPI upstream sources configured"
)
# Normalize package name (PEP 503)
normalized_package = re.sub(r'[-_.]+', '-', resolve_request.package).lower()
# Query the Simple API to find the download URL
download_url = None
matched_filename = None
last_error = None
for source in all_sources:
try:
headers = {"User-Agent": "Orchard-CacheResolver/1.0"}
# Build auth if needed
if hasattr(source, 'auth_type'):
if source.auth_type == "bearer":
password = source.get_password() if hasattr(source, 'get_password') else getattr(source, 'password', None)
if password:
headers["Authorization"] = f"Bearer {password}"
elif source.auth_type == "api_key":
custom_headers = source.get_headers() if hasattr(source, 'get_headers') else {}
if custom_headers:
headers.update(custom_headers)
auth = None
if hasattr(source, 'auth_type') and source.auth_type == "basic":
username = getattr(source, 'username', None)
if username:
password = source.get_password() if hasattr(source, 'get_password') else getattr(source, 'password', '')
auth = (username, password or '')
source_url = getattr(source, 'url', '')
package_url = source_url.rstrip('/') + f'/simple/{normalized_package}/'
timeout = httpx.Timeout(connect=30.0, read=60.0)
with httpx.Client(timeout=timeout, follow_redirects=True) as client:
response = client.get(package_url, headers=headers, auth=auth)
if response.status_code == 404:
last_error = f"Package not found in {getattr(source, 'name', 'source')}"
continue
if response.status_code != 200:
last_error = f"HTTP {response.status_code} from {getattr(source, 'name', 'source')}"
continue
# Parse HTML to find the version
html = response.text
# Look for links containing the version
# Pattern: href="...{package}-{version}...#sha256=..."
version_pattern = re.escape(resolve_request.version)
link_pattern = rf'href="([^"]+{normalized_package}[^"]*{version_pattern}[^"]*)"'
matches = re.findall(link_pattern, html, re.IGNORECASE)
if not matches:
# Try with original package name
link_pattern = rf'href="([^"]+{re.escape(resolve_request.package)}[^"]*{version_pattern}[^"]*)"'
matches = re.findall(link_pattern, html, re.IGNORECASE)
if matches:
# Prefer .tar.gz or .whl files
for match in matches:
url = match.split('#')[0] # Remove hash fragment
if url.endswith('.tar.gz') or url.endswith('.whl'):
download_url = url
# Extract filename
matched_filename = url.split('/')[-1]
break
if not download_url:
# Use first match
download_url = matches[0].split('#')[0]
matched_filename = download_url.split('/')[-1]
break
last_error = f"Version {resolve_request.version} not found for {resolve_request.package}"
except httpx.ConnectError as e:
last_error = f"Connection failed: {e}"
logger.warning(f"Cache resolve: failed to connect to {getattr(source, 'url', 'source')}: {e}")
except httpx.TimeoutException as e:
last_error = f"Timeout: {e}"
logger.warning(f"Cache resolve: timeout connecting to {getattr(source, 'url', 'source')}: {e}")
except Exception as e:
last_error = str(e)
logger.warning(f"Cache resolve: error: {e}")
if not download_url:
raise HTTPException(
status_code=404,
detail=f"Could not find {resolve_request.package}=={resolve_request.version}: {last_error}"
)
# Now cache the artifact using the existing cache_artifact logic
# Construct a CacheRequest
cache_request = CacheRequest(
url=download_url,
source_type="pypi",
package_name=normalized_package,
tag=matched_filename or resolve_request.version,
user_project=resolve_request.user_project,
user_package=resolve_request.user_package,
user_tag=resolve_request.user_tag,
)
# Call the cache logic
return cache_artifact(
request=request,
cache_request=cache_request,
db=db,
storage=storage,
current_user=current_user,
)
# --- Upstream Sources Admin API --- # --- Upstream Sources Admin API ---
from .schemas import ( from .schemas import (

View File

@@ -33,6 +33,7 @@ class ProjectResponse(BaseModel):
name: str name: str
description: Optional[str] description: Optional[str]
is_public: bool is_public: bool
is_system: bool = False
created_at: datetime created_at: datetime
updated_at: datetime updated_at: datetime
created_by: str created_by: str
@@ -1432,4 +1433,41 @@ class CacheResponse(BaseModel):
user_reference: Optional[str] = None # e.g., "my-app/npm-deps:lodash-4.17.21" user_reference: Optional[str] = None # e.g., "my-app/npm-deps:lodash-4.17.21"
class CacheResolveRequest(BaseModel):
"""Request to cache an artifact by package coordinates (no URL required).
The server will construct the appropriate URL based on source_type and
configured upstream sources.
"""
source_type: str
package: str
version: str
user_project: Optional[str] = None
user_package: Optional[str] = None
user_tag: Optional[str] = None
@field_validator('source_type')
@classmethod
def validate_source_type(cls, v: str) -> str:
if v not in SOURCE_TYPES:
raise ValueError(f"source_type must be one of: {', '.join(SOURCE_TYPES)}")
return v
@field_validator('package')
@classmethod
def validate_package(cls, v: str) -> str:
v = v.strip()
if not v:
raise ValueError("package cannot be empty")
return v
@field_validator('version')
@classmethod
def validate_version(cls, v: str) -> str:
v = v.strip()
if not v:
raise ValueError("version cannot be empty")
return v

View File

@@ -0,0 +1 @@
# Scripts package

View File

@@ -0,0 +1,262 @@
#!/usr/bin/env python3
"""
Backfill script to extract dependencies from cached PyPI packages.
This script scans all artifacts in the _pypi project and extracts
Requires-Dist metadata from wheel and sdist files that don't already
have dependencies recorded.
Usage:
# From within the container:
python -m scripts.backfill_pypi_dependencies
# Or with docker exec:
docker exec orchard_orchard-server_1 python -m scripts.backfill_pypi_dependencies
# Dry run (preview only):
docker exec orchard_orchard-server_1 python -m scripts.backfill_pypi_dependencies --dry-run
"""
import argparse
import logging
import re
import sys
import tarfile
import zipfile
from io import BytesIO
from typing import List, Optional, Tuple
# Add parent directory to path for imports
sys.path.insert(0, "/app")
from sqlalchemy import create_engine
from sqlalchemy.orm import sessionmaker
from backend.app.config import get_settings
from backend.app.models import (
Artifact,
ArtifactDependency,
Package,
Project,
Tag,
)
from backend.app.storage import get_storage
logging.basicConfig(
level=logging.INFO,
format="%(asctime)s - %(levelname)s - %(message)s",
)
logger = logging.getLogger(__name__)
def parse_requires_dist(requires_dist: str) -> Tuple[Optional[str], Optional[str]]:
"""Parse a Requires-Dist line into (package_name, version_constraint)."""
# Remove any environment markers (after semicolon)
if ";" in requires_dist:
requires_dist = requires_dist.split(";")[0].strip()
# Match patterns like "package (>=1.0)" or "package>=1.0" or "package"
match = re.match(
r"^([a-zA-Z0-9][-a-zA-Z0-9._]*)\s*(?:\(([^)]+)\)|([<>=!~][^\s;]+))?",
requires_dist.strip(),
)
if not match:
return None, None
package_name = match.group(1)
version_constraint = match.group(2) or match.group(3)
# Normalize package name (PEP 503)
normalized_name = re.sub(r"[-_.]+", "-", package_name).lower()
if version_constraint:
version_constraint = version_constraint.strip()
return normalized_name, version_constraint
def extract_requires_from_metadata(metadata_content: str) -> List[Tuple[str, Optional[str]]]:
"""Extract all Requires-Dist entries from METADATA/PKG-INFO content."""
dependencies = []
for line in metadata_content.split("\n"):
if line.startswith("Requires-Dist:"):
value = line[len("Requires-Dist:"):].strip()
pkg_name, version = parse_requires_dist(value)
if pkg_name:
dependencies.append((pkg_name, version))
return dependencies
def extract_metadata_from_wheel(content: bytes) -> Optional[str]:
"""Extract METADATA file content from a wheel (zip) file."""
try:
with zipfile.ZipFile(BytesIO(content)) as zf:
for name in zf.namelist():
if name.endswith(".dist-info/METADATA"):
return zf.read(name).decode("utf-8", errors="replace")
except Exception as e:
logger.warning(f"Failed to extract metadata from wheel: {e}")
return None
def extract_metadata_from_sdist(content: bytes) -> Optional[str]:
"""Extract PKG-INFO file content from a source distribution (.tar.gz)."""
try:
with tarfile.open(fileobj=BytesIO(content), mode="r:gz") as tf:
for member in tf.getmembers():
if member.name.endswith("/PKG-INFO") and member.name.count("/") == 1:
f = tf.extractfile(member)
if f:
return f.read().decode("utf-8", errors="replace")
except Exception as e:
logger.warning(f"Failed to extract metadata from sdist: {e}")
return None
def extract_dependencies(content: bytes, filename: str) -> List[Tuple[str, Optional[str]]]:
"""Extract dependencies from a PyPI package file."""
metadata = None
if filename.endswith(".whl"):
metadata = extract_metadata_from_wheel(content)
elif filename.endswith(".tar.gz"):
metadata = extract_metadata_from_sdist(content)
if metadata:
return extract_requires_from_metadata(metadata)
return []
def backfill_dependencies(dry_run: bool = False):
"""Main backfill function."""
settings = get_settings()
# Create database connection
engine = create_engine(settings.database_url)
Session = sessionmaker(bind=engine)
db = Session()
# Create storage client
storage = get_storage()
try:
# Find the _pypi project
pypi_project = db.query(Project).filter(Project.name == "_pypi").first()
if not pypi_project:
logger.info("No _pypi project found. Nothing to backfill.")
return
# Get all packages in _pypi
packages = db.query(Package).filter(Package.project_id == pypi_project.id).all()
logger.info(f"Found {len(packages)} packages in _pypi project")
total_artifacts = 0
artifacts_with_deps = 0
artifacts_processed = 0
dependencies_added = 0
for package in packages:
# Get all tags (each tag points to an artifact)
tags = db.query(Tag).filter(Tag.package_id == package.id).all()
for tag in tags:
total_artifacts += 1
filename = tag.name
# Skip non-package files (like .metadata files)
if not (filename.endswith(".whl") or filename.endswith(".tar.gz")):
continue
# Check if this artifact already has dependencies
existing_deps = db.query(ArtifactDependency).filter(
ArtifactDependency.artifact_id == tag.artifact_id
).count()
if existing_deps > 0:
artifacts_with_deps += 1
continue
# Get the artifact
artifact = db.query(Artifact).filter(Artifact.id == tag.artifact_id).first()
if not artifact:
logger.warning(f"Artifact {tag.artifact_id} not found for tag {filename}")
continue
logger.info(f"Processing {package.name}/{filename}...")
if dry_run:
logger.info(f" [DRY RUN] Would extract dependencies from {filename}")
artifacts_processed += 1
continue
# Download the artifact from S3
try:
content = storage.get(artifact.s3_key)
except Exception as e:
logger.error(f" Failed to download {filename}: {e}")
continue
# Extract dependencies
deps = extract_dependencies(content, filename)
if deps:
logger.info(f" Found {len(deps)} dependencies")
for dep_name, dep_version in deps:
# Check if already exists (race condition protection)
existing = db.query(ArtifactDependency).filter(
ArtifactDependency.artifact_id == tag.artifact_id,
ArtifactDependency.dependency_project == "_pypi",
ArtifactDependency.dependency_package == dep_name,
).first()
if not existing:
dep = ArtifactDependency(
artifact_id=tag.artifact_id,
dependency_project="_pypi",
dependency_package=dep_name,
version_constraint=dep_version if dep_version else "*",
)
db.add(dep)
dependencies_added += 1
logger.info(f" + {dep_name} {dep_version or '*'}")
db.commit()
else:
logger.info(f" No dependencies found")
artifacts_processed += 1
logger.info("")
logger.info("=" * 50)
logger.info("Backfill complete!")
logger.info(f" Total artifacts: {total_artifacts}")
logger.info(f" Already had deps: {artifacts_with_deps}")
logger.info(f" Processed: {artifacts_processed}")
logger.info(f" Dependencies added: {dependencies_added}")
if dry_run:
logger.info(" (DRY RUN - no changes made)")
finally:
db.close()
def main():
parser = argparse.ArgumentParser(
description="Backfill dependencies for cached PyPI packages"
)
parser.add_argument(
"--dry-run",
action="store_true",
help="Preview what would be done without making changes",
)
args = parser.parse_args()
backfill_dependencies(dry_run=args.dry_run)
if __name__ == "__main__":
main()

View File

@@ -128,7 +128,9 @@ class TestProjectListingFilters:
assert response.status_code == 200 assert response.status_code == 200
data = response.json() data = response.json()
names = [p["name"] for p in data["items"]] # Filter out system projects (names starting with "_") as they may have
# collation-specific sort behavior and aren't part of the test data
names = [p["name"] for p in data["items"] if not p["name"].startswith("_")]
assert names == sorted(names) assert names == sorted(names)

View File

@@ -0,0 +1,137 @@
"""Integration tests for PyPI transparent proxy."""
import os
import pytest
import httpx
def get_base_url():
"""Get the base URL for the Orchard server from environment."""
return os.environ.get("ORCHARD_TEST_URL", "http://localhost:8080")
class TestPyPIProxyEndpoints:
"""Tests for PyPI proxy endpoints.
These endpoints are public (no auth required) since pip needs to use them.
"""
@pytest.mark.integration
def test_pypi_simple_index(self):
"""Test that /pypi/simple/ returns HTML response."""
with httpx.Client(base_url=get_base_url(), timeout=30.0) as client:
response = client.get("/pypi/simple/")
# Returns 200 if sources configured, 503 if not
assert response.status_code in (200, 503)
if response.status_code == 200:
assert "text/html" in response.headers.get("content-type", "")
else:
assert "No PyPI upstream sources configured" in response.json()["detail"]
@pytest.mark.integration
def test_pypi_package_endpoint(self):
"""Test that /pypi/simple/{package}/ returns appropriate response."""
with httpx.Client(base_url=get_base_url(), timeout=30.0) as client:
response = client.get("/pypi/simple/requests/")
# Returns 200 if sources configured and package found,
# 404 if package not found, 503 if no sources
assert response.status_code in (200, 404, 503)
if response.status_code == 200:
assert "text/html" in response.headers.get("content-type", "")
elif response.status_code == 404:
assert "not found" in response.json()["detail"].lower()
else: # 503
assert "No PyPI upstream sources configured" in response.json()["detail"]
@pytest.mark.integration
def test_pypi_download_missing_upstream_param(self):
"""Test that /pypi/simple/{package}/{filename} requires upstream param."""
with httpx.Client(base_url=get_base_url(), timeout=30.0) as client:
response = client.get("/pypi/simple/requests/requests-2.31.0.tar.gz")
assert response.status_code == 400
assert "upstream" in response.json()["detail"].lower()
class TestPyPILinkRewriting:
"""Tests for URL rewriting in PyPI proxy responses."""
def test_rewrite_package_links(self):
"""Test that download links are rewritten to go through proxy."""
from app.pypi_proxy import _rewrite_package_links
html = '''
<html>
<body>
<a href="https://files.pythonhosted.org/packages/ab/cd/requests-2.31.0.tar.gz#sha256=abc123">requests-2.31.0.tar.gz</a>
<a href="https://files.pythonhosted.org/packages/ef/gh/requests-2.31.0-py3-none-any.whl#sha256=def456">requests-2.31.0-py3-none-any.whl</a>
</body>
</html>
'''
# upstream_base_url is used to resolve relative URLs (not needed here since URLs are absolute)
result = _rewrite_package_links(
html,
"http://localhost:8080",
"requests",
"https://pypi.org/simple/requests/"
)
# Links should be rewritten to go through our proxy
assert "/pypi/simple/requests/requests-2.31.0.tar.gz?upstream=" in result
assert "/pypi/simple/requests/requests-2.31.0-py3-none-any.whl?upstream=" in result
# Original URLs should be encoded in upstream param
assert "files.pythonhosted.org" in result
# Hash fragments should be preserved
assert "#sha256=abc123" in result
assert "#sha256=def456" in result
def test_rewrite_relative_links(self):
"""Test that relative URLs are resolved to absolute URLs."""
from app.pypi_proxy import _rewrite_package_links
# Artifactory-style relative URLs
html = '''
<html>
<body>
<a href="../../packages/ab/cd/requests-2.31.0.tar.gz#sha256=abc123">requests-2.31.0.tar.gz</a>
</body>
</html>
'''
result = _rewrite_package_links(
html,
"https://orchard.example.com",
"requests",
"https://artifactory.example.com/api/pypi/pypi-remote/simple/requests/"
)
# The relative URL should be resolved to absolute
# ../../packages/ab/cd/... from /api/pypi/pypi-remote/simple/requests/ resolves to /api/pypi/pypi-remote/packages/ab/cd/...
assert "upstream=https%3A%2F%2Fartifactory.example.com%2Fapi%2Fpypi%2Fpypi-remote%2Fpackages" in result
# Hash fragment should be preserved
assert "#sha256=abc123" in result
class TestPyPIPackageNormalization:
"""Tests for PyPI package name normalization."""
@pytest.mark.integration
def test_package_name_normalized(self):
"""Test that package names are normalized per PEP 503.
Different capitalizations/separators should all be valid paths.
The endpoint normalizes to lowercase with hyphens before lookup.
"""
with httpx.Client(base_url=get_base_url(), timeout=30.0) as client:
# Test various name formats - all should be valid endpoint paths
for package_name in ["Requests", "some_package", "some-package"]:
response = client.get(f"/pypi/simple/{package_name}/")
# 200 = found, 404 = not found, 503 = no sources configured
assert response.status_code in (200, 404, 503), \
f"Unexpected status {response.status_code} for {package_name}"
# Verify response is appropriate for the status code
if response.status_code == 200:
assert "text/html" in response.headers.get("content-type", "")
elif response.status_code == 503:
assert "No PyPI upstream sources configured" in response.json()["detail"]

View File

@@ -46,8 +46,6 @@ import {
UpstreamSourceCreate, UpstreamSourceCreate,
UpstreamSourceUpdate, UpstreamSourceUpdate,
UpstreamSourceTestResult, UpstreamSourceTestResult,
CacheSettings,
CacheSettingsUpdate,
} from './types'; } from './types';
const API_BASE = '/api/v1'; const API_BASE = '/api/v1';
@@ -748,21 +746,3 @@ export async function testUpstreamSource(id: string): Promise<UpstreamSourceTest
}); });
return handleResponse<UpstreamSourceTestResult>(response); return handleResponse<UpstreamSourceTestResult>(response);
} }
// Cache Settings Admin API
export async function getCacheSettings(): Promise<CacheSettings> {
const response = await fetch(`${API_BASE}/admin/cache-settings`, {
credentials: 'include',
});
return handleResponse<CacheSettings>(response);
}
export async function updateCacheSettings(data: CacheSettingsUpdate): Promise<CacheSettings> {
const response = await fetch(`${API_BASE}/admin/cache-settings`, {
method: 'PUT',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify(data),
credentials: 'include',
});
return handleResponse<CacheSettings>(response);
}

View File

@@ -272,7 +272,7 @@
.footer { .footer {
background: var(--bg-secondary); background: var(--bg-secondary);
border-top: 1px solid var(--border-primary); border-top: 1px solid var(--border-primary);
padding: 24px 0; padding: 12px 0;
} }
.footer-content { .footer-content {

View File

@@ -34,74 +34,6 @@
margin-bottom: 1rem; margin-bottom: 1rem;
} }
/* Settings Section */
.settings-section {
background: var(--bg-secondary);
border: 1px solid var(--border-color);
border-radius: 8px;
padding: 1.5rem;
margin-bottom: 2rem;
}
.settings-grid {
display: flex;
flex-direction: column;
gap: 1rem;
}
.setting-item {
display: flex;
justify-content: space-between;
align-items: center;
padding: 1rem;
background: var(--bg-primary);
border: 1px solid var(--border-color);
border-radius: 4px;
}
.toggle-label {
display: flex;
flex-direction: column;
gap: 0.25rem;
}
.setting-name {
font-weight: 500;
color: var(--text-primary);
display: flex;
align-items: center;
gap: 0.5rem;
}
.setting-description {
font-size: 0.85rem;
color: var(--text-secondary);
}
.toggle-button {
padding: 0.5rem 1rem;
border: none;
border-radius: 4px;
cursor: pointer;
font-weight: 500;
min-width: 100px;
}
.toggle-button.on {
background-color: #28a745;
color: white;
}
.toggle-button.off {
background-color: #dc3545;
color: white;
}
.toggle-button:disabled {
opacity: 0.6;
cursor: not-allowed;
}
/* Sources Section */ /* Sources Section */
.sources-section { .sources-section {
background: var(--bg-secondary); background: var(--bg-secondary);
@@ -133,7 +65,7 @@
.sources-table th, .sources-table th,
.sources-table td { .sources-table td {
padding: 0.75rem 1rem; padding: 0.75rem 1rem;
text-align: left; text-align: center;
border-bottom: 1px solid var(--border-color); border-bottom: 1px solid var(--border-color);
} }
@@ -159,6 +91,11 @@
white-space: nowrap; white-space: nowrap;
} }
/* Name column should be left-aligned */
.sources-table td:first-child {
text-align: left;
}
.url-cell { .url-cell {
font-family: monospace; font-family: monospace;
font-size: 0.9rem; font-size: 0.9rem;
@@ -166,6 +103,7 @@
overflow: hidden; overflow: hidden;
text-overflow: ellipsis; text-overflow: ellipsis;
white-space: nowrap; white-space: nowrap;
text-align: left;
} }
/* Badges */ /* Badges */
@@ -194,6 +132,12 @@
color: #c62828; color: #c62828;
} }
.coming-soon-badge {
color: #9e9e9e;
font-style: italic;
font-size: 0.85em;
}
/* Actions */ /* Actions */
.actions-cell { .actions-cell {
white-space: nowrap; white-space: nowrap;
@@ -207,35 +151,37 @@
margin-right: 0; margin-right: 0;
} }
.test-result { .test-cell {
display: inline-flex; text-align: center;
align-items: center; width: 2rem;
gap: 0.25rem;
padding: 0.2rem 0.5rem;
border-radius: 4px;
font-size: 0.75rem;
font-weight: 500;
white-space: nowrap;
} }
.test-result.success { .test-dot {
background-color: #e8f5e9; font-size: 1rem;
cursor: default;
}
.test-dot.success {
color: #2e7d32; color: #2e7d32;
} }
.test-result.failure { .test-dot.failure {
background-color: #ffebee;
color: #c62828; color: #c62828;
cursor: pointer; cursor: pointer;
} }
.test-result.failure:hover { .test-dot.failure:hover {
background-color: #ffcdd2; color: #b71c1c;
} }
.test-result.testing { .test-dot.testing {
background-color: #e3f2fd;
color: #1976d2; color: #1976d2;
animation: pulse 1s infinite;
}
@keyframes pulse {
0%, 100% { opacity: 1; }
50% { opacity: 0.4; }
} }
/* Error Modal */ /* Error Modal */
@@ -309,10 +255,22 @@
} }
.btn-sm { .btn-sm {
padding: 0.25rem 0.5rem; padding: 0.25rem 0.75rem;
font-size: 0.8rem; font-size: 0.8rem;
} }
.btn-secondary {
background-color: var(--bg-tertiary);
border-color: var(--border-color);
color: var(--text-primary);
font-weight: 500;
}
.btn-secondary:hover {
background-color: var(--bg-secondary);
border-color: var(--text-secondary);
}
.empty-message { .empty-message {
color: var(--text-secondary); color: var(--text-secondary);
font-style: italic; font-style: italic;
@@ -406,9 +364,14 @@
.form-actions { .form-actions {
display: flex; display: flex;
justify-content: flex-end; justify-content: space-between;
gap: 0.5rem; align-items: center;
margin-top: 1.5rem; margin-top: 1.5rem;
padding-top: 1rem; padding-top: 1rem;
border-top: 1px solid var(--border-color); border-top: 1px solid var(--border-color);
} }
.form-actions-right {
display: flex;
gap: 0.5rem;
}

View File

@@ -7,13 +7,12 @@ import {
updateUpstreamSource, updateUpstreamSource,
deleteUpstreamSource, deleteUpstreamSource,
testUpstreamSource, testUpstreamSource,
getCacheSettings,
updateCacheSettings,
} from '../api'; } from '../api';
import { UpstreamSource, CacheSettings, SourceType, AuthType } from '../types'; import { UpstreamSource, SourceType, AuthType } from '../types';
import './AdminCachePage.css'; import './AdminCachePage.css';
const SOURCE_TYPES: SourceType[] = ['npm', 'pypi', 'maven', 'docker', 'helm', 'nuget', 'deb', 'rpm', 'generic']; const SOURCE_TYPES: SourceType[] = ['npm', 'pypi', 'maven', 'docker', 'helm', 'nuget', 'deb', 'rpm', 'generic'];
const SUPPORTED_SOURCE_TYPES: Set<SourceType> = new Set(['pypi', 'generic']);
const AUTH_TYPES: AuthType[] = ['none', 'basic', 'bearer', 'api_key']; const AUTH_TYPES: AuthType[] = ['none', 'basic', 'bearer', 'api_key'];
function AdminCachePage() { function AdminCachePage() {
@@ -25,11 +24,6 @@ function AdminCachePage() {
const [loadingSources, setLoadingSources] = useState(true); const [loadingSources, setLoadingSources] = useState(true);
const [sourcesError, setSourcesError] = useState<string | null>(null); const [sourcesError, setSourcesError] = useState<string | null>(null);
// Cache settings state
const [settings, setSettings] = useState<CacheSettings | null>(null);
const [loadingSettings, setLoadingSettings] = useState(true);
const [settingsError, setSettingsError] = useState<string | null>(null);
// Create/Edit form state // Create/Edit form state
const [showForm, setShowForm] = useState(false); const [showForm, setShowForm] = useState(false);
const [editingSource, setEditingSource] = useState<UpstreamSource | null>(null); const [editingSource, setEditingSource] = useState<UpstreamSource | null>(null);
@@ -53,9 +47,6 @@ function AdminCachePage() {
// Delete confirmation state // Delete confirmation state
const [deletingId, setDeletingId] = useState<string | null>(null); const [deletingId, setDeletingId] = useState<string | null>(null);
// Settings update state
const [updatingSettings, setUpdatingSettings] = useState(false);
// Success message // Success message
const [successMessage, setSuccessMessage] = useState<string | null>(null); const [successMessage, setSuccessMessage] = useState<string | null>(null);
@@ -72,7 +63,6 @@ function AdminCachePage() {
useEffect(() => { useEffect(() => {
if (user && user.is_admin) { if (user && user.is_admin) {
loadSources(); loadSources();
loadSettings();
} }
}, [user]); }, [user]);
@@ -96,19 +86,6 @@ function AdminCachePage() {
} }
} }
async function loadSettings() {
setLoadingSettings(true);
setSettingsError(null);
try {
const data = await getCacheSettings();
setSettings(data);
} catch (err) {
setSettingsError(err instanceof Error ? err.message : 'Failed to load settings');
} finally {
setLoadingSettings(false);
}
}
function openCreateForm() { function openCreateForm() {
setEditingSource(null); setEditingSource(null);
setFormData({ setFormData({
@@ -255,30 +232,6 @@ function AdminCachePage() {
setShowErrorModal(true); setShowErrorModal(true);
} }
async function handleSettingsToggle(field: 'auto_create_system_projects') {
if (!settings) return;
// Check if env override is active
const isOverridden = field === 'auto_create_system_projects' && settings.auto_create_system_projects_env_override !== null;
if (isOverridden) {
alert('This setting is overridden by an environment variable and cannot be changed via UI.');
return;
}
setUpdatingSettings(true);
try {
const update = { [field]: !settings[field] };
const newSettings = await updateCacheSettings(update);
setSettings(newSettings);
setSuccessMessage(`Setting "${field}" updated`);
} catch (err) {
setSettingsError(err instanceof Error ? err.message : 'Failed to update settings');
} finally {
setUpdatingSettings(false);
}
}
if (authLoading) { if (authLoading) {
return <div className="admin-cache-page">Loading...</div>; return <div className="admin-cache-page">Loading...</div>;
} }
@@ -293,49 +246,13 @@ function AdminCachePage() {
return ( return (
<div className="admin-cache-page"> <div className="admin-cache-page">
<h1>Cache Management</h1> <h1>Upstream Sources</h1>
{successMessage && <div className="success-message">{successMessage}</div>} {successMessage && <div className="success-message">{successMessage}</div>}
{/* Cache Settings Section */}
<section className="settings-section">
<h2>Global Settings</h2>
{loadingSettings ? (
<p>Loading settings...</p>
) : settingsError ? (
<div className="error-message">{settingsError}</div>
) : settings ? (
<div className="settings-grid">
<div className="setting-item">
<label className="toggle-label">
<span className="setting-name">
Auto-create System Projects
{settings.auto_create_system_projects_env_override !== null && (
<span className="env-badge" title="Overridden by environment variable">
ENV
</span>
)}
</span>
<span className="setting-description">
Automatically create system projects (_npm, _pypi, etc.) on first cache request.
</span>
</label>
<button
className={`toggle-button ${settings.auto_create_system_projects ? 'on' : 'off'}`}
onClick={() => handleSettingsToggle('auto_create_system_projects')}
disabled={updatingSettings || settings.auto_create_system_projects_env_override !== null}
>
{settings.auto_create_system_projects ? 'Enabled' : 'Disabled'}
</button>
</div>
</div>
) : null}
</section>
{/* Upstream Sources Section */} {/* Upstream Sources Section */}
<section className="sources-section"> <section className="sources-section">
<div className="section-header"> <div className="section-header">
<h2>Upstream Sources</h2>
<button className="btn btn-primary" onClick={openCreateForm}> <button className="btn btn-primary" onClick={openCreateForm}>
Add Source Add Source
</button> </button>
@@ -356,7 +273,6 @@ function AdminCachePage() {
<th>URL</th> <th>URL</th>
<th>Priority</th> <th>Priority</th>
<th>Status</th> <th>Status</th>
<th>Source</th>
<th>Test</th> <th>Test</th>
<th>Actions</th> <th>Actions</th>
</tr> </tr>
@@ -366,66 +282,50 @@ function AdminCachePage() {
<tr key={source.id} className={source.enabled ? '' : 'disabled-row'}> <tr key={source.id} className={source.enabled ? '' : 'disabled-row'}>
<td> <td>
<span className="source-name">{source.name}</span> <span className="source-name">{source.name}</span>
{source.source === 'env' && (
<span className="env-badge" title="Defined via environment variable">ENV</span>
)}
</td> </td>
<td>{source.source_type}</td> <td>
<td className="url-cell">{source.url}</td> {source.source_type}
{!SUPPORTED_SOURCE_TYPES.has(source.source_type) && (
<span className="coming-soon-badge"> (coming soon)</span>
)}
</td>
<td className="url-cell" title={source.url}>{source.url}</td>
<td>{source.priority}</td> <td>{source.priority}</td>
<td> <td>
<span className={`status-badge ${source.enabled ? 'enabled' : 'disabled'}`}> <span className={`status-badge ${source.enabled ? 'enabled' : 'disabled'}`}>
{source.enabled ? 'Enabled' : 'Disabled'} {source.enabled ? 'Enabled' : 'Disabled'}
</span> </span>
</td> </td>
<td> <td className="test-cell">
{source.source === 'env' ? (
<span className="env-badge" title="Defined via environment variable">
ENV
</span>
) : (
'Database'
)}
</td>
<td>
{testingId === source.id ? ( {testingId === source.id ? (
<span className="test-result testing">Testing...</span> <span className="test-dot testing" title="Testing..."></span>
) : testResults[source.id] ? ( ) : testResults[source.id] ? (
testResults[source.id].success ? ( testResults[source.id].success ? (
<span className="test-result success" title={testResults[source.id].message}> <span className="test-dot success" title={testResults[source.id].message}></span>
OK
</span>
) : ( ) : (
<span <span
className="test-result failure" className="test-dot failure"
title="Click to see details" title="Click to see error"
onClick={() => showError(source.name, testResults[source.id].message)} onClick={() => showError(source.name, testResults[source.id].message)}
> ></span>
Error
</span>
) )
) : ( ) : null}
<span className="test-result" style={{ opacity: 0.5 }}></span>
)}
</td> </td>
<td className="actions-cell"> <td className="actions-cell">
<button <button
className="btn btn-sm" className="btn btn-sm btn-secondary"
onClick={() => handleTest(source)} onClick={() => handleTest(source)}
disabled={testingId === source.id} disabled={testingId === source.id}
> >
Test Test
</button> </button>
{source.source !== 'env' && ( {source.source !== 'env' && (
<> <button className="btn btn-sm btn-secondary" onClick={() => openEditForm(source)}>
<button className="btn btn-sm" onClick={() => openEditForm(source)}> Edit
Edit </button>
</button>
<button
className="btn btn-sm btn-danger"
onClick={() => handleDelete(source)}
disabled={deletingId === source.id}
>
{deletingId === source.id ? 'Deleting...' : 'Delete'}
</button>
</>
)} )}
</td> </td>
</tr> </tr>
@@ -465,7 +365,7 @@ function AdminCachePage() {
> >
{SOURCE_TYPES.map((type) => ( {SOURCE_TYPES.map((type) => (
<option key={type} value={type}> <option key={type} value={type}>
{type} {type}{!SUPPORTED_SOURCE_TYPES.has(type) ? ' (coming soon)' : ''}
</option> </option>
))} ))}
</select> </select>
@@ -561,12 +461,27 @@ function AdminCachePage() {
)} )}
<div className="form-actions"> <div className="form-actions">
<button type="button" className="btn" onClick={() => setShowForm(false)}> {editingSource && (
Cancel <button
</button> type="button"
<button type="submit" className="btn btn-primary" disabled={isSaving}> className="btn btn-danger"
{isSaving ? 'Saving...' : editingSource ? 'Update' : 'Create'} onClick={() => {
</button> handleDelete(editingSource);
setShowForm(false);
}}
disabled={deletingId === editingSource.id}
>
{deletingId === editingSource.id ? 'Deleting...' : 'Delete'}
</button>
)}
<div className="form-actions-right">
<button type="button" className="btn" onClick={() => setShowForm(false)}>
Cancel
</button>
<button type="submit" className="btn btn-primary" disabled={isSaving}>
{isSaving ? 'Saving...' : editingSource ? 'Update' : 'Create'}
</button>
</div>
</div> </div>
</form> </form>
</div> </div>

View File

@@ -249,7 +249,7 @@ function Home() {
key: 'created_by', key: 'created_by',
header: 'Owner', header: 'Owner',
className: 'cell-owner', className: 'cell-owner',
render: (project) => project.created_by, render: (project) => project.team_name || project.created_by,
}, },
...(user ...(user
? [ ? [

View File

@@ -642,6 +642,11 @@ tr:hover .copy-btn {
padding: 20px; padding: 20px;
} }
/* Ensure file modal needs higher z-index when opened from deps modal */
.modal-overlay:has(.ensure-file-modal) {
z-index: 1100;
}
.ensure-file-modal { .ensure-file-modal {
background: var(--bg-secondary); background: var(--bg-secondary);
border: 1px solid var(--border-primary); border: 1px solid var(--border-primary);
@@ -793,4 +798,194 @@ tr:hover .copy-btn {
.ensure-file-modal { .ensure-file-modal {
max-height: 90vh; max-height: 90vh;
} }
.action-menu-dropdown {
right: 0;
left: auto;
}
}
/* Header upload button */
.header-upload-btn {
margin-left: auto;
}
/* Tag/Version cell */
.tag-version-cell {
display: flex;
flex-direction: column;
gap: 4px;
}
.tag-version-cell .version-badge {
font-size: 0.75rem;
color: var(--text-muted);
}
/* Icon buttons */
.btn-icon {
display: flex;
align-items: center;
justify-content: center;
width: 32px;
height: 32px;
padding: 0;
background: transparent;
border: 1px solid transparent;
border-radius: var(--radius-sm);
color: var(--text-secondary);
cursor: pointer;
transition: all var(--transition-fast);
}
.btn-icon:hover {
background: var(--bg-hover);
color: var(--text-primary);
}
/* Action menu */
.action-buttons {
display: flex;
align-items: center;
gap: 4px;
}
.action-menu {
position: relative;
}
/* Action menu backdrop for click-outside */
.action-menu-backdrop {
position: fixed;
top: 0;
left: 0;
right: 0;
bottom: 0;
z-index: 999;
}
.action-menu-dropdown {
position: fixed;
z-index: 1000;
min-width: 180px;
padding: 4px 0;
background: var(--bg-secondary);
border: 1px solid var(--border-primary);
border-radius: var(--radius-md);
box-shadow: 0 4px 12px rgba(0, 0, 0, 0.15);
}
.action-menu-dropdown button {
display: block;
width: 100%;
padding: 8px 12px;
background: none;
border: none;
text-align: left;
font-size: 0.875rem;
color: var(--text-primary);
cursor: pointer;
transition: background var(--transition-fast);
}
.action-menu-dropdown button:hover {
background: var(--bg-hover);
}
/* Upload Modal */
.upload-modal,
.create-tag-modal {
background: var(--bg-secondary);
border-radius: var(--radius-lg);
width: 90%;
max-width: 500px;
max-height: 90vh;
overflow: hidden;
}
.modal-header {
display: flex;
align-items: center;
justify-content: space-between;
padding: 16px 20px;
border-bottom: 1px solid var(--border-primary);
}
.modal-header h3 {
margin: 0;
font-size: 1.125rem;
font-weight: 600;
}
.modal-body {
padding: 20px;
}
.modal-description {
margin-bottom: 16px;
color: var(--text-secondary);
font-size: 0.875rem;
}
.modal-actions {
display: flex;
justify-content: flex-end;
gap: 12px;
margin-top: 20px;
padding-top: 16px;
border-top: 1px solid var(--border-primary);
}
/* Dependencies Modal */
.deps-modal {
background: var(--bg-secondary);
border-radius: var(--radius-lg);
width: 90%;
max-width: 600px;
max-height: 80vh;
overflow: hidden;
display: flex;
flex-direction: column;
}
.deps-modal .modal-body {
overflow-y: auto;
flex: 1;
}
.deps-modal-controls {
display: flex;
gap: 8px;
margin-bottom: 16px;
}
/* Artifact ID Modal */
.artifact-id-modal {
background: var(--bg-secondary);
border-radius: var(--radius-lg);
width: 90%;
max-width: 500px;
}
.artifact-id-display {
display: flex;
align-items: center;
gap: 12px;
padding: 16px;
background: var(--bg-tertiary);
border-radius: var(--radius-md);
border: 1px solid var(--border-primary);
}
.artifact-id-display code {
font-family: 'JetBrains Mono', 'Fira Code', 'Consolas', monospace;
font-size: 0.8125rem;
color: var(--text-primary);
word-break: break-all;
flex: 1;
}
.artifact-id-display .copy-btn {
opacity: 1;
flex-shrink: 0;
} }

View File

@@ -63,12 +63,17 @@ function PackagePage() {
const [accessDenied, setAccessDenied] = useState(false); const [accessDenied, setAccessDenied] = useState(false);
const [uploadTag, setUploadTag] = useState(''); const [uploadTag, setUploadTag] = useState('');
const [uploadSuccess, setUploadSuccess] = useState<string | null>(null); const [uploadSuccess, setUploadSuccess] = useState<string | null>(null);
const [artifactIdInput, setArtifactIdInput] = useState('');
const [accessLevel, setAccessLevel] = useState<AccessLevel | null>(null); const [accessLevel, setAccessLevel] = useState<AccessLevel | null>(null);
const [createTagName, setCreateTagName] = useState(''); const [createTagName, setCreateTagName] = useState('');
const [createTagArtifactId, setCreateTagArtifactId] = useState(''); const [createTagArtifactId, setCreateTagArtifactId] = useState('');
const [createTagLoading, setCreateTagLoading] = useState(false); const [createTagLoading, setCreateTagLoading] = useState(false);
// UI state
const [showUploadModal, setShowUploadModal] = useState(false);
const [showCreateTagModal, setShowCreateTagModal] = useState(false);
const [openMenuId, setOpenMenuId] = useState<string | null>(null);
const [menuPosition, setMenuPosition] = useState<{ top: number; left: number } | null>(null);
// Dependencies state // Dependencies state
const [selectedTag, setSelectedTag] = useState<TagDetail | null>(null); const [selectedTag, setSelectedTag] = useState<TagDetail | null>(null);
const [dependencies, setDependencies] = useState<Dependency[]>([]); const [dependencies, setDependencies] = useState<Dependency[]>([]);
@@ -86,6 +91,13 @@ function PackagePage() {
// Dependency graph modal state // Dependency graph modal state
const [showGraph, setShowGraph] = useState(false); const [showGraph, setShowGraph] = useState(false);
// Dependencies modal state
const [showDepsModal, setShowDepsModal] = useState(false);
// Artifact ID modal state
const [showArtifactIdModal, setShowArtifactIdModal] = useState(false);
const [viewArtifactId, setViewArtifactId] = useState<string | null>(null);
// Ensure file modal state // Ensure file modal state
const [showEnsureFile, setShowEnsureFile] = useState(false); const [showEnsureFile, setShowEnsureFile] = useState(false);
const [ensureFileContent, setEnsureFileContent] = useState<string | null>(null); const [ensureFileContent, setEnsureFileContent] = useState<string | null>(null);
@@ -96,6 +108,9 @@ function PackagePage() {
// Derived permissions // Derived permissions
const canWrite = accessLevel === 'write' || accessLevel === 'admin'; const canWrite = accessLevel === 'write' || accessLevel === 'admin';
// Detect system projects (convention: name starts with "_")
const isSystemProject = projectName?.startsWith('_') ?? false;
// Get params from URL // Get params from URL
const page = parseInt(searchParams.get('page') || '1', 10); const page = parseInt(searchParams.get('page') || '1', 10);
const search = searchParams.get('search') || ''; const search = searchParams.get('search') || '';
@@ -323,92 +338,212 @@ function PackagePage() {
setSelectedTag(tag); setSelectedTag(tag);
}; };
const columns = [ const handleMenuOpen = (e: React.MouseEvent, tagId: string) => {
{ e.stopPropagation();
key: 'name', if (openMenuId === tagId) {
header: 'Tag', setOpenMenuId(null);
sortable: true, setMenuPosition(null);
render: (t: TagDetail) => ( } else {
<strong const rect = e.currentTarget.getBoundingClientRect();
className={`tag-name-link ${selectedTag?.id === t.id ? 'selected' : ''}`} setMenuPosition({ top: rect.bottom + 4, left: rect.right - 180 });
onClick={() => handleTagSelect(t)} setOpenMenuId(tagId);
style={{ cursor: 'pointer' }} }
};
// System projects show Version first, regular projects show Tag first
const columns = isSystemProject
? [
// System project columns: Version first, then Filename
{
key: 'version',
header: 'Version',
sortable: true,
render: (t: TagDetail) => (
<strong
className={`tag-name-link ${selectedTag?.id === t.id ? 'selected' : ''}`}
onClick={() => handleTagSelect(t)}
style={{ cursor: 'pointer' }}
>
<span className="version-badge">{t.version || t.name}</span>
</strong>
),
},
{
key: 'artifact_original_name',
header: 'Filename',
className: 'cell-truncate',
render: (t: TagDetail) => (
<span title={t.artifact_original_name || t.name}>{t.artifact_original_name || t.name}</span>
),
},
{
key: 'artifact_size',
header: 'Size',
render: (t: TagDetail) => <span>{formatBytes(t.artifact_size)}</span>,
},
{
key: 'created_at',
header: 'Cached',
sortable: true,
render: (t: TagDetail) => (
<span>{new Date(t.created_at).toLocaleDateString()}</span>
),
},
{
key: 'actions',
header: '',
render: (t: TagDetail) => (
<div className="action-buttons">
<a
href={getDownloadUrl(projectName!, packageName!, t.name)}
className="btn btn-icon"
download
title="Download"
>
<svg width="16" height="16" viewBox="0 0 24 24" fill="none" stroke="currentColor" strokeWidth="2">
<path d="M21 15v4a2 2 0 0 1-2 2H5a2 2 0 0 1-2-2v-4" />
<polyline points="7 10 12 15 17 10" />
<line x1="12" y1="15" x2="12" y2="3" />
</svg>
</a>
<button
className="btn btn-icon"
onClick={(e) => handleMenuOpen(e, t.id)}
title="More actions"
>
<svg width="16" height="16" viewBox="0 0 24 24" fill="none" stroke="currentColor" strokeWidth="2">
<circle cx="12" cy="12" r="1" />
<circle cx="12" cy="5" r="1" />
<circle cx="12" cy="19" r="1" />
</svg>
</button>
</div>
),
},
]
: [
// Regular project columns: Tag, Version, Filename
{
key: 'name',
header: 'Tag',
sortable: true,
render: (t: TagDetail) => (
<strong
className={`tag-name-link ${selectedTag?.id === t.id ? 'selected' : ''}`}
onClick={() => handleTagSelect(t)}
style={{ cursor: 'pointer' }}
>
{t.name}
</strong>
),
},
{
key: 'version',
header: 'Version',
render: (t: TagDetail) => (
<span className="version-badge">{t.version || '—'}</span>
),
},
{
key: 'artifact_original_name',
header: 'Filename',
className: 'cell-truncate',
render: (t: TagDetail) => (
<span title={t.artifact_original_name || undefined}>{t.artifact_original_name || '—'}</span>
),
},
{
key: 'artifact_size',
header: 'Size',
render: (t: TagDetail) => <span>{formatBytes(t.artifact_size)}</span>,
},
{
key: 'created_at',
header: 'Created',
sortable: true,
render: (t: TagDetail) => (
<span title={`by ${t.created_by}`}>{new Date(t.created_at).toLocaleDateString()}</span>
),
},
{
key: 'actions',
header: '',
render: (t: TagDetail) => (
<div className="action-buttons">
<a
href={getDownloadUrl(projectName!, packageName!, t.name)}
className="btn btn-icon"
download
title="Download"
>
<svg width="16" height="16" viewBox="0 0 24 24" fill="none" stroke="currentColor" strokeWidth="2">
<path d="M21 15v4a2 2 0 0 1-2 2H5a2 2 0 0 1-2-2v-4" />
<polyline points="7 10 12 15 17 10" />
<line x1="12" y1="15" x2="12" y2="3" />
</svg>
</a>
<button
className="btn btn-icon"
onClick={(e) => handleMenuOpen(e, t.id)}
title="More actions"
>
<svg width="16" height="16" viewBox="0 0 24 24" fill="none" stroke="currentColor" strokeWidth="2">
<circle cx="12" cy="12" r="1" />
<circle cx="12" cy="5" r="1" />
<circle cx="12" cy="19" r="1" />
</svg>
</button>
</div>
),
},
];
// Find the tag for the open menu
const openMenuTag = tags.find(t => t.id === openMenuId);
// Close menu when clicking outside
const handleClickOutside = () => {
if (openMenuId) {
setOpenMenuId(null);
setMenuPosition(null);
}
};
// Render dropdown menu as a portal-like element
const renderActionMenu = () => {
if (!openMenuId || !menuPosition || !openMenuTag) return null;
const t = openMenuTag;
return (
<div
className="action-menu-backdrop"
onClick={handleClickOutside}
>
<div
className="action-menu-dropdown"
style={{ top: menuPosition.top, left: menuPosition.left }}
onClick={(e) => e.stopPropagation()}
> >
{t.name} <button onClick={() => { setViewArtifactId(t.artifact_id); setShowArtifactIdModal(true); setOpenMenuId(null); setMenuPosition(null); }}>
</strong> View Artifact ID
), </button>
}, <button onClick={() => { navigator.clipboard.writeText(t.artifact_id); setOpenMenuId(null); setMenuPosition(null); }}>
{ Copy Artifact ID
key: 'version', </button>
header: 'Version', <button onClick={() => { fetchEnsureFileForTag(t.name); setOpenMenuId(null); setMenuPosition(null); }}>
render: (t: TagDetail) => ( View Ensure File
<span className="version-badge">{t.version || '-'}</span> </button>
), {canWrite && !isSystemProject && (
}, <button onClick={() => { setCreateTagArtifactId(t.artifact_id); setShowCreateTagModal(true); setOpenMenuId(null); setMenuPosition(null); }}>
{ Create/Update Tag
key: 'artifact_id', </button>
header: 'Artifact ID', )}
render: (t: TagDetail) => ( <button onClick={() => { handleTagSelect(t); setShowDepsModal(true); setOpenMenuId(null); setMenuPosition(null); }}>
<div className="artifact-id-cell"> View Dependencies
<code className="artifact-id">{t.artifact_id.substring(0, 12)}...</code>
<CopyButton text={t.artifact_id} />
</div>
),
},
{
key: 'artifact_size',
header: 'Size',
render: (t: TagDetail) => <span>{formatBytes(t.artifact_size)}</span>,
},
{
key: 'artifact_content_type',
header: 'Type',
render: (t: TagDetail) => (
<span className="content-type">{t.artifact_content_type || '-'}</span>
),
},
{
key: 'artifact_original_name',
header: 'Filename',
className: 'cell-truncate',
render: (t: TagDetail) => (
<span title={t.artifact_original_name || undefined}>{t.artifact_original_name || '-'}</span>
),
},
{
key: 'created_at',
header: 'Created',
sortable: true,
render: (t: TagDetail) => (
<div className="created-cell">
<span>{new Date(t.created_at).toLocaleString()}</span>
<span className="created-by">by {t.created_by}</span>
</div>
),
},
{
key: 'actions',
header: 'Actions',
render: (t: TagDetail) => (
<div className="action-buttons">
<button
className="btn btn-secondary btn-small"
onClick={() => fetchEnsureFileForTag(t.name)}
title="View orchard.ensure file"
>
Ensure
</button> </button>
<a
href={getDownloadUrl(projectName!, packageName!, t.name)}
className="btn btn-secondary btn-small"
download
>
Download
</a>
</div> </div>
), </div>
}, );
]; };
if (loading && !tagsData) { if (loading && !tagsData) {
return <div className="loading">Loading...</div>; return <div className="loading">Loading...</div>;
@@ -451,6 +586,19 @@ function PackagePage() {
<div className="page-header__title-row"> <div className="page-header__title-row">
<h1>{packageName}</h1> <h1>{packageName}</h1>
{pkg && <Badge variant="default">{pkg.format}</Badge>} {pkg && <Badge variant="default">{pkg.format}</Badge>}
{user && canWrite && !isSystemProject && (
<button
className="btn btn-primary btn-small header-upload-btn"
onClick={() => setShowUploadModal(true)}
>
<svg width="14" height="14" viewBox="0 0 24 24" fill="none" stroke="currentColor" strokeWidth="2" style={{ marginRight: '6px' }}>
<path d="M21 15v4a2 2 0 0 1-2 2H5a2 2 0 0 1-2-2v-4" />
<polyline points="17 8 12 3 7 8" />
<line x1="12" y1="3" x2="12" y2="15" />
</svg>
Upload
</button>
)}
</div> </div>
{pkg?.description && <p className="description">{pkg.description}</p>} {pkg?.description && <p className="description">{pkg.description}</p>}
<div className="page-header__meta"> <div className="page-header__meta">
@@ -468,14 +616,14 @@ function PackagePage() {
</div> </div>
{pkg && (pkg.tag_count !== undefined || pkg.artifact_count !== undefined) && ( {pkg && (pkg.tag_count !== undefined || pkg.artifact_count !== undefined) && (
<div className="package-header-stats"> <div className="package-header-stats">
{pkg.tag_count !== undefined && ( {!isSystemProject && pkg.tag_count !== undefined && (
<span className="stat-item"> <span className="stat-item">
<strong>{pkg.tag_count}</strong> tags <strong>{pkg.tag_count}</strong> tags
</span> </span>
)} )}
{pkg.artifact_count !== undefined && ( {pkg.artifact_count !== undefined && (
<span className="stat-item"> <span className="stat-item">
<strong>{pkg.artifact_count}</strong> artifacts <strong>{pkg.artifact_count}</strong> {isSystemProject ? 'versions' : 'artifacts'}
</span> </span>
)} )}
{pkg.total_size !== undefined && pkg.total_size > 0 && ( {pkg.total_size !== undefined && pkg.total_size > 0 && (
@@ -483,7 +631,7 @@ function PackagePage() {
<strong>{formatBytes(pkg.total_size)}</strong> total <strong>{formatBytes(pkg.total_size)}</strong> total
</span> </span>
)} )}
{pkg.latest_tag && ( {!isSystemProject && pkg.latest_tag && (
<span className="stat-item"> <span className="stat-item">
Latest: <strong className="accent">{pkg.latest_tag}</strong> Latest: <strong className="accent">{pkg.latest_tag}</strong>
</span> </span>
@@ -496,44 +644,9 @@ function PackagePage() {
{error && <div className="error-message">{error}</div>} {error && <div className="error-message">{error}</div>}
{uploadSuccess && <div className="success-message">{uploadSuccess}</div>} {uploadSuccess && <div className="success-message">{uploadSuccess}</div>}
{user && (
<div className="upload-section card">
<h3>Upload Artifact</h3>
{canWrite ? (
<div className="upload-form">
<div className="form-group">
<label htmlFor="upload-tag">Tag (optional)</label>
<input
id="upload-tag"
type="text"
value={uploadTag}
onChange={(e) => setUploadTag(e.target.value)}
placeholder="v1.0.0, latest, stable..."
/>
</div>
<DragDropUpload
projectName={projectName!}
packageName={packageName!}
tag={uploadTag || undefined}
onUploadComplete={handleUploadComplete}
onUploadError={handleUploadError}
/>
</div>
) : (
<DragDropUpload
projectName={projectName!}
packageName={packageName!}
disabled={true}
disabledReason="You have read-only access to this project and cannot upload artifacts."
onUploadComplete={handleUploadComplete}
onUploadError={handleUploadError}
/>
)}
</div>
)}
<div className="section-header"> <div className="section-header">
<h2>Tags / Versions</h2> <h2>{isSystemProject ? 'Versions' : 'Tags / Versions'}</h2>
</div> </div>
<div className="list-controls"> <div className="list-controls">
@@ -577,110 +690,6 @@ function PackagePage() {
/> />
)} )}
{/* Dependencies Section */}
{tags.length > 0 && (
<div className="dependencies-section card">
<div className="dependencies-header">
<h3>Dependencies</h3>
<div className="dependencies-controls">
{selectedTag && (
<>
<button
className="btn btn-secondary btn-small"
onClick={fetchEnsureFile}
disabled={ensureFileLoading}
title="View orchard.ensure file"
>
<svg width="14" height="14" viewBox="0 0 24 24" fill="none" stroke="currentColor" strokeWidth="2" style={{ marginRight: '6px' }}>
<path d="M14 2H6a2 2 0 0 0-2 2v16a2 2 0 0 0 2 2h12a2 2 0 0 0 2-2V8z"></path>
<polyline points="14 2 14 8 20 8"></polyline>
<line x1="16" y1="13" x2="8" y2="13"></line>
<line x1="16" y1="17" x2="8" y2="17"></line>
<polyline points="10 9 9 9 8 9"></polyline>
</svg>
{ensureFileLoading ? 'Loading...' : 'View Ensure File'}
</button>
<button
className="btn btn-secondary btn-small"
onClick={() => setShowGraph(true)}
title="View full dependency tree"
>
<svg width="14" height="14" viewBox="0 0 24 24" fill="none" stroke="currentColor" strokeWidth="2" style={{ marginRight: '6px' }}>
<circle cx="12" cy="12" r="3"></circle>
<circle cx="4" cy="4" r="2"></circle>
<circle cx="20" cy="4" r="2"></circle>
<circle cx="4" cy="20" r="2"></circle>
<circle cx="20" cy="20" r="2"></circle>
<line x1="9.5" y1="9.5" x2="5.5" y2="5.5"></line>
<line x1="14.5" y1="9.5" x2="18.5" y2="5.5"></line>
<line x1="9.5" y1="14.5" x2="5.5" y2="18.5"></line>
<line x1="14.5" y1="14.5" x2="18.5" y2="18.5"></line>
</svg>
View Graph
</button>
</>
)}
</div>
</div>
<div className="dependencies-tag-select">
{selectedTag && (
<select
className="tag-selector"
value={selectedTag.id}
onChange={(e) => {
const tag = tags.find(t => t.id === e.target.value);
if (tag) setSelectedTag(tag);
}}
>
{tags.map(t => (
<option key={t.id} value={t.id}>
{t.name}{t.version ? ` (${t.version})` : ''}
</option>
))}
</select>
)}
</div>
{depsLoading ? (
<div className="deps-loading">Loading dependencies...</div>
) : depsError ? (
<div className="deps-error">{depsError}</div>
) : dependencies.length === 0 ? (
<div className="deps-empty">
{selectedTag ? (
<span><strong>{selectedTag.name}</strong> has no dependencies</span>
) : (
<span>No dependencies</span>
)}
</div>
) : (
<div className="deps-list">
<div className="deps-summary">
<strong>{selectedTag?.name}</strong> has {dependencies.length} {dependencies.length === 1 ? 'dependency' : 'dependencies'}:
</div>
<ul className="deps-items">
{dependencies.map((dep) => (
<li key={dep.id} className="dep-item">
<Link
to={`/project/${dep.project}/${dep.package}`}
className="dep-link"
>
{dep.project}/{dep.package}
</Link>
<span className="dep-constraint">
@ {dep.version || dep.tag}
</span>
<span className="dep-status dep-status--ok" title="Package exists">
&#10003;
</span>
</li>
))}
</ul>
</div>
)}
</div>
)}
{/* Used By (Reverse Dependencies) Section */} {/* Used By (Reverse Dependencies) Section */}
<div className="used-by-section card"> <div className="used-by-section card">
<h3>Used By</h3> <h3>Used By</h3>
@@ -737,78 +746,6 @@ function PackagePage() {
)} )}
</div> </div>
<div className="download-by-id-section card">
<h3>Download by Artifact ID</h3>
<div className="download-by-id-form">
<input
type="text"
value={artifactIdInput}
onChange={(e) => setArtifactIdInput(e.target.value.toLowerCase().replace(/[^a-f0-9]/g, '').slice(0, 64))}
placeholder="Enter SHA256 artifact ID (64 hex characters)"
className="artifact-id-input"
/>
<a
href={artifactIdInput.length === 64 ? getDownloadUrl(projectName!, packageName!, `artifact:${artifactIdInput}`) : '#'}
className={`btn btn-primary ${artifactIdInput.length !== 64 ? 'btn-disabled' : ''}`}
download
onClick={(e) => {
if (artifactIdInput.length !== 64) {
e.preventDefault();
}
}}
>
Download
</a>
</div>
{artifactIdInput.length > 0 && artifactIdInput.length !== 64 && (
<p className="validation-hint">Artifact ID must be exactly 64 hex characters ({artifactIdInput.length}/64)</p>
)}
</div>
{user && canWrite && (
<div className="create-tag-section card">
<h3>Create / Update Tag</h3>
<p className="section-description">Point a tag at any existing artifact by its ID</p>
<form onSubmit={handleCreateTag} className="create-tag-form">
<div className="form-row">
<div className="form-group">
<label htmlFor="create-tag-name">Tag Name</label>
<input
id="create-tag-name"
type="text"
value={createTagName}
onChange={(e) => setCreateTagName(e.target.value)}
placeholder="latest, stable, v1.0.0..."
disabled={createTagLoading}
/>
</div>
<div className="form-group form-group--wide">
<label htmlFor="create-tag-artifact">Artifact ID</label>
<input
id="create-tag-artifact"
type="text"
value={createTagArtifactId}
onChange={(e) => setCreateTagArtifactId(e.target.value.toLowerCase().replace(/[^a-f0-9]/g, '').slice(0, 64))}
placeholder="SHA256 hash (64 hex characters)"
className="artifact-id-input"
disabled={createTagLoading}
/>
</div>
<button
type="submit"
className="btn btn-primary"
disabled={createTagLoading || !createTagName.trim() || createTagArtifactId.length !== 64}
>
{createTagLoading ? 'Creating...' : 'Create Tag'}
</button>
</div>
{createTagArtifactId.length > 0 && createTagArtifactId.length !== 64 && (
<p className="validation-hint">Artifact ID must be exactly 64 hex characters ({createTagArtifactId.length}/64)</p>
)}
</form>
</div>
)}
<div className="usage-section card"> <div className="usage-section card">
<h3>Usage</h3> <h3>Usage</h3>
<p>Download artifacts using:</p> <p>Download artifacts using:</p>
@@ -831,6 +768,118 @@ function PackagePage() {
/> />
)} )}
{/* Upload Modal */}
{showUploadModal && (
<div className="modal-overlay" onClick={() => setShowUploadModal(false)}>
<div className="upload-modal" onClick={(e) => e.stopPropagation()}>
<div className="modal-header">
<h3>Upload Artifact</h3>
<button
className="modal-close"
onClick={() => setShowUploadModal(false)}
title="Close"
>
<svg width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" strokeWidth="2">
<line x1="18" y1="6" x2="6" y2="18"></line>
<line x1="6" y1="6" x2="18" y2="18"></line>
</svg>
</button>
</div>
<div className="modal-body">
<div className="form-group">
<label htmlFor="upload-tag">Tag (optional)</label>
<input
id="upload-tag"
type="text"
value={uploadTag}
onChange={(e) => setUploadTag(e.target.value)}
placeholder="v1.0.0, latest, stable..."
/>
</div>
<DragDropUpload
projectName={projectName!}
packageName={packageName!}
tag={uploadTag || undefined}
onUploadComplete={(result) => {
handleUploadComplete(result);
setShowUploadModal(false);
setUploadTag('');
}}
onUploadError={handleUploadError}
/>
</div>
</div>
</div>
)}
{/* Create/Update Tag Modal */}
{showCreateTagModal && (
<div className="modal-overlay" onClick={() => setShowCreateTagModal(false)}>
<div className="create-tag-modal" onClick={(e) => e.stopPropagation()}>
<div className="modal-header">
<h3>Create / Update Tag</h3>
<button
className="modal-close"
onClick={() => { setShowCreateTagModal(false); setCreateTagName(''); setCreateTagArtifactId(''); }}
title="Close"
>
<svg width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" strokeWidth="2">
<line x1="18" y1="6" x2="6" y2="18"></line>
<line x1="6" y1="6" x2="18" y2="18"></line>
</svg>
</button>
</div>
<div className="modal-body">
<p className="modal-description">Point a tag at an artifact by its ID</p>
<form onSubmit={(e) => { handleCreateTag(e); setShowCreateTagModal(false); }}>
<div className="form-group">
<label htmlFor="modal-tag-name">Tag Name</label>
<input
id="modal-tag-name"
type="text"
value={createTagName}
onChange={(e) => setCreateTagName(e.target.value)}
placeholder="latest, stable, v1.0.0..."
disabled={createTagLoading}
/>
</div>
<div className="form-group">
<label htmlFor="modal-artifact-id">Artifact ID</label>
<input
id="modal-artifact-id"
type="text"
value={createTagArtifactId}
onChange={(e) => setCreateTagArtifactId(e.target.value.toLowerCase().replace(/[^a-f0-9]/g, '').slice(0, 64))}
placeholder="SHA256 hash (64 hex characters)"
className="artifact-id-input"
disabled={createTagLoading}
/>
{createTagArtifactId.length > 0 && createTagArtifactId.length !== 64 && (
<p className="validation-hint">{createTagArtifactId.length}/64 characters</p>
)}
</div>
<div className="modal-actions">
<button
type="button"
className="btn btn-secondary"
onClick={() => { setShowCreateTagModal(false); setCreateTagName(''); setCreateTagArtifactId(''); }}
>
Cancel
</button>
<button
type="submit"
className="btn btn-primary"
disabled={createTagLoading || !createTagName.trim() || createTagArtifactId.length !== 64}
>
{createTagLoading ? 'Creating...' : 'Create Tag'}
</button>
</div>
</form>
</div>
</div>
</div>
)}
{/* Ensure File Modal */} {/* Ensure File Modal */}
{showEnsureFile && ( {showEnsureFile && (
<div className="modal-overlay" onClick={() => setShowEnsureFile(false)}> <div className="modal-overlay" onClick={() => setShowEnsureFile(false)}>
@@ -872,6 +921,107 @@ function PackagePage() {
</div> </div>
</div> </div>
)} )}
{/* Dependencies Modal */}
{showDepsModal && selectedTag && (
<div className="modal-overlay" onClick={() => setShowDepsModal(false)}>
<div className="deps-modal" onClick={(e) => e.stopPropagation()}>
<div className="modal-header">
<h3>Dependencies for {selectedTag.version || selectedTag.name}</h3>
<button
className="modal-close"
onClick={() => setShowDepsModal(false)}
title="Close"
>
<svg width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" strokeWidth="2">
<line x1="18" y1="6" x2="6" y2="18"></line>
<line x1="6" y1="6" x2="18" y2="18"></line>
</svg>
</button>
</div>
<div className="modal-body">
<div className="deps-modal-controls">
<button
className="btn btn-secondary btn-small"
onClick={fetchEnsureFile}
disabled={ensureFileLoading}
>
View Ensure File
</button>
<button
className="btn btn-secondary btn-small"
onClick={() => { setShowDepsModal(false); setShowGraph(true); }}
>
View Graph
</button>
</div>
{depsLoading ? (
<div className="deps-loading">Loading dependencies...</div>
) : depsError ? (
<div className="deps-error">{depsError}</div>
) : dependencies.length === 0 ? (
<div className="deps-empty">No dependencies</div>
) : (
<div className="deps-list">
<div className="deps-summary">
{dependencies.length} {dependencies.length === 1 ? 'dependency' : 'dependencies'}:
</div>
<ul className="deps-items">
{dependencies.map((dep) => (
<li key={dep.id} className="dep-item">
<Link
to={`/project/${dep.project}/${dep.package}`}
className="dep-link"
onClick={() => setShowDepsModal(false)}
>
{dep.project}/{dep.package}
</Link>
<span className="dep-constraint">
@ {dep.version || dep.tag}
</span>
<span className="dep-status dep-status--ok" title="Package exists">
&#10003;
</span>
</li>
))}
</ul>
</div>
)}
</div>
</div>
</div>
)}
{/* Artifact ID Modal */}
{showArtifactIdModal && viewArtifactId && (
<div className="modal-overlay" onClick={() => setShowArtifactIdModal(false)}>
<div className="artifact-id-modal" onClick={(e) => e.stopPropagation()}>
<div className="modal-header">
<h3>Artifact ID</h3>
<button
className="modal-close"
onClick={() => setShowArtifactIdModal(false)}
title="Close"
>
<svg width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" strokeWidth="2">
<line x1="18" y1="6" x2="6" y2="18"></line>
<line x1="6" y1="6" x2="18" y2="18"></line>
</svg>
</button>
</div>
<div className="modal-body">
<p className="modal-description">SHA256 hash identifying this artifact:</p>
<div className="artifact-id-display">
<code>{viewArtifactId}</code>
<CopyButton text={viewArtifactId} />
</div>
</div>
</div>
</div>
)}
{/* Action Menu Dropdown */}
{renderActionMenu()}
</div> </div>
); );
} }

View File

@@ -214,7 +214,7 @@ function ProjectPage() {
</div> </div>
</div> </div>
<div className="page-header__actions"> <div className="page-header__actions">
{canAdmin && !project.team_id && ( {canAdmin && !project.team_id && !project.is_system && (
<button <button
className="btn btn-secondary" className="btn btn-secondary"
onClick={() => navigate(`/project/${projectName}/settings`)} onClick={() => navigate(`/project/${projectName}/settings`)}
@@ -227,11 +227,11 @@ function ProjectPage() {
Settings Settings
</button> </button>
)} )}
{canWrite ? ( {canWrite && !project.is_system ? (
<button className="btn btn-primary" onClick={() => setShowForm(!showForm)}> <button className="btn btn-primary" onClick={() => setShowForm(!showForm)}>
{showForm ? 'Cancel' : '+ New Package'} {showForm ? 'Cancel' : '+ New Package'}
</button> </button>
) : user ? ( ) : user && !project.is_system ? (
<span className="text-muted" title="You have read-only access to this project"> <span className="text-muted" title="You have read-only access to this project">
Read-only access Read-only access
</span> </span>
@@ -294,18 +294,20 @@ function ProjectPage() {
placeholder="Filter packages..." placeholder="Filter packages..."
className="list-controls__search" className="list-controls__search"
/> />
<select {!project?.is_system && (
className="list-controls__select" <select
value={format} className="list-controls__select"
onChange={(e) => handleFormatChange(e.target.value)} value={format}
> onChange={(e) => handleFormatChange(e.target.value)}
<option value="">All formats</option> >
{FORMAT_OPTIONS.map((f) => ( <option value="">All formats</option>
<option key={f} value={f}> {FORMAT_OPTIONS.map((f) => (
{f} <option key={f} value={f}>
</option> {f}
))} </option>
</select> ))}
</select>
)}
</div> </div>
{hasActiveFilters && ( {hasActiveFilters && (
@@ -341,19 +343,19 @@ function ProjectPage() {
className: 'cell-description', className: 'cell-description',
render: (pkg) => pkg.description || '—', render: (pkg) => pkg.description || '—',
}, },
{ ...(!project?.is_system ? [{
key: 'format', key: 'format',
header: 'Format', header: 'Format',
render: (pkg) => <Badge variant="default">{pkg.format}</Badge>, render: (pkg: Package) => <Badge variant="default">{pkg.format}</Badge>,
}, }] : []),
{ ...(!project?.is_system ? [{
key: 'tag_count', key: 'tag_count',
header: 'Tags', header: 'Tags',
render: (pkg) => pkg.tag_count ?? '—', render: (pkg: Package) => pkg.tag_count ?? '—',
}, }] : []),
{ {
key: 'artifact_count', key: 'artifact_count',
header: 'Artifacts', header: project?.is_system ? 'Versions' : 'Artifacts',
render: (pkg) => pkg.artifact_count ?? '—', render: (pkg) => pkg.artifact_count ?? '—',
}, },
{ {
@@ -362,12 +364,12 @@ function ProjectPage() {
render: (pkg) => render: (pkg) =>
pkg.total_size !== undefined && pkg.total_size > 0 ? formatBytes(pkg.total_size) : '—', pkg.total_size !== undefined && pkg.total_size > 0 ? formatBytes(pkg.total_size) : '—',
}, },
{ ...(!project?.is_system ? [{
key: 'latest_tag', key: 'latest_tag',
header: 'Latest', header: 'Latest',
render: (pkg) => render: (pkg: Package) =>
pkg.latest_tag ? <strong style={{ color: 'var(--accent-primary)' }}>{pkg.latest_tag}</strong> : '—', pkg.latest_tag ? <strong style={{ color: 'var(--accent-primary)' }}>{pkg.latest_tag}</strong> : '—',
}, }] : []),
{ {
key: 'created_at', key: 'created_at',
header: 'Created', header: 'Created',

View File

@@ -557,15 +557,3 @@ export interface UpstreamSourceTestResult {
source_id: string; source_id: string;
source_name: string; source_name: string;
} }
// Cache Settings types
export interface CacheSettings {
auto_create_system_projects: boolean;
auto_create_system_projects_env_override: boolean | null;
created_at: string | null;
updated_at: string | null;
}
export interface CacheSettingsUpdate {
auto_create_system_projects?: boolean;
}