13 Commits

Author SHA1 Message Date
Mondo Diaz
c31a147e1f fix: treat bare version constraints as exact match
When resolving dependencies like certifi@2025.10.5, the bare version
string "2025.10.5" was being rejected as an invalid SpecifierSet and
falling back to wildcard, which fetched the latest version instead.

Now bare versions starting with a digit are automatically prefixed
with "==" to create an exact match constraint.
2026-02-04 17:02:02 -06:00
Mondo Diaz
6cf487b224 fix: add security checks and tests for code review
Security:
- Add authorization checks to list_packages, update_package, delete_package endpoints
- Add MAX_TOTAL_ARTIFACTS limit (1000) to prevent memory exhaustion during dependency resolution
- Add TooManyArtifactsError exception for proper error handling

UI:
- Display reverse dependency errors in PackagePage
- Add warning display for failed dependency fetches in DependencyGraph

Tests:
- Add unit tests for metadata extraction (deb, wheel, tarball, jar)
- Add unit tests for rate limit configuration
- Add unit tests for PyPI registry client
2026-02-04 16:19:16 -06:00
Mondo Diaz
7140c9f4f2 fix: filter platform-specific and extra dependencies in PyPI proxy
The dependency parser was stripping environment markers but not checking
if they indicated optional or platform-specific packages. This caused
packages like jaraco.path to pull in pyobjc (324 sub-packages) even on
non-macOS systems.

Changes:
- Filter dependencies with 'extra ==' markers (optional extras)
- Filter dependencies with 'sys_platform' or 'platform_system' markers
- Add diagnostic logging for depth exceeded errors
- Add unit tests for dependency filtering

Fixes tensorflow dependency resolution exceeding max depth.
2026-02-04 14:56:59 -06:00
Mondo Diaz
6957bb26e9 chore: increase MAX_DEPENDENCY_DEPTH from 50 to 100 2026-02-04 14:06:06 -06:00
Mondo Diaz
561b82da92 chore: remove unused auto_fetch_max_depth config setting 2026-02-04 14:04:35 -06:00
Mondo Diaz
0fb69a6aaa feat: remove fetch depth limit for dependency resolution
Real package managers (pip, npm, Maven) don't have depth limits - they
resolve the full dependency tree. We have other safeguards:
- Loop prevention via fetch_attempted set
- Timeout via auto_fetch_timeout setting
- Dependency trees are finite
2026-02-04 13:55:53 -06:00
Mondo Diaz
f1ac43c1cb fix: use lenient conflict handling for dependency resolution
Instead of failing with 409 on version conflicts, use "first version wins"
strategy. This allows resolution to succeed for complex dependency trees
like tensorflow where transitive dependencies may have overlapping but
not identical version requirements.

The resolver now:
- Checks if an already-resolved version satisfies a new constraint
- If yes, reuses the existing version
- If no, logs the mismatch and uses the first-encountered version

This matches pip's behavior of picking a working version rather than
failing on theoretical conflicts.
2026-02-04 13:45:15 -06:00
Mondo Diaz
23ffbada00 feat: increase auto_fetch_max_depth from 3 to 10 2026-02-04 13:38:57 -06:00
Mondo Diaz
2ea3a39416 fix: prevent false circular dependency detection on self-dependencies
When packages like pytest have extras (e.g., pytest[testing]) that depend
on the base package, the resolution was incorrectly detecting this as a
circular dependency.

Added additional check to skip dependencies that resolve to an artifact
already in the visiting set, preventing the false cycle detection while
still catching real circular dependencies.
2026-02-04 13:19:03 -06:00
Mondo Diaz
7bfec020c8 feat: change auto_fetch default to true
Auto-fetching missing dependencies from upstream is the more useful default
behavior. Users who need fast, network-free resolution can explicitly set
auto_fetch=false.

Artifacts are content-addressed by SHA256, so reproducibility concerns don't
apply - the same version always produces the same artifact.
2026-02-04 12:23:01 -06:00
Mondo Diaz
6b9863f9c3 fix: fetch root artifact from upstream when missing in auto_fetch mode
When auto_fetch=true and the root artifact doesn't exist locally in a
system project (_pypi), now attempts to fetch it from upstream before
starting dependency resolution. Also fixed a bug where fetched_artifacts
was being redeclared, which would lose the root artifact from the list.
2026-02-04 12:18:44 -06:00
Mondo Diaz
5cff4092e3 feat: add auto-fetch for missing dependencies from upstream registries
Add auto_fetch parameter to dependency resolution endpoint that fetches
missing dependencies from upstream registries (PyPI) when resolving.

- Add RegistryClient abstraction with PyPIRegistryClient implementation
- Extract fetch_and_cache_pypi_package() for reuse
- Add resolve_dependencies_with_fetch() async function
- Extend MissingDependency schema with fetch_attempted/fetch_error
- Add fetched list to DependencyResolutionResponse
- Add auto_fetch_max_depth config setting (default: 3)
- Remove Usage section from Package page UI
- Add 6 integration tests for auto-fetch functionality
2026-02-04 12:01:49 -06:00
Mondo Diaz
b82bd1c85a fix: remove dead code and security issue from code review
- Remove unused _get_pypi_upstream_sources_cached function (never called)
- Remove unused CacheService import and get_cache helper
- Remove unused cache parameter from pypi_download_file
- Fix asyncio.get_event_loop() deprecation - use get_running_loop()
- Note: The caching implementation was incomplete but the other
  performance improvements (connection pooling, batch DB ops) remain
2026-02-04 10:57:32 -06:00
17 changed files with 2293 additions and 179 deletions

View File

@@ -7,6 +7,17 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
## [Unreleased]
### Added
- Added auto-fetch capability to dependency resolution endpoint
- `GET /api/v1/project/{project}/{package}/+/{ref}/resolve?auto_fetch=true` fetches missing dependencies from upstream registries
- PyPI registry client queries PyPI JSON API to resolve version constraints
- Fetched artifacts are cached and included in response `fetched` field
- Missing dependencies show `fetch_attempted` and `fetch_error` status
- Configurable max fetch depth via `ORCHARD_AUTO_FETCH_MAX_DEPTH` (default: 3)
- Added `backend/app/registry_client.py` with extensible registry client abstraction
- `RegistryClient` ABC for implementing upstream registry clients
- `PyPIRegistryClient` implementation using PyPI JSON API
- `get_registry_client()` factory function for future npm/maven support
- Added `fetch_and_cache_pypi_package()` reusable function for PyPI package fetching
- Added HTTP connection pooling infrastructure for improved PyPI proxy performance
- `HttpClientManager` with configurable pool size, timeouts, and thread pool executor
- Eliminates per-request connection overhead (~100-500ms → ~5ms)
@@ -33,6 +44,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- Added `POST /api/v1/cache/resolve` endpoint to cache packages by coordinates instead of URL (#108)
### Changed
- Removed Usage section from Package page (curl command examples)
- PyPI proxy now uses shared HTTP connection pool instead of per-request clients
- PyPI proxy now caches upstream source configuration in Redis
- Dependency storage now uses batch INSERT instead of individual queries

View File

@@ -89,6 +89,10 @@ class Settings(BaseSettings):
pypi_cache_max_depth: int = 10 # Maximum recursion depth for dependency caching
pypi_cache_max_attempts: int = 3 # Maximum retry attempts for failed cache tasks
# Auto-fetch configuration for dependency resolution
auto_fetch_dependencies: bool = False # Server default for auto_fetch parameter
auto_fetch_timeout: int = 300 # Total timeout for auto-fetch resolution in seconds
# JWT Authentication settings (optional, for external identity providers)
jwt_enabled: bool = False # Enable JWT token validation
jwt_secret: str = "" # Secret key for HS256, or leave empty for RS256 with JWKS

View File

@@ -11,11 +11,18 @@ Handles:
"""
import re
import logging
import yaml
from typing import List, Dict, Any, Optional, Set, Tuple
from typing import List, Dict, Any, Optional, Set, Tuple, TYPE_CHECKING
from sqlalchemy.orm import Session
from sqlalchemy import and_
if TYPE_CHECKING:
from .storage import S3Storage
from .registry_client import RegistryClient
logger = logging.getLogger(__name__)
# Import packaging for PEP 440 version matching
try:
from packaging.specifiers import SpecifierSet, InvalidSpecifier
@@ -102,9 +109,17 @@ class DependencyDepthExceededError(DependencyError):
super().__init__(f"Dependency resolution exceeded maximum depth of {max_depth}")
class TooManyArtifactsError(DependencyError):
"""Raised when dependency resolution resolves too many artifacts."""
def __init__(self, max_artifacts: int):
self.max_artifacts = max_artifacts
super().__init__(f"Dependency resolution exceeded maximum of {max_artifacts} artifacts")
# Safety limits to prevent DoS attacks
MAX_DEPENDENCY_DEPTH = 50 # Maximum levels of nested dependencies
MAX_DEPENDENCY_DEPTH = 100 # Maximum levels of nested dependencies
MAX_DEPENDENCIES_PER_ARTIFACT = 200 # Maximum direct dependencies per artifact
MAX_TOTAL_ARTIFACTS = 1000 # Maximum total artifacts in resolution to prevent memory issues
def parse_ensure_file(content: bytes) -> EnsureFileContent:
@@ -325,6 +340,33 @@ def _is_version_constraint(version_str: str) -> bool:
return any(op in version_str for op in ['>=', '<=', '!=', '~=', '>', '<', '==', '*'])
def _version_satisfies_constraint(version: str, constraint: str) -> bool:
"""
Check if a version satisfies a constraint.
Args:
version: A version string (e.g., '1.26.0')
constraint: A version constraint (e.g., '>=1.20', '>=1.20,<2.0', '*')
Returns:
True if the version satisfies the constraint, False otherwise
"""
if not HAS_PACKAGING:
return False
# Wildcard matches everything
if constraint == '*' or not constraint:
return True
try:
spec = SpecifierSet(constraint)
v = Version(version)
return v in spec
except (InvalidSpecifier, InvalidVersion):
# If we can't parse, assume it doesn't match
return False
def _resolve_version_constraint(
db: Session,
package: Package,
@@ -696,6 +738,8 @@ def resolve_dependencies(
current_path: Dict[str, str] = {}
# Resolution order (topological)
resolution_order: List[str] = []
# Track resolution path for debugging
resolution_path_sync: List[str] = []
def _resolve_recursive(
artifact_id: str,
@@ -707,12 +751,16 @@ def resolve_dependencies(
depth: int = 0,
):
"""Recursively resolve dependencies with cycle/conflict detection."""
pkg_key = f"{proj_name}/{pkg_name}"
# Safety limit: prevent DoS through deeply nested dependencies
if depth > MAX_DEPENDENCY_DEPTH:
logger.error(
f"Dependency depth exceeded at {pkg_key} (depth={depth}). "
f"Resolution path: {' -> '.join(resolution_path_sync[-20:])}"
)
raise DependencyDepthExceededError(MAX_DEPENDENCY_DEPTH)
pkg_key = f"{proj_name}/{pkg_name}"
# Cycle detection (at artifact level)
if artifact_id in visiting:
# Build cycle path from current_path
@@ -720,34 +768,25 @@ def resolve_dependencies(
cycle = [cycle_start, pkg_key]
raise CircularDependencyError(cycle)
# Conflict detection - check if we've seen this package before with a different version
# Version conflict handling - use first resolved version (lenient mode)
if pkg_key in version_requirements:
existing_versions = {r["version"] for r in version_requirements[pkg_key]}
if version_or_tag not in existing_versions:
# Conflict detected - same package, different version
requirements = version_requirements[pkg_key] + [
{"version": version_or_tag, "required_by": required_by}
]
raise DependencyConflictError([
DependencyConflict(
project=proj_name,
package=pkg_name,
requirements=[
{
"version": r["version"],
"required_by": [{"path": r["required_by"]}] if r["required_by"] else []
}
for r in requirements
],
)
])
# Same version already resolved - skip
if artifact_id in visited:
return
# Different version requested - log and use existing (first wins)
existing = version_requirements[pkg_key][0]["version"]
logger.debug(
f"Version mismatch for {pkg_key}: using {existing} "
f"(also requested: {version_or_tag} by {required_by})"
)
# Already resolved this package - skip
return
if artifact_id in visited:
return
# Track path for debugging (only after early-return checks)
resolution_path_sync.append(f"{pkg_key}@{version_or_tag}")
visiting.add(artifact_id)
current_path[artifact_id] = pkg_key
@@ -799,6 +838,10 @@ def resolve_dependencies(
if dep_artifact_id == artifact_id:
continue
# Skip if this artifact is already being visited (would cause cycle)
if dep_artifact_id in visiting:
continue
_resolve_recursive(
dep_artifact_id,
dep.dependency_project,
@@ -812,6 +855,11 @@ def resolve_dependencies(
visiting.remove(artifact_id)
del current_path[artifact_id]
visited.add(artifact_id)
resolution_path_sync.pop()
# Check total artifacts limit
if len(resolution_order) >= MAX_TOTAL_ARTIFACTS:
raise TooManyArtifactsError(MAX_TOTAL_ARTIFACTS)
# Add to resolution order (dependencies before dependents)
resolution_order.append(artifact_id)
@@ -848,6 +896,417 @@ def resolve_dependencies(
},
resolved=resolved_list,
missing=missing_dependencies,
fetched=[], # No fetching in sync version
total_size=total_size,
artifact_count=len(resolved_list),
)
# System project mapping for auto-fetch
SYSTEM_PROJECT_REGISTRY_MAP = {
"_pypi": "pypi",
"_npm": "npm",
"_maven": "maven",
}
async def resolve_dependencies_with_fetch(
db: Session,
project_name: str,
package_name: str,
ref: str,
base_url: str,
storage: "S3Storage",
registry_clients: Dict[str, "RegistryClient"],
) -> DependencyResolutionResponse:
"""
Resolve all dependencies for an artifact recursively, fetching missing ones from upstream.
This async version extends the basic resolution with auto-fetch capability:
when a missing dependency is from a system project (e.g., _pypi), it attempts
to fetch the package from the corresponding upstream registry.
If the root artifact itself doesn't exist in a system project, it will also
be fetched from upstream before resolution begins.
Args:
db: Database session
project_name: Project name
package_name: Package name
ref: Version reference (or artifact:hash)
base_url: Base URL for download URLs
storage: S3 storage for caching fetched artifacts
registry_clients: Map of system project to registry client {"_pypi": PyPIRegistryClient}
Returns:
DependencyResolutionResponse with all resolved artifacts and fetch status
Raises:
DependencyNotFoundError: If the root artifact cannot be found (even after fetch attempt)
CircularDependencyError: If circular dependencies are detected
"""
# Track fetched artifacts for response
fetched_artifacts: List[ResolvedArtifact] = []
# Check if project exists
project = db.query(Project).filter(Project.name == project_name).first()
# If project doesn't exist and it's a system project pattern, we can't auto-create it
if not project:
raise DependencyNotFoundError(project_name, package_name, ref)
# Check if package exists
package = db.query(Package).filter(
Package.project_id == project.id,
Package.name == package_name,
).first()
# Try to resolve the root artifact
root_artifact_id = None
root_version = None
root_size = None
# Handle artifact: prefix for direct artifact ID references
if ref.startswith("artifact:"):
artifact_id = ref[9:]
artifact = db.query(Artifact).filter(Artifact.id == artifact_id).first()
if artifact:
root_artifact_id = artifact.id
root_version = artifact_id[:12]
root_size = artifact.size
elif package:
# Try to resolve by version/constraint
resolved = _resolve_dependency_to_artifact(
db, project_name, package_name, ref
)
if resolved:
root_artifact_id, root_version, root_size = resolved
# If root artifact not found and this is a system project, try to fetch it
if root_artifact_id is None and project_name in SYSTEM_PROJECT_REGISTRY_MAP:
logger.info(
f"Root artifact {project_name}/{package_name}@{ref} not found, "
"attempting to fetch from upstream"
)
client = registry_clients.get(project_name)
if client:
try:
# Resolve the version constraint from upstream
version_info = await client.resolve_constraint(package_name, ref)
if version_info:
# Fetch and cache the package
fetch_result = await client.fetch_package(
package_name, version_info, db, storage
)
if fetch_result:
logger.info(
f"Successfully fetched root artifact {package_name}=="
f"{fetch_result.version} (artifact {fetch_result.artifact_id[:12]})"
)
root_artifact_id = fetch_result.artifact_id
root_version = fetch_result.version
root_size = fetch_result.size
# Add to fetched list
fetched_artifacts.append(ResolvedArtifact(
artifact_id=fetch_result.artifact_id,
project=project_name,
package=package_name,
version=fetch_result.version,
size=fetch_result.size,
download_url=f"{base_url}/api/v1/project/{project_name}/{package_name}/+/{fetch_result.version}",
))
except Exception as e:
logger.warning(f"Failed to fetch root artifact {package_name}: {e}")
# If still no root artifact, raise error
if root_artifact_id is None:
raise DependencyNotFoundError(project_name, package_name, ref)
# Track state
resolved_artifacts: Dict[str, ResolvedArtifact] = {}
missing_dependencies: List[MissingDependency] = []
# Note: fetched_artifacts was already initialized above (line 911)
# and may already contain the root artifact if it was fetched from upstream
version_requirements: Dict[str, List[Dict[str, Any]]] = {}
visiting: Set[str] = set()
visited: Set[str] = set()
current_path: Dict[str, str] = {}
resolution_order: List[str] = []
# Track fetch attempts to prevent loops
fetch_attempted: Set[str] = set() # "project/package@constraint"
async def _try_fetch_dependency(
dep_project: str,
dep_package: str,
constraint: str,
required_by: str,
) -> Optional[Tuple[str, str, int]]:
"""
Try to fetch a missing dependency from upstream registry.
Returns (artifact_id, version, size) if successful, None otherwise.
"""
# Only fetch from system projects
registry_type = SYSTEM_PROJECT_REGISTRY_MAP.get(dep_project)
if not registry_type:
logger.debug(
f"Not a system project, skipping fetch: {dep_project}/{dep_package}"
)
return None
# Build fetch key for loop prevention
fetch_key = f"{dep_project}/{dep_package}@{constraint}"
if fetch_key in fetch_attempted:
logger.debug(f"Already attempted fetch for {fetch_key}")
return None
fetch_attempted.add(fetch_key)
# Get registry client
client = registry_clients.get(dep_project)
if not client:
logger.debug(f"No registry client for {dep_project}")
return None
try:
# Resolve version constraint
version_info = await client.resolve_constraint(dep_package, constraint)
if not version_info:
logger.info(
f"No version of {dep_package} matches constraint '{constraint}' on upstream"
)
return None
# Fetch and cache the package
fetch_result = await client.fetch_package(
dep_package, version_info, db, storage
)
if not fetch_result:
logger.warning(f"Failed to fetch {dep_package}=={version_info.version}")
return None
logger.info(
f"Successfully fetched {dep_package}=={version_info.version} "
f"(artifact {fetch_result.artifact_id[:12]})"
)
# Add to fetched list for response
fetched_artifacts.append(ResolvedArtifact(
artifact_id=fetch_result.artifact_id,
project=dep_project,
package=dep_package,
version=fetch_result.version,
size=fetch_result.size,
download_url=f"{base_url}/api/v1/project/{dep_project}/{dep_package}/+/{fetch_result.version}",
))
return (fetch_result.artifact_id, fetch_result.version, fetch_result.size)
except Exception as e:
logger.warning(f"Error fetching {dep_package}: {e}")
return None
# Track resolution path for debugging
resolution_path: List[str] = []
async def _resolve_recursive_async(
artifact_id: str,
proj_name: str,
pkg_name: str,
version_or_tag: str,
size: int,
required_by: Optional[str],
depth: int = 0,
):
"""Recursively resolve dependencies with fetch capability."""
pkg_key = f"{proj_name}/{pkg_name}"
if depth > MAX_DEPENDENCY_DEPTH:
logger.error(
f"Dependency depth exceeded at {pkg_key} (depth={depth}). "
f"Resolution path: {' -> '.join(resolution_path[-20:])}"
)
raise DependencyDepthExceededError(MAX_DEPENDENCY_DEPTH)
# Cycle detection
if artifact_id in visiting:
cycle_start = current_path.get(artifact_id, pkg_key)
cycle = [cycle_start, pkg_key]
raise CircularDependencyError(cycle)
# Version conflict handling - use first resolved version (lenient mode)
if pkg_key in version_requirements:
existing_versions = {r["version"] for r in version_requirements[pkg_key]}
if version_or_tag not in existing_versions:
# Different version requested - log and use existing (first wins)
existing = version_requirements[pkg_key][0]["version"]
logger.debug(
f"Version mismatch for {pkg_key}: using {existing} "
f"(also requested: {version_or_tag} by {required_by})"
)
# Already resolved this package - skip
return
if artifact_id in visited:
return
# Track path for debugging (only after early-return checks)
resolution_path.append(f"{pkg_key}@{version_or_tag}")
visiting.add(artifact_id)
current_path[artifact_id] = pkg_key
if pkg_key not in version_requirements:
version_requirements[pkg_key] = []
version_requirements[pkg_key].append({
"version": version_or_tag,
"required_by": required_by,
})
# Get dependencies
deps = db.query(ArtifactDependency).filter(
ArtifactDependency.artifact_id == artifact_id
).all()
for dep in deps:
# Skip self-dependencies (common with PyPI extras like pytest[testing] -> pytest)
dep_proj_normalized = dep.dependency_project.lower()
dep_pkg_normalized = _normalize_pypi_package_name(dep.dependency_package)
curr_proj_normalized = proj_name.lower()
curr_pkg_normalized = _normalize_pypi_package_name(pkg_name)
if dep_proj_normalized == curr_proj_normalized and dep_pkg_normalized == curr_pkg_normalized:
logger.debug(
f"Skipping self-dependency: {pkg_key} -> {dep.dependency_project}/{dep.dependency_package}"
)
continue
# Also check if this dependency would resolve to the current artifact
# (handles cases where package names differ but resolve to same artifact)
resolved_dep = _resolve_dependency_to_artifact(
db,
dep.dependency_project,
dep.dependency_package,
dep.version_constraint,
)
if not resolved_dep:
# Try to fetch from upstream if it's a system project
fetched = await _try_fetch_dependency(
dep.dependency_project,
dep.dependency_package,
dep.version_constraint,
pkg_key,
)
if fetched:
resolved_dep = fetched
else:
# Still missing - add to missing list with fetch status
fetch_key = f"{dep.dependency_project}/{dep.dependency_package}@{dep.version_constraint}"
was_attempted = fetch_key in fetch_attempted
missing_dependencies.append(MissingDependency(
project=dep.dependency_project,
package=dep.dependency_package,
constraint=dep.version_constraint,
required_by=pkg_key,
fetch_attempted=was_attempted,
))
continue
dep_artifact_id, dep_version, dep_size = resolved_dep
# Skip if resolved to same artifact (self-dependency at artifact level)
if dep_artifact_id == artifact_id:
logger.debug(
f"Skipping self-dependency (same artifact): {pkg_key} -> "
f"{dep.dependency_project}/{dep.dependency_package} (artifact {dep_artifact_id[:12]})"
)
continue
# Skip if this artifact is already being visited (would cause cycle)
if dep_artifact_id in visiting:
logger.debug(
f"Skipping dependency already in resolution stack: {pkg_key} -> "
f"{dep.dependency_project}/{dep.dependency_package} (artifact {dep_artifact_id[:12]})"
)
continue
# Check if we've already resolved this package to a different version
dep_pkg_key = f"{dep.dependency_project}/{dep.dependency_package}"
if dep_pkg_key in version_requirements:
existing_version = version_requirements[dep_pkg_key][0]["version"]
if existing_version != dep_version:
# Different version resolved - check if existing satisfies new constraint
if HAS_PACKAGING and _version_satisfies_constraint(existing_version, dep.version_constraint):
logger.debug(
f"Reusing existing version {existing_version} for {dep_pkg_key} "
f"(satisfies constraint {dep.version_constraint})"
)
continue
else:
logger.debug(
f"Version conflict for {dep_pkg_key}: have {existing_version}, "
f"need {dep.version_constraint} (resolved to {dep_version})"
)
# Don't raise error - just use the first version we resolved
# This is more lenient than strict conflict detection
continue
await _resolve_recursive_async(
dep_artifact_id,
dep.dependency_project,
dep.dependency_package,
dep_version,
dep_size,
pkg_key,
depth + 1,
)
visiting.remove(artifact_id)
del current_path[artifact_id]
visited.add(artifact_id)
resolution_path.pop()
# Check total artifacts limit
if len(resolution_order) >= MAX_TOTAL_ARTIFACTS:
raise TooManyArtifactsError(MAX_TOTAL_ARTIFACTS)
resolution_order.append(artifact_id)
resolved_artifacts[artifact_id] = ResolvedArtifact(
artifact_id=artifact_id,
project=proj_name,
package=pkg_name,
version=version_or_tag,
size=size,
download_url=f"{base_url}/api/v1/project/{proj_name}/{pkg_name}/+/{version_or_tag}",
)
# Start resolution from root
await _resolve_recursive_async(
root_artifact_id,
project_name,
package_name,
root_version,
root_size,
None,
)
# Build response in topological order
resolved_list = [resolved_artifacts[aid] for aid in resolution_order]
total_size = sum(r.size for r in resolved_list)
return DependencyResolutionResponse(
requested={
"project": project_name,
"package": package_name,
"ref": ref,
},
resolved=resolved_list,
missing=missing_dependencies,
fetched=fetched_artifacts,
total_size=total_size,
artifact_count=len(resolved_list),
)

View File

@@ -150,7 +150,7 @@ class HttpClientManager:
if not self._executor:
raise RuntimeError("HttpClientManager not started. Call startup() first.")
loop = asyncio.get_event_loop()
loop = asyncio.get_running_loop()
return await loop.run_in_executor(self._executor, func, *args)
@property

View File

@@ -27,7 +27,6 @@ from .models import UpstreamSource, CachedUrl, Artifact, Project, Package, Packa
from .storage import S3Storage, get_storage
from .config import get_env_upstream_sources, get_settings
from .http_client import HttpClientManager
from .cache_service import CacheService, CacheCategory
from .db_utils import ArtifactRepository
logger = logging.getLogger(__name__)
@@ -40,11 +39,6 @@ def get_http_client(request: Request) -> HttpClientManager:
return request.app.state.http_client
def get_cache(request: Request) -> CacheService:
"""Get CacheService from app state."""
return request.app.state.cache
# Timeout configuration for proxy requests
PROXY_CONNECT_TIMEOUT = 30.0
PROXY_READ_TIMEOUT = 60.0
@@ -53,17 +47,36 @@ PROXY_READ_TIMEOUT = 60.0
def _parse_requires_dist(requires_dist: str) -> Tuple[str, Optional[str]]:
"""Parse a Requires-Dist line into (package_name, version_constraint).
Filters out optional/extra dependencies and platform-specific dependencies
to avoid pulling in unnecessary packages during dependency resolution.
Examples:
"requests (>=2.25.0)" -> ("requests", ">=2.25.0")
"typing-extensions; python_version < '3.8'" -> ("typing-extensions", None)
"numpy>=1.21.0" -> ("numpy", ">=1.21.0")
"certifi" -> ("certifi", None)
"pytest; extra == 'test'" -> (None, None) # Filtered: extra dependency
"pyobjc; sys_platform == 'darwin'" -> (None, None) # Filtered: platform-specific
Returns:
Tuple of (normalized_package_name, version_constraint or None)
Returns (None, None) for dependencies that should be filtered out.
"""
# Remove any environment markers (after semicolon)
# Check for and filter environment markers (after semicolon)
if ';' in requires_dist:
marker_part = requires_dist.split(';', 1)[1].lower()
# Filter out extra/optional dependencies - these are not core dependencies
# Examples: "pytest; extra == 'test'", "sphinx; extra == 'docs'"
if 'extra' in marker_part:
return None, None
# Filter out platform-specific dependencies to avoid cross-platform bloat
# Examples: "pyobjc; sys_platform == 'darwin'", "pywin32; sys_platform == 'win32'"
if 'sys_platform' in marker_part or 'platform_system' in marker_part:
return None, None
# Strip the marker for remaining dependencies (like python_version constraints)
requires_dist = requires_dist.split(';')[0].strip()
# Match patterns like "package (>=1.0)" or "package>=1.0" or "package"
@@ -254,62 +267,6 @@ def _extract_pypi_version(filename: str) -> Optional[str]:
return None
async def _get_pypi_upstream_sources_cached(
db: Session,
cache: CacheService,
) -> list[UpstreamSource]:
"""
Get PyPI upstream sources with caching.
Sources are cached for cache_ttl_upstream seconds to avoid
repeated database queries on every request.
"""
cache_key = "sources"
# Try cache first
cached = await cache.get(CacheCategory.UPSTREAM_SOURCES, cache_key, protocol="pypi")
if cached:
source_data = json.loads(cached.decode())
# Reconstruct UpstreamSource-like objects from cached data
# We cache just the essential fields needed for requests
return [type('CachedSource', (), d)() for d in source_data]
# Query database
db_sources = (
db.query(UpstreamSource)
.filter(UpstreamSource.source_type == "pypi", UpstreamSource.enabled == True)
.order_by(UpstreamSource.priority)
.all()
)
# Combine with env sources
env_sources = [s for s in get_env_upstream_sources() if s.source_type == "pypi"]
all_sources = list(db_sources) + list(env_sources)
all_sources = sorted(all_sources, key=lambda s: s.priority)
# Cache the essential fields
if all_sources and cache.enabled:
cache_data = [
{
"name": s.name,
"url": s.url,
"priority": s.priority,
"auth_type": getattr(s, "auth_type", "none"),
"username": getattr(s, "username", None),
"password": getattr(s, "password", None),
}
for s in all_sources
]
await cache.set(
CacheCategory.UPSTREAM_SOURCES,
cache_key,
json.dumps(cache_data).encode(),
protocol="pypi",
)
return all_sources
def _get_pypi_upstream_sources(db: Session) -> list[UpstreamSource]:
"""Get all enabled upstream sources configured for PyPI."""
# Get database sources
@@ -634,6 +591,258 @@ async def pypi_package_versions(
)
async def fetch_and_cache_pypi_package(
db: Session,
storage: S3Storage,
http_client: httpx.AsyncClient,
package_name: str,
filename: str,
download_url: str,
expected_sha256: Optional[str] = None,
) -> Optional[dict]:
"""
Fetch a PyPI package from upstream and cache it in Orchard.
This is the core caching logic extracted from pypi_download_file() for reuse
by the registry client during auto-fetch dependency resolution.
Args:
db: Database session
storage: S3 storage instance
http_client: Async HTTP client for making requests
package_name: Normalized package name (e.g., 'requests')
filename: Package filename (e.g., 'requests-2.31.0-py3-none-any.whl')
download_url: Full URL to download from upstream
expected_sha256: Optional SHA256 to verify download integrity
Returns:
Dict with artifact_id, size, version, already_cached if successful.
None if the fetch failed.
"""
# Normalize package name
normalized_name = re.sub(r'[-_.]+', '-', package_name).lower()
# Check if we already have this URL cached
url_hash = hashlib.sha256(download_url.encode()).hexdigest()
cached_url = db.query(CachedUrl).filter(CachedUrl.url_hash == url_hash).first()
if cached_url:
# Already cached - return existing artifact info
artifact = db.query(Artifact).filter(Artifact.id == cached_url.artifact_id).first()
if artifact:
version = _extract_pypi_version(filename)
logger.info(f"PyPI fetch: {filename} already cached (artifact {artifact.id[:12]})")
return {
"artifact_id": artifact.id,
"size": artifact.size,
"version": version,
"already_cached": True,
}
# Get upstream sources for auth headers
sources = _get_pypi_upstream_sources(db)
matched_source = sources[0] if sources else None
headers = {"User-Agent": "Orchard-PyPI-Proxy/1.0"}
if matched_source:
headers.update(_build_auth_headers(matched_source))
auth = _get_basic_auth(matched_source) if matched_source else None
download_timeout = httpx.Timeout(connect=30.0, read=300.0, write=300.0, pool=30.0)
try:
logger.info(f"PyPI fetch: downloading {filename} from {download_url}")
response = await http_client.get(
download_url,
headers=headers,
auth=auth,
timeout=download_timeout,
)
# Handle redirects manually
redirect_count = 0
while response.status_code in (301, 302, 303, 307, 308) and redirect_count < 5:
redirect_url = response.headers.get('location')
if not redirect_url:
break
if not redirect_url.startswith('http'):
redirect_url = urljoin(download_url, redirect_url)
logger.debug(f"PyPI fetch: following redirect to {redirect_url}")
# Don't send auth to different hosts
redirect_headers = {"User-Agent": "Orchard-PyPI-Proxy/1.0"}
redirect_auth = None
if urlparse(redirect_url).netloc == urlparse(download_url).netloc:
redirect_headers.update(headers)
redirect_auth = auth
response = await http_client.get(
redirect_url,
headers=redirect_headers,
auth=redirect_auth,
follow_redirects=False,
timeout=download_timeout,
)
redirect_count += 1
if response.status_code != 200:
error_detail = _parse_upstream_error(response)
logger.warning(f"PyPI fetch: upstream returned {response.status_code} for {filename}: {error_detail}")
return None
content_type = response.headers.get('content-type', 'application/octet-stream')
# Stream to temp file to avoid loading large packages into memory
tmp_path = None
try:
with tempfile.NamedTemporaryFile(delete=False, suffix=f"_{filename}") as tmp_file:
tmp_path = tmp_file.name
async for chunk in response.aiter_bytes(chunk_size=65536):
tmp_file.write(chunk)
# Store in S3 from temp file (computes hash and deduplicates automatically)
with open(tmp_path, 'rb') as f:
result = storage.store(f)
sha256 = result.sha256
size = result.size
# Verify hash if expected
if expected_sha256 and sha256 != expected_sha256.lower():
logger.error(
f"PyPI fetch: hash mismatch for {filename}: "
f"expected {expected_sha256[:12]}, got {sha256[:12]}"
)
return None
# Extract dependencies from the temp file
extracted_deps = _extract_dependencies_from_file(tmp_path, filename)
if extracted_deps:
logger.info(f"PyPI fetch: extracted {len(extracted_deps)} dependencies from {filename}")
logger.info(f"PyPI fetch: downloaded {filename}, {size} bytes, sha256={sha256[:12]}")
finally:
# Clean up temp file
if tmp_path and os.path.exists(tmp_path):
os.unlink(tmp_path)
# Check if artifact already exists
existing = db.query(Artifact).filter(Artifact.id == sha256).first()
if existing:
existing.ref_count += 1
db.flush()
else:
new_artifact = Artifact(
id=sha256,
original_name=filename,
content_type=content_type,
size=size,
ref_count=1,
created_by="pypi-proxy",
s3_key=result.s3_key,
checksum_md5=result.md5,
checksum_sha1=result.sha1,
s3_etag=result.s3_etag,
)
db.add(new_artifact)
db.flush()
# Create/get system project and package
system_project = db.query(Project).filter(Project.name == "_pypi").first()
if not system_project:
system_project = Project(
name="_pypi",
description="System project for cached PyPI packages",
is_public=True,
is_system=True,
created_by="pypi-proxy",
)
db.add(system_project)
db.flush()
elif not system_project.is_system:
system_project.is_system = True
db.flush()
package = db.query(Package).filter(
Package.project_id == system_project.id,
Package.name == normalized_name,
).first()
if not package:
package = Package(
project_id=system_project.id,
name=normalized_name,
description=f"PyPI package: {normalized_name}",
format="pypi",
)
db.add(package)
db.flush()
# Extract and create version
version = _extract_pypi_version(filename)
if version and not filename.endswith('.metadata'):
existing_version = db.query(PackageVersion).filter(
PackageVersion.package_id == package.id,
PackageVersion.version == version,
).first()
if not existing_version:
pkg_version = PackageVersion(
package_id=package.id,
artifact_id=sha256,
version=version,
version_source="filename",
created_by="pypi-proxy",
)
db.add(pkg_version)
# Cache the URL mapping
existing_cached = db.query(CachedUrl).filter(CachedUrl.url_hash == url_hash).first()
if not existing_cached:
cached_url_record = CachedUrl(
url_hash=url_hash,
url=download_url,
artifact_id=sha256,
)
db.add(cached_url_record)
# Store extracted dependencies using batch operation
if extracted_deps:
seen_deps: dict[str, str] = {}
for dep_name, dep_version in extracted_deps:
if dep_name not in seen_deps:
seen_deps[dep_name] = dep_version if dep_version else "*"
deps_to_store = [
("_pypi", dep_name, dep_version)
for dep_name, dep_version in seen_deps.items()
]
repo = ArtifactRepository(db)
inserted = repo.batch_upsert_dependencies(sha256, deps_to_store)
if inserted > 0:
logger.debug(f"Stored {inserted} dependencies for {sha256[:12]}...")
db.commit()
return {
"artifact_id": sha256,
"size": size,
"version": version,
"already_cached": False,
}
except httpx.ConnectError as e:
logger.warning(f"PyPI fetch: connection failed for {filename}: {e}")
return None
except httpx.TimeoutException as e:
logger.warning(f"PyPI fetch: timeout for {filename}: {e}")
return None
except Exception as e:
logger.exception(f"PyPI fetch: error downloading {filename}")
return None
@router.get("/simple/{package_name}/{filename}")
async def pypi_download_file(
request: Request,
@@ -643,7 +852,6 @@ async def pypi_download_file(
db: Session = Depends(get_db),
storage: S3Storage = Depends(get_storage),
http_client: HttpClientManager = Depends(get_http_client),
cache: CacheService = Depends(get_cache),
):
"""
Download a package file, caching it in Orchard.

View File

@@ -0,0 +1,426 @@
"""
Registry client abstraction for upstream package registries.
Provides a pluggable interface for fetching packages from upstream registries
(PyPI, npm, Maven, etc.) during dependency resolution with auto-fetch enabled.
"""
import hashlib
import logging
import os
import re
import tempfile
from abc import ABC, abstractmethod
from dataclasses import dataclass
from typing import List, Optional, TYPE_CHECKING
from urllib.parse import urljoin, urlparse
import httpx
from packaging.specifiers import SpecifierSet, InvalidSpecifier
from packaging.version import Version, InvalidVersion
from sqlalchemy.orm import Session
if TYPE_CHECKING:
from .storage import S3Storage
from .http_client import HttpClientManager
logger = logging.getLogger(__name__)
@dataclass
class VersionInfo:
"""Information about a package version from an upstream registry."""
version: str
download_url: str
filename: str
sha256: Optional[str] = None
size: Optional[int] = None
content_type: Optional[str] = None
@dataclass
class FetchResult:
"""Result of fetching a package from upstream."""
artifact_id: str # SHA256 hash
size: int
version: str
filename: str
already_cached: bool = False
class RegistryClient(ABC):
"""Abstract base class for upstream registry clients."""
@property
@abstractmethod
def source_type(self) -> str:
"""Return the source type this client handles (e.g., 'pypi', 'npm')."""
pass
@abstractmethod
async def get_available_versions(self, package_name: str) -> List[str]:
"""
Get all available versions of a package from upstream.
Args:
package_name: The normalized package name
Returns:
List of version strings, sorted from oldest to newest
"""
pass
@abstractmethod
async def resolve_constraint(
self, package_name: str, constraint: str
) -> Optional[VersionInfo]:
"""
Find the best version matching a constraint.
Args:
package_name: The normalized package name
constraint: Version constraint (e.g., '>=1.9', '<2.0,>=1.5', '*')
Returns:
VersionInfo with download URL, or None if no matching version found
"""
pass
@abstractmethod
async def fetch_package(
self,
package_name: str,
version_info: VersionInfo,
db: Session,
storage: "S3Storage",
) -> Optional[FetchResult]:
"""
Fetch and cache a package from upstream.
Args:
package_name: The normalized package name
version_info: Version details including download URL
db: Database session for creating records
storage: S3 storage for caching the artifact
Returns:
FetchResult with artifact_id, or None if fetch failed
"""
pass
class PyPIRegistryClient(RegistryClient):
"""PyPI registry client using the JSON API."""
# Timeout configuration for PyPI requests
CONNECT_TIMEOUT = 30.0
READ_TIMEOUT = 60.0
DOWNLOAD_TIMEOUT = 300.0 # Longer timeout for file downloads
def __init__(
self,
http_client: httpx.AsyncClient,
upstream_sources: List,
pypi_api_url: str = "https://pypi.org/pypi",
):
"""
Initialize PyPI registry client.
Args:
http_client: Shared async HTTP client
upstream_sources: List of configured upstream sources for auth
pypi_api_url: Base URL for PyPI JSON API
"""
self.client = http_client
self.sources = upstream_sources
self.api_url = pypi_api_url
@property
def source_type(self) -> str:
return "pypi"
def _normalize_package_name(self, name: str) -> str:
"""Normalize a PyPI package name per PEP 503."""
return re.sub(r"[-_.]+", "-", name).lower()
def _get_auth_headers(self) -> dict:
"""Get authentication headers from configured sources."""
headers = {"User-Agent": "Orchard-Registry-Client/1.0"}
if self.sources:
source = self.sources[0]
if hasattr(source, "auth_type"):
if source.auth_type == "bearer":
password = (
source.get_password()
if hasattr(source, "get_password")
else getattr(source, "password", None)
)
if password:
headers["Authorization"] = f"Bearer {password}"
elif source.auth_type == "api_key":
custom_headers = (
source.get_headers()
if hasattr(source, "get_headers")
else {}
)
if custom_headers:
headers.update(custom_headers)
return headers
def _get_basic_auth(self) -> Optional[tuple]:
"""Get basic auth credentials if configured."""
if self.sources:
source = self.sources[0]
if hasattr(source, "auth_type") and source.auth_type == "basic":
username = getattr(source, "username", None)
if username:
password = (
source.get_password()
if hasattr(source, "get_password")
else getattr(source, "password", "")
)
return (username, password or "")
return None
async def get_available_versions(self, package_name: str) -> List[str]:
"""Get all available versions from PyPI JSON API."""
normalized = self._normalize_package_name(package_name)
url = f"{self.api_url}/{normalized}/json"
headers = self._get_auth_headers()
auth = self._get_basic_auth()
timeout = httpx.Timeout(self.READ_TIMEOUT, connect=self.CONNECT_TIMEOUT)
try:
response = await self.client.get(
url, headers=headers, auth=auth, timeout=timeout
)
if response.status_code == 404:
logger.debug(f"Package {normalized} not found on PyPI")
return []
if response.status_code != 200:
logger.warning(
f"PyPI API returned {response.status_code} for {normalized}"
)
return []
data = response.json()
releases = data.get("releases", {})
# Filter to valid versions and sort
versions = []
for v in releases.keys():
try:
Version(v)
versions.append(v)
except InvalidVersion:
continue
versions.sort(key=lambda x: Version(x))
return versions
except httpx.RequestError as e:
logger.warning(f"Failed to query PyPI for {normalized}: {e}")
return []
except Exception as e:
logger.warning(f"Error parsing PyPI response for {normalized}: {e}")
return []
async def resolve_constraint(
self, package_name: str, constraint: str
) -> Optional[VersionInfo]:
"""Find best version matching constraint from PyPI."""
normalized = self._normalize_package_name(package_name)
url = f"{self.api_url}/{normalized}/json"
headers = self._get_auth_headers()
auth = self._get_basic_auth()
timeout = httpx.Timeout(self.READ_TIMEOUT, connect=self.CONNECT_TIMEOUT)
try:
response = await self.client.get(
url, headers=headers, auth=auth, timeout=timeout
)
if response.status_code == 404:
logger.debug(f"Package {normalized} not found on PyPI")
return None
if response.status_code != 200:
logger.warning(
f"PyPI API returned {response.status_code} for {normalized}"
)
return None
data = response.json()
releases = data.get("releases", {})
# Handle wildcard - return latest version
if constraint == "*":
latest_version = data.get("info", {}).get("version")
if latest_version and latest_version in releases:
return self._get_version_info(
normalized, latest_version, releases[latest_version]
)
return None
# Parse constraint
# If constraint looks like a bare version (no operator), treat as exact match
# e.g., "2025.10.5" -> "==2025.10.5"
effective_constraint = constraint
if constraint and constraint[0].isdigit():
effective_constraint = f"=={constraint}"
logger.debug(
f"Bare version '{constraint}' for {normalized}, "
f"treating as exact match '{effective_constraint}'"
)
try:
specifier = SpecifierSet(effective_constraint)
except InvalidSpecifier:
# Invalid constraint - treat as wildcard
logger.warning(
f"Invalid version constraint '{constraint}' for {normalized}, "
"treating as wildcard"
)
latest_version = data.get("info", {}).get("version")
if latest_version and latest_version in releases:
return self._get_version_info(
normalized, latest_version, releases[latest_version]
)
return None
# Find matching versions
matching = []
for v_str, files in releases.items():
if not files: # Skip versions with no files
continue
try:
v = Version(v_str)
if v in specifier:
matching.append((v_str, v, files))
except InvalidVersion:
continue
if not matching:
logger.debug(
f"No versions of {normalized} match constraint '{constraint}'"
)
return None
# Sort by version and return highest match
matching.sort(key=lambda x: x[1], reverse=True)
best_version, _, best_files = matching[0]
return self._get_version_info(normalized, best_version, best_files)
except httpx.RequestError as e:
logger.warning(f"Failed to query PyPI for {normalized}: {e}")
return None
except Exception as e:
logger.warning(f"Error resolving {normalized}@{constraint}: {e}")
return None
def _get_version_info(
self, package_name: str, version: str, files: List[dict]
) -> Optional[VersionInfo]:
"""Extract download info from PyPI release files."""
if not files:
return None
# Prefer wheel over sdist
wheel_file = None
sdist_file = None
for f in files:
filename = f.get("filename", "")
if filename.endswith(".whl"):
# Prefer platform-agnostic wheels
if "py3-none-any" in filename or wheel_file is None:
wheel_file = f
elif filename.endswith(".tar.gz") and sdist_file is None:
sdist_file = f
selected = wheel_file or sdist_file
if not selected:
# Fall back to first available file
selected = files[0]
return VersionInfo(
version=version,
download_url=selected.get("url", ""),
filename=selected.get("filename", ""),
sha256=selected.get("digests", {}).get("sha256"),
size=selected.get("size"),
content_type="application/zip"
if selected.get("filename", "").endswith(".whl")
else "application/gzip",
)
async def fetch_package(
self,
package_name: str,
version_info: VersionInfo,
db: Session,
storage: "S3Storage",
) -> Optional[FetchResult]:
"""Fetch and cache a PyPI package."""
# Import here to avoid circular imports
from .pypi_proxy import fetch_and_cache_pypi_package
normalized = self._normalize_package_name(package_name)
logger.info(
f"Fetching {normalized}=={version_info.version} from upstream PyPI"
)
result = await fetch_and_cache_pypi_package(
db=db,
storage=storage,
http_client=self.client,
package_name=normalized,
filename=version_info.filename,
download_url=version_info.download_url,
expected_sha256=version_info.sha256,
)
if result is None:
return None
return FetchResult(
artifact_id=result["artifact_id"],
size=result["size"],
version=version_info.version,
filename=version_info.filename,
already_cached=result.get("already_cached", False),
)
def get_registry_client(
source_type: str,
http_client: httpx.AsyncClient,
upstream_sources: List,
) -> Optional[RegistryClient]:
"""
Factory function to get a registry client for a source type.
Args:
source_type: The registry type ('pypi', 'npm', etc.)
http_client: Shared async HTTP client
upstream_sources: List of configured upstream sources
Returns:
RegistryClient for the source type, or None if not supported
"""
if source_type == "pypi":
# Filter to PyPI sources
pypi_sources = [s for s in upstream_sources if getattr(s, "source_type", "") == "pypi"]
return PyPIRegistryClient(http_client, pypi_sources)
# Future: Add npm, maven, etc.
logger.debug(f"No registry client available for source type: {source_type}")
return None

View File

@@ -141,11 +141,13 @@ from .dependencies import (
get_reverse_dependencies,
check_circular_dependencies,
resolve_dependencies,
resolve_dependencies_with_fetch,
InvalidEnsureFileError,
CircularDependencyError,
DependencyConflictError,
DependencyNotFoundError,
DependencyDepthExceededError,
TooManyArtifactsError,
)
from .config import get_settings, get_env_upstream_sources
from .checksum import (
@@ -2665,10 +2667,10 @@ def list_packages(
format: Optional[str] = Query(default=None, description="Filter by package format"),
platform: Optional[str] = Query(default=None, description="Filter by platform"),
db: Session = Depends(get_db),
current_user: Optional[User] = Depends(get_current_user_optional),
):
project = db.query(Project).filter(Project.name == project_name).first()
if not project:
raise HTTPException(status_code=404, detail="Project not found")
# Check read access (handles private project visibility)
project = check_project_access(db, project_name, current_user, "read")
# Validate sort field
valid_sort_fields = {
@@ -2949,13 +2951,13 @@ def update_package(
package_update: PackageUpdate,
request: Request,
db: Session = Depends(get_db),
current_user: Optional[User] = Depends(get_current_user_optional),
):
"""Update a package's metadata."""
user_id = get_user_id(request)
project = db.query(Project).filter(Project.name == project_name).first()
if not project:
raise HTTPException(status_code=404, detail="Project not found")
# Check write access to project
project = check_project_access(db, project_name, current_user, "write")
package = (
db.query(Package)
@@ -3032,6 +3034,7 @@ def delete_package(
package_name: str,
request: Request,
db: Session = Depends(get_db),
current_user: Optional[User] = Depends(get_current_user_optional),
):
"""
Delete a package and all its versions.
@@ -3042,9 +3045,8 @@ def delete_package(
"""
user_id = get_user_id(request)
project = db.query(Project).filter(Project.name == project_name).first()
if not project:
raise HTTPException(status_code=404, detail="Project not found")
# Check write access to project (deletion requires write permission)
project = check_project_access(db, project_name, current_user, "write")
package = (
db.query(Package)
@@ -7025,12 +7027,17 @@ def get_package_reverse_dependencies(
response_model=DependencyResolutionResponse,
tags=["dependencies"],
)
def resolve_artifact_dependencies(
async def resolve_artifact_dependencies(
project_name: str,
package_name: str,
ref: str,
request: Request,
auto_fetch: bool = Query(
True,
description="Fetch missing dependencies from upstream registries (e.g., PyPI). Set to false for fast, network-free resolution."
),
db: Session = Depends(get_db),
storage: S3Storage = Depends(get_storage),
current_user: Optional[User] = Depends(get_current_user_optional),
):
"""
@@ -7039,6 +7046,16 @@ def resolve_artifact_dependencies(
Returns a flat list of all artifacts needed, in topological order
(dependencies before dependents). Includes download URLs for each artifact.
**Parameters:**
- **auto_fetch**: When true (default), attempts to fetch missing dependencies from
upstream registries (PyPI for _pypi project packages). Set to false for
fast, network-free resolution when all dependencies are already cached.
**Response Fields:**
- **resolved**: All artifacts in dependency order with download URLs
- **missing**: Dependencies that couldn't be resolved (with fetch status if auto_fetch=true)
- **fetched**: Artifacts that were fetched from upstream during this request
**Error Responses:**
- 404: Artifact or dependency not found
- 409: Circular dependency or version conflict detected
@@ -7050,7 +7067,38 @@ def resolve_artifact_dependencies(
base_url = str(request.base_url).rstrip("/")
try:
return resolve_dependencies(db, project_name, package_name, ref, base_url)
if auto_fetch:
# Use async resolution with auto-fetch
from .registry_client import get_registry_client
from .pypi_proxy import _get_pypi_upstream_sources
settings = get_settings()
# Get HTTP client from app state
http_client = request.app.state.http_client.get_client()
# Get upstream sources for registry clients
pypi_sources = _get_pypi_upstream_sources(db)
# Build registry clients
registry_clients = {}
pypi_client = get_registry_client("pypi", http_client, pypi_sources)
if pypi_client:
registry_clients["_pypi"] = pypi_client
return await resolve_dependencies_with_fetch(
db=db,
project_name=project_name,
package_name=package_name,
ref=ref,
base_url=base_url,
storage=storage,
registry_clients=registry_clients,
)
else:
# Fast, synchronous resolution without network calls
return resolve_dependencies(db, project_name, package_name, ref, base_url)
except DependencyNotFoundError as e:
raise HTTPException(
status_code=404,
@@ -7090,6 +7138,15 @@ def resolve_artifact_dependencies(
"max_depth": e.max_depth,
}
)
except TooManyArtifactsError as e:
raise HTTPException(
status_code=400,
detail={
"error": "too_many_artifacts",
"message": str(e),
"max_artifacts": e.max_artifacts,
}
)
# --- Upstream Caching Routes ---

View File

@@ -892,6 +892,8 @@ class MissingDependency(BaseModel):
package: str
constraint: Optional[str] = None
required_by: Optional[str] = None
fetch_attempted: bool = False # True if auto-fetch was attempted
fetch_error: Optional[str] = None # Error message if fetch failed
class DependencyResolutionResponse(BaseModel):
@@ -899,6 +901,7 @@ class DependencyResolutionResponse(BaseModel):
requested: Dict[str, str] # project, package, ref
resolved: List[ResolvedArtifact]
missing: List[MissingDependency] = []
fetched: List[ResolvedArtifact] = [] # Artifacts fetched from upstream during resolution
total_size: int
artifact_count: int

View File

@@ -873,10 +873,14 @@ class TestCircularDependencyDetection:
class TestConflictDetection:
"""Tests for #81: Dependency Conflict Detection and Reporting"""
"""Tests for dependency conflict handling.
The resolver uses "first version wins" strategy for version conflicts,
allowing resolution to succeed rather than failing with an error.
"""
@pytest.mark.integration
def test_detect_version_conflict(
def test_version_conflict_uses_first_version(
self, integration_client, test_project, unique_test_id
):
"""Test conflict when two deps require different versions of same package."""
@@ -968,21 +972,19 @@ class TestConflictDetection:
)
assert response.status_code == 200
# Try to resolve app - should report conflict
# Try to resolve app - with lenient conflict handling, this should succeed
# The resolver uses "first version wins" strategy for conflicting versions
response = integration_client.get(
f"/api/v1/project/{test_project}/{pkg_app}/+/1.0.0/resolve"
)
assert response.status_code == 409
assert response.status_code == 200
data = response.json()
# Error details are nested in "detail" for HTTPException
detail = data.get("detail", data)
assert detail.get("error") == "dependency_conflict"
assert len(detail.get("conflicts", [])) > 0
# Verify conflict details
conflict = detail["conflicts"][0]
assert conflict["package"] == pkg_common
assert len(conflict["requirements"]) == 2
# Resolution should succeed with first-encountered version of common
assert data["artifact_count"] >= 1
# Find the common package in resolved list
common_resolved = [r for r in data["resolved"] if r["package"] == pkg_common]
assert len(common_resolved) == 1 # Only one version should be included
finally:
for pkg in [pkg_app, pkg_lib_a, pkg_lib_b, pkg_common]:
@@ -1067,3 +1069,277 @@ class TestConflictDetection:
finally:
for pkg in [pkg_app, pkg_lib_a, pkg_lib_b, pkg_common]:
integration_client.delete(f"/api/v1/project/{test_project}/packages/{pkg}")
class TestAutoFetchDependencies:
"""Tests for auto-fetch functionality in dependency resolution.
These tests verify:
- Resolution with auto_fetch=true (default) fetches missing dependencies from upstream
- Resolution with auto_fetch=false skips network calls for fast resolution
- Proper handling of missing/non-existent packages
- Response schema includes fetched artifacts list
"""
@pytest.mark.integration
def test_resolve_auto_fetch_true_is_default(
self, integration_client, test_package, unique_test_id
):
"""Test that auto_fetch=true is the default (no fetch needed when all deps cached)."""
project_name, package_name = test_package
# Upload a simple artifact without dependencies
content = unique_content("autofetch-default", unique_test_id, "nodeps")
files = {"file": ("default.tar.gz", BytesIO(content), "application/gzip")}
response = integration_client.post(
f"/api/v1/project/{project_name}/{package_name}/upload",
files=files,
data={"version": f"v1.0.0-{unique_test_id}"},
)
assert response.status_code == 200
# Resolve without auto_fetch param (should default to false)
response = integration_client.get(
f"/api/v1/project/{project_name}/{package_name}/+/v1.0.0-{unique_test_id}/resolve"
)
assert response.status_code == 200
data = response.json()
# Should have empty fetched list
assert data.get("fetched", []) == []
assert data["artifact_count"] == 1
@pytest.mark.integration
def test_resolve_auto_fetch_explicit_false(
self, integration_client, test_package, unique_test_id
):
"""Test that auto_fetch=false works explicitly."""
project_name, package_name = test_package
content = unique_content("autofetch-explicit-false", unique_test_id, "nodeps")
files = {"file": ("explicit.tar.gz", BytesIO(content), "application/gzip")}
response = integration_client.post(
f"/api/v1/project/{project_name}/{package_name}/upload",
files=files,
data={"version": f"v2.0.0-{unique_test_id}"},
)
assert response.status_code == 200
# Resolve with explicit auto_fetch=false
response = integration_client.get(
f"/api/v1/project/{project_name}/{package_name}/+/v2.0.0-{unique_test_id}/resolve",
params={"auto_fetch": "false"},
)
assert response.status_code == 200
data = response.json()
assert data.get("fetched", []) == []
@pytest.mark.integration
def test_resolve_auto_fetch_true_no_missing_deps(
self, integration_client, test_project, unique_test_id
):
"""Test that auto_fetch=true works when all deps are already cached."""
pkg_a = f"fetch-a-{unique_test_id}"
pkg_b = f"fetch-b-{unique_test_id}"
for pkg in [pkg_a, pkg_b]:
response = integration_client.post(
f"/api/v1/project/{test_project}/packages",
json={"name": pkg}
)
assert response.status_code == 200
try:
# Upload B (no deps)
content_b = unique_content("B", unique_test_id, "fetch")
files = {"file": ("b.tar.gz", BytesIO(content_b), "application/gzip")}
response = integration_client.post(
f"/api/v1/project/{test_project}/{pkg_b}/upload",
files=files,
data={"version": "1.0.0"},
)
assert response.status_code == 200
# Upload A (depends on B)
ensure_a = yaml.dump({
"dependencies": [
{"project": test_project, "package": pkg_b, "version": "1.0.0"}
]
})
content_a = unique_content("A", unique_test_id, "fetch")
files = {
"file": ("a.tar.gz", BytesIO(content_a), "application/gzip"),
"ensure": ("orchard.ensure", BytesIO(ensure_a.encode()), "application/x-yaml"),
}
response = integration_client.post(
f"/api/v1/project/{test_project}/{pkg_a}/upload",
files=files,
data={"version": "1.0.0"},
)
assert response.status_code == 200
# Resolve with auto_fetch=true - should work since deps are cached
response = integration_client.get(
f"/api/v1/project/{test_project}/{pkg_a}/+/1.0.0/resolve",
params={"auto_fetch": "true"},
)
assert response.status_code == 200
data = response.json()
# Should resolve successfully
assert data["artifact_count"] == 2
# Nothing fetched since everything was cached
assert len(data.get("fetched", [])) == 0
# No missing deps
assert len(data.get("missing", [])) == 0
finally:
for pkg in [pkg_a, pkg_b]:
integration_client.delete(f"/api/v1/project/{test_project}/packages/{pkg}")
@pytest.mark.integration
def test_resolve_missing_dep_with_auto_fetch_false(
self, integration_client, test_package, unique_test_id
):
"""Test that missing deps are reported when auto_fetch=false."""
project_name, package_name = test_package
# Create _pypi system project if it doesn't exist
response = integration_client.get("/api/v1/projects/_pypi")
if response.status_code == 404:
response = integration_client.post(
"/api/v1/projects",
json={"name": "_pypi", "description": "System project for PyPI packages"}
)
# May fail if already exists or can't create - that's ok
# Upload artifact with dependency on _pypi package that doesn't exist locally
ensure_content = yaml.dump({
"dependencies": [
{"project": "_pypi", "package": "nonexistent-pkg-xyz123", "version": ">=1.0.0"}
]
})
content = unique_content("missing-pypi", unique_test_id, "dep")
files = {
"file": ("missing-pypi-dep.tar.gz", BytesIO(content), "application/gzip"),
"ensure": ("orchard.ensure", BytesIO(ensure_content.encode()), "application/x-yaml"),
}
response = integration_client.post(
f"/api/v1/project/{project_name}/{package_name}/upload",
files=files,
data={"version": f"v3.0.0-{unique_test_id}"},
)
# Upload should succeed - validation is loose for system projects
if response.status_code == 200:
# Resolve without auto_fetch - should report missing
response = integration_client.get(
f"/api/v1/project/{project_name}/{package_name}/+/v3.0.0-{unique_test_id}/resolve",
params={"auto_fetch": "false"},
)
assert response.status_code == 200
data = response.json()
# Should have missing dependencies
assert len(data.get("missing", [])) >= 1
# Verify missing dependency structure
missing = data["missing"][0]
assert missing["project"] == "_pypi"
assert missing["package"] == "nonexistent-pkg-xyz123"
# Without auto_fetch, these should be false/None
assert missing.get("fetch_attempted", False) is False
@pytest.mark.integration
def test_resolve_response_schema_has_fetched_field(
self, integration_client, test_package, unique_test_id
):
"""Test that the resolve response always includes the fetched field."""
project_name, package_name = test_package
content = unique_content("schema-check", unique_test_id, "nodeps")
files = {"file": ("schema.tar.gz", BytesIO(content), "application/gzip")}
response = integration_client.post(
f"/api/v1/project/{project_name}/{package_name}/upload",
files=files,
data={"version": f"v4.0.0-{unique_test_id}"},
)
assert response.status_code == 200
# Check both auto_fetch modes include fetched field
for auto_fetch in ["false", "true"]:
response = integration_client.get(
f"/api/v1/project/{project_name}/{package_name}/+/v4.0.0-{unique_test_id}/resolve",
params={"auto_fetch": auto_fetch},
)
assert response.status_code == 200
data = response.json()
# Required fields
assert "requested" in data
assert "resolved" in data
assert "missing" in data
assert "fetched" in data # New field
assert "total_size" in data
assert "artifact_count" in data
# Types
assert isinstance(data["fetched"], list)
assert isinstance(data["missing"], list)
@pytest.mark.integration
def test_missing_dep_schema_has_fetch_fields(
self, integration_client, test_package, unique_test_id
):
"""Test that missing dependency entries have fetch_attempted and fetch_error fields."""
project_name, package_name = test_package
# Create a dependency on a non-existent package in a real project
dep_project_name = f"dep-test-{unique_test_id}"
response = integration_client.post(
"/api/v1/projects", json={"name": dep_project_name}
)
assert response.status_code == 200
try:
ensure_content = yaml.dump({
"dependencies": [
{"project": dep_project_name, "package": "nonexistent-pkg", "version": "1.0.0"}
]
})
content = unique_content("missing-schema", unique_test_id, "check")
files = {
"file": ("missing-schema.tar.gz", BytesIO(content), "application/gzip"),
"ensure": ("orchard.ensure", BytesIO(ensure_content.encode()), "application/x-yaml"),
}
response = integration_client.post(
f"/api/v1/project/{project_name}/{package_name}/upload",
files=files,
data={"version": f"v5.0.0-{unique_test_id}"},
)
assert response.status_code == 200
# Resolve
response = integration_client.get(
f"/api/v1/project/{project_name}/{package_name}/+/v5.0.0-{unique_test_id}/resolve",
params={"auto_fetch": "true"},
)
assert response.status_code == 200
data = response.json()
# Should have missing dependencies
assert len(data.get("missing", [])) >= 1
# Check schema for missing dependency
missing = data["missing"][0]
assert "project" in missing
assert "package" in missing
assert "constraint" in missing
assert "required_by" in missing
# New fields
assert "fetch_attempted" in missing
assert "fetch_error" in missing # May be None
finally:
integration_client.delete(f"/api/v1/projects/{dep_project_name}")

View File

@@ -0,0 +1,243 @@
"""Unit tests for metadata extraction functionality."""
import io
import gzip
import tarfile
import zipfile
import pytest
from app.metadata import (
extract_metadata,
extract_deb_metadata,
extract_wheel_metadata,
extract_tarball_metadata,
extract_jar_metadata,
parse_deb_control,
)
class TestDebMetadata:
"""Tests for Debian package metadata extraction."""
def test_parse_deb_control_basic(self):
"""Test parsing a basic control file."""
control = """Package: my-package
Version: 1.2.3
Architecture: amd64
Maintainer: Test <test@example.com>
Description: A test package
"""
result = parse_deb_control(control)
assert result["package_name"] == "my-package"
assert result["version"] == "1.2.3"
assert result["architecture"] == "amd64"
assert result["format"] == "deb"
def test_parse_deb_control_with_epoch(self):
"""Test parsing version with epoch."""
control = """Package: another-pkg
Version: 2:1.0.0-1
"""
result = parse_deb_control(control)
assert result["version"] == "2:1.0.0-1"
assert result["package_name"] == "another-pkg"
assert result["format"] == "deb"
def test_extract_deb_metadata_invalid_magic(self):
"""Test that invalid ar magic returns empty dict."""
file = io.BytesIO(b"not an ar archive")
result = extract_deb_metadata(file)
assert result == {}
def test_extract_deb_metadata_valid_ar_no_control(self):
"""Test ar archive without control.tar returns empty."""
# Create minimal ar archive with just debian-binary
ar_data = b"!<arch>\n"
ar_data += b"debian-binary/ 0 0 0 100644 4 `\n"
ar_data += b"2.0\n"
file = io.BytesIO(ar_data)
result = extract_deb_metadata(file)
# Should return empty since no control.tar found
assert result == {} or "version" not in result
class TestWheelMetadata:
"""Tests for Python wheel metadata extraction."""
def _create_wheel_with_metadata(self, metadata_content: str) -> io.BytesIO:
"""Helper to create a wheel file with given METADATA content."""
buf = io.BytesIO()
with zipfile.ZipFile(buf, 'w') as zf:
zf.writestr('package-1.0.0.dist-info/METADATA', metadata_content)
buf.seek(0)
return buf
def test_extract_wheel_version(self):
"""Test extracting version from wheel METADATA."""
metadata = """Metadata-Version: 2.1
Name: my-package
Version: 2.3.4
Summary: A test package
"""
file = self._create_wheel_with_metadata(metadata)
result = extract_wheel_metadata(file)
assert result.get("version") == "2.3.4"
assert result.get("package_name") == "my-package"
assert result.get("format") == "wheel"
def test_extract_wheel_no_version(self):
"""Test wheel without version field."""
metadata = """Metadata-Version: 2.1
Name: no-version-pkg
"""
file = self._create_wheel_with_metadata(metadata)
result = extract_wheel_metadata(file)
assert "version" not in result
assert result.get("package_name") == "no-version-pkg"
assert result.get("format") == "wheel"
def test_extract_wheel_invalid_zip(self):
"""Test that invalid zip returns format-only dict."""
file = io.BytesIO(b"not a zip file")
result = extract_wheel_metadata(file)
assert result == {"format": "wheel"}
def test_extract_wheel_no_metadata_file(self):
"""Test wheel without METADATA file returns format-only dict."""
buf = io.BytesIO()
with zipfile.ZipFile(buf, 'w') as zf:
zf.writestr('some_file.py', 'print("hello")')
buf.seek(0)
result = extract_wheel_metadata(buf)
assert result == {"format": "wheel"}
class TestTarballMetadata:
"""Tests for tarball metadata extraction from filename."""
def test_extract_version_from_filename_standard(self):
"""Test standard package-version.tar.gz format."""
file = io.BytesIO(b"") # Content doesn't matter for filename extraction
result = extract_tarball_metadata(file, "mypackage-1.2.3.tar.gz")
assert result.get("version") == "1.2.3"
assert result.get("package_name") == "mypackage"
assert result.get("format") == "tarball"
def test_extract_version_with_v_prefix(self):
"""Test version with v prefix."""
file = io.BytesIO(b"")
result = extract_tarball_metadata(file, "package-v2.0.0.tar.gz")
assert result.get("version") == "2.0.0"
assert result.get("package_name") == "package"
assert result.get("format") == "tarball"
def test_extract_version_underscore_separator(self):
"""Test package_version format."""
file = io.BytesIO(b"")
result = extract_tarball_metadata(file, "my_package_3.1.4.tar.gz")
assert result.get("version") == "3.1.4"
assert result.get("package_name") == "my_package"
assert result.get("format") == "tarball"
def test_extract_version_complex(self):
"""Test complex version string."""
file = io.BytesIO(b"")
result = extract_tarball_metadata(file, "package-1.0.0-beta.1.tar.gz")
# The regex handles versions with suffix like -beta_1
assert result.get("format") == "tarball"
# May or may not extract version depending on regex match
if "version" in result:
assert result.get("package_name") == "package"
def test_extract_no_version_in_filename(self):
"""Test filename without version returns format-only dict."""
file = io.BytesIO(b"")
result = extract_tarball_metadata(file, "package.tar.gz")
# Should return format but no version
assert result.get("version") is None
assert result.get("format") == "tarball"
class TestJarMetadata:
"""Tests for JAR/Java metadata extraction."""
def _create_jar_with_manifest(self, manifest_content: str) -> io.BytesIO:
"""Helper to create a JAR file with given MANIFEST.MF content."""
buf = io.BytesIO()
with zipfile.ZipFile(buf, 'w') as zf:
zf.writestr('META-INF/MANIFEST.MF', manifest_content)
buf.seek(0)
return buf
def test_extract_jar_version_from_manifest(self):
"""Test extracting version from MANIFEST.MF."""
manifest = """Manifest-Version: 1.0
Implementation-Title: my-library
Implementation-Version: 4.5.6
"""
file = self._create_jar_with_manifest(manifest)
result = extract_jar_metadata(file)
assert result.get("version") == "4.5.6"
assert result.get("package_name") == "my-library"
assert result.get("format") == "jar"
def test_extract_jar_bundle_version(self):
"""Test extracting OSGi Bundle-Version."""
manifest = """Manifest-Version: 1.0
Bundle-Version: 2.1.0
Bundle-Name: Test Bundle
"""
file = self._create_jar_with_manifest(manifest)
result = extract_jar_metadata(file)
# Bundle-Version is stored in bundle_version, not version
assert result.get("bundle_version") == "2.1.0"
assert result.get("bundle_name") == "Test Bundle"
assert result.get("format") == "jar"
def test_extract_jar_invalid_zip(self):
"""Test that invalid JAR returns format-only dict."""
file = io.BytesIO(b"not a jar file")
result = extract_jar_metadata(file)
assert result == {"format": "jar"}
class TestExtractMetadataDispatch:
"""Tests for the main extract_metadata dispatcher function."""
def test_dispatch_to_wheel(self):
"""Test that .whl files use wheel extractor."""
buf = io.BytesIO()
with zipfile.ZipFile(buf, 'w') as zf:
zf.writestr('pkg-1.0.dist-info/METADATA', 'Version: 1.0.0\nName: pkg')
buf.seek(0)
result = extract_metadata(buf, "package-1.0.0-py3-none-any.whl")
assert result.get("version") == "1.0.0"
assert result.get("package_name") == "pkg"
assert result.get("format") == "wheel"
def test_dispatch_to_tarball(self):
"""Test that .tar.gz files use tarball extractor."""
file = io.BytesIO(b"")
result = extract_metadata(file, "mypackage-2.3.4.tar.gz")
assert result.get("version") == "2.3.4"
assert result.get("package_name") == "mypackage"
assert result.get("format") == "tarball"
def test_dispatch_unknown_extension(self):
"""Test that unknown extensions return empty dict."""
file = io.BytesIO(b"some content")
result = extract_metadata(file, "unknown.xyz")
assert result == {}
def test_file_position_reset_after_extraction(self):
"""Test that file position is reset to start after extraction."""
buf = io.BytesIO()
with zipfile.ZipFile(buf, 'w') as zf:
zf.writestr('pkg-1.0.dist-info/METADATA', 'Version: 1.0.0\nName: pkg')
buf.seek(0)
extract_metadata(buf, "package.whl")
# File should be back at position 0
assert buf.tell() == 0

View File

@@ -0,0 +1,85 @@
"""Unit tests for PyPI proxy functionality."""
import pytest
from app.pypi_proxy import _parse_requires_dist
class TestParseRequiresDist:
"""Tests for _parse_requires_dist function."""
def test_simple_package(self):
"""Test parsing a simple package name."""
name, version = _parse_requires_dist("numpy")
assert name == "numpy"
assert version is None
def test_package_with_version(self):
"""Test parsing package with version constraint."""
name, version = _parse_requires_dist("numpy>=1.21.0")
assert name == "numpy"
assert version == ">=1.21.0"
def test_package_with_parenthesized_version(self):
"""Test parsing package with parenthesized version."""
name, version = _parse_requires_dist("requests (>=2.25.0)")
assert name == "requests"
assert version == ">=2.25.0"
def test_package_with_python_version_marker(self):
"""Test that python_version markers are preserved but marker stripped."""
name, version = _parse_requires_dist("typing-extensions; python_version < '3.8'")
assert name == "typing-extensions"
assert version is None
def test_filters_extra_dependencies(self):
"""Test that extra dependencies are filtered out."""
# Extra dependencies should return (None, None)
name, version = _parse_requires_dist("pytest; extra == 'test'")
assert name is None
assert version is None
name, version = _parse_requires_dist("sphinx; extra == 'docs'")
assert name is None
assert version is None
def test_filters_platform_specific_darwin(self):
"""Test that macOS-specific dependencies are filtered out."""
name, version = _parse_requires_dist("pyobjc; sys_platform == 'darwin'")
assert name is None
assert version is None
def test_filters_platform_specific_win32(self):
"""Test that Windows-specific dependencies are filtered out."""
name, version = _parse_requires_dist("pywin32; sys_platform == 'win32'")
assert name is None
assert version is None
def test_filters_platform_system_marker(self):
"""Test that platform_system markers are filtered out."""
name, version = _parse_requires_dist("jaraco-windows; platform_system == 'Windows'")
assert name is None
assert version is None
def test_normalizes_package_name(self):
"""Test that package names are normalized (PEP 503)."""
name, version = _parse_requires_dist("Typing_Extensions>=3.7.4")
assert name == "typing-extensions"
assert version == ">=3.7.4"
def test_complex_version_constraint(self):
"""Test parsing complex version constraints."""
name, version = _parse_requires_dist("gast!=0.5.0,!=0.5.1,!=0.5.2,>=0.2.1")
assert name == "gast"
assert version == "!=0.5.0,!=0.5.1,!=0.5.2,>=0.2.1"
def test_version_range(self):
"""Test parsing version range constraints."""
name, version = _parse_requires_dist("grpcio<2.0,>=1.24.3")
assert name == "grpcio"
assert version == "<2.0,>=1.24.3"
def test_tilde_version(self):
"""Test parsing tilde version constraints."""
name, version = _parse_requires_dist("tensorboard~=2.20.0")
assert name == "tensorboard"
assert version == "~=2.20.0"

View File

@@ -0,0 +1,65 @@
"""Unit tests for rate limiting configuration."""
import os
import pytest
class TestRateLimitConfiguration:
"""Tests for rate limit configuration."""
def test_default_login_rate_limit(self):
"""Test default login rate limit is 5/minute."""
# Import fresh to get default value
import importlib
import app.rate_limit as rate_limit_module
# Save original env value
original = os.environ.get("ORCHARD_LOGIN_RATE_LIMIT")
try:
# Clear env variable to test default
if "ORCHARD_LOGIN_RATE_LIMIT" in os.environ:
del os.environ["ORCHARD_LOGIN_RATE_LIMIT"]
# Reload module to pick up new env
importlib.reload(rate_limit_module)
assert rate_limit_module.LOGIN_RATE_LIMIT == "5/minute"
finally:
# Restore original env value
if original is not None:
os.environ["ORCHARD_LOGIN_RATE_LIMIT"] = original
importlib.reload(rate_limit_module)
def test_custom_login_rate_limit(self):
"""Test custom login rate limit from environment."""
import importlib
import app.rate_limit as rate_limit_module
# Save original env value
original = os.environ.get("ORCHARD_LOGIN_RATE_LIMIT")
try:
# Set custom rate limit
os.environ["ORCHARD_LOGIN_RATE_LIMIT"] = "10/minute"
# Reload module to pick up new env
importlib.reload(rate_limit_module)
assert rate_limit_module.LOGIN_RATE_LIMIT == "10/minute"
finally:
# Restore original env value
if original is not None:
os.environ["ORCHARD_LOGIN_RATE_LIMIT"] = original
else:
if "ORCHARD_LOGIN_RATE_LIMIT" in os.environ:
del os.environ["ORCHARD_LOGIN_RATE_LIMIT"]
importlib.reload(rate_limit_module)
def test_limiter_exists(self):
"""Test that limiter object is created."""
from app.rate_limit import limiter
assert limiter is not None
# Limiter should have a key_func set
assert limiter._key_func is not None

View File

@@ -0,0 +1,300 @@
"""Unit tests for registry client functionality."""
import pytest
from unittest.mock import AsyncMock, MagicMock, patch
import httpx
from packaging.specifiers import SpecifierSet
from app.registry_client import (
PyPIRegistryClient,
VersionInfo,
FetchResult,
get_registry_client,
)
class TestPyPIRegistryClient:
"""Tests for PyPI registry client."""
@pytest.fixture
def mock_http_client(self):
"""Create a mock async HTTP client."""
return AsyncMock(spec=httpx.AsyncClient)
@pytest.fixture
def client(self, mock_http_client):
"""Create a PyPI registry client with mocked HTTP."""
return PyPIRegistryClient(
http_client=mock_http_client,
upstream_sources=[],
pypi_api_url="https://pypi.org/pypi",
)
def test_source_type(self, client):
"""Test source_type returns 'pypi'."""
assert client.source_type == "pypi"
def test_normalize_package_name(self, client):
"""Test package name normalization per PEP 503."""
assert client._normalize_package_name("My_Package") == "my-package"
assert client._normalize_package_name("my.package") == "my-package"
assert client._normalize_package_name("my-package") == "my-package"
assert client._normalize_package_name("MY-PACKAGE") == "my-package"
assert client._normalize_package_name("my__package") == "my-package"
assert client._normalize_package_name("my..package") == "my-package"
@pytest.mark.asyncio
async def test_get_available_versions_success(self, client, mock_http_client):
"""Test fetching available versions from PyPI."""
mock_response = MagicMock()
mock_response.status_code = 200
mock_response.json.return_value = {
"releases": {
"1.0.0": [{"packagetype": "bdist_wheel"}],
"1.1.0": [{"packagetype": "bdist_wheel"}],
"2.0.0": [{"packagetype": "bdist_wheel"}],
}
}
mock_http_client.get.return_value = mock_response
versions = await client.get_available_versions("test-package")
assert "1.0.0" in versions
assert "1.1.0" in versions
assert "2.0.0" in versions
mock_http_client.get.assert_called_once()
@pytest.mark.asyncio
async def test_get_available_versions_empty(self, client, mock_http_client):
"""Test handling package with no releases."""
mock_response = MagicMock()
mock_response.status_code = 200
mock_response.json.return_value = {"releases": {}}
mock_http_client.get.return_value = mock_response
versions = await client.get_available_versions("empty-package")
assert versions == []
@pytest.mark.asyncio
async def test_get_available_versions_404(self, client, mock_http_client):
"""Test handling non-existent package."""
mock_response = MagicMock()
mock_response.status_code = 404
mock_http_client.get.return_value = mock_response
versions = await client.get_available_versions("nonexistent")
assert versions == []
@pytest.mark.asyncio
async def test_resolve_constraint_wildcard(self, client, mock_http_client):
"""Test resolving wildcard constraint returns latest."""
mock_response = MagicMock()
mock_response.status_code = 200
mock_response.json.return_value = {
"info": {"version": "2.0.0"},
"releases": {
"1.0.0": [
{
"packagetype": "bdist_wheel",
"url": "https://files.pythonhosted.org/test-1.0.0.whl",
"filename": "test-1.0.0.whl",
"digests": {"sha256": "abc123"},
"size": 1000,
}
],
"2.0.0": [
{
"packagetype": "bdist_wheel",
"url": "https://files.pythonhosted.org/test-2.0.0.whl",
"filename": "test-2.0.0.whl",
"digests": {"sha256": "def456"},
"size": 2000,
}
],
},
}
mock_http_client.get.return_value = mock_response
result = await client.resolve_constraint("test-package", "*")
assert result is not None
assert result.version == "2.0.0"
@pytest.mark.asyncio
async def test_resolve_constraint_specific_version(self, client, mock_http_client):
"""Test resolving specific version constraint."""
mock_response = MagicMock()
mock_response.status_code = 200
mock_response.json.return_value = {
"releases": {
"1.0.0": [
{
"packagetype": "bdist_wheel",
"url": "https://files.pythonhosted.org/test-1.0.0.whl",
"filename": "test-1.0.0.whl",
"digests": {"sha256": "abc123"},
"size": 1000,
}
],
"2.0.0": [
{
"packagetype": "bdist_wheel",
"url": "https://files.pythonhosted.org/test-2.0.0.whl",
"filename": "test-2.0.0.whl",
}
],
},
}
mock_http_client.get.return_value = mock_response
result = await client.resolve_constraint("test-package", ">=1.0.0,<2.0.0")
assert result is not None
assert result.version == "1.0.0"
@pytest.mark.asyncio
async def test_resolve_constraint_no_match(self, client, mock_http_client):
"""Test resolving constraint with no matching version."""
mock_response = MagicMock()
mock_response.status_code = 200
mock_response.json.return_value = {
"releases": {
"1.0.0": [
{
"packagetype": "bdist_wheel",
"url": "https://files.pythonhosted.org/test-1.0.0.whl",
"filename": "test-1.0.0.whl",
}
],
},
}
mock_http_client.get.return_value = mock_response
result = await client.resolve_constraint("test-package", ">=5.0.0")
assert result is None
@pytest.mark.asyncio
async def test_resolve_constraint_bare_version(self, client, mock_http_client):
"""Test resolving bare version string as exact match."""
mock_response = MagicMock()
mock_response.status_code = 200
mock_response.json.return_value = {
"info": {"version": "2.0.0"},
"releases": {
"1.0.0": [
{
"packagetype": "bdist_wheel",
"url": "https://files.pythonhosted.org/test-1.0.0.whl",
"filename": "test-1.0.0.whl",
"digests": {"sha256": "abc123"},
"size": 1000,
}
],
"2.0.0": [
{
"packagetype": "bdist_wheel",
"url": "https://files.pythonhosted.org/test-2.0.0.whl",
"filename": "test-2.0.0.whl",
"digests": {"sha256": "def456"},
"size": 2000,
}
],
},
}
mock_http_client.get.return_value = mock_response
# Bare version "1.0.0" should resolve to exactly 1.0.0, not latest
result = await client.resolve_constraint("test-package", "1.0.0")
assert result is not None
assert result.version == "1.0.0"
class TestVersionInfo:
"""Tests for VersionInfo dataclass."""
def test_create_version_info(self):
"""Test creating VersionInfo with all fields."""
info = VersionInfo(
version="1.0.0",
download_url="https://example.com/pkg-1.0.0.whl",
filename="pkg-1.0.0.whl",
sha256="abc123",
size=5000,
content_type="application/zip",
)
assert info.version == "1.0.0"
assert info.download_url == "https://example.com/pkg-1.0.0.whl"
assert info.filename == "pkg-1.0.0.whl"
assert info.sha256 == "abc123"
assert info.size == 5000
def test_create_version_info_minimal(self):
"""Test creating VersionInfo with only required fields."""
info = VersionInfo(
version="1.0.0",
download_url="https://example.com/pkg.whl",
filename="pkg.whl",
)
assert info.sha256 is None
assert info.size is None
class TestFetchResult:
"""Tests for FetchResult dataclass."""
def test_create_fetch_result(self):
"""Test creating FetchResult."""
result = FetchResult(
artifact_id="abc123def456",
size=10000,
version="2.0.0",
filename="pkg-2.0.0.whl",
already_cached=True,
)
assert result.artifact_id == "abc123def456"
assert result.size == 10000
assert result.version == "2.0.0"
assert result.already_cached is True
def test_fetch_result_default_not_cached(self):
"""Test FetchResult defaults to not cached."""
result = FetchResult(
artifact_id="xyz",
size=100,
version="1.0.0",
filename="pkg.whl",
)
assert result.already_cached is False
class TestGetRegistryClient:
"""Tests for registry client factory function."""
def test_get_pypi_client(self):
"""Test getting PyPI client."""
mock_client = MagicMock()
mock_sources = []
client = get_registry_client("pypi", mock_client, mock_sources)
assert isinstance(client, PyPIRegistryClient)
def test_get_unsupported_client(self):
"""Test getting unsupported registry type returns None."""
mock_client = MagicMock()
client = get_registry_client("npm", mock_client, [])
assert client is None
def test_get_unknown_client(self):
"""Test getting unknown registry type returns None."""
mock_client = MagicMock()
client = get_registry_client("unknown", mock_client, [])
assert client is None

View File

@@ -227,6 +227,21 @@
line-height: 1.5;
}
.graph-warning {
display: flex;
align-items: center;
gap: 8px;
padding: 8px 16px;
background: rgba(245, 158, 11, 0.1);
border-top: 1px solid rgba(245, 158, 11, 0.3);
color: var(--warning-color, #f59e0b);
font-size: 0.875rem;
}
.graph-warning svg {
flex-shrink: 0;
}
/* Missing Dependencies */
.missing-dependencies {
border-top: 1px solid var(--border-primary);

View File

@@ -106,6 +106,7 @@ function DependencyGraph({ projectName, packageName, tagName, onClose }: Depende
const [loading, setLoading] = useState(true);
const [error, setError] = useState<string | null>(null);
const [warning, setWarning] = useState<string | null>(null);
const [resolution, setResolution] = useState<DependencyResolutionResponse | null>(null);
const [nodes, setNodes, onNodesChange] = useNodesState<NodeData>([]);
const [edges, setEdges, onEdgesChange] = useEdgesState([]);
@@ -127,16 +128,24 @@ function DependencyGraph({ projectName, packageName, tagName, onClose }: Depende
// Fetch dependencies for each artifact
const depsMap = new Map<string, Dependency[]>();
const failedFetches: string[] = [];
for (const artifact of resolutionData.resolved) {
try {
const deps = await getArtifactDependencies(artifact.artifact_id);
depsMap.set(artifact.artifact_id, deps.dependencies);
} catch {
} catch (err) {
console.warn(`Failed to fetch dependencies for ${artifact.package}:`, err);
failedFetches.push(artifact.package);
depsMap.set(artifact.artifact_id, []);
}
}
// Report warning if some fetches failed
if (failedFetches.length > 0) {
setWarning(`Could not load dependency details for: ${failedFetches.slice(0, 3).join(', ')}${failedFetches.length > 3 ? ` and ${failedFetches.length - 3} more` : ''}`);
}
// Find the root artifact
const rootArtifact = resolutionData.resolved.find(
a => a.project === resolutionData.requested.project &&
@@ -324,6 +333,17 @@ function DependencyGraph({ projectName, packageName, tagName, onClose }: Depende
)}
</div>
{warning && (
<div className="graph-warning">
<svg width="16" height="16" viewBox="0 0 24 24" fill="none" stroke="currentColor" strokeWidth="2">
<path d="M10.29 3.86L1.82 18a2 2 0 0 0 1.71 3h16.94a2 2 0 0 0 1.71-3L13.71 3.86a2 2 0 0 0-3.42 0z"></path>
<line x1="12" y1="9" x2="12" y2="13"></line>
<line x1="12" y1="17" x2="12.01" y2="17"></line>
</svg>
<span>{warning}</span>
</div>
)}
{resolution && resolution.missing && resolution.missing.length > 0 && (
<div className="missing-dependencies">
<h3>Not Cached ({resolution.missing.length})</h3>

View File

@@ -185,56 +185,6 @@ h2 {
color: var(--warning-color, #f59e0b);
}
/* Usage Section */
.usage-section {
margin-top: 32px;
background: var(--bg-secondary);
}
.usage-section h3 {
margin-bottom: 12px;
color: var(--text-primary);
font-size: 1rem;
font-weight: 600;
}
.usage-section p {
color: var(--text-secondary);
margin-bottom: 12px;
font-size: 0.875rem;
}
.usage-section pre {
background: #0d0d0f;
border: 1px solid var(--border-primary);
padding: 16px 20px;
border-radius: var(--radius-md);
overflow-x: auto;
margin-bottom: 16px;
}
.usage-section code {
font-family: 'JetBrains Mono', 'Fira Code', 'Consolas', monospace;
font-size: 0.8125rem;
color: #e2e8f0;
}
/* Syntax highlighting for code blocks */
.usage-section pre {
position: relative;
}
.usage-section pre::before {
content: 'bash';
position: absolute;
top: 8px;
right: 12px;
font-size: 0.6875rem;
color: var(--text-muted);
text-transform: uppercase;
letter-spacing: 0.05em;
}
/* Copy button for code blocks (optional enhancement) */
.code-block {
position: relative;

View File

@@ -78,7 +78,7 @@ function PackagePage() {
// Reverse dependencies state
const [reverseDeps, setReverseDeps] = useState<DependentInfo[]>([]);
const [reverseDepsLoading, setReverseDepsLoading] = useState(false);
const [_reverseDepsError, setReverseDepsError] = useState<string | null>(null);
const [reverseDepsError, setReverseDepsError] = useState<string | null>(null);
const [reverseDepsPage, setReverseDepsPage] = useState(1);
const [reverseDepsTotal, setReverseDepsTotal] = useState(0);
const [reverseDepsHasMore, setReverseDepsHasMore] = useState(false);
@@ -647,10 +647,13 @@ function PackagePage() {
/>
)}
{/* Used By (Reverse Dependencies) Section - only show if there are reverse deps */}
{reverseDeps.length > 0 && (
{/* Used By (Reverse Dependencies) Section - only show if there are reverse deps or error */}
{(reverseDeps.length > 0 || reverseDepsError) && (
<div className="used-by-section card">
<h3>Used By</h3>
{reverseDepsError && (
<div className="error-message">{reverseDepsError}</div>
)}
<div className="reverse-deps-list">
<div className="deps-summary">
{reverseDepsTotal} {reverseDepsTotal === 1 ? 'package depends' : 'packages depend'} on this:
@@ -696,18 +699,6 @@ function PackagePage() {
</div>
)}
<div className="usage-section card">
<h3>Usage</h3>
<p>Download artifacts using:</p>
<pre>
<code>curl -O {window.location.origin}/api/v1/project/{projectName}/{packageName}/+/latest</code>
</pre>
<p>Or with a specific version:</p>
<pre>
<code>curl -O {window.location.origin}/api/v1/project/{projectName}/{packageName}/+/1.0.0</code>
</pre>
</div>
{/* Dependency Graph Modal */}
{showGraph && selectedArtifact && (
<DependencyGraph