Real package managers (pip, npm, Maven) don't have depth limits - they
resolve the full dependency tree. We have other safeguards:
- Loop prevention via fetch_attempted set
- Timeout via auto_fetch_timeout setting
- Dependency trees are finite
Instead of failing with 409 on version conflicts, use "first version wins"
strategy. This allows resolution to succeed for complex dependency trees
like tensorflow where transitive dependencies may have overlapping but
not identical version requirements.
The resolver now:
- Checks if an already-resolved version satisfies a new constraint
- If yes, reuses the existing version
- If no, logs the mismatch and uses the first-encountered version
This matches pip's behavior of picking a working version rather than
failing on theoretical conflicts.
When packages like pytest have extras (e.g., pytest[testing]) that depend
on the base package, the resolution was incorrectly detecting this as a
circular dependency.
Added additional check to skip dependencies that resolve to an artifact
already in the visiting set, preventing the false cycle detection while
still catching real circular dependencies.
Auto-fetching missing dependencies from upstream is the more useful default
behavior. Users who need fast, network-free resolution can explicitly set
auto_fetch=false.
Artifacts are content-addressed by SHA256, so reproducibility concerns don't
apply - the same version always produces the same artifact.
When auto_fetch=true and the root artifact doesn't exist locally in a
system project (_pypi), now attempts to fetch it from upstream before
starting dependency resolution. Also fixed a bug where fetched_artifacts
was being redeclared, which would lose the root artifact from the list.
- Remove unused _get_pypi_upstream_sources_cached function (never called)
- Remove unused CacheService import and get_cache helper
- Remove unused cache parameter from pypi_download_file
- Fix asyncio.get_event_loop() deprecation - use get_running_loop()
- Note: The caching implementation was incomplete but the other
performance improvements (connection pooling, batch DB ops) remain
- Fix import in test_db_utils.py: use app.models instead of backend.app.models
- Update health endpoint test to expect 'ok' status and infrastructure keys
- Add CHANGELOG entries for PyPI proxy performance improvements
Add dependency injection helper functions for HttpClientManager
and CacheService, along with imports for the new infrastructure
modules (http_client, cache_service, db_utils).
Add optimized database operations for artifact storage:
- Atomic upserts using ON CONFLICT for artifact creation
- Batch inserts for dependencies to eliminate N+1 queries
- Joined queries for cached URL lookups
- All methods include comprehensive unit tests
Add HttpClientManager class for managing httpx.AsyncClient pools with
FastAPI lifespan integration. Features include:
- Default shared connection pool for general requests
- Configurable max connections, keep-alive, and timeouts
- Dedicated thread pool for blocking I/O operations
- Graceful startup/shutdown lifecycle management
- Per-upstream client isolation support (for future use)
Includes comprehensive unit tests covering initialization, startup,
shutdown, client retrieval, blocking operations, idempotency, and
error handling.
Comprehensive design for:
- HTTP connection pooling with lifecycle management
- Redis caching layer (TTL for discovery, permanent for immutable)
- Abstract PackageProxyBase for multi-protocol support (npm, Maven)
- Database query optimization with batch operations
- Dependency resolution caching for ensure files
- Observability via health endpoints
Maintains hermetic build guarantees: artifact content and extracted
metadata are immutable, only discovery data uses TTL-based caching.
Wheel METADATA files can list the same dependency multiple times under
different extras (e.g., bokeh appears under [docs] and [bokeh-tests]).
This caused unique constraint violations when storing dependencies.
Fix by deduplicating extracted deps before DB insertion.
Adds the tag removal migration to the inline migrations in database.py:
- Drops tag-related triggers and functions
- Removes tag_constraint column from artifact_dependencies
- Makes version_constraint NOT NULL
- Drops tags and tag_history tables
- Renames uploads.tag_name to version
Re-adds the dependency extraction that was accidentally removed with the
proactive caching feature. Now when a PyPI package is cached:
1. Extract METADATA from wheel or PKG-INFO from sdist
2. Parse Requires-Dist lines for dependencies
3. Store in artifact_dependencies table
This restores the dependency graph functionality for PyPI packages.
Adds ORCHARD_PYPI_DOWNLOAD_MODE setting (default: "redirect"):
- "redirect": Redirect pip to S3 presigned URL - reduces pod bandwidth
- "proxy": Stream through Orchard pod - for environments where clients can't reach S3
In redirect mode, Orchard only handles metadata requests and upstream fetches.
All file transfers go directly from S3 to the client.
The list_package_artifacts endpoint was only querying artifacts via the
Upload table. PyPI proxy creates PackageVersion records but not Upload
records, so cached packages would show stats (size, version count) but
no artifacts in the listing.
Now queries artifacts from both Upload and PackageVersion tables using
a union, so PyPI-cached packages display their artifacts correctly.
Large packages like TensorFlow (~600MB) caused read timeouts because the
entire file was loaded into memory before responding to the client. Now
the file is stored to S3 first, then streamed back using StreamingResponse.
- Add sha256 field to list_package_artifacts response (artifact ID is SHA256)
- Add version field to PackageArtifactResponse schema
- Add version field to frontend PackageArtifact type
- Update getArtifactVersion to prefer direct version field
- Fix upload response to return actual version (not requested version)
when artifact already has a version in the package
- Update ref_count tests to use multiple packages (one version per
artifact per package design constraint)
- Remove allow_public_internet references from upstream caching tests
- Update consistency check test to not assert global system health
- Add versions field to artifact schemas
- Fix dependencies resolution to handle removed tag constraint
- Update CacheRequest test to use version field
- Fix upload_test_file calls that still used tag parameter
- Update artifact history test to check versions instead of tags
- Update artifact stats tests to check versions instead of tags
- Fix garbage collection tests to delete versions instead of tags
- Remove TestGlobalTags class (endpoint removed)
- Update project/package stats tests to check version_count
- Fix upload_test_file fixture in test_download_verification
- Remove Tag/TagHistory model tests from unit tests
- Update CacheSettings tests to remove allow_public_internet field
- Replace tag= with version= in upload_test_file calls
- Update test assertions to use versions instead of tags
- Remove tests for tag: prefix downloads (now uses version:)
- Update dependency tests for version-only schema
Tags were mutable aliases that caused confusion alongside the immutable
version system. This removes tags entirely, keeping only PackageVersion
for artifact references.
Changes:
- Remove tags and tag_history tables (migration 012)
- Remove Tag model, TagRepository, and 6 tag API endpoints
- Update cache system to create versions instead of tags
- Update frontend to display versions instead of tags
- Remove tag-related schemas and types
- Update artifact cleanup service for version-based ref_count
The artifacts endpoint only supports sorting by: created_at, size, original_name
But the frontend was defaulting to 'name' (from the old tags endpoint).
- Change default sort from 'name' to 'created_at'
- Change default order from 'asc' to 'desc' (newest first)
- Remove sortable flag from version/tags columns (not DB fields)
- Add sortable flag to original_name and size columns
The circular dependency error '_pypi/psutil → _pypi/psutil' occurred because
dependencies with extras like 'psutil[test]' weren't being recognized as
self-dependencies. The comparison 'psutil[test] != psutil' failed.
- Add _normalize_pypi_package_name() helper that strips extras brackets
and normalizes separators per PEP 503
- Update _detect_package_cycle to use normalized names for cycle detection
- Update check_circular_dependencies to use normalized initial path
- Simplify self-dependency check in resolve_dependencies to use helper
- Add artifact: prefix handling in resolve_dependencies for direct artifact
ID references, enabling dependency resolution for tagless artifacts
- Refactor PackagePage from tag-based to artifact-based data display
- Add PackageArtifact type with tags array for artifact-centric API responses
- Update download URLs to use artifact:ID prefix when no tags exist
- Conditionally show "View Ensure File" only when artifact has tags
