Add upstream caching infrastructure and refactor CI pipeline
Upstream Caching (Epic #68-#75, #105): - Add upstream_sources and cache_settings tables with migrations - Add cache management API endpoints (CRUD for sources, settings) - Add environment variable overrides for upstream sources and cache settings - Add encryption module for storing credentials securely - Add frontend Admin Cache Management page - Add is_system field to projects for system cache distinction - Add purge_seed_data for transitioning to production-like environments CI Pipeline Refactoring: - Remove reset jobs (reset_stage_pre, reset_stage) - Add ephemeral orchard-test deployment for main branch testing - Run integration tests on ephemeral deployment before promoting to stage - Stage is now long-running pre-prod (smoke tests only) - Disable prosper_setup for tag pipelines
This commit is contained in:
70
CHANGELOG.md
70
CHANGELOG.md
@@ -7,6 +7,76 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
||||
|
||||
## [Unreleased]
|
||||
### Added
|
||||
- Added frontend system projects visual distinction (#105)
|
||||
- "Cache" badge for system projects in project list
|
||||
- "System Cache" badge on project detail page
|
||||
- Added `is_system` field to Project type
|
||||
- Added frontend admin page for upstream sources and cache settings (#75)
|
||||
- New `/admin/cache` page accessible from user menu (admin only)
|
||||
- Upstream sources table with create/edit/delete/test connectivity
|
||||
- Cache settings section with air-gap mode and auto-create system projects toggles
|
||||
- Visual indicators for env-defined sources (locked, cannot be modified)
|
||||
- Environment variable override badges when settings are overridden
|
||||
- API client functions for all cache admin operations
|
||||
- Added environment variable overrides for cache configuration (#74)
|
||||
- `ORCHARD_CACHE_ALLOW_PUBLIC_INTERNET` - Override allow_public_internet (air-gap mode)
|
||||
- `ORCHARD_CACHE_AUTO_CREATE_SYSTEM_PROJECTS` - Override auto_create_system_projects
|
||||
- `ORCHARD_UPSTREAM__{NAME}__*` - Define upstream sources via env vars
|
||||
- Env-defined sources appear in API with `source: "env"` marker
|
||||
- Env-defined sources cannot be modified/deleted via API (400 error)
|
||||
- Cache settings response includes `*_env_override` fields when overridden
|
||||
- 7 unit tests for env var parsing and configuration
|
||||
- Added Global Cache Settings Admin API (#73)
|
||||
- `GET /api/v1/admin/cache-settings` - Retrieve current cache settings
|
||||
- `PUT /api/v1/admin/cache-settings` - Update cache settings (partial updates)
|
||||
- Admin-only access with audit logging
|
||||
- Controls `allow_public_internet` (air-gap mode) and `auto_create_system_projects`
|
||||
- 7 integration tests for settings management
|
||||
- Added Upstream Sources Admin API for managing cache sources (#72)
|
||||
- `GET /api/v1/admin/upstream-sources` - List sources with filtering
|
||||
- `POST /api/v1/admin/upstream-sources` - Create source with auth configuration
|
||||
- `GET /api/v1/admin/upstream-sources/{id}` - Get source details
|
||||
- `PUT /api/v1/admin/upstream-sources/{id}` - Update source (partial updates)
|
||||
- `DELETE /api/v1/admin/upstream-sources/{id}` - Delete source
|
||||
- `POST /api/v1/admin/upstream-sources/{id}/test` - Test connectivity
|
||||
- Admin-only access with audit logging
|
||||
- Credentials never exposed (only has_password/has_headers flags)
|
||||
- 13 integration tests for all CRUD operations
|
||||
- Added system project restrictions and management (#71)
|
||||
- System projects (`_npm`, `_pypi`, etc.) cannot be deleted (returns 403)
|
||||
- System projects cannot be made private (must remain public)
|
||||
- `GET /api/v1/system-projects` endpoint to list all system cache projects
|
||||
- 5 integration tests for system project restrictions
|
||||
- Added Cache API endpoint for fetching and storing artifacts from upstream URLs (#70)
|
||||
- `POST /api/v1/cache` endpoint to cache artifacts from upstream registries
|
||||
- URL parsing helpers to extract package name/version from npm, PyPI, Maven URLs
|
||||
- Automatic system project creation (`_npm`, `_pypi`, `_maven`, etc.)
|
||||
- URL-to-artifact provenance tracking via `cached_urls` table
|
||||
- Optional user project cross-referencing for custom organization
|
||||
- Cache hit returns existing artifact without re-fetching
|
||||
- Air-gap mode enforcement (blocks public URLs when disabled)
|
||||
- Hash verification for downloaded artifacts
|
||||
- 21 unit tests for URL parsing and cache endpoint
|
||||
- Added HTTP client for fetching artifacts from upstream sources (#69)
|
||||
- `UpstreamClient` class in `backend/app/upstream.py` with streaming downloads
|
||||
- SHA256 hash computation while streaming (doesn't load large files into memory)
|
||||
- Auth support: none, basic auth, bearer token, API key (custom headers)
|
||||
- URL-to-source matching by URL prefix with priority ordering
|
||||
- Configuration options: timeouts, retries with exponential backoff, redirect limits, max file size
|
||||
- Air-gap mode enforcement via `allow_public_internet` setting
|
||||
- Response header capture for provenance tracking
|
||||
- Proper error handling with custom exception types
|
||||
- Connection test method for upstream source validation
|
||||
- 33 unit tests for client functionality
|
||||
- Added upstream artifact caching schema for hermetic builds (#68)
|
||||
- `upstream_sources` table for configuring upstream registries (npm, PyPI, Maven, etc.)
|
||||
- `cache_settings` table for global settings including air-gap mode
|
||||
- `cached_urls` table for URL-to-artifact provenance tracking
|
||||
- `is_system` column on projects for system cache projects (_npm, _pypi, etc.)
|
||||
- Support for multiple auth types: none, basic auth, bearer token, API key
|
||||
- Fernet encryption for credentials using `ORCHARD_CACHE_ENCRYPTION_KEY`
|
||||
- Default upstream sources seeded (npm-public, pypi-public, maven-central, docker-hub) - disabled by default
|
||||
- Migration `010_upstream_caching.sql`
|
||||
- Added team-based multi-tenancy for organizing projects and collaboration (#88-#104)
|
||||
- Teams serve as organizational containers for projects
|
||||
- Users can belong to multiple teams with different roles (owner, admin, member)
|
||||
|
||||
Reference in New Issue
Block a user