bef16d884b
Comprehensive design for: - HTTP connection pooling with lifecycle management - Redis caching layer (TTL for discovery, permanent for immutable) - Abstract PackageProxyBase for multi-protocol support (npm, Maven) - Database query optimization with batch operations - Dependency resolution caching for ensure files - Observability via health endpoints Maintains hermetic build guarantees: artifact content and extracted metadata are immutable, only discovery data uses TTL-based caching.
229 lines
9.6 KiB
Markdown
229 lines
9.6 KiB
Markdown
# PyPI Proxy Performance & Multi-Protocol Architecture Design
|
|
|
|
**Date:** 2026-02-04
|
|
**Status:** Approved
|
|
**Branch:** fix/pypi-proxy-timeout
|
|
|
|
## Overview
|
|
|
|
Comprehensive infrastructure overhaul to address latency, throughput, and resource consumption issues in the PyPI proxy, while establishing a foundation for npm, Maven, and other package protocols.
|
|
|
|
## Goals
|
|
|
|
1. **Reduce latency** - Eliminate per-request connection overhead, cache aggressively
|
|
2. **Increase throughput** - Handle hundreds of concurrent requests without degradation
|
|
3. **Lower resource usage** - Connection pooling, efficient DB queries, proper async I/O
|
|
4. **Enable multi-protocol** - Abstract base class ready for npm/Maven/etc.
|
|
5. **Maintain hermetic builds** - Immutable artifact content and metadata, mutable discovery data
|
|
|
|
## Architecture
|
|
|
|
```
|
|
┌─────────────────────────────────────────────────────────────────────┐
|
|
│ FastAPI Application │
|
|
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │
|
|
│ │ PyPI Proxy │ │ npm Proxy │ │ Maven Proxy │ │ (future) │ │
|
|
│ │ Router │ │ Router │ │ Router │ │ │ │
|
|
│ └──────┬──────┘ └──────┬──────┘ └──────┬──────┘ └─────────────┘ │
|
|
│ │ │ │ │
|
|
│ └────────────────┼────────────────┘ │
|
|
│ ▼ │
|
|
│ ┌───────────────────────┐ │
|
|
│ │ PackageProxyBase │ ← Abstract base class │
|
|
│ │ - check_cache() │ │
|
|
│ │ - fetch_upstream() │ │
|
|
│ │ - store_artifact() │ │
|
|
│ │ - serve_artifact() │ │
|
|
│ └───────────┬───────────┘ │
|
|
│ │ │
|
|
│ ┌────────────────┼────────────────┐ │
|
|
│ ▼ ▼ ▼ │
|
|
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │
|
|
│ │ HttpClient │ │ CacheService│ │ ThreadPool │ │
|
|
│ │ Manager │ │ (Redis) │ │ Executor │ │
|
|
│ └─────────────┘ └─────────────┘ └─────────────┘ │
|
|
│ │ │ │ │
|
|
└─────────┼────────────────┼────────────────┼──────────────────────────┘
|
|
▼ ▼ ▼
|
|
┌──────────┐ ┌──────────┐ ┌──────────────┐
|
|
│ Upstream │ │ Redis │ │ S3/MinIO │
|
|
│ Sources │ │ │ │ │
|
|
└──────────┘ └──────────┘ └──────────────┘
|
|
```
|
|
|
|
## Components
|
|
|
|
### 1. HttpClientManager
|
|
|
|
Manages httpx.AsyncClient pools with FastAPI lifespan integration.
|
|
|
|
**Features:**
|
|
- Default pool for general requests
|
|
- Per-upstream pools for sources needing specific config/auth
|
|
- Graceful shutdown drains in-flight requests
|
|
- Dedicated thread pool for blocking operations
|
|
|
|
**Configuration:**
|
|
```bash
|
|
ORCHARD_HTTP_MAX_CONNECTIONS=100 # Default pool size
|
|
ORCHARD_HTTP_KEEPALIVE_CONNECTIONS=20 # Keep-alive connections
|
|
ORCHARD_HTTP_CONNECT_TIMEOUT=30 # Connection timeout (seconds)
|
|
ORCHARD_HTTP_READ_TIMEOUT=60 # Read timeout (seconds)
|
|
ORCHARD_HTTP_WORKER_THREADS=32 # Thread pool size
|
|
```
|
|
|
|
**File:** `backend/app/http_client.py`
|
|
|
|
### 2. CacheService (Redis Layer)
|
|
|
|
Redis-backed caching with category-aware TTL and invalidation.
|
|
|
|
**Cache Categories:**
|
|
|
|
| Category | TTL | Invalidation | Purpose |
|
|
|----------|-----|--------------|---------|
|
|
| ARTIFACT_METADATA | Forever | Never (immutable) | Artifact info by SHA256 |
|
|
| ARTIFACT_DEPENDENCIES | Forever | Never (immutable) | Extracted deps by SHA256 |
|
|
| DEPENDENCY_RESOLUTION | Forever | Manual/refresh param | Resolution results |
|
|
| UPSTREAM_SOURCES | 1 hour | On DB change | Upstream config |
|
|
| PACKAGE_INDEX | 5 min | TTL only | PyPI/npm index pages |
|
|
| PACKAGE_VERSIONS | 5 min | TTL only | Version listings |
|
|
|
|
**Key format:** `orchard:{category}:{protocol}:{identifier}`
|
|
|
|
**Configuration:**
|
|
```bash
|
|
ORCHARD_REDIS_HOST=redis
|
|
ORCHARD_REDIS_PORT=6379
|
|
ORCHARD_REDIS_DB=0
|
|
ORCHARD_CACHE_TTL_INDEX=300 # Package index: 5 minutes
|
|
ORCHARD_CACHE_TTL_VERSIONS=300 # Version listings: 5 minutes
|
|
ORCHARD_CACHE_TTL_UPSTREAM=3600 # Upstream config: 1 hour
|
|
```
|
|
|
|
**File:** `backend/app/cache_service.py`
|
|
|
|
### 3. PackageProxyBase
|
|
|
|
Abstract base class defining the cache→fetch→store→serve flow.
|
|
|
|
**Abstract methods (protocol-specific):**
|
|
- `get_protocol_name()` - Return 'pypi', 'npm', 'maven'
|
|
- `get_system_project_name()` - Return '_pypi', '_npm'
|
|
- `rewrite_index_html()` - Rewrite upstream index to Orchard URLs
|
|
- `extract_metadata()` - Extract deps from package file
|
|
- `parse_package_url()` - Parse URL into package/version/filename
|
|
|
|
**Concrete methods (shared):**
|
|
- `serve_index()` - Serve package index with caching
|
|
- `serve_artifact()` - Full cache→fetch→store→serve flow
|
|
|
|
**File:** `backend/app/proxy_base.py`
|
|
|
|
### 4. ArtifactRepository (DB Optimization)
|
|
|
|
Optimized database operations eliminating N+1 queries.
|
|
|
|
**Key methods:**
|
|
- `get_or_create_artifact()` - Atomic upsert via ON CONFLICT
|
|
- `batch_upsert_dependencies()` - Single INSERT for all deps
|
|
- `get_cached_url_with_artifact()` - Joined query for cache lookup
|
|
|
|
**Query reduction:**
|
|
|
|
| Operation | Before | After |
|
|
|-----------|--------|-------|
|
|
| Cache hit check | 2 queries | 1 query (joined) |
|
|
| Store artifact | 3-4 queries | 1 query (upsert) |
|
|
| Store 50 deps | 50+ queries | 1 query (batch) |
|
|
|
|
**Configuration:**
|
|
```bash
|
|
ORCHARD_DATABASE_POOL_SIZE=20 # Base connections (up from 5)
|
|
ORCHARD_DATABASE_MAX_OVERFLOW=30 # Burst capacity (up from 10)
|
|
ORCHARD_DATABASE_POOL_TIMEOUT=30 # Wait timeout
|
|
ORCHARD_DATABASE_POOL_PRE_PING=false # Disable in prod for performance
|
|
```
|
|
|
|
**File:** `backend/app/db_utils.py`
|
|
|
|
### 5. Dependency Resolution Caching
|
|
|
|
Cache resolution results for ensure files and API queries.
|
|
|
|
**Cache key:** Hash of (artifact_id, max_depth, include_optional)
|
|
|
|
**Invalidation:** Manual only (immutable artifact deps mean cached resolutions stay valid)
|
|
|
|
**Refresh:** `?refresh=true` parameter forces fresh resolution
|
|
|
|
**File:** Updates to `backend/app/dependencies.py`
|
|
|
|
### 6. FastAPI Integration
|
|
|
|
Lifespan-managed infrastructure with dependency injection.
|
|
|
|
**Startup:**
|
|
1. Initialize HttpClientManager (connection pools)
|
|
2. Initialize CacheService (Redis connection)
|
|
3. Load upstream source configs
|
|
|
|
**Shutdown:**
|
|
1. Drain in-flight HTTP requests
|
|
2. Close Redis connections
|
|
3. Shutdown thread pool
|
|
|
|
**Health endpoint additions:**
|
|
- Database connection status
|
|
- Redis ping
|
|
- HTTP pool active/max connections
|
|
- Thread pool active/max workers
|
|
|
|
**File:** Updates to `backend/app/main.py`
|
|
|
|
## Files Summary
|
|
|
|
**New files:**
|
|
- `backend/app/http_client.py` - HttpClientManager
|
|
- `backend/app/cache_service.py` - CacheService
|
|
- `backend/app/proxy_base.py` - PackageProxyBase
|
|
- `backend/app/db_utils.py` - ArtifactRepository
|
|
|
|
**Modified files:**
|
|
- `backend/app/config.py` - New settings
|
|
- `backend/app/main.py` - Lifespan integration
|
|
- `backend/app/pypi_proxy.py` - Refactor to use base class
|
|
- `backend/app/dependencies.py` - Resolution caching
|
|
- `backend/app/routes.py` - Health endpoint, DI
|
|
|
|
## Hermetic Build Guarantees
|
|
|
|
**Immutable (cached forever):**
|
|
- Artifact content (by SHA256)
|
|
- Extracted dependencies for a specific artifact
|
|
- Dependency resolution results
|
|
|
|
**Mutable (TTL + event invalidation):**
|
|
- Package index listings
|
|
- Version discovery
|
|
- Upstream source configuration
|
|
|
|
Once an artifact is cached with SHA256 `abc123` and dependencies extracted, that data never changes.
|
|
|
|
## Performance Expectations
|
|
|
|
| Metric | Before | After |
|
|
|--------|--------|-------|
|
|
| HTTP connection setup | Per request (~100-500ms) | Pooled (~5ms) |
|
|
| Cache hit (index page) | N/A | ~5ms (Redis) |
|
|
| Store 50 dependencies | ~500ms (50 queries) | ~10ms (1 query) |
|
|
| Dependency resolution (cached) | N/A | ~5ms |
|
|
| Concurrent request capacity | ~15 (DB pool) | ~50 (configurable) |
|
|
|
|
## Testing Requirements
|
|
|
|
- Unit tests for each new component
|
|
- Integration tests for full proxy flow
|
|
- Load tests to verify pool sizing
|
|
- Cache hit/miss verification tests
|