diff --git a/README.md b/README.md index 0216818..87764f0 100644 --- a/README.md +++ b/README.md @@ -28,6 +28,16 @@ Orchard is a centralized binary artifact storage system that provides content-ad - **Web UI** - React-based interface for managing artifacts - **Docker Compose Setup** - Easy local development environment - **Helm Chart** - Kubernetes deployment with PostgreSQL, MinIO, and Redis subcharts +- **Multipart Upload** - Automatic multipart upload for files larger than 100MB +- **Resumable Uploads** - API for resumable uploads with part-by-part upload support +- **Range Requests** - HTTP range request support for partial downloads +- **Format-Specific Metadata** - Automatic extraction of metadata from package formats: + - `.deb` - Debian packages (name, version, architecture, maintainer) + - `.rpm` - RPM packages (name, version, release, architecture) + - `.tar.gz/.tgz` - Tarballs (name, version from filename) + - `.whl` - Python wheels (name, version, author) + - `.jar` - Java JARs (manifest info, Maven coordinates) + - `.zip` - ZIP files (file count, uncompressed size) ### API Endpoints @@ -41,12 +51,25 @@ Orchard is a centralized binary artifact storage system that provides content-ad | `GET` | `/api/v1/project/:project/packages` | List packages in a project | | `POST` | `/api/v1/project/:project/packages` | Create a new package | | `POST` | `/api/v1/project/:project/:package/upload` | Upload an artifact | -| `GET` | `/api/v1/project/:project/:package/+/:ref` | Download an artifact | +| `GET` | `/api/v1/project/:project/:package/+/:ref` | Download an artifact (supports Range header) | +| `HEAD` | `/api/v1/project/:project/:package/+/:ref` | Get artifact metadata without downloading | | `GET` | `/api/v1/project/:project/:package/tags` | List all tags | | `POST` | `/api/v1/project/:project/:package/tags` | Create a tag | | `GET` | `/api/v1/project/:project/:package/consumers` | List consumers of a package | | `GET` | `/api/v1/artifact/:id` | Get artifact metadata by hash | +#### Resumable Upload Endpoints + +For large files, use the resumable upload API: + +| Method | Endpoint | Description | +|--------|----------|-------------| +| `POST` | `/api/v1/project/:project/:package/upload/init` | Initialize resumable upload | +| `PUT` | `/api/v1/project/:project/:package/upload/:upload_id/part/:part_number` | Upload a part | +| `POST` | `/api/v1/project/:project/:package/upload/:upload_id/complete` | Complete upload | +| `DELETE` | `/api/v1/project/:project/:package/upload/:upload_id` | Abort upload | +| `GET` | `/api/v1/project/:project/:package/upload/:upload_id/status` | Get upload status | + ### Reference Formats When downloading artifacts, the `:ref` parameter supports multiple formats: @@ -146,10 +169,43 @@ Response: "size": 1048576, "project": "my-project", "package": "releases", - "tag": "v1.0.0" + "tag": "v1.0.0", + "format_metadata": { + "format": "tarball", + "package_name": "app", + "version": "1.0.0" + }, + "deduplicated": false } ``` +### Resumable Upload (for large files) + +For files larger than 100MB, use the resumable upload API: + +```bash +# 1. Initialize upload (client must compute SHA256 hash first) +curl -X POST http://localhost:8080/api/v1/project/my-project/releases/upload/init \ + -H "Content-Type: application/json" \ + -d '{ + "expected_hash": "a3f5d8e12b4c67890abcdef1234567890abcdef1234567890abcdef12345678", + "filename": "large-file.tar.gz", + "size": 524288000, + "tag": "v2.0.0" + }' + +# Response: {"upload_id": "abc123", "already_exists": false, "chunk_size": 10485760} + +# 2. Upload parts (10MB chunks recommended) +curl -X PUT http://localhost:8080/api/v1/project/my-project/releases/upload/abc123/part/1 \ + --data-binary @chunk1.bin + +# 3. Complete the upload +curl -X POST http://localhost:8080/api/v1/project/my-project/releases/upload/abc123/complete \ + -H "Content-Type: application/json" \ + -d '{"tag": "v2.0.0"}' +``` + ### Download an Artifact ```bash @@ -161,6 +217,12 @@ curl -O http://localhost:8080/api/v1/project/my-project/releases/+/artifact:a3f5 # Using the short URL pattern curl -O http://localhost:8080/project/my-project/releases/+/latest + +# Partial download (range request) +curl -H "Range: bytes=0-1023" http://localhost:8080/api/v1/project/my-project/releases/+/v1.0.0 + +# Check file info without downloading (HEAD request) +curl -I http://localhost:8080/api/v1/project/my-project/releases/+/v1.0.0 ``` ### Create a Tag @@ -185,12 +247,13 @@ orchard/ │ ├── app/ │ │ ├── __init__.py │ │ ├── config.py # Pydantic settings -│ │ ├── database.py # SQLAlchemy setup +│ │ ├── database.py # SQLAlchemy setup and migrations │ │ ├── main.py # FastAPI application +│ │ ├── metadata.py # Format-specific metadata extraction │ │ ├── models.py # SQLAlchemy models │ │ ├── routes.py # API endpoints │ │ ├── schemas.py # Pydantic schemas -│ │ └── storage.py # S3 storage layer +│ │ └── storage.py # S3 storage layer with multipart support │ └── requirements.txt ├── frontend/ │ ├── src/ @@ -278,9 +341,8 @@ The following features are planned but not yet implemented: - [ ] Automated update propagation - [ ] OIDC/SAML authentication - [ ] API key management -- [ ] Package format detection -- [ ] Multipart upload for large files - [ ] Redis caching layer +- [ ] Garbage collection for orphaned artifacts ## License