diff --git a/README.md b/README.md new file mode 100644 index 0000000..70dc5bc --- /dev/null +++ b/README.md @@ -0,0 +1,247 @@ +# Orchard + +**Content-Addressable Storage System** + +Orchard is a centralized binary artifact storage system that provides content-addressable storage with automatic deduplication, flexible access control, and multi-format package support. Like an orchard that cultivates and distributes fruit, Orchard nurtures and distributes the products of software builds. + +## Features + +### Currently Implemented + +- **Content-Addressable Storage** - Artifacts are stored and referenced by their SHA256 hash, ensuring deduplication and data integrity +- **Grove/Tree/Fruit Hierarchy** - Organized storage structure: + - **Grove** - Top-level project container + - **Tree** - Named package within a grove + - **Fruit** - Specific artifact instance identified by SHA256 +- **Grafts (Tags/Versions)** - Alias system for referencing artifacts by human-readable names (e.g., `v1.0.0`, `latest`, `stable`) +- **S3-Compatible Backend** - Uses MinIO (or any S3-compatible storage) for artifact storage +- **PostgreSQL Metadata** - Relational database for metadata, access control, and audit trails +- **REST API** - Full HTTP API for all operations +- **Web UI** - Browser-based interface for managing artifacts +- **Docker Compose Setup** - Easy local development environment + +### API Endpoints + +| Method | Endpoint | Description | +|--------|----------|-------------| +| `GET` | `/` | Web UI | +| `GET` | `/health` | Health check | +| `GET` | `/api/v1/groves` | List all groves | +| `POST` | `/api/v1/groves` | Create a new grove | +| `GET` | `/api/v1/groves/:grove` | Get grove details | +| `GET` | `/api/v1/grove/:grove/trees` | List trees in a grove | +| `POST` | `/api/v1/grove/:grove/trees` | Create a new tree | +| `POST` | `/api/v1/grove/:grove/:tree/cultivate` | Upload an artifact | +| `GET` | `/api/v1/grove/:grove/:tree/+/:ref` | Download an artifact | +| `GET` | `/api/v1/grove/:grove/:tree/grafts` | List all tags/versions | +| `POST` | `/api/v1/grove/:grove/:tree/graft` | Create a tag | +| `GET` | `/api/v1/grove/:grove/:tree/consumers` | List consumers of a tree | +| `GET` | `/api/v1/fruit/:id` | Get fruit metadata by hash | + +### Reference Formats + +When downloading artifacts, the `:ref` parameter supports multiple formats: + +- `latest` - Tag name directly +- `v1.0.0` - Version tag +- `tag:stable` - Explicit tag reference +- `version:2024.1` - Version reference +- `fruit:a3f5d8e12b4c6789...` - Direct SHA256 hash reference + +## Quick Start + +### Prerequisites + +- Docker and Docker Compose + +### Running Locally + +```bash +# Start all services +docker-compose up -d + +# View logs +docker-compose logs -f orchard-server + +# Stop services +docker-compose down +``` + +### Services + +| Service | Port | Description | +|---------|------|-------------| +| orchard-server | 8080 | Main API server and Web UI | +| postgres | 5432 | PostgreSQL database | +| minio | 9000 | S3-compatible object storage | +| minio (console) | 9001 | MinIO web console | +| redis | 6379 | Cache (for future use) | + +### Access Points + +- **Web UI**: http://localhost:8080 +- **API**: http://localhost:8080/api/v1 +- **MinIO Console**: http://localhost:9001 (user: `minioadmin`, pass: `minioadmin`) + +## Usage Examples + +### Create a Grove + +```bash +curl -X POST http://localhost:8080/api/v1/groves \ + -H "Content-Type: application/json" \ + -d '{"name": "my-project", "description": "My project artifacts", "is_public": true}' +``` + +### Create a Tree + +```bash +curl -X POST http://localhost:8080/api/v1/grove/my-project/trees \ + -H "Content-Type: application/json" \ + -d '{"name": "releases", "description": "Release builds"}' +``` + +### Upload an Artifact (Cultivate) + +```bash +curl -X POST http://localhost:8080/api/v1/grove/my-project/releases/cultivate \ + -F "file=@./build/app-v1.0.0.tar.gz" \ + -F "tag=v1.0.0" +``` + +Response: +```json +{ + "fruit_id": "a3f5d8e12b4c67890abcdef1234567890abcdef1234567890abcdef12345678", + "size": 1048576, + "grove": "my-project", + "tree": "releases", + "tag": "v1.0.0" +} +``` + +### Download an Artifact (Harvest) + +```bash +# By tag +curl -O http://localhost:8080/api/v1/grove/my-project/releases/+/v1.0.0 + +# By fruit ID +curl -O http://localhost:8080/api/v1/grove/my-project/releases/+/fruit:a3f5d8e12b4c6789... + +# Using the spec-compliant URL pattern +curl -O http://localhost:8080/grove/my-project/releases/+/latest +``` + +### Create a Tag (Graft) + +```bash +curl -X POST http://localhost:8080/api/v1/grove/my-project/releases/graft \ + -H "Content-Type: application/json" \ + -d '{"name": "stable", "fruit_id": "a3f5d8e12b4c6789..."}' +``` + +### Search by Fruit ID + +```bash +curl http://localhost:8080/api/v1/fruit/a3f5d8e12b4c67890abcdef1234567890abcdef1234567890abcdef12345678 +``` + +## Project Structure + +``` +orchard/ +├── cmd/ +│ └── orchard-server/ +│ └── main.go # Application entrypoint +├── internal/ +│ ├── api/ +│ │ ├── handlers.go # HTTP request handlers +│ │ ├── router.go # Route definitions +│ │ └── static/ # Web UI assets +│ │ ├── index.html +│ │ ├── style.css +│ │ └── app.js +│ ├── config/ +│ │ └── config.go # Configuration management +│ ├── models/ +│ │ └── models.go # Data structures +│ └── storage/ +│ ├── database.go # PostgreSQL operations +│ └── s3.go # S3 storage operations +├── migrations/ +│ └── 001_initial.sql # Database schema +├── Dockerfile # Multi-stage Docker build +├── docker-compose.yml # Local development stack +├── config.yaml # Default configuration +├── Makefile # Build automation +├── go.mod # Go module definition +└── go.sum # Dependency checksums +``` + +## Configuration + +Configuration can be provided via `config.yaml` or environment variables prefixed with `ORCHARD_`: + +| Environment Variable | Description | Default | +|---------------------|-------------|---------| +| `ORCHARD_SERVER_HOST` | Server bind address | `0.0.0.0` | +| `ORCHARD_SERVER_PORT` | Server port | `8080` | +| `ORCHARD_DATABASE_HOST` | PostgreSQL host | `localhost` | +| `ORCHARD_DATABASE_PORT` | PostgreSQL port | `5432` | +| `ORCHARD_DATABASE_USER` | PostgreSQL user | `orchard` | +| `ORCHARD_DATABASE_PASSWORD` | PostgreSQL password | - | +| `ORCHARD_DATABASE_DBNAME` | PostgreSQL database | `orchard` | +| `ORCHARD_S3_ENDPOINT` | S3 endpoint URL | - | +| `ORCHARD_S3_REGION` | S3 region | `us-east-1` | +| `ORCHARD_S3_BUCKET` | S3 bucket name | `orchard-artifacts` | +| `ORCHARD_S3_ACCESS_KEY_ID` | S3 access key | - | +| `ORCHARD_S3_SECRET_ACCESS_KEY` | S3 secret key | - | + +## Database Schema + +### Core Tables + +- **groves** - Top-level project containers +- **trees** - Packages within groves +- **fruits** - Content-addressable artifacts (SHA256) +- **grafts** - Tags/aliases pointing to fruits +- **graft_history** - Audit trail for tag changes +- **harvests** - Upload event records +- **consumers** - Dependency tracking +- **access_permissions** - Grove-level access control +- **api_keys** - Programmatic access tokens +- **audit_logs** - Immutable operation logs + +## Terminology + +| Orchard Term | Traditional Term | Description | +|--------------|------------------|-------------| +| Grove | Project | Top-level organizational unit | +| Tree | Package | Named collection of related artifacts | +| Fruit | Instance | Specific content identified by SHA256 | +| Seed | Dependency | Required package specification | +| Harvest | Download/Fetch | Retrieve dependencies | +| Cultivate | Upload/Publish | Store new artifact | +| Graft | Alias/Tag | Alternative name for content | +| Prune | Clean/GC | Remove unused local cache | + +## Future Work + +The following features from the specification are planned but not yet implemented: + +- [ ] CLI tool (`orchard` command) +- [ ] `orchard.ensure` file parsing +- [ ] Lock file generation (`orchard.lock`) +- [ ] Export/Import for air-gapped systems +- [ ] Consumer notification (pollination) +- [ ] Automated update propagation +- [ ] OIDC/SAML authentication +- [ ] API key management +- [ ] Package format detection +- [ ] Multipart upload for large files +- [ ] Redis caching layer + +## License + +Internal use only.