Commit Graph

24 Commits

Author SHA1 Message Date
Mondo Diaz
939192f425 Add integration tests for stats endpoints and fix JSON report serialization 2026-01-05 15:11:15 -06:00
Mondo Diaz
c977d1d465 Fix API/frontend type mismatch for dashboard
- Backend: Change 'id' to 'artifact_id' in most_referenced_artifacts response
- Backend: Add content_type field to referenced artifacts
- Frontend: Add orphaned_size_bytes to Stats interface
- Frontend: Add missing fields to DeduplicationStats interface
2026-01-05 15:05:41 -06:00
Mondo Diaz
c79b10cbc5 Add comprehensive stats endpoints and reporting features
Backend stats endpoints:
- GET /api/v1/project/:project/packages/:package/stats - per-package stats
- GET /api/v1/artifact/:id/stats - artifact reference statistics
- GET /api/v1/stats/cross-project - cross-project deduplication detection
- GET /api/v1/stats/timeline - time-based metrics (daily/weekly/monthly)
- GET /api/v1/stats/export - CSV/JSON export
- GET /api/v1/stats/report - markdown/JSON summary report generation

Enhanced existing endpoints:
- Added storage_saved_bytes and deduplication_ratio to project stats
- Added date range filtering via from_date/to_date params

New schemas:
- PackageStatsResponse
- ArtifactStatsResponse
- CrossProjectDeduplicationResponse
- TimeBasedStatsResponse
- StatsReportResponse
2026-01-05 14:57:47 -06:00
Mondo Diaz
e215ecabcd Add S3 configuration options and improved error handling
- Add s3_verify_ssl config option for SSL/TLS verification
- Add s3_connect_timeout and s3_read_timeout config options
- Add s3_max_retries config option with adaptive retry mode
- Add S3StorageUnavailableError for backend availability issues
- Add HashCollisionError for detecting extremely rare hash collisions
- Add hash collision detection by comparing file sizes on dedup
- Handle network interruption and timeout errors explicitly
- Update routes.py to handle new exception types with appropriate HTTP codes
2026-01-05 14:46:18 -06:00
Mondo Diaz
7c31b6a244 Add integration tests for deduplication and ref_count
- Add test_integration_uploads.py with 12 tests for duplicate upload scenarios
- Add test_ref_count.py with 7 tests for ref_count management
- Fix ArtifactDetailResponse to include sha256 and checksum fields
- Fix health check SQL warning by wrapping in text()
- Update tests to use unique content per test run for idempotency
2026-01-05 14:29:12 -06:00
Mondo Diaz
109677e43a Add storage abstraction, stats endpoints, garbage collection, and test infrastructure
- Add StorageBackend protocol for backend-agnostic storage interface
- Add health check with storage and database connectivity verification
- Add garbage collection endpoints for orphaned artifacts (ref_count=0)
- Add deduplication statistics endpoints (/api/v1/stats, /stats/storage, /stats/deduplication)
- Add per-project statistics endpoint
- Add verify_integrity method for post-upload hash validation
- Set up pytest infrastructure with mock S3 client
- Add unit tests for hash calculation and duplicate detection
2026-01-05 11:16:46 -06:00
Mondo Diaz
dbe78ded2f Fix double-counting: let SQL triggers manage ref_count
Previously both Python code AND SQL triggers were incrementing/decrementing
ref_count, causing inconsistent values. Now:

- New artifacts start with ref_count=0 (triggers increment on tag creation)
- Tag INSERT/UPDATE/DELETE triggers handle all ref_count changes
- Python code only logs operations, doesn't manipulate ref_count
- Delete endpoints rely on cascade + triggers for ref_count management

Tested: new uploads, duplicates, tag deletion, package deletion (cascade),
project deletion (cascade) - all ref_count values now correct.
2026-01-05 10:18:23 -06:00
Mondo Diaz
865812af98 Add ref_count management for deletions with atomic operations and error handling
- Add DELETE endpoints for tags, packages, and projects with proper ref_count
  decrements for all affected artifacts
- Implement atomic ref_count operations using SELECT FOR UPDATE row-level locking
  to prevent race conditions
- Add custom storage exceptions (HashComputationError, S3ExistenceCheckError,
  S3UploadError) with retry logic for S3 existence checks
- Handle race conditions in upload by locking artifact row before modification
- Add comprehensive logging for all ref_count changes and deduplication events
- Include ref_count in upload response schema
2026-01-05 10:04:59 -06:00
Mondo Diaz
2df97ae94a Add presigned URL support for direct S3 downloads (#48) 2025-12-15 16:06:51 -06:00
Mondo Diaz
3fd2747ae4 Store SHA256 checksums with artifacts and add multiple hash support 2025-12-15 14:47:30 -06:00
Mondo Diaz
b52c8840f1 Add schema enhancements for uploads, artifacts, and audit tracking 2025-12-12 15:23:50 -06:00
Mondo Diaz
9604540dd3 Implement database storage layer 2025-12-12 12:45:33 -06:00
Mondo Diaz
096887d4da Add global search and filtering enhancements 2025-12-12 12:12:46 -06:00
Mondo Diaz
7d80bef39a Fix: restore enhanced tags API endpoints 2025-12-12 10:57:27 -06:00
Mondo Diaz
5d0122fc36 Revert "Add API endpoints for listing tagged versions and artifacts"
This reverts commit 54e33e67ce.
2025-12-12 10:33:21 -06:00
Mondo Diaz
2b5bc60a69 Add API endpoints for listing tagged versions and artifacts 2025-12-12 10:23:40 -06:00
Mondo Diaz
dea03c4a12 Implement Backend API to List Packages within a Project 2025-12-11 18:47:46 -06:00
Mondo Diaz
c119ab4a04 Implement backend upload/download API enhancements 2025-12-11 18:05:08 -06:00
Mondo Diaz
e9404a4425 Merge branch 'feature/projects-api-pagination-search' into 'main'
Add pagination and search to projects API

See merge request esv/bsf/bsf-integration/orchard/orchard-mvp!1
2025-12-11 15:03:42 -06:00
Mondo Diaz
b896ad1fad Add pagination and search to projects API 2025-12-11 15:03:41 -06:00
Mondo Diaz
c08d1082eb Add development mode with automatic test data seeding 2025-12-11 14:36:22 -06:00
Mondo Diaz
2e2f17ae37 Fix remaining old terminology references
- backend/app/main.py: grove/ -> project/ in SPA route check
- frontend/vite.config.ts: /grove -> /project proxy
- helm/orchard/templates/NOTES.txt: Updated API examples
- migrations/001_initial.sql: Updated table/column names
2025-12-08 10:41:26 -06:00
Mondo Diaz
ff7df9eb3f Rename terminology to industry standard terms
- Grove → Project
- Tree → Package
- Fruit → Artifact
- Graft → Tag
- Cultivate → Upload
- Harvest → Download

Updated across:
- Backend models, schemas, and routes
- Frontend types, API client, and components
- README documentation
- API endpoints now use /project/:project/packages pattern
2025-12-08 10:38:44 -06:00
Mondo Diaz
2261bfc830 Rewrite from Go + vanilla JS to Python (FastAPI) + React (TypeScript)
- Backend: Python 3.12 with FastAPI, SQLAlchemy, boto3
- Frontend: React 18 with TypeScript, Vite build tooling
- Updated Dockerfile for multi-stage Node + Python build
- Updated CI pipeline for Python backend
- Removed old Go code (cmd/, internal/, go.mod, go.sum)
- Updated README with new tech stack documentation
2025-12-05 17:16:43 -06:00