bitforge/orchard
3
0
Fork 0
Code Projects Wiki Activity
88 Commits 19 Branches 0 Tags
ee14dbe87816e307c03ee8879c1b8951b144afc0
Commit Graph

88 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Mondo Diaz
ee14dbe878 Remove unused imports 
- routes.py: Remove unused re, hashlib, ArtifactResponse imports
- storage.py: Remove unused ABC, abstractmethod imports (using Protocol instead)
 2026-01-05 15:47:22 -06:00
Mondo Diaz
af66fd5845 Add security fixes: SHA256 hash validation and streaming file size enforcement 
- Add field_validator to ResumableUploadInitRequest to validate expected_hash
  is a valid 64-character lowercase hex SHA256 hash (normalizes to lowercase)
- Add FileSizeExceededError exception for file size limit violations
- Enforce file size limits in storage layer during streaming (prevents
  Content-Length header spoofing)
- Add FileSizeExceededError handler in upload endpoint returning HTTP 413
- Add node_modules and frontend/dist to .gitignore
 2026-01-05 15:43:19 -06:00
Mondo Diaz
55a38ad850 Add deduplication design doc, file size limits, and validation tests 
- Add max_file_size (10GB) and min_file_size (1 byte) config options
- Add file size validation to regular and resumable upload endpoints
- Create comprehensive deduplication design document covering:
  - SHA256 algorithm selection rationale and migration path
  - Content-addressable storage model
  - S3 key derivation and prefix sharding
  - Duplicate detection workflow
  - Reference counting lifecycle
  - Edge cases and error handling
  - Collision detection strategy
  - Performance considerations
  - Operations runbook
- Add tests for empty file rejection and file size validation
 2026-01-05 15:35:21 -06:00
Mondo Diaz
32115fc1c5 Add integration tests for garbage collection endpoints 2026-01-05 15:24:46 -06:00
Mondo Diaz
4c2e21295f Add comprehensive ref_count tests and fix resumable upload double-counting bug 
- Add tests for cascade deletion ref_count (package/project delete)
- Add tests for tag update ref_count adjustments
- Fix resumable upload bug where ref_count was incremented manually AND by SQL trigger
- ref_count is now exclusively managed by SQL triggers on tag INSERT/DELETE/UPDATE
 2026-01-05 15:19:05 -06:00
Mondo Diaz
7d4091221a Update CHANGELOG for issues 33, 34, 35 2026-01-05 15:11:48 -06:00
Mondo Diaz
939192f425 Add integration tests for stats endpoints and fix JSON report serialization 2026-01-05 15:11:15 -06:00
Mondo Diaz
c977d1d465 Fix API/frontend type mismatch for dashboard 
- Backend: Change 'id' to 'artifact_id' in most_referenced_artifacts response
- Backend: Add content_type field to referenced artifacts
- Frontend: Add orphaned_size_bytes to Stats interface
- Frontend: Add missing fields to DeduplicationStats interface
 2026-01-05 15:05:41 -06:00
Mondo Diaz
e37e1892b2 Add Dashboard UI for storage and deduplication statistics 
New components:
- Dashboard.tsx - main dashboard page with stats overview
- Dashboard.css - responsive styling with dark theme

Features:
- Storage overview cards (total used, saved, ratio, percentage)
- Artifact statistics cards
- Deduplication effectiveness visualization with progress bars
- Circular progress indicator for deduplication rate
- Top referenced artifacts table

Updated files:
- api.ts - added getStats, getDeduplicationStats, getTimelineStats, getCrossProjectStats
- types.ts - added Stats, DeduplicationStats, TimelineStats interfaces
- App.tsx - added /dashboard route
- Layout.tsx - added Dashboard navigation link
 2026-01-05 15:02:30 -06:00
Mondo Diaz
c79b10cbc5 Add comprehensive stats endpoints and reporting features 
Backend stats endpoints:
- GET /api/v1/project/:project/packages/:package/stats - per-package stats
- GET /api/v1/artifact/:id/stats - artifact reference statistics
- GET /api/v1/stats/cross-project - cross-project deduplication detection
- GET /api/v1/stats/timeline - time-based metrics (daily/weekly/monthly)
- GET /api/v1/stats/export - CSV/JSON export
- GET /api/v1/stats/report - markdown/JSON summary report generation

Enhanced existing endpoints:
- Added storage_saved_bytes and deduplication_ratio to project stats
- Added date range filtering via from_date/to_date params

New schemas:
- PackageStatsResponse
- ArtifactStatsResponse
- CrossProjectDeduplicationResponse
- TimeBasedStatsResponse
- StatsReportResponse
 2026-01-05 14:57:47 -06:00
Mondo Diaz
e215ecabcd Add S3 configuration options and improved error handling 
- Add s3_verify_ssl config option for SSL/TLS verification
- Add s3_connect_timeout and s3_read_timeout config options
- Add s3_max_retries config option with adaptive retry mode
- Add S3StorageUnavailableError for backend availability issues
- Add HashCollisionError for detecting extremely rare hash collisions
- Add hash collision detection by comparing file sizes on dedup
- Handle network interruption and timeout errors explicitly
- Update routes.py to handle new exception types with appropriate HTTP codes
 2026-01-05 14:46:18 -06:00
Mondo Diaz
eca291d194 Add S3 verification and failure cleanup tests 
- Add test_s3_bucket_single_object_after_duplicates to verify only one S3 object exists
- Add tests for upload failure scenarios (invalid project/package, empty file)
- Add tests for orphaned S3 objects and database records cleanup
- Add S3 direct access helpers (list_s3_objects_by_hash, s3_object_exists, etc.)
- Fix conftest.py to use setdefault for env vars (don't override container config)

All 52 tests now pass.
 2026-01-05 14:39:22 -06:00
Mondo Diaz
7c31b6a244 Add integration tests for deduplication and ref_count 
- Add test_integration_uploads.py with 12 tests for duplicate upload scenarios
- Add test_ref_count.py with 7 tests for ref_count management
- Fix ArtifactDetailResponse to include sha256 and checksum fields
- Fix health check SQL warning by wrapping in text()
- Update tests to use unique content per test run for idempotency
 2026-01-05 14:29:12 -06:00
Mondo Diaz
d2abfe671a Update CHANGELOG for issues #33, #34, #35, #36 2026-01-05 11:31:55 -06:00
Mondo Diaz
109677e43a Add storage abstraction, stats endpoints, garbage collection, and test infrastructure 
- Add StorageBackend protocol for backend-agnostic storage interface
- Add health check with storage and database connectivity verification
- Add garbage collection endpoints for orphaned artifacts (ref_count=0)
- Add deduplication statistics endpoints (/api/v1/stats, /stats/storage, /stats/deduplication)
- Add per-project statistics endpoint
- Add verify_integrity method for post-upload hash validation
- Set up pytest infrastructure with mock S3 client
- Add unit tests for hash calculation and duplicate detection
 2026-01-05 11:16:46 -06:00
Mondo Diaz
dbe78ded2f Fix double-counting: let SQL triggers manage ref_count 
Previously both Python code AND SQL triggers were incrementing/decrementing
ref_count, causing inconsistent values. Now:

- New artifacts start with ref_count=0 (triggers increment on tag creation)
- Tag INSERT/UPDATE/DELETE triggers handle all ref_count changes
- Python code only logs operations, doesn't manipulate ref_count
- Delete endpoints rely on cascade + triggers for ref_count management

Tested: new uploads, duplicates, tag deletion, package deletion (cascade),
project deletion (cascade) - all ref_count values now correct.
 2026-01-05 10:18:23 -06:00
Mondo Diaz
865812af98 Add ref_count management for deletions with atomic operations and error handling 
- Add DELETE endpoints for tags, packages, and projects with proper ref_count
  decrements for all affected artifacts
- Implement atomic ref_count operations using SELECT FOR UPDATE row-level locking
  to prevent race conditions
- Add custom storage exceptions (HashComputationError, S3ExistenceCheckError,
  S3UploadError) with retry logic for S3 existence checks
- Handle race conditions in upload by locking artifact row before modification
- Add comprehensive logging for all ref_count changes and deduplication events
- Include ref_count in upload response schema
 2026-01-05 10:04:59 -06:00
Mondo Diaz
66622caf5d Add AGENTS.md to gitignore for OpenCode compatibility 2026-01-05 09:24:33 -06:00
Mondo Diaz
96d79e4127 Merge branch 'fix/helm-minio-ingress-rename' into 'main' 
Fix Helm chart: rename minio.ingress to minioIngress to avoid subchart conflict

See merge request esv/bsf/bsf-integration/orchard/orchard-mvp!19
 2025-12-16 12:51:42 -06:00
Mondo Diaz
accba9e404 Fix Helm chart: rename minio.ingress to minioIngress to avoid subchart conflict 2025-12-16 12:51:41 -06:00
Dane Moss
64e420fb58 Merge branch 'cut_new_release' into 'main' 
release new image with presigned URL support for direct s3 downloads as default download mode

See merge request esv/bsf/bsf-integration/orchard/orchard-mvp!18
 2025-12-15 15:30:38 -07:00
dane.moss
994f166de8 release new image with presigned URL support for direct s3 downloads as default download mode 2025-12-15 15:17:53 -07:00
Mondo Diaz
8999552949 Merge branch 'feature/presigned-url-downloads' into 'main' 
Add presigned URL support for direct S3 downloads (#48)

Closes #48

See merge request esv/bsf/bsf-integration/orchard/orchard-mvp!17
 2025-12-15 16:06:51 -06:00
Mondo Diaz
2df97ae94a Add presigned URL support for direct S3 downloads (#48) 2025-12-15 16:06:51 -06:00
Mondo Diaz
caa0c5af0c Merge branch 'feature/store-sha256-checksums' into 'main' 
Store SHA256 checksums with artifacts and add multiple hash support

Closes #25

See merge request esv/bsf/bsf-integration/orchard/orchard-mvp!16
 2025-12-15 14:47:31 -06:00
Mondo Diaz
3fd2747ae4 Store SHA256 checksums with artifacts and add multiple hash support 2025-12-15 14:47:30 -06:00
Mondo Diaz
96367da448 Merge branch 'feature/integrity-verification-design' into 'main' 
Add integrity verification workflow design document

Closes #24

See merge request esv/bsf/bsf-integration/orchard/orchard-mvp!15
 2025-12-15 14:00:32 -06:00
Mondo Diaz
2686fdcb89 Add integrity verification workflow design document 2025-12-15 14:00:32 -06:00
Dane Moss
0eb2deb4ca Merge branch 'update_urls' into 'main' 
update URLs to point to BSF

Closes #46

See merge request esv/bsf/bsf-integration/orchard/orchard-mvp!14
 2025-12-15 11:30:07 -07:00
Dane Moss
3fe421f31d update URLs to point to BSF 2025-12-15 11:30:07 -07:00
Mondo Diaz
68660eacf6 Merge branch 'feature/schema-enhancements' into 'main' 
Add schema enhancements for uploads, artifacts, and audit tracking

Closes #16

See merge request esv/bsf/bsf-integration/orchard/orchard-mvp!13
 2025-12-12 15:23:50 -06:00
Mondo Diaz
b52c8840f1 Add schema enhancements for uploads, artifacts, and audit tracking 2025-12-12 15:23:50 -06:00
Dane Moss
4afcdf5cda Merge branch 'add_prosper_config' into 'main' 
Add prosper config

Closes #45

See merge request esv/bsf/bsf-integration/orchard/orchard-mvp!12
 2025-12-12 13:52:27 -07:00
Dane Moss
bc3da14d50 Add prosper config 2025-12-12 13:52:27 -07:00
Mondo Diaz
2843335f6d Document curl -OJ flag for correct download filenames 
- Update download examples to use -OJ flag
- Add note explaining curl download flags (-O, -J, -OJ, -o)
- Add example for saving to a specific filename
 2025-12-12 13:53:15 -06:00
Mondo Diaz
2097865874 Remove redundant search bar from Home, rename page filters 
- Remove SearchInput from Home page (use GlobalSearch in header instead)
- Rename "Search packages..." to "Filter packages..." on ProjectPage
- Rename "Search tags..." to "Filter tags..." on PackagePage
- Update FilterChip labels from "Search" to "Filter"

This differentiates the global search (header) from page-level filtering.
 2025-12-12 12:55:31 -06:00
Mondo Diaz
0e1474bf6c Merge branch 'feature/database-storage-layer' into 'main' 
Implement database storage layer

Closes #17

See merge request esv/bsf/bsf-integration/orchard/orchard-mvp!11
 2025-12-12 12:45:33 -06:00
Mondo Diaz
9604540dd3 Implement database storage layer 2025-12-12 12:45:33 -06:00
Mondo Diaz
a6df5aba5a Merge branch 'feature/search-filtering-enhancements' into 'main' 
Add global search and filtering enhancements

Closes #6

See merge request esv/bsf/bsf-integration/orchard/orchard-mvp!10
 2025-12-12 12:12:46 -06:00
Mondo Diaz
096887d4da Add global search and filtering enhancements 2025-12-12 12:12:46 -06:00
Mondo Diaz
7d80bef39a Fix: restore enhanced tags API endpoints 2025-12-12 10:57:27 -06:00
Mondo Diaz
96198dc127 Merge branch 'fix/restore-merged-features' 2025-12-12 10:55:19 -06:00
Mondo Diaz
fd06dfb3ce Reapply "Add API endpoints for listing tagged versions and artifacts" 
This reverts commit 11852adc66.
 2025-12-12 10:55:15 -06:00
Mondo Diaz
11852adc66 Revert "Add API endpoints for listing tagged versions and artifacts" 
This reverts commit 54e33e67ce.
 2025-12-12 10:49:55 -06:00
Mondo Diaz
21555d64a3 Merge branch 'fix/restore-merged-features' into 'main' 
fix merge issue

See merge request esv/bsf/bsf-integration/orchard/orchard-mvp!9
 2025-12-12 10:48:55 -06:00
Mondo Diaz
b83f19aa52 fix merge issue 2025-12-12 10:48:55 -06:00
Mondo Diaz
5d0122fc36 Revert "Add API endpoints for listing tagged versions and artifacts" 
This reverts commit 54e33e67ce.
 2025-12-12 10:33:21 -06:00
Mondo Diaz
81b423e0ea Merge branch 'feature/frontend-hierarchy-components' into 'main' 
Develop Frontend Components for Project, Package, and Instance Views

Closes #5

See merge request esv/bsf/bsf-integration/orchard/orchard-mvp!8
 2025-12-12 10:23:45 -06:00
Mondo Diaz
e89947f3d3 Develop Frontend Components for Project, Package, and Instance Views 2025-12-12 10:23:44 -06:00
Mondo Diaz
459867abdb Merge branch 'feature/list-tagged-versions-api' into 'main' 
Add API endpoints for listing tagged versions and artifacts

Closes #4

See merge request esv/bsf/bsf-integration/orchard/orchard-mvp!7
 2025-12-12 10:23:40 -06:00