bitforge/cf-uploader
3
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Update readme, add script

Browse Source
This commit is contained in:
pratik
2025-10-22 09:31:42 -05:00
parent 72708bbf91
commit 304a781c19
3 changed files with 429 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files

4
.claude/settings.local.json
View File

@@ -6,7 +6,9 @@
"Bash(gradle wrapper:*)", "Bash(gradle wrapper:*)",
"Bash(./gradlew build:*)", "Bash(./gradlew build:*)",
"Bash(./gradlew clean build:*)", "Bash(./gradlew clean build:*)",
"Bash(jar:*)" "Bash(jar:*)",
"Bash(git log:*)",
"Bash(dir /B *.sh *.md)"
], ],
"deny": [], "deny": [],
"ask": [] "ask": []

147
README.md
View File

@@ -28,7 +28,16 @@ The application will start on `http://localhost:8080`
## API Endpoints ## API Endpoints
### 1. Deploy Application to Cloud Foundry ### Deployment Options
This service provides two deployment methods:
1. **Traditional Upload** (`/api/cf/deploy`) - Single request deployment (may hit nginx size/timeout limits)
2. **Chunked Upload** (`/api/cf/upload/*`) - Chunked upload with async deployment (recommended for production)
---
### Option 1: Traditional Deploy (Simple, may timeout with large files)
**Endpoint:** `POST /api/cf/deploy` **Endpoint:** `POST /api/cf/deploy`
@@ -75,6 +84,137 @@ curl -X POST http://localhost:8080/api/cf/deploy \
--- ---
### Option 2: Chunked Upload with Async Deployment (Recommended for Production)
**Why use chunked upload?**
- Bypasses nginx `client_max_body_size` restrictions
- Avoids nginx timeout issues during deployment
- Better for large JAR files (>10MB)
- More resilient to network interruptions
#### Quick Start: Using the Deployment Script
The easiest way to deploy using chunked upload is with the provided bash script:
1. **Configure the script:**
```bash
# Edit deploy-chunked.sh and set your values:
API_BASE="https://your-cf-deployer.example.com/api/cf"
JAR_FILE="./your-app.jar"
MANIFEST_FILE="./manifest.yml"
CF_API_ENDPOINT="https://api.cf.example.com"
CF_USERNAME="your-username"
CF_PASSWORD="your-password"
CF_ORGANIZATION="your-org"
CF_SPACE="your-space"
CF_APP_NAME="your-app"
```
2. **Make it executable and run:**
```bash
chmod +x deploy-chunked.sh
./deploy-chunked.sh
```
The script handles:
- File chunking (1MB chunks by default)
- Progress tracking
- Async deployment
- Status polling
- Error handling
#### Manual cURL Commands (For Custom Integration)
**Step 1: Initialize Upload Session**
```bash
SESSION_ID=$(curl -s -X POST "http://localhost:8080/api/cf/upload/init" \
-H "Content-Type: application/json" \
-d '{
"apiEndpoint": "https://api.cf.example.com",
"username": "your-username",
"password": "your-password",
"organization": "your-org",
"space": "your-space",
"appName": "my-app",
"skipSslValidation": false
}' | grep -o '"uploadSessionId":"[^"]*' | cut -d'"' -f4)
echo "Session ID: $SESSION_ID"
```
**Step 2: Upload JAR File in Chunks**
```bash
# Split your JAR into 1MB chunks and upload each
CHUNK_SIZE=1048576 # 1MB
FILE="./app.jar"
TOTAL_CHUNKS=$(( ($(stat -c%s "$FILE") + CHUNK_SIZE - 1) / CHUNK_SIZE ))
# Split file and upload each chunk
split -b $CHUNK_SIZE "$FILE" chunk_
CHUNK_INDEX=0
for chunk_file in chunk_*; do
curl -X POST "http://localhost:8080/api/cf/upload/chunk" \
-F "uploadSessionId=$SESSION_ID" \
-F "fileType=jarFile" \
-F "chunkIndex=$CHUNK_INDEX" \
-F "totalChunks=$TOTAL_CHUNKS" \
-F "fileName=app.jar" \
-F "chunk=@$chunk_file"
CHUNK_INDEX=$((CHUNK_INDEX + 1))
done
rm chunk_* # Cleanup
```
**Step 3: Upload Manifest**
```bash
curl -X POST "http://localhost:8080/api/cf/upload/chunk" \
-F "uploadSessionId=$SESSION_ID" \
-F "fileType=manifest" \
-F "chunkIndex=0" \
-F "totalChunks=1" \
-F "fileName=manifest.yml" \
-F "chunk=@./manifest.yml"
```
**Step 4: Start Async Deployment**
```bash
curl -X POST "http://localhost:8080/api/cf/upload/finalize?uploadSessionId=$SESSION_ID&async=true"
# Response: {"uploadSessionId":"...","status":"IN_PROGRESS","message":"Deployment started...","progress":0}
```
**Step 5: Poll Deployment Status**
```bash
# Check deployment status every 5 seconds
while true; do
STATUS=$(curl -s "http://localhost:8080/api/cf/deployment/status/$SESSION_ID")
echo "$STATUS"
# Check if completed or failed
if echo "$STATUS" | grep -q '"status":"COMPLETED"'; then
echo "Deployment successful!"
break
elif echo "$STATUS" | grep -q '"status":"FAILED"'; then
echo "Deployment failed!"
break
fi
sleep 5
done
```
**For more details, see:**
- `deploy-chunked.sh` - Complete working script
- `CHUNKED_UPLOAD_GUIDE.md` - Detailed API documentation
- `TIMEOUT_SOLUTION.md` - Architecture and design details
- `CHUNK_SIZE_GUIDE.md` - Chunk size recommendations
- `MEMORY_FIX.md` - JVM memory configuration for Tanzu
---
### 2. List Applications ### 2. List Applications
**Endpoint:** `POST /api/cf/apps` **Endpoint:** `POST /api/cf/apps`
@@ -261,15 +401,18 @@ cf.cli.path=
## Features ## Features
- **Application Deployment**: Deploy JAR files to Cloud Foundry with manifest support - **Application Deployment**: Deploy JAR files to Cloud Foundry with manifest support
- **Chunked Upload Support**: Upload large files in chunks to bypass nginx size restrictions (NEW)
- **Async Deployment**: Non-blocking deployment with status polling to avoid timeout issues (NEW)
- **Application Management**: List apps, view details, and access logs - **Application Management**: List apps, view details, and access logs
- **Route Management**: List all routes in your CF space - **Route Management**: List all routes in your CF space
- **Automatic CF CLI Management**: Bundled CF CLI binaries for Linux, macOS, and Windows - **Automatic CF CLI Management**: Bundled CF CLI binaries for Linux, macOS, and Windows
- **Secure Password Handling**: Passwords are masked in all log output - **Secure Password Handling**: Passwords are masked in all log output
- **Comprehensive Logging**: Detailed DEBUG-level logging for troubleshooting deployments - **Comprehensive Logging**: Detailed DEBUG-level logging for troubleshooting deployments
- **Configurable Timeouts**: Adjustable timeout for long-running deployments (default: 600s) - **Configurable Timeouts**: Adjustable timeout for long-running deployments (default: 600s)
- **Large File Support**: Multipart file upload support up to 500MB - **Large File Support**: Chunked uploads support files of any size
- **Automatic Cleanup**: Temporary files are automatically cleaned up after operations - **Automatic Cleanup**: Temporary files are automatically cleaned up after operations
- **Error Handling**: Comprehensive exception handling with detailed error messages - **Error Handling**: Comprehensive exception handling with detailed error messages
- **Production Ready**: Memory-optimized for Tanzu deployments with low-memory instances
## Error Handling ## Error Handling

281
deploy-chunked.sh Normal file
View File

@@ -0,0 +1,281 @@
#!/bin/bash
#############################################################################
# CF Deployer - Chunked Upload Deployment Script
#
# This script deploys a Java application to Cloud Foundry using chunked
# uploads to bypass nginx size restrictions and async deployment to avoid
# timeout issues.
#
# Usage:
# ./deploy-chunked.sh
#
# Configuration:
# Edit the variables below to match your environment
#############################################################################
set -e # Exit on error
#############################################################################
# CONFIGURATION - Update these values for your deployment
#############################################################################
# API endpoint
API_BASE="http://localhost:8080/api/cf"
# Files to deploy
JAR_FILE="./app.jar"
MANIFEST_FILE="./manifest.yml"
# Chunk size (bytes)
# Recommended: 1MB (1048576) for Tanzu with memory constraints
# Options: 512KB (524288), 1MB (1048576), 2MB (2097152), 5MB (5242880)
CHUNK_SIZE=1048576 # 1MB
# Cloud Foundry configuration
CF_API_ENDPOINT="https://api.cf.example.com"
CF_USERNAME="your-username"
CF_PASSWORD="your-password"
CF_ORGANIZATION="your-org"
CF_SPACE="your-space"
CF_APP_NAME="your-app"
CF_SKIP_SSL="false" # Use "true" for self-signed certificates
# Polling configuration
POLL_INTERVAL=5 # seconds between status checks
MAX_WAIT=600 # maximum wait time in seconds (10 minutes)
#############################################################################
# SCRIPT - Do not modify below this line
#############################################################################
# Colors for output
RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[1;33m'
BLUE='\033[0;34m'
NC='\033[0m' # No Color
# Helper functions
log_info() {
echo -e "${BLUE}[INFO]${NC} $1"
}
log_success() {
echo -e "${GREEN}[SUCCESS]${NC} $1"
}
log_error() {
echo -e "${RED}[ERROR]${NC} $1"
}
log_warning() {
echo -e "${YELLOW}[WARNING]${NC} $1"
}
# Check if files exist
if [ ! -f "$JAR_FILE" ]; then
log_error "JAR file not found: $JAR_FILE"
exit 1
fi
if [ ! -f "$MANIFEST_FILE" ]; then
log_error "Manifest file not found: $MANIFEST_FILE"
exit 1
fi
# Check if curl is available
if ! command -v curl &> /dev/null; then
log_error "curl is not installed. Please install curl to use this script."
exit 1
fi
# Build CF configuration JSON
CF_CONFIG=$(cat <<EOF
{
"apiEndpoint": "$CF_API_ENDPOINT",
"username": "$CF_USERNAME",
"password": "$CF_PASSWORD",
"organization": "$CF_ORGANIZATION",
"space": "$CF_SPACE",
"appName": "$CF_APP_NAME",
"skipSslValidation": $CF_SKIP_SSL
}
EOF
)
log_info "Starting deployment of $CF_APP_NAME to Cloud Foundry"
log_info "JAR: $JAR_FILE"
log_info "Manifest: $MANIFEST_FILE"
log_info "Chunk size: $CHUNK_SIZE bytes ($(($CHUNK_SIZE / 1024))KB)"
echo ""
#############################################################################
# STEP 1: Initialize Upload Session
#############################################################################
log_info "Step 1/5: Initializing upload session..."
INIT_RESPONSE=$(curl -s -X POST "$API_BASE/upload/init" \
-H "Content-Type: application/json" \
-d "$CF_CONFIG")
if [ $? -ne 0 ]; then
log_error "Failed to initialize upload session"
echo "$INIT_RESPONSE"
exit 1
fi
SESSION_ID=$(echo "$INIT_RESPONSE" | grep -o '"uploadSessionId":"[^"]*' | cut -d'"' -f4)
if [ -z "$SESSION_ID" ]; then
log_error "Failed to get session ID from response:"
echo "$INIT_RESPONSE"
exit 1
fi
log_success "Upload session created: $SESSION_ID"
echo ""
#############################################################################
# FUNCTION: Upload file in chunks
#############################################################################
upload_file_in_chunks() {
local file_path=$1
local file_type=$2
local file_name=$(basename "$file_path")
# Get file size (cross-platform)
if [[ "$OSTYPE" == "darwin"* ]]; then
local file_size=$(stat -f%z "$file_path")
else
local file_size=$(stat -c%s "$file_path")
fi
local total_chunks=$(( ($file_size + $CHUNK_SIZE - 1) / $CHUNK_SIZE ))
local file_size_mb=$(echo "scale=2; $file_size / 1048576" | bc)
log_info "Uploading $file_type: $file_name (${file_size_mb}MB, $total_chunks chunks)"
# Create temporary directory for chunks
local temp_dir=$(mktemp -d)
# Split file into chunks
split -b $CHUNK_SIZE "$file_path" "$temp_dir/chunk_"
local chunk_index=0
for chunk_file in "$temp_dir"/chunk_*; do
printf " Chunk %3d/%3d... " "$((chunk_index + 1))" "$total_chunks"
RESPONSE=$(curl -s -X POST "$API_BASE/upload/chunk" \
-F "uploadSessionId=$SESSION_ID" \
-F "fileType=$file_type" \
-F "chunkIndex=$chunk_index" \
-F "totalChunks=$total_chunks" \
-F "fileName=$file_name" \
-F "chunk=@$chunk_file")
SUCCESS=$(echo "$RESPONSE" | grep -o '"success":[^,}]*' | cut -d':' -f2)
if [ "$SUCCESS" != "true" ]; then
echo -e "${RED}FAILED${NC}"
log_error "Failed to upload chunk $((chunk_index + 1))/$total_chunks"
echo "$RESPONSE"
rm -rf "$temp_dir"
exit 1
fi
echo -e "${GREEN}OK${NC}"
chunk_index=$((chunk_index + 1))
done
# Cleanup temporary directory
rm -rf "$temp_dir"
log_success "$file_type upload completed ($total_chunks chunks)"
echo ""
}
#############################################################################
# STEP 2: Upload JAR file
#############################################################################
log_info "Step 2/5: Uploading JAR file..."
upload_file_in_chunks "$JAR_FILE" "jarFile"
#############################################################################
# STEP 3: Upload manifest file
#############################################################################
log_info "Step 3/5: Uploading manifest file..."
upload_file_in_chunks "$MANIFEST_FILE" "manifest"
#############################################################################
# STEP 4: Start async deployment
#############################################################################
log_info "Step 4/5: Starting async deployment..."
FINALIZE_RESPONSE=$(curl -s -X POST "$API_BASE/upload/finalize?uploadSessionId=$SESSION_ID&async=true")
STATUS=$(echo "$FINALIZE_RESPONSE" | grep -o '"status":"[^"]*' | cut -d'"' -f4)
if [ "$STATUS" != "IN_PROGRESS" ]; then
log_error "Failed to start deployment. Status: $STATUS"
echo "$FINALIZE_RESPONSE"
exit 1
fi
log_success "Deployment started successfully"
log_info "Step 5/5: Polling deployment status (max wait: ${MAX_WAIT}s)..."
echo ""
#############################################################################
# STEP 5: Poll deployment status
#############################################################################
elapsed=0
last_message=""
while [ $elapsed -lt $MAX_WAIT ]; do
sleep $POLL_INTERVAL
elapsed=$((elapsed + POLL_INTERVAL))
STATUS_RESPONSE=$(curl -s "$API_BASE/deployment/status/$SESSION_ID" 2>/dev/null)
if [ $? -ne 0 ]; then
log_warning "Failed to fetch status, retrying..."
continue
fi
CURRENT_STATUS=$(echo "$STATUS_RESPONSE" | grep -o '"status":"[^"]*' | cut -d'"' -f4)
MESSAGE=$(echo "$STATUS_RESPONSE" | grep -o '"message":"[^"]*' | cut -d'"' -f4)
PROGRESS=$(echo "$STATUS_RESPONSE" | grep -o '"progress":[0-9]*' | cut -d':' -f2)
# Only print if message changed to reduce clutter
if [ "$MESSAGE" != "$last_message" ]; then
printf " [%3ds] Status: %-15s Progress: %3s%% - %s\n" \
"$elapsed" "$CURRENT_STATUS" "${PROGRESS:-0}" "$MESSAGE"
last_message="$MESSAGE"
fi
# Check if deployment completed
if [ "$CURRENT_STATUS" = "COMPLETED" ]; then
echo ""
log_success "Deployment completed successfully!"
echo ""
log_info "Deployment details:"
echo "$STATUS_RESPONSE" | python -m json.tool 2>/dev/null || echo "$STATUS_RESPONSE"
exit 0
fi
# Check if deployment failed
if [ "$CURRENT_STATUS" = "FAILED" ]; then
echo ""
log_error "Deployment failed!"
echo ""
log_info "Error details:"
echo "$STATUS_RESPONSE" | python -m json.tool 2>/dev/null || echo "$STATUS_RESPONSE"
exit 1
fi
done
# Timeout reached
echo ""
log_warning "Deployment timeout reached after ${MAX_WAIT}s"
log_info "The deployment may still be running. Check status manually:"
echo " curl $API_BASE/deployment/status/$SESSION_ID"
exit 1