282 lines
7.1 KiB
Markdown
282 lines
7.1 KiB
Markdown
# Nginx Timeout Solution
|
|
|
|
## The Problem
|
|
|
|
### Chunking Solves Upload Timeouts ✅
|
|
- Each chunk upload completes in seconds
|
|
- Well under nginx's 30-second timeout
|
|
- **No problem here!**
|
|
|
|
### But Deployment Still Times Out ❌
|
|
The `/upload/finalize` endpoint can take 3-5+ minutes because:
|
|
1. CF login
|
|
2. CF push (staging, building, deploying)
|
|
3. CF logout
|
|
|
|
This **WILL** hit nginx's 30-second timeout!
|
|
|
|
## The Solution: Async Deployment
|
|
|
|
Instead of waiting for deployment to complete, we return immediately and let the client poll for status.
|
|
|
|
### Flow Comparison
|
|
|
|
**Before (Times Out):**
|
|
```
|
|
Client → finalize → [waits 5 minutes] → ⏱️ NGINX TIMEOUT after 30s
|
|
```
|
|
|
|
**After (Works):**
|
|
```
|
|
Client → finalize → ✅ Returns immediately (202 Accepted)
|
|
Client → poll status every 5s → IN_PROGRESS
|
|
Client → poll status → IN_PROGRESS
|
|
Client → poll status → COMPLETED ✅
|
|
```
|
|
|
|
## Updated API
|
|
|
|
### 1. Initialize Upload (unchanged)
|
|
```bash
|
|
POST /api/cf/upload/init
|
|
```
|
|
|
|
### 2. Upload Chunks (unchanged)
|
|
```bash
|
|
POST /api/cf/upload/chunk
|
|
```
|
|
|
|
### 3. Finalize Upload (NEW: async by default)
|
|
```bash
|
|
POST /api/cf/upload/finalize?uploadSessionId={sessionId}&async=true
|
|
```
|
|
|
|
**Response (202 Accepted):**
|
|
```json
|
|
{
|
|
"uploadSessionId": "550e8400-e29b-41d4-a716-446655440000",
|
|
"status": "IN_PROGRESS",
|
|
"message": "Deployment started. Use /deployment/status endpoint to check progress.",
|
|
"progress": 0
|
|
}
|
|
```
|
|
|
|
### 4. Poll Deployment Status (NEW)
|
|
```bash
|
|
GET /api/cf/deployment/status/{uploadSessionId}
|
|
```
|
|
|
|
**Response while deploying:**
|
|
```json
|
|
{
|
|
"uploadSessionId": "550e8400-e29b-41d4-a716-446655440000",
|
|
"status": "IN_PROGRESS",
|
|
"message": "Logging into Cloud Foundry...",
|
|
"progress": 10
|
|
}
|
|
```
|
|
|
|
**Response when complete:**
|
|
```json
|
|
{
|
|
"uploadSessionId": "550e8400-e29b-41d4-a716-446655440000",
|
|
"status": "COMPLETED",
|
|
"message": "Deployment completed successfully",
|
|
"output": "[full CF CLI output]",
|
|
"progress": 100
|
|
}
|
|
```
|
|
|
|
**Response if failed:**
|
|
```json
|
|
{
|
|
"uploadSessionId": "550e8400-e29b-41d4-a716-446655440000",
|
|
"status": "FAILED",
|
|
"message": "Deployment failed: ...",
|
|
"error": "[error details]",
|
|
"progress": 0
|
|
}
|
|
```
|
|
|
|
## Updated Bash Script
|
|
|
|
```bash
|
|
#!/bin/bash
|
|
|
|
API_BASE="http://your-app.example.com/api/cf"
|
|
JAR_FILE="hf.jar"
|
|
MANIFEST_FILE="manifest.yml"
|
|
CHUNK_SIZE=1048576 # 1MB
|
|
|
|
CF_CONFIG='{
|
|
"apiEndpoint": "https://api.cf.example.com",
|
|
"username": "your-username",
|
|
"password": "your-password",
|
|
"organization": "your-org",
|
|
"space": "your-space",
|
|
"appName": "your-app",
|
|
"skipSslValidation": false
|
|
}'
|
|
|
|
echo "=== Step 1: Initialize Upload Session ==="
|
|
INIT_RESPONSE=$(curl -s -X POST "$API_BASE/upload/init" \
|
|
-H "Content-Type: application/json" \
|
|
-d "$CF_CONFIG")
|
|
|
|
SESSION_ID=$(echo $INIT_RESPONSE | grep -o '"uploadSessionId":"[^"]*' | cut -d'"' -f4)
|
|
echo "Session created: $SESSION_ID"
|
|
|
|
# Function to upload file in chunks
|
|
upload_file_in_chunks() {
|
|
local file_path=$1
|
|
local file_type=$2
|
|
local file_name=$(basename "$file_path")
|
|
local file_size=$(stat -f%z "$file_path" 2>/dev/null || stat -c%s "$file_path")
|
|
local total_chunks=$(( ($file_size + $CHUNK_SIZE - 1) / $CHUNK_SIZE ))
|
|
|
|
echo ""
|
|
echo "=== Uploading $file_type: $file_name ($total_chunks chunks) ==="
|
|
|
|
local temp_dir=$(mktemp -d)
|
|
split -b $CHUNK_SIZE "$file_path" "$temp_dir/chunk_"
|
|
|
|
local chunk_index=0
|
|
for chunk_file in "$temp_dir"/chunk_*; do
|
|
printf "Uploading chunk %3d/%3d... " "$((chunk_index + 1))" "$total_chunks"
|
|
|
|
RESPONSE=$(curl -s -X POST "$API_BASE/upload/chunk" \
|
|
-F "uploadSessionId=$SESSION_ID" \
|
|
-F "fileType=$file_type" \
|
|
-F "chunkIndex=$chunk_index" \
|
|
-F "totalChunks=$total_chunks" \
|
|
-F "fileName=$file_name" \
|
|
-F "chunk=@$chunk_file")
|
|
|
|
SUCCESS=$(echo $RESPONSE | grep -o '"success":[^,]*' | cut -d':' -f2)
|
|
|
|
if [ "$SUCCESS" != "true" ]; then
|
|
echo "FAILED"
|
|
echo "$RESPONSE"
|
|
rm -rf "$temp_dir"
|
|
exit 1
|
|
fi
|
|
|
|
echo "OK"
|
|
chunk_index=$((chunk_index + 1))
|
|
done
|
|
|
|
rm -rf "$temp_dir"
|
|
echo "$file_type upload completed"
|
|
}
|
|
|
|
# Step 2: Upload JAR file
|
|
upload_file_in_chunks "$JAR_FILE" "jarFile"
|
|
|
|
# Step 3: Upload manifest file
|
|
upload_file_in_chunks "$MANIFEST_FILE" "manifest"
|
|
|
|
# Step 4: Start async deployment
|
|
echo ""
|
|
echo "=== Step 4: Starting deployment (async) ==="
|
|
FINALIZE_RESPONSE=$(curl -s -X POST "$API_BASE/upload/finalize?uploadSessionId=$SESSION_ID&async=true")
|
|
|
|
STATUS=$(echo $FINALIZE_RESPONSE | grep -o '"status":"[^"]*' | cut -d'"' -f4)
|
|
|
|
if [ "$STATUS" != "IN_PROGRESS" ]; then
|
|
echo "Failed to start deployment:"
|
|
echo "$FINALIZE_RESPONSE"
|
|
exit 1
|
|
fi
|
|
|
|
echo "Deployment started. Polling for status..."
|
|
|
|
# Step 5: Poll deployment status
|
|
POLL_INTERVAL=5 # seconds
|
|
MAX_WAIT=600 # 10 minutes max
|
|
|
|
elapsed=0
|
|
while [ $elapsed -lt $MAX_WAIT ]; do
|
|
sleep $POLL_INTERVAL
|
|
elapsed=$((elapsed + POLL_INTERVAL))
|
|
|
|
STATUS_RESPONSE=$(curl -s "$API_BASE/deployment/status/$SESSION_ID")
|
|
CURRENT_STATUS=$(echo $STATUS_RESPONSE | grep -o '"status":"[^"]*' | cut -d'"' -f4)
|
|
MESSAGE=$(echo $STATUS_RESPONSE | grep -o '"message":"[^"]*' | cut -d'"' -f4)
|
|
PROGRESS=$(echo $STATUS_RESPONSE | grep -o '"progress":[0-9]*' | cut -d':' -f2)
|
|
|
|
printf "\r[%3ds] Status: %-15s Progress: %3s%% - %s" \
|
|
"$elapsed" "$CURRENT_STATUS" "${PROGRESS:-0}" "$MESSAGE"
|
|
|
|
if [ "$CURRENT_STATUS" = "COMPLETED" ]; then
|
|
echo ""
|
|
echo ""
|
|
echo "=== Deployment successful! ==="
|
|
echo "$STATUS_RESPONSE" | jq '.' 2>/dev/null || echo "$STATUS_RESPONSE"
|
|
exit 0
|
|
elif [ "$CURRENT_STATUS" = "FAILED" ]; then
|
|
echo ""
|
|
echo ""
|
|
echo "=== Deployment failed ==="
|
|
echo "$STATUS_RESPONSE" | jq '.' 2>/dev/null || echo "$STATUS_RESPONSE"
|
|
exit 1
|
|
fi
|
|
done
|
|
|
|
echo ""
|
|
echo "=== Deployment timeout after ${MAX_WAIT}s ==="
|
|
echo "Check status manually: curl $API_BASE/deployment/status/$SESSION_ID"
|
|
exit 1
|
|
```
|
|
|
|
## Status Values
|
|
|
|
| Status | Description |
|
|
|--------|-------------|
|
|
| `PENDING` | Upload session created but deployment not started |
|
|
| `IN_PROGRESS` | Deployment is currently running |
|
|
| `COMPLETED` | Deployment finished successfully |
|
|
| `FAILED` | Deployment failed with errors |
|
|
|
|
## Nginx Configuration
|
|
|
|
With async deployment, nginx timeout is **not an issue**:
|
|
|
|
```nginx
|
|
server {
|
|
listen 80;
|
|
server_name your-app.example.com;
|
|
|
|
# Each chunk upload completes quickly
|
|
client_max_body_size 10m;
|
|
|
|
# Standard timeouts work fine now
|
|
proxy_read_timeout 60s; # Chunks complete in <5s
|
|
proxy_connect_timeout 10s;
|
|
proxy_send_timeout 60s;
|
|
|
|
location /api/cf/ {
|
|
proxy_pass http://cf-deployer-backend:8080;
|
|
proxy_set_header Host $host;
|
|
proxy_set_header X-Real-IP $remote_addr;
|
|
}
|
|
}
|
|
```
|
|
|
|
## For Backwards Compatibility
|
|
|
|
If you want synchronous deployment (will timeout on nginx!):
|
|
|
|
```bash
|
|
# Synchronous (old way - may timeout)
|
|
curl -X POST "$API_BASE/upload/finalize?uploadSessionId=$SESSION_ID&async=false"
|
|
```
|
|
|
|
**Default is async=true** to avoid timeout issues.
|
|
|
|
## Summary
|
|
|
|
✅ **Chunk uploads**: Complete in seconds, no timeout
|
|
✅ **Finalize endpoint**: Returns immediately (async), no timeout
|
|
✅ **Status polling**: Each poll completes in milliseconds, no timeout
|
|
✅ **Total solution**: Works with standard 30-second nginx timeout!
|