init
This commit is contained in:
465
DEPLOYMENT.md
Normal file
465
DEPLOYMENT.md
Normal file
@@ -0,0 +1,465 @@
|
||||
# Deployment Guide
|
||||
|
||||
This guide covers deploying the Test Artifact Data Lake in various environments.
|
||||
|
||||
## Table of Contents
|
||||
- [Local Development](#local-development)
|
||||
- [Docker Compose](#docker-compose)
|
||||
- [Kubernetes/Helm](#kuberneteshelm)
|
||||
- [AWS Deployment](#aws-deployment)
|
||||
- [Self-Hosted Deployment](#self-hosted-deployment)
|
||||
- [GitLab CI/CD](#gitlab-cicd)
|
||||
|
||||
---
|
||||
|
||||
## Local Development
|
||||
|
||||
### Prerequisites
|
||||
- Python 3.11+
|
||||
- PostgreSQL 15+
|
||||
- MinIO or AWS S3 access
|
||||
|
||||
### Steps
|
||||
|
||||
1. **Create virtual environment:**
|
||||
```bash
|
||||
python -m venv venv
|
||||
source venv/bin/activate # On Windows: venv\Scripts\activate
|
||||
```
|
||||
|
||||
2. **Install dependencies:**
|
||||
```bash
|
||||
pip install -r requirements.txt
|
||||
```
|
||||
|
||||
3. **Set up PostgreSQL:**
|
||||
```bash
|
||||
createdb datalake
|
||||
```
|
||||
|
||||
4. **Configure environment:**
|
||||
```bash
|
||||
cp .env.example .env
|
||||
# Edit .env with your configuration
|
||||
```
|
||||
|
||||
5. **Run the application:**
|
||||
```bash
|
||||
python -m uvicorn app.main:app --reload
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Docker Compose
|
||||
|
||||
### Quick Start
|
||||
|
||||
1. **Start all services:**
|
||||
```bash
|
||||
docker-compose up -d
|
||||
```
|
||||
|
||||
2. **Check logs:**
|
||||
```bash
|
||||
docker-compose logs -f api
|
||||
```
|
||||
|
||||
3. **Stop services:**
|
||||
```bash
|
||||
docker-compose down
|
||||
```
|
||||
|
||||
### Services Included
|
||||
- PostgreSQL (port 5432)
|
||||
- MinIO (port 9000, console 9001)
|
||||
- API (port 8000)
|
||||
|
||||
### Customization
|
||||
|
||||
Edit `docker-compose.yml` to:
|
||||
- Change port mappings
|
||||
- Adjust resource limits
|
||||
- Add environment variables
|
||||
- Configure volumes
|
||||
|
||||
---
|
||||
|
||||
## Kubernetes/Helm
|
||||
|
||||
### Prerequisites
|
||||
- Kubernetes cluster (1.24+)
|
||||
- Helm 3.x
|
||||
- kubectl configured
|
||||
|
||||
### Installation
|
||||
|
||||
1. **Add dependencies (if using PostgreSQL/MinIO from Bitnami):**
|
||||
```bash
|
||||
helm repo add bitnami https://charts.bitnami.com/bitnami
|
||||
helm repo update
|
||||
```
|
||||
|
||||
2. **Install with default values:**
|
||||
```bash
|
||||
helm install datalake ./helm \
|
||||
--namespace datalake \
|
||||
--create-namespace
|
||||
```
|
||||
|
||||
3. **Custom installation:**
|
||||
```bash
|
||||
helm install datalake ./helm \
|
||||
--namespace datalake \
|
||||
--create-namespace \
|
||||
--set image.repository=your-registry/datalake \
|
||||
--set image.tag=1.0.0 \
|
||||
--set ingress.enabled=true \
|
||||
--set ingress.hosts[0].host=datalake.yourdomain.com
|
||||
```
|
||||
|
||||
### Configuration Options
|
||||
|
||||
**Image:**
|
||||
```bash
|
||||
--set image.repository=your-registry/datalake
|
||||
--set image.tag=1.0.0
|
||||
--set image.pullPolicy=Always
|
||||
```
|
||||
|
||||
**Resources:**
|
||||
```bash
|
||||
--set resources.requests.cpu=1000m
|
||||
--set resources.requests.memory=1Gi
|
||||
--set resources.limits.cpu=2000m
|
||||
--set resources.limits.memory=2Gi
|
||||
```
|
||||
|
||||
**Autoscaling:**
|
||||
```bash
|
||||
--set autoscaling.enabled=true
|
||||
--set autoscaling.minReplicas=3
|
||||
--set autoscaling.maxReplicas=10
|
||||
--set autoscaling.targetCPUUtilizationPercentage=80
|
||||
```
|
||||
|
||||
**Ingress:**
|
||||
```bash
|
||||
--set ingress.enabled=true
|
||||
--set ingress.className=nginx
|
||||
--set ingress.hosts[0].host=datalake.example.com
|
||||
--set ingress.hosts[0].paths[0].path=/
|
||||
--set ingress.hosts[0].paths[0].pathType=Prefix
|
||||
```
|
||||
|
||||
### Upgrade
|
||||
|
||||
```bash
|
||||
helm upgrade datalake ./helm \
|
||||
--namespace datalake \
|
||||
--set image.tag=1.1.0
|
||||
```
|
||||
|
||||
### Uninstall
|
||||
|
||||
```bash
|
||||
helm uninstall datalake --namespace datalake
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## AWS Deployment
|
||||
|
||||
### Using AWS S3 Storage
|
||||
|
||||
1. **Create S3 bucket:**
|
||||
```bash
|
||||
aws s3 mb s3://your-test-artifacts-bucket
|
||||
```
|
||||
|
||||
2. **Create IAM user with S3 access:**
|
||||
```bash
|
||||
aws iam create-user --user-name datalake-service
|
||||
aws iam attach-user-policy --user-name datalake-service \
|
||||
--policy-arn arn:aws:iam::aws:policy/AmazonS3FullAccess
|
||||
```
|
||||
|
||||
3. **Generate access keys:**
|
||||
```bash
|
||||
aws iam create-access-key --user-name datalake-service
|
||||
```
|
||||
|
||||
4. **Deploy with Helm:**
|
||||
```bash
|
||||
helm install datalake ./helm \
|
||||
--namespace datalake \
|
||||
--create-namespace \
|
||||
--set config.storageBackend=s3 \
|
||||
--set aws.enabled=true \
|
||||
--set aws.accessKeyId=YOUR_ACCESS_KEY \
|
||||
--set aws.secretAccessKey=YOUR_SECRET_KEY \
|
||||
--set aws.region=us-east-1 \
|
||||
--set aws.bucketName=your-test-artifacts-bucket \
|
||||
--set minio.enabled=false
|
||||
```
|
||||
|
||||
### Using EKS
|
||||
|
||||
1. **Create EKS cluster:**
|
||||
```bash
|
||||
eksctl create cluster \
|
||||
--name datalake-cluster \
|
||||
--region us-east-1 \
|
||||
--nodegroup-name standard-workers \
|
||||
--node-type t3.medium \
|
||||
--nodes 3
|
||||
```
|
||||
|
||||
2. **Configure kubectl:**
|
||||
```bash
|
||||
aws eks update-kubeconfig --name datalake-cluster --region us-east-1
|
||||
```
|
||||
|
||||
3. **Deploy application:**
|
||||
```bash
|
||||
helm install datalake ./helm \
|
||||
--namespace datalake \
|
||||
--create-namespace \
|
||||
--set config.storageBackend=s3
|
||||
```
|
||||
|
||||
### Using RDS for PostgreSQL
|
||||
|
||||
```bash
|
||||
helm install datalake ./helm \
|
||||
--namespace datalake \
|
||||
--create-namespace \
|
||||
--set postgresql.enabled=false \
|
||||
--set config.databaseUrl="postgresql://user:pass@your-rds-endpoint:5432/datalake"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Self-Hosted Deployment
|
||||
|
||||
### Using MinIO
|
||||
|
||||
1. **Deploy MinIO:**
|
||||
```bash
|
||||
helm install minio bitnami/minio \
|
||||
--namespace datalake \
|
||||
--create-namespace \
|
||||
--set auth.rootUser=admin \
|
||||
--set auth.rootPassword=adminpassword \
|
||||
--set persistence.size=100Gi
|
||||
```
|
||||
|
||||
2. **Deploy application:**
|
||||
```bash
|
||||
helm install datalake ./helm \
|
||||
--namespace datalake \
|
||||
--set config.storageBackend=minio \
|
||||
--set minio.enabled=false \
|
||||
--set minio.endpoint=minio:9000 \
|
||||
--set minio.accessKey=admin \
|
||||
--set minio.secretKey=adminpassword
|
||||
```
|
||||
|
||||
### On-Premise Kubernetes
|
||||
|
||||
1. **Prepare persistent volumes:**
|
||||
```yaml
|
||||
apiVersion: v1
|
||||
kind: PersistentVolume
|
||||
metadata:
|
||||
name: datalake-postgres-pv
|
||||
spec:
|
||||
capacity:
|
||||
storage: 20Gi
|
||||
accessModes:
|
||||
- ReadWriteOnce
|
||||
hostPath:
|
||||
path: /data/postgres
|
||||
```
|
||||
|
||||
2. **Deploy with local storage:**
|
||||
```bash
|
||||
helm install datalake ./helm \
|
||||
--namespace datalake \
|
||||
--create-namespace \
|
||||
--set postgresql.persistence.storageClass=local-storage \
|
||||
--set minio.persistence.storageClass=local-storage
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## GitLab CI/CD
|
||||
|
||||
### Setup
|
||||
|
||||
1. **Configure GitLab variables:**
|
||||
|
||||
Go to Settings → CI/CD → Variables and add:
|
||||
|
||||
| Variable | Description | Protected | Masked |
|
||||
|----------|-------------|-----------|---------|
|
||||
| `CI_REGISTRY_USER` | Docker registry username | No | No |
|
||||
| `CI_REGISTRY_PASSWORD` | Docker registry password | No | Yes |
|
||||
| `KUBE_CONFIG_DEV` | Base64 kubeconfig for dev | No | Yes |
|
||||
| `KUBE_CONFIG_STAGING` | Base64 kubeconfig for staging | Yes | Yes |
|
||||
| `KUBE_CONFIG_PROD` | Base64 kubeconfig for prod | Yes | Yes |
|
||||
|
||||
2. **Encode kubeconfig:**
|
||||
```bash
|
||||
cat ~/.kube/config | base64 -w 0
|
||||
```
|
||||
|
||||
### Pipeline Stages
|
||||
|
||||
1. **Test**: Runs on all branches and MRs
|
||||
2. **Build**: Builds Docker image on main/develop/tags
|
||||
3. **Deploy**: Manual deployment to dev/staging/prod
|
||||
|
||||
### Deployment Flow
|
||||
|
||||
**Development:**
|
||||
```bash
|
||||
git push origin develop
|
||||
# Manually trigger deploy:dev job in GitLab
|
||||
```
|
||||
|
||||
**Staging:**
|
||||
```bash
|
||||
git push origin main
|
||||
# Manually trigger deploy:staging job in GitLab
|
||||
```
|
||||
|
||||
**Production:**
|
||||
```bash
|
||||
git tag v1.0.0
|
||||
git push origin v1.0.0
|
||||
# Manually trigger deploy:prod job in GitLab
|
||||
```
|
||||
|
||||
### Customizing Pipeline
|
||||
|
||||
Edit `.gitlab-ci.yml` to:
|
||||
- Add more test stages
|
||||
- Change deployment namespaces
|
||||
- Adjust Helm values per environment
|
||||
- Add security scanning
|
||||
- Configure rollback procedures
|
||||
|
||||
---
|
||||
|
||||
## Monitoring
|
||||
|
||||
### Health Checks
|
||||
|
||||
```bash
|
||||
# Kubernetes
|
||||
kubectl get pods -n datalake
|
||||
kubectl logs -f -n datalake deployment/datalake
|
||||
|
||||
# Direct
|
||||
curl http://localhost:8000/health
|
||||
```
|
||||
|
||||
### Metrics
|
||||
|
||||
Add Prometheus monitoring:
|
||||
```bash
|
||||
helm install datalake ./helm \
|
||||
--set metrics.enabled=true \
|
||||
--set serviceMonitor.enabled=true
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Backup and Recovery
|
||||
|
||||
### Database Backup
|
||||
|
||||
```bash
|
||||
# PostgreSQL
|
||||
kubectl exec -n datalake deployment/datalake-postgresql -- \
|
||||
pg_dump -U user datalake > backup.sql
|
||||
|
||||
# Restore
|
||||
kubectl exec -i -n datalake deployment/datalake-postgresql -- \
|
||||
psql -U user datalake < backup.sql
|
||||
```
|
||||
|
||||
### Storage Backup
|
||||
|
||||
**S3:**
|
||||
```bash
|
||||
aws s3 sync s3://your-bucket s3://backup-bucket
|
||||
```
|
||||
|
||||
**MinIO:**
|
||||
```bash
|
||||
mc mirror minio/test-artifacts backup/test-artifacts
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Pod Not Starting
|
||||
```bash
|
||||
kubectl describe pod -n datalake <pod-name>
|
||||
kubectl logs -n datalake <pod-name>
|
||||
```
|
||||
|
||||
### Database Connection Issues
|
||||
```bash
|
||||
kubectl exec -it -n datalake deployment/datalake -- \
|
||||
psql $DATABASE_URL
|
||||
```
|
||||
|
||||
### Storage Issues
|
||||
```bash
|
||||
# Check MinIO
|
||||
kubectl port-forward -n datalake svc/minio 9000:9000
|
||||
# Access http://localhost:9000
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Security Considerations
|
||||
|
||||
1. **Use secrets management:**
|
||||
- Kubernetes Secrets
|
||||
- AWS Secrets Manager
|
||||
- HashiCorp Vault
|
||||
|
||||
2. **Enable TLS:**
|
||||
- Configure ingress with TLS certificates
|
||||
- Use cert-manager for automatic certificates
|
||||
|
||||
3. **Network policies:**
|
||||
- Restrict pod-to-pod communication
|
||||
- Limit external access
|
||||
|
||||
4. **RBAC:**
|
||||
- Configure Kubernetes RBAC
|
||||
- Limit service account permissions
|
||||
|
||||
---
|
||||
|
||||
## Performance Tuning
|
||||
|
||||
### Database
|
||||
- Increase connection pool size
|
||||
- Add database indexes
|
||||
- Configure autovacuum
|
||||
|
||||
### API
|
||||
- Increase replica count
|
||||
- Configure horizontal pod autoscaling
|
||||
- Adjust resource requests/limits
|
||||
|
||||
### Storage
|
||||
- Use CDN for frequently accessed files
|
||||
- Configure S3 Transfer Acceleration
|
||||
- Optimize MinIO deployment
|
||||
Reference in New Issue
Block a user