> ## Documentation Index
> Fetch the complete documentation index at: https://mintlify.com/BerriAI/litellm/llms.txt
> Use this file to discover all available pages before exploring further.

# Deploy to Fly.io

> Deploy LiteLLM globally with Fly.io's edge compute platform for low-latency worldwide access

## Overview

Fly.io deploys LiteLLM containers close to your users in 30+ regions worldwide, providing:

* **Low latency**: Edge compute near users
* **Global distribution**: Automatic multi-region deployment
* **Auto-scaling**: Scale based on demand
* **Private networking**: WireGuard-based secure networking

## Quick Start

<Steps>
  <Step title="Install Fly CLI">
    ```bash theme={null}
    # macOS
    brew install flyctl

    # Linux
    curl -L https://fly.io/install.sh | sh

    # Windows (PowerShell)
    pwsh -Command "iwr https://fly.io/install.ps1 -useb | iex"
    ```
  </Step>

  <Step title="Login to Fly">
    ```bash theme={null}
    flyctl auth login
    ```
  </Step>

  <Step title="Clone LiteLLM">
    ```bash theme={null}
    git clone https://github.com/BerriAI/litellm.git
    cd litellm
    ```
  </Step>

  <Step title="Launch Application">
    ```bash theme={null}
    flyctl launch --name litellm-proxy
    ```

    Fly will:

    * Detect Dockerfile
    * Create `fly.toml` configuration
    * Prompt for region selection
    * Ask to create PostgreSQL database
  </Step>

  <Step title="Set Environment Variables">
    ```bash theme={null}
    # Set master key
    flyctl secrets set LITELLM_MASTER_KEY=sk-1234

    # Set provider API keys
    flyctl secrets set OPENAI_API_KEY=sk-proj-...
    flyctl secrets set ANTHROPIC_API_KEY=sk-ant-...
    ```
  </Step>

  <Step title="Deploy">
    ```bash theme={null}
    flyctl deploy
    ```
  </Step>
</Steps>

## Configuration File

### Create fly.toml

Fly CLI generates a basic config, but customize for LiteLLM:

```toml fly.toml theme={null}
app = "litellm-proxy"
primary_region = "iad"  # Washington, DC

[build]
  dockerfile = "Dockerfile"

[http_service]
  internal_port = 4000
  force_https = true
  auto_stop_machines = true
  auto_start_machines = true
  min_machines_running = 2  # HA: always 2 instances
  processes = ["app"]

[[services]]
  protocol = "tcp"
  internal_port = 4000

  [[services.ports]]
    port = 80
    handlers = ["http"]
    force_https = true

  [[services.ports]]
    port = 443
    handlers = ["tls", "http"]

  [services.concurrency]
    type = "connections"
    hard_limit = 250
    soft_limit = 200

  [[services.tcp_checks]]
    interval = "15s"
    timeout = "10s"
    grace_period = "30s"

  [[services.http_checks]]
    interval = 30000
    timeout = 10000
    grace_period = "40s"
    method = "GET"
    path = "/health/liveliness"
    protocol = "http"

[env]
  STORE_MODEL_IN_DB = "True"
  PORT = "4000"

[[vm]]
  memory = "2gb"
  cpu_kind = "shared"
  cpus = 2
```

### Multi-Region Deployment

Deploy to multiple regions for global coverage:

```toml theme={null}
app = "litellm-proxy"
primary_region = "iad"

# Deploy to multiple regions
[regions]
  iad = {}  # Washington, DC (primary)
  lhr = {}  # London
  fra = {}  # Frankfurt
  sin = {}  # Singapore
  syd = {}  # Sydney

[http_service]
  internal_port = 4000
  force_https = true
  min_machines_running = 1  # Per region
  auto_stop_machines = false  # Keep running
```

Deploy:

```bash theme={null}
flyctl deploy --region iad,lhr,fra,sin,syd
```

<Tip>
  Fly.io routes requests to the nearest region automatically using Anycast DNS.
</Tip>

## Database Setup

### Fly PostgreSQL

<Steps>
  <Step title="Create Database Cluster">
    ```bash theme={null}
    flyctl postgres create \
      --name litellm-db \
      --region iad \
      --initial-cluster-size 2 \
      --vm-size shared-cpu-2x \
      --volume-size 10
    ```

    Options:

    * **Regions:** Deploy in 2-3 regions for HA
    * **VM size:** `shared-cpu-1x`, `shared-cpu-2x`, `performance-1x`
    * **Volume:** 10GB minimum, scales as needed
  </Step>

  <Step title="Attach Database to App">
    ```bash theme={null}
    flyctl postgres attach litellm-db --app litellm-proxy
    ```

    This sets `DATABASE_URL` environment variable automatically.
  </Step>

  <Step title="Verify Connection">
    ```bash theme={null}
    flyctl ssh console --app litellm-proxy

    # Inside container
    psql $DATABASE_URL
    ```
  </Step>
</Steps>

### Database Replication

For multi-region HA:

```bash theme={null}
# Create primary database
flyctl postgres create --name litellm-db-primary --region iad

# Add read replicas in other regions
flyctl postgres create \
  --name litellm-db-replica-lhr \
  --region lhr \
  --fork-from litellm-db-primary

flyctl postgres create \
  --name litellm-db-replica-sin \
  --region sin \
  --fork-from litellm-db-primary
```

Configure read replicas:

```toml theme={null}
[env]
  DATABASE_URL = "postgres://..."  # Primary (writes)
  DATABASE_REPLICA_URL = "postgres://..."  # Read replica
```

## Private Networking

Fly.io provides private IPv6 networking via WireGuard:

### Connect Services

```bash theme={null}
# Services communicate via internal DNS
litellm-proxy.internal       # Your app
litellm-db.internal           # PostgreSQL
litellm-redis.internal        # Redis (if deployed)
```

Connection strings:

```bash theme={null}
# PostgreSQL
DATABASE_URL=postgresql://user:pass@litellm-db.internal:5432/litellm

# Redis
REDIS_URL=redis://litellm-redis.internal:6379
```

### Connect External Services

For managed services (AWS RDS, etc.), use Fly Proxy:

```bash theme={null}
# Create proxy to external database
flyctl proxy postgres://user:pass@aws-rds-endpoint.com:5432/litellm
```

## Configuration File Deployment

### Method 1: Bake into Image

Add to Dockerfile:

```dockerfile theme={null}
FROM ghcr.io/berriai/litellm:main-stable

# Copy config into image
COPY config.yaml /app/config.yaml

ENTRYPOINT ["litellm"]
CMD ["--config", "/app/config.yaml", "--port", "4000"]
```

### Method 2: Fly Secrets

Store config as secret:

```bash theme={null}
# Create config.yaml
cat > config.yaml << 'EOF'
model_list:
  - model_name: gpt-4o
    litellm_params:
      model: gpt-4o
      api_key: os.environ/OPENAI_API_KEY
general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY
EOF

# Store as secret (base64 encoded)
flyctl secrets set CONFIG_YAML="$(cat config.yaml | base64)"
```

Update entrypoint to decode:

```bash theme={null}
echo $CONFIG_YAML | base64 -d > /app/config.yaml
litellm --config /app/config.yaml --port 4000
```

### Method 3: Fly Volumes

<Warning>
  Volumes are region-specific and not replicated. Use for local data only.
</Warning>

```bash theme={null}
# Create volume
flyctl volumes create litellm_data --region iad --size 10

# Mount in fly.toml
[[mounts]]
  source = "litellm_data"
  destination = "/data"
```

## Secrets Management

### Set Secrets

```bash theme={null}
# Set individual secrets
flyctl secrets set LITELLM_MASTER_KEY=sk-1234
flyctl secrets set OPENAI_API_KEY=sk-proj-...
flyctl secrets set ANTHROPIC_API_KEY=sk-ant-...

# Set from file
flyctl secrets import < secrets.txt
```

**secrets.txt:**

```bash theme={null}
LITELLM_MASTER_KEY=sk-1234
OPENAI_API_KEY=sk-proj-...
ANTHROPIC_API_KEY=sk-ant-...
AZURE_API_KEY=your-key
AZURE_API_BASE=https://your-resource.openai.azure.com
```

### View Secrets

```bash theme={null}
# List secret names (values hidden)
flyctl secrets list

# Unset secret
flyctl secrets unset OPENAI_API_KEY
```

## Custom Domains

<Steps>
  <Step title="Add Certificate">
    ```bash theme={null}
    flyctl certs create api.yourdomain.com
    ```
  </Step>

  <Step title="Get DNS Records">
    ```bash theme={null}
    flyctl certs show api.yourdomain.com
    ```

    Shows required DNS records (CNAME or A/AAAA).
  </Step>

  <Step title="Update DNS">
    Add to your DNS provider:

    ```
    Type:  CNAME
    Name:  api
    Value: litellm-proxy.fly.dev
    TTL:   Auto
    ```
  </Step>

  <Step title="Verify">
    ```bash theme={null}
    flyctl certs check api.yourdomain.com
    ```

    Fly automatically provisions SSL certificate.
  </Step>
</Steps>

## Scaling

### Autoscaling Configuration

```toml fly.toml theme={null}
[http_service]
  internal_port = 4000
  force_https = true
  
  # Autoscaling
  auto_stop_machines = true
  auto_start_machines = true
  min_machines_running = 2
  max_machines_running = 10

[http_service.concurrency]
  type = "requests"
  hard_limit = 100
  soft_limit = 80
```

### Manual Scaling

```bash theme={null}
# Scale to specific count
flyctl scale count 5

# Scale by region
flyctl scale count iad=2 lhr=2 sin=1

# Scale VM resources
flyctl scale vm shared-cpu-2x --memory 4096

# View current scale
flyctl scale show
```

### Resource Presets

| Preset           | CPUs | Memory    | Use Case          |
| ---------------- | ---- | --------- | ----------------- |
| `shared-cpu-1x`  | 1    | 256MB-2GB | Development       |
| `shared-cpu-2x`  | 2    | 512MB-4GB | Small production  |
| `shared-cpu-4x`  | 4    | 1GB-8GB   | Medium production |
| `performance-1x` | 1    | 2GB-8GB   | Dedicated CPU     |
| `performance-2x` | 2    | 4GB-16GB  | High performance  |

## Monitoring and Metrics

### Fly Metrics

```bash theme={null}
# View dashboard
flyctl dashboard

# Live monitoring
flyctl status

# Resource usage
flyctl metrics

# View logs
flyctl logs

# Follow logs
flyctl logs -f
```

### Prometheus Integration

Expose metrics to Fly's Prometheus:

```toml fly.toml theme={null}
[[metrics]]
  port = 4000
  path = "/metrics"
```

Access metrics:

```bash theme={null}
flyctl prometheus
```

### External Monitoring

Add observability providers:

```bash theme={null}
# Datadog
flyctl secrets set \
  USE_DDTRACE=true \
  DD_API_KEY=your-key \
  DD_SITE=datadoghq.com

# Langfuse
flyctl secrets set \
  LANGFUSE_PUBLIC_KEY=pk-... \
  LANGFUSE_SECRET_KEY=sk-... \
  LANGFUSE_HOST=https://cloud.langfuse.com
```

## Deployment Strategies

### Blue-Green Deployment

```bash theme={null}
# Deploy new version without replacing old
flyctl deploy --strategy bluegreen

# Verify new version
curl https://litellm-proxy.fly.dev/health

# If successful, Fly automatically switches traffic
# If issues, rollback:
flyctl releases rollback
```

### Canary Deployment

```bash theme={null}
# Deploy to 10% of machines
flyctl deploy --strategy canary:10

# Monitor metrics
flyctl metrics

# Promote to 100%
flyctl deploy --strategy immediate
```

### Rolling Deployment (Default)

```bash theme={null}
# Replace machines one at a time
flyctl deploy --strategy rolling
```

## High Availability Setup

### Multi-Region with Load Balancing

```toml fly.toml theme={null}
app = "litellm-proxy"
primary_region = "iad"

[http_service]
  internal_port = 4000
  force_https = true
  min_machines_running = 2  # Per region

# Deploy to multiple regions
[services.regions]
  iad = { min_machines = 2 }  # US East
  lhr = { min_machines = 2 }  # Europe
  nrt = { min_machines = 1 }  # Asia

[[vm]]
  memory = "2gb"
  cpu_kind = "performance"
  cpus = 2
```

Deploy:

```bash theme={null}
flyctl deploy --ha
```

### Health Checks

```toml theme={null}
[[services.http_checks]]
  interval = "30s"
  timeout = "10s"
  grace_period = "40s"
  method = "GET"
  path = "/health/liveliness"
  protocol = "http"
  
  # Headers
  [services.http_checks.headers]
    Authorization = "Bearer sk-1234"

[[services.tcp_checks]]
  interval = "15s"
  timeout = "10s"
  grace_period = "30s"
```

## Cost Optimization

### Pricing Overview

```yaml theme={null}
Fly.io Pricing (Pay-as-you-go):
  Compute:
    shared-cpu-1x: $0.0000008/sec ($2.07/month)
    shared-cpu-2x: $0.0000016/sec ($4.15/month)
  
  Memory:
    256MB: $0.0000002/sec ($0.52/month)
    Per GB: $0.0000008/sec ($2.07/month)
  
  Bandwidth:
    First 100GB: Free
    Over 100GB: $0.02/GB
  
  PostgreSQL:
    shared-cpu-1x + 10GB: ~$5/month
    performance-1x + 50GB: ~$30/month
```

### Free Allowance

```yaml theme={null}
Free Tier (Hobby Plan):
  - 3 shared-cpu-1x VMs (256MB each)
  - Up to 160GB storage
  - 100GB outbound bandwidth
  
Good for: Development and testing
```

### Optimization Tips

<Steps>
  <Step title="Use Autoscaling">
    Stop machines when idle:

    ```toml theme={null}
    auto_stop_machines = true
    auto_start_machines = true
    ```
  </Step>

  <Step title="Right-Size Resources">
    Start small, scale up based on metrics:

    ```bash theme={null}
    flyctl scale vm shared-cpu-1x --memory 1024
    ```
  </Step>

  <Step title="Use Regional Routing">
    Deploy only in regions with actual traffic.
  </Step>

  <Step title="Optimize Images">
    Use multi-stage builds, minimize layers:

    ```dockerfile theme={null}
    FROM cgr.dev/chainguard/wolfi-base AS runtime
    # Minimal runtime dependencies
    ```
  </Step>
</Steps>

## Troubleshooting

### Deployment Failures

```bash theme={null}
# Check deployment status
flyctl status

# View logs
flyctl logs

# Common issues:

# 1. Health check failing
Error: Health checks failed
Solution: Increase grace_period or fix /health endpoint

# 2. Out of memory
Error: OOM Killed
Solution: Increase VM memory
  flyctl scale vm shared-cpu-2x --memory 2048

# 3. Port binding error
Error: listen tcp :4000: bind: address already in use
Solution: Use PORT environment variable
```

### Database Connection Issues

```bash theme={null}
# Test database connectivity
flyctl ssh console
psql $DATABASE_URL

# Check database status
flyctl postgres db list --app litellm-db

# View database logs
flyctl logs --app litellm-db
```

### SSH into Machine

```bash theme={null}
# SSH into running machine
flyctl ssh console

# SSH into specific machine
flyctl ssh console -s <machine-id>

# Run command
flyctl ssh console -C "ls -la /app"
```

### Restart Machines

```bash theme={null}
# Restart all machines
flyctl apps restart

# Restart specific machine
flyctl machine restart <machine-id>
```

## Security Best Practices

### Network Security

```toml fly.toml theme={null}
# Force HTTPS
[http_service]
  force_https = true

# Internal services only
[[services]]
  internal_port = 4000
  protocol = "tcp"
  
  # No public ports
```

### Secrets Rotation

```bash theme={null}
# Update secrets without downtime
flyctl secrets set LITELLM_MASTER_KEY=new-key

# Fly automatically restarts machines with new secrets
```

### Private Networking

```bash theme={null}
# Keep sensitive services internal
# Use .internal DNS for service-to-service communication
DATABASE_URL=postgresql://user:pass@litellm-db.internal:5432/db
```

## Next Steps

<CardGroup cols={2}>
  <Card title="Monitoring" icon="chart-line" href="/deployment/monitoring">
    Set up comprehensive observability
  </Card>

  <Card title="High Availability" icon="server" href="/deployment/high-availability">
    Multi-region HA deployment patterns
  </Card>

  <Card title="Security" icon="shield" href="/deployment/security">
    Harden your deployment
  </Card>

  <Card title="Performance" icon="gauge" href="/deployment/performance">
    Optimize for global traffic
  </Card>
</CardGroup>