Docker Deployment

Deploy LiteLLM using Docker and Docker Compose with PostgreSQL and monitoring

Quick Start

The fastest way to deploy LiteLLM is using Docker Compose with the official image.

1. Pull the Docker Image

   docker pull ghcr.io/berriai/litellm:main-latest
   

   Stable releases use the -stable tag (undergoes 12-hour load tests):

   docker pull ghcr.io/berriai/litellm:main-stable
   
2. Create Configuration File

   Create a config.yaml file with your model configuration:

   config.yaml

   model_list:
     - model_name: gpt-4o
       litellm_params:
         model: gpt-4o
         api_key: os.environ/OPENAI_API_KEY
     
     - model_name: claude-sonnet-4
       litellm_params:
         model: anthropic/claude-sonnet-4-20250514
         api_key: os.environ/ANTHROPIC_API_KEY
   
   general_settings:
     master_key: os.environ/LITELLM_MASTER_KEY
     database_url: os.environ/DATABASE_URL
   
3. Run with Docker Compose

   Use the provided docker-compose.yml for a complete setup with PostgreSQL and Prometheus:

   docker-compose.yml

   services:
     litellm:
       image: ghcr.io/berriai/litellm:main-stable
       ports:
         - "4000:4000"
       environment:
         DATABASE_URL: "postgresql://llmproxy:dbpassword9090@db:5432/litellm"
         STORE_MODEL_IN_DB: "True" # Allows adding models via UI
       volumes:
         - ./config.yaml:/app/config.yaml
       command:
         - "--config=/app/config.yaml"
       depends_on:
         - db
       healthcheck:
         test:
           - CMD-SHELL
           - python3 -c "import urllib.request; urllib.request.urlopen('http://localhost:4000/health/liveliness')"
         interval: 30s
         timeout: 10s
         retries: 3
         start_period: 40s
   
     db:
       image: postgres:16
       restart: always
       environment:
         POSTGRES_DB: litellm
         POSTGRES_USER: llmproxy
         POSTGRES_PASSWORD: dbpassword9090
       ports:
         - "5432:5432"
       volumes:
         - postgres_data:/var/lib/postgresql/data
       healthcheck:
         test: ["CMD-SHELL", "pg_isready -d litellm -U llmproxy"]
         interval: 1s
         timeout: 5s
         retries: 10
   
     prometheus:
       image: prom/prometheus
       volumes:
         - prometheus_data:/prometheus
         - ./prometheus.yml:/etc/prometheus/prometheus.yml
       ports:
         - "9090:9090"
       command:
         - "--config.file=/etc/prometheus/prometheus.yml"
         - "--storage.tsdb.path=/prometheus"
         - "--storage.tsdb.retention.time=15d"
       restart: always
   
   volumes:
     prometheus_data:
       driver: local
     postgres_data:
       name: litellm_postgres_data
   

   Start all services:

   docker-compose up -d
   
4. Verify Deployment

   Check health status:

   curl http://localhost:4000/health/liveliness
   

   Access the admin UI at http://localhost:4000/ui

Docker Image Details

Multi-Stage Build

The Dockerfile uses a multi-stage build for security and size optimization:

# Base images from Chainguard (minimal, security-focused)
ARG LITELLM_BUILD_IMAGE=cgr.dev/chainguard/wolfi-base
ARG LITELLM_RUNTIME_IMAGE=cgr.dev/chainguard/wolfi-base

# Builder stage
FROM $LITELLM_BUILD_IMAGE AS builder
WORKDIR /app

# Install build dependencies
RUN apk add --no-cache bash gcc py3-pip python3 python3-dev openssl openssl-dev
RUN python -m pip install build

# Copy source and build
COPY . .
RUN ./docker/build_admin_ui.sh
RUN rm -rf dist/* && python -m build
RUN pip install dist/*.whl
RUN pip wheel --no-cache-dir --wheel-dir=/wheels/ -r requirements.txt

# Runtime stage
FROM $LITELLM_RUNTIME_IMAGE AS runtime
WORKDIR /app

# Install runtime dependencies (includes Node.js for Prisma)
RUN apk add --no-cache bash openssl tzdata nodejs npm python3 py3-pip libsndfile

# Copy built artifacts
COPY --from=builder /app/dist/*.whl .
COPY --from=builder /wheels/ /wheels/
RUN pip install *.whl /wheels/* --no-index --find-links=/wheels/

# Generate Prisma client
RUN prisma generate --schema=./litellm/proxy/schema.prisma

EXPOSE 4000/tcp
ENTRYPOINT ["docker/prod_entrypoint.sh"]
CMD ["--port", "4000"]

Environment Variables

Security Best Practice: Use environment variables for sensitive data instead of hardcoding in config files.

Variable	Description	Required
DATABASE_URL	PostgreSQL connection string	Yes (for production)
LITELLM_MASTER_KEY	Master key for proxy authentication	Recommended
STORE_MODEL_IN_DB	Enable model management via UI	No
OPENAI_API_KEY	OpenAI API key	If using OpenAI
ANTHROPIC_API_KEY	Anthropic API key	If using Anthropic
USE_DDTRACE	Enable Datadog tracing	No
SEPARATE_HEALTH_APP	Run health checks on separate port	No

Standalone Docker Run

For simple deployments without Docker Compose:

# Without database (in-memory)
docker run -d \
  --name litellm \
  -p 4000:4000 \
  -e LITELLM_MASTER_KEY=sk-1234 \
  -e OPENAI_API_KEY=your-key \
  ghcr.io/berriai/litellm:main-stable \
  --model gpt-4o

# With external database
docker run -d \
  --name litellm \
  -p 4000:4000 \
  -e DATABASE_URL=postgresql://user:pass@host:5432/litellm \
  -e LITELLM_MASTER_KEY=sk-1234 \
  -v $(pwd)/config.yaml:/app/config.yaml \
  ghcr.io/berriai/litellm:main-stable \
  --config /app/config.yaml

Database Setup

LiteLLM uses Prisma with PostgreSQL for persistence.

Database Schema

The schema includes tables for:

• API Keys (LiteLLM_VerificationToken) - Virtual keys with budgets and rate limits
• Teams (LiteLLM_TeamTable) - Team management and spend tracking
• Users (LiteLLM_UserTable) - User authentication and authorization
• Models (LiteLLM_ProxyModelTable) - Model configurations
• Spend Logs (LiteLLM_SpendLogs) - Request logs and cost tracking
• Organizations (LiteLLM_OrganizationTable) - Multi-tenant support

Migrations

Migrations run automatically on container startup. The proxy takes 15-20 seconds to fully initialize.

The entrypoint script (docker/prod_entrypoint.sh) handles migrations:

#!/bin/sh
if [ "$SEPARATE_HEALTH_APP" = "1" ]; then
    export LITELLM_ARGS="$@"
    exec supervisord -c /etc/supervisord.conf
fi

if [ "$USE_DDTRACE" = "true" ]; then
    export DD_TRACE_OPENAI_ENABLED="False"
    exec ddtrace-run litellm "$@"
else
    exec litellm "$@"
fi

Prisma generates the client and applies migrations during the build process.

Prometheus Configuration

Create prometheus.yml for metrics collection:

prometheus.yml

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'litellm'
    static_configs:
      - targets: ['litellm:4000']

LiteLLM exposes metrics at /metrics endpoint.

Volume Persistence

Always use named volumes for production databases to prevent data loss.

volumes:
  postgres_data:
    name: litellm_postgres_data
    driver: local
  prometheus_data:
    driver: local

Health Checks

LiteLLM provides multiple health check endpoints:

• /health/liveliness - Basic service health
• /health/readiness - Ready to accept traffic
• /health - Combined health status

Health check configuration:

healthcheck:
  test: ["CMD-SHELL", "python3 -c \"import urllib.request; urllib.request.urlopen('http://localhost:4000/health/liveliness')\""]
  interval: 30s
  timeout: 10s
  retries: 3
  start_period: 40s  # Allow time for migrations

Upgrading

# Pull latest stable image
docker-compose pull

# Restart services (migrations run automatically)
docker-compose up -d

# Check logs
docker-compose logs -f litellm

Troubleshooting

Container won’t start

# Check logs
docker logs litellm

# Common issues:
# 1. Database not ready - increase start_period in healthcheck
# 2. Invalid config.yaml - validate YAML syntax
# 3. Missing environment variables - check .env file

Database connection errors

# Test database connectivity
docker exec -it litellm_db psql -U llmproxy -d litellm

# Check DATABASE_URL format:
# postgresql://username:password@hostname:5432/database

Performance issues

# Check resource usage
docker stats litellm

# Increase container resources
docker run --memory=2g --cpus=2 ...

Next Steps

Kubernetes

Scale with Kubernetes and Helm charts

Monitoring

Set up observability and alerting

Security

Secure your deployment

High Availability

Deploy for production at scale