Quick Start Guide

Get started with LiteLLM Proxy in 5 minutes

Installation

Install LiteLLM with proxy dependencies:

pip install 'litellm[proxy]'

Basic Setup

1. Create Configuration File

   Create a config.yaml file with your model configurations:

   config.yaml

   model_list:
     - model_name: gpt-3.5-turbo
       litellm_params:
         model: openai/gpt-3.5-turbo
         api_key: os.environ/OPENAI_API_KEY
     
     - model_name: claude-3-sonnet
       litellm_params:
         model: anthropic/claude-3-5-sonnet-20241022
         api_key: os.environ/ANTHROPIC_API_KEY
   
   general_settings:
     master_key: sk-1234  # Change this to a secure key
   

   Never commit your master_key to version control. Use environment variables in production.

2. Set Environment Variables

   Export your provider API keys:

   export OPENAI_API_KEY="sk-..."
   export ANTHROPIC_API_KEY="sk-ant-..."
   
3. Start the Proxy

   Run the proxy server:

   litellm --config config.yaml
   

   The proxy will start on http://0.0.0.0:4000 by default.

   Custom Port

   litellm --config config.yaml --port 8000
   
4. Test the Proxy

   Make a test request using curl:

   curl -X POST 'http://localhost:4000/chat/completions' \
     -H 'Content-Type: application/json' \
     -H 'Authorization: Bearer sk-1234' \
     -d '{
       "model": "gpt-3.5-turbo",
       "messages": [
         {"role": "user", "content": "Hello, how are you?"}
       ]
     }'
   

Generate a Virtual Key

Create an API key for your application:

curl -X POST 'http://localhost:4000/key/generate' \
  -H 'Content-Type: application/json' \
  -H 'Authorization: Bearer sk-1234' \
  -d '{
    "models": ["gpt-3.5-turbo", "claude-3-sonnet"],
    "max_budget": 10.0,
    "duration": "30d"
  }'

Response:

{
  "key": "sk-1234567890abcdef",
  "key_name": null,
  "expires": "2024-04-15T10:30:00Z",
  "models": ["gpt-3.5-turbo", "claude-3-sonnet"],
  "max_budget": 10.0
}

Use the Virtual Key

Now use the generated key instead of the master key:

Python

import openai

client = openai.OpenAI(
    api_key="sk-1234567890abcdef",
    base_url="http://localhost:4000"
)

response = client.chat.completions.create(
    model="gpt-3.5-turbo",
    messages=[{"role": "user", "content": "Hello!"}]
)
print(response.choices[0].message.content)

Access the Admin UI

Open your browser to http://localhost:4000/ui to access the admin dashboard.

Login with your master key (sk-1234) to:

• View all virtual keys
• Create and manage keys
• Monitor usage and spending
• View request logs
• Manage teams and users

Load Balancing Example

Configure multiple deployments for automatic load balancing:

config.yaml

model_list:
  # OpenAI deployment 1
  - model_name: gpt-4
    litellm_params:
      model: openai/gpt-4
      api_key: os.environ/OPENAI_API_KEY
      rpm: 480

  # OpenAI deployment 2 (backup)
  - model_name: gpt-4
    litellm_params:
      model: openai/gpt-4
      api_key: os.environ/OPENAI_API_KEY_2
      rpm: 480

  # Azure fallback
  - model_name: gpt-4
    litellm_params:
      model: azure/gpt-4
      api_key: os.environ/AZURE_API_KEY
      api_base: os.environ/AZURE_API_BASE
      api_version: "2024-02-15-preview"

router_settings:
  routing_strategy: usage-based-routing-v2
  enable_pre_call_checks: true

litellm_settings:
  num_retries: 3
  context_window_fallbacks: [{"gpt-4": ["claude-3-opus"]}]

Now requests to gpt-4 will automatically load balance across all three deployments, with fallback to Claude if all GPT-4 deployments fail.

Database Setup (Optional)

For persistent storage of keys, users, and spend data:

1. Set up PostgreSQL

   # Using Docker
   docker run -d \
     --name litellm-db \
     -e POSTGRES_DB=litellm \
     -e POSTGRES_USER=llmproxy \
     -e POSTGRES_PASSWORD=dbpassword9090 \
     -p 5432:5432 \
     postgres:16
   
2. Configure Database URL

   export DATABASE_URL="postgresql://llmproxy:dbpassword9090@localhost:5432/litellm"
   
3. Enable DB Storage

   Update your config.yaml:

   general_settings:
     master_key: sk-1234
     store_model_in_db: true
     database_url: os.environ/DATABASE_URL
   
4. Start Proxy

   The proxy will automatically run database migrations on startup:

   litellm --config config.yaml
   

Health Check

Verify the proxy is running:

curl http://localhost:4000/health

Response:

{
  "status": "healthy",
  "db": "connected",
  "cache": "connected",
  "litellm_version": "1.x.x"
}

Next Steps

Docker Deployment

Deploy with Docker for production

Configuration Options

Explore all configuration settings

Virtual Keys

Advanced key management

Budget Alerts

Set up budget monitoring

Common Issues

Port Already in Use

If port 4000 is already in use, specify a different port:

litellm --config config.yaml --port 8080

API Key Not Found

Make sure environment variables are exported:

echo $OPENAI_API_KEY  # Should show your key

Database Connection Failed

Verify the DATABASE_URL format:

postgresql://username:password@host:port/database