Deploy to Railway

Deploy LiteLLM to Railway with automatic provisioning and GitHub integration

Quick Deploy

Deploy LiteLLM to Railway with one click:

1. Click Deploy Button

   Click the “Deploy on Railway” button above to open the template.

2. Configure Template

   Railway will prompt you to set:

   • Project name
   • Environment variables (API keys)
   • Region preference
3. Deploy

   Railway automatically:

   • Creates a new project
   • Provisions PostgreSQL database
   • Builds and deploys LiteLLM
   • Generates a public URL
4. Access Service

   Once deployed, Railway provides:

   • Public URL: https://litellm-production.up.railway.app
   • Custom domain: Add your own domain in settings

Manual Railway Deployment

From GitHub Repository

1. Create New Project

   1. Go to Railway Dashboard
   2. Click New Project
   3. Select Deploy from GitHub repo
   4. Choose BerriAI/litellm or your fork
2. Configure Service

   Railway auto-detects Docker configuration from the repository.

   Detected settings:

   • Runtime: Docker
   • Build: Uses /Dockerfile
   • Start command: Automatic from ENTRYPOINT
3. Add PostgreSQL

   1. Click New Service in your project
   2. Select Database → PostgreSQL
   3. Railway provisions database and injects DATABASE_URL
4. Set Environment Variables

   Add required variables (see configuration section)

5. Deploy

   Railway automatically builds and deploys on push to main branch.

From Docker Image

1. Create New Project

   1. Click New Project
   2. Select Deploy a Docker Image
2. Specify Image

   ghcr.io/berriai/litellm:main-stable
   
3. Configure Command

   Add custom start command:

   --port $PORT --config /app/config.yaml
   

Database Setup

Railway PostgreSQL

Railway automatically sets DATABASE_URL when you add PostgreSQL to your project.

1. Add Database

   1. In your project, click New
   2. Select Database → PostgreSQL
   3. Railway creates database with:
      • 1GB storage (Hobby plan)
      • Automatic backups
      • Private networking
2. Connection String

   Railway automatically injects:

   DATABASE_URL=postgresql://postgres:password@host:5432/railway
   

   No manual configuration needed!

3. Enable Model Storage

   Add to your service:

   STORE_MODEL_IN_DB=True
   

Database Migrations

Migrations run automatically on container start using Prisma.

First deployment takes 15-20 seconds while Prisma runs migrations and generates the client.

Environment Configuration

Required Variables

In Railway service settings, add:

# Authentication
LITELLM_MASTER_KEY=sk-1234  # Generate a secure key

# Database (auto-injected if using Railway PostgreSQL)
# DATABASE_URL=postgresql://...  

# Enable UI model management
STORE_MODEL_IN_DB=True

Provider API Keys

OpenAI

OPENAI_API_KEY=sk-proj-...

Anthropic

ANTHROPIC_API_KEY=sk-ant-...

Google Gemini

GEMINI_API_KEY=AIza...

Azure OpenAI

AZURE_API_KEY=your-key
AZURE_API_BASE=https://your-resource.openai.azure.com
AZURE_API_VERSION=2024-02-15-preview

Optional Configuration

# Redis for caching
REDIS_HOST=redis.railway.internal
REDIS_PORT=6379
REDIS_PASSWORD=your-password

# Observability
LANGFUSE_PUBLIC_KEY=pk-...
LANGFUSE_SECRET_KEY=sk-...

# Debug mode
LITELLM_LOG=DEBUG

# Custom port (Railway sets this automatically)
# PORT=4000

Don’t set PORT manually - Railway injects this automatically and maps to your service.

Configuration with config.yaml

Mount Configuration File

1. Create Config in Repo

   Add config.yaml to your repository:

   config.yaml

   model_list:
     - model_name: gpt-4o
       litellm_params:
         model: gpt-4o
         api_key: os.environ/OPENAI_API_KEY
     
     - model_name: claude-sonnet-4
       litellm_params:
         model: anthropic/claude-sonnet-4-20250514
         api_key: os.environ/ANTHROPIC_API_KEY
   
   general_settings:
     master_key: os.environ/LITELLM_MASTER_KEY
     database_url: os.environ/DATABASE_URL
   
2. Update Start Command

   In Railway service settings:

   litellm --config /app/config.yaml --port $PORT
   
3. Redeploy

   Push to GitHub or trigger manual deploy in Railway dashboard.

Use Railway Variables

1. Create Variable Groups

   Organize variables by environment:

   • production - Live environment
   • staging - Testing environment
   • development - Local development
2. Reference in Config

   model_list:
     - model_name: gpt-4o
       litellm_params:
         model: gpt-4o
         api_key: os.environ/OPENAI_API_KEY
         api_base: os.environ/OPENAI_API_BASE  # Optional override
   
   general_settings:
     master_key: os.environ/LITELLM_MASTER_KEY
     database_url: os.environ/DATABASE_URL
   

Custom Domain

1. Generate Railway Domain

   Railway automatically provides:

   https://litellm-production-xxxx.up.railway.app
   
2. Add Custom Domain

   1. In service settings, go to Settings → Networking
   2. Click Add Custom Domain
   3. Enter your domain: api.yourdomain.com
3. Configure DNS

   Add a CNAME record:

   Type:  CNAME
   Name:  api
   Value: litellm-production-xxxx.up.railway.app
   TTL:   Auto
   
4. Wait for SSL

   Railway automatically provisions SSL certificate (2-5 minutes).

Railway provides automatic SSL for all domains using Let’s Encrypt.

Redis for Caching

Add Redis Service

1. Add Redis to Project

   1. Click New in your project
   2. Select Database → Redis
   3. Railway provisions Redis with:
      • 100MB storage (Hobby plan)
      • Private networking
      • Auto-injected connection details
2. Configure LiteLLM

   Railway auto-injects Redis variables:

   REDIS_URL=redis://default:[e-mail]:6379
   

   Or use individual variables:

   REDIS_HOST=redis.railway.internal
   REDIS_PORT=6379
   REDIS_PASSWORD=auto-generated
   
3. Enable Caching

   Update config.yaml:

   general_settings:
     cache: true
     redis_host: os.environ/REDIS_HOST
     redis_port: os.environ/REDIS_PORT
     redis_password: os.environ/REDIS_PASSWORD
   

Private Networking

Railway provides free private networking between services in the same project.

Services can communicate using:

# Private DNS (recommended)
postgresql://user:[e-mail]:5432/db
redis://redis.railway.internal:6379

# Public URLs (avoid for internal communication)
https://service-name.up.railway.app

Deployment Strategies

Continuous Deployment

Railway automatically deploys on Git push:

git add .
git commit -m "Update LiteLLM configuration"
git push origin main

Railway will:

1. Detect the push via webhook
2. Build Docker image
3. Run health checks
4. Deploy with zero downtime

Manual Deployment

1. Go to service in Railway dashboard
2. Click Deploy → Redeploy
3. Or Trigger Deploy for specific commit

Rollback

1. View Deployments

   Click Deployments tab in service dashboard

2. Select Previous Version

   Click on successful deployment

3. Rollback

   Click Redeploy this version

Monitoring and Observability

Railway Observability

Built-in monitoring includes:

• Logs: Real-time log streaming
• Metrics: CPU, Memory, Network, Disk
• Deployments: Build and deploy history
• Events: Service lifecycle events

View Logs

# In Railway dashboard
1. Click on your service
2. Go to "Logs" tab
3. Filter by:
   - Time range
   - Log level (ERROR, INFO, DEBUG)
   - Search text

Metrics Dashboard

Access in Metrics tab:

• CPU usage: % utilization over time
• Memory usage: MB used / available
• Network: Inbound/outbound traffic
• Request count: Requests per second

External Monitoring

Add monitoring services:

Langfuse

LANGFUSE_PUBLIC_KEY=pk-...
LANGFUSE_SECRET_KEY=sk-...
LANGFUSE_HOST=https://cloud.langfuse.com

Add to config.yaml:

litellm_settings:
  success_callback: ["langfuse"]

Datadog

USE_DDTRACE=true
DD_API_KEY=your-key
DD_SITE=datadoghq.com
DD_SERVICE=litellm-proxy
DD_ENV=production

Sentry

SENTRY_DSN=https://[e-mail]/...
SENTRY_ENVIRONMENT=production

Scaling

Vertical Scaling

Upgrade service resources:

1. Go to Settings → Resources
2. Select plan:
   • Hobby: 512MB RAM, 0.5 vCPU ($5/month)
   • Pro: 8GB RAM, 8 vCPU ($20/month)

Horizontal Scaling

Railway Hobby plan supports 1 instance only. Upgrade to Pro for multiple instances.

Pro plan features:

• Multiple service instances
• Load balancing
• Zero-downtime deployments
• Vertical and horizontal scaling

Cost Management

Pricing Overview

Hobby Plan ($5/month):
  - 512MB RAM, 0.5 vCPU per service
  - $5 free credit (covers small deployments)
  - 100GB bandwidth
  - Unlimited projects

Pro Plan (Usage-based):
  - $20/month base
  - Pay for resources used
  - 8GB RAM, 8 vCPU available
  - Priority support

Cost Optimization Tips

1. Use Shared Database

   Share PostgreSQL between staging/development environments.

2. Remove Unused Services

   Delete stopped or unnecessary services to avoid charges.

3. Set Resource Limits

   Configure memory and CPU limits to control costs:

   # In Railway service settings
   Memory Limit: 1GB
   CPU Limit: 1 vCPU
   
4. Use Sleep Mode

   Enable auto-sleep for development environments (Hobby plan only).

Troubleshooting

Build Failures

# Check build logs in Deployments tab

# Common issues:

# 1. Out of memory during build
Error: Docker build failed - OOM
Solution: Upgrade to Pro plan or optimize Dockerfile

# 2. npm/pip installation timeout
Error: Command timed out after 15 minutes
Solution: Use Docker layer caching, pre-build dependencies

# 3. File not found
Error: COPY failed: file not found: config.yaml
Solution: Ensure file is committed to Git

Service Won’t Start

# Check logs for startup errors

# Missing DATABASE_URL
Error: DATABASE_URL not set
Solution: Add PostgreSQL service to project

# Port binding error
Error: Port 4000 already in use
Solution: Use $PORT variable, don't hardcode port

# Config file error
Error: Failed to load config.yaml
Solution: Validate YAML syntax, check file path

Database Connection Issues

# Use private networking
# ✅ Correct:
DATABASE_URL=postgresql://user:[e-mail]:5432/db

# ❌ Wrong (public URL, slower and less secure):
DATABASE_URL=postgresql://user:[e-mail]:5432/db

# Test connection
railway run psql $DATABASE_URL

High Memory Usage

# Check metrics in Railway dashboard
# If memory usage > 80%:

# 1. Upgrade to larger instance
# 2. Enable Redis caching to reduce memory load
# 3. Optimize model configurations
# 4. Reduce concurrent requests

Railway CLI

Installation

# Install Railway CLI
npm install -g @railway/cli

# Login
railway login

# Link to project
railway link

Common Commands

# View logs
railway logs

# Open service in browser
railway open

# Run command with Railway environment
railway run litellm --config config.yaml

# Deploy manually
railway up

# View environment variables
railway variables

# Set environment variable
railway variables set OPENAI_API_KEY=sk-...

# SSH into service
railway shell

Comparison: Railway vs Render

Feature	Railway	Render
Free tier	$5 credit	90-day PostgreSQL trial
Pricing	Usage-based	Fixed tiers
Auto-scaling	Pro plan	Yes
Private networking	✅ Free	❌
CLI	✅	❌
Build speed	Fast (layer caching)	Medium
Zero-downtime deploys	✅	✅

Next Steps

Fly.io Deployment

Deploy closer to users with edge compute

Monitoring

Set up observability and alerts

Security

Secure your deployment

Performance

Optimize for production workloads