> ## Documentation Index
> Fetch the complete documentation index at: https://mintlify.com/BerriAI/litellm/llms.txt
> Use this file to discover all available pages before exploring further.

# Vision (Multimodal)

> Process images and vision inputs across providers

## Overview

Vision models can analyze images alongside text. LiteLLM provides a unified interface for vision capabilities across multiple providers including OpenAI, Anthropic, Google, and more.

## Quick Start

```python theme={null}
from litellm import completion

response = completion(
    model="gpt-4o",
    messages=[{
        "role": "user",
        "content": [
            {"type": "text", "text": "What's in this image?"},
            {"type": "image_url", "image_url": {"url": "https://example.com/image.jpg"}}
        ]
    }]
)

print(response.choices[0].message.content)
```

## Image Input Methods

<Tabs>
  <Tab title="URL">
    Reference images via URL.

    ```python theme={null}
    from litellm import completion

    response = completion(
        model="gpt-4o",
        messages=[{
            "role": "user",
            "content": [
                {"type": "text", "text": "Describe this image"},
                {
                    "type": "image_url",
                    "image_url": {
                        "url": "https://example.com/photo.jpg"
                    }
                }
            ]
        }]
    )
    ```
  </Tab>

  <Tab title="Base64">
    Embed images as base64 data.

    ```python theme={null}
    import base64
    from litellm import completion

    # Read and encode image
    with open("image.jpg", "rb") as image_file:
        base64_image = base64.b64encode(image_file.read()).decode('utf-8')

    response = completion(
        model="gpt-4o",
        messages=[{
            "role": "user",
            "content": [
                {"type": "text", "text": "What's in this image?"},
                {
                    "type": "image_url",
                    "image_url": {
                        "url": f"data:image/jpeg;base64,{base64_image}"
                    }
                }
            ]
        }]
    )
    ```
  </Tab>

  <Tab title="Local File">
    LiteLLM can automatically convert local files.

    ```python theme={null}
    from litellm import completion
    import litellm

    response = completion(
        model="gpt-4o",
        messages=[{
            "role": "user",
            "content": [
                {"type": "text", "text": "Analyze this image"},
                {
                    "type": "image_url",
                    "image_url": {"url": "file://path/to/image.jpg"}
                }
            ]
        }]
    )
    ```
  </Tab>
</Tabs>

## Multiple Images

Process multiple images in a single request.

```python theme={null}
from litellm import completion

response = completion(
    model="gpt-4o",
    messages=[{
        "role": "user",
        "content": [
            {"type": "text", "text": "Compare these two images"},
            {"type": "image_url", "image_url": {"url": "https://example.com/image1.jpg"}},
            {"type": "image_url", "image_url": {"url": "https://example.com/image2.jpg"}}
        ]
    }]
)

print(response.choices[0].message.content)
```

## Provider Support

<Tabs>
  <Tab title="OpenAI">
    GPT-4o and GPT-4 Turbo with vision.

    ```python theme={null}
    from litellm import completion

    # GPT-4o - Best vision model
    response = completion(
        model="gpt-4o",
        messages=[{
            "role": "user",
            "content": [
                {"type": "text", "text": "What's in this image?"},
                {"type": "image_url", "image_url": {"url": "https://..."}}
            ]
        }]
    )

    # GPT-4o-mini - Faster and cheaper
    response = completion(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": [...]}]
    )
    ```
  </Tab>

  <Tab title="Anthropic">
    Claude models with vision support.

    ```python theme={null}
    response = completion(
        model="anthropic/claude-3.5-sonnet",
        messages=[{
            "role": "user",
            "content": [
                {"type": "text", "text": "Describe this image"},
                {"type": "image_url", "image_url": {"url": "https://..."}}
            ]
        }]
    )
    ```
  </Tab>

  <Tab title="Google Gemini">
    Gemini models with native multimodal support.

    ```python theme={null}
    response = completion(
        model="gemini/gemini-1.5-pro",
        messages=[{
            "role": "user",
            "content": [
                {"type": "text", "text": "What's in this image?"},
                {"type": "image_url", "image_url": {"url": "https://..."}}
            ]
        }]
    )
    ```
  </Tab>

  <Tab title="Ollama">
    Local vision models like LLaVA.

    ```python theme={null}
    response = completion(
        model="ollama/llava",
        messages=[{
            "role": "user",
            "content": [
                {"type": "text", "text": "Describe this image"},
                {"type": "image_url", "image_url": {"url": "https://..."}}
            ]
        }],
        api_base="http://localhost:11434"
    )
    ```
  </Tab>
</Tabs>

## Image Detail Level

Control how the model processes images (OpenAI models).

```python theme={null}
from litellm import completion

response = completion(
    model="gpt-4o",
    messages=[{
        "role": "user",
        "content": [
            {"type": "text", "text": "Analyze this image in detail"},
            {
                "type": "image_url",
                "image_url": {
                    "url": "https://example.com/image.jpg",
                    "detail": "high"  # "low", "high", or "auto"
                }
            }
        ]
    }]
)
```

* **low**: Faster, lower cost, less detail (512x512)
* **high**: Slower, higher cost, more detail (2048x2048)
* **auto**: Model decides based on image size

## Common Use Cases

<Tabs>
  <Tab title="Image Description">
    ```python theme={null}
    from litellm import completion

    response = completion(
        model="gpt-4o",
        messages=[{
            "role": "user",
            "content": [
                {"type": "text", "text": "Provide a detailed description of this image"},
                {"type": "image_url", "image_url": {"url": "https://..."}}
            ]
        }]
    )
    ```
  </Tab>

  <Tab title="OCR / Text Extraction">
    ```python theme={null}
    response = completion(
        model="gpt-4o",
        messages=[{
            "role": "user",
            "content": [
                {"type": "text", "text": "Extract all text from this image"},
                {"type": "image_url", "image_url": {"url": "https://..."}}
            ]
        }]
    )
    ```
  </Tab>

  <Tab title="Object Detection">
    ```python theme={null}
    response = completion(
        model="gpt-4o",
        messages=[{
            "role": "user",
            "content": [
                {"type": "text", "text": "List all objects you can identify in this image"},
                {"type": "image_url", "image_url": {"url": "https://..."}}
            ]
        }]
    )
    ```
  </Tab>

  <Tab title="Image Comparison">
    ```python theme={null}
    response = completion(
        model="gpt-4o",
        messages=[{
            "role": "user",
            "content": [
                {"type": "text", "text": "What are the differences between these images?"},
                {"type": "image_url", "image_url": {"url": "https://.../before.jpg"}},
                {"type": "image_url", "image_url": {"url": "https://.../after.jpg"}}
            ]
        }]
    )
    ```
  </Tab>

  <Tab title="Chart Analysis">
    ```python theme={null}
    response = completion(
        model="gpt-4o",
        messages=[{
            "role": "user",
            "content": [
                {"type": "text", "text": "Analyze this chart and provide key insights"},
                {"type": "image_url", "image_url": {"url": "https://.../chart.png"}}
            ]
        }]
    )
    ```
  </Tab>
</Tabs>

## Streaming with Vision

```python theme={null}
from litellm import completion

response = completion(
    model="gpt-4o",
    messages=[{
        "role": "user",
        "content": [
            {"type": "text", "text": "Describe this image in detail"},
            {"type": "image_url", "image_url": {"url": "https://..."}}
        ]
    }],
    stream=True
)

for chunk in response:
    if chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="")
```

## Multi-turn Vision Conversations

```python theme={null}
from litellm import completion

messages = [
    {
        "role": "user",
        "content": [
            {"type": "text", "text": "What's in this image?"},
            {"type": "image_url", "image_url": {"url": "https://.../room.jpg"}}
        ]
    }
]

# First response
response = completion(model="gpt-4o", messages=messages)
messages.append(response.choices[0].message)

# Follow-up question
messages.append({
    "role": "user",
    "content": "What color is the furniture?"
})

response = completion(model="gpt-4o", messages=messages)
print(response.choices[0].message.content)
```

## JSON Mode with Vision

Get structured output from image analysis.

```python theme={null}
from litellm import completion
import json

response = completion(
    model="gpt-4o",
    messages=[{
        "role": "user",
        "content": [
            {
                "type": "text",
                "text": "Extract product information from this image as JSON with fields: name, price, description"
            },
            {"type": "image_url", "image_url": {"url": "https://.../product.jpg"}}
        ]
    }],
    response_format={"type": "json_object"}
)

data = json.loads(response.choices[0].message.content)
print(data)
```

## Error Handling

```python theme={null}
from litellm import completion
from litellm.exceptions import BadRequestError, APIError

try:
    response = completion(
        model="gpt-4o",
        messages=[{
            "role": "user",
            "content": [
                {"type": "text", "text": "Describe this"},
                {"type": "image_url", "image_url": {"url": "invalid-url"}}
            ]
        }]
    )
except BadRequestError as e:
    print(f"Invalid image URL or format: {e}")
except APIError as e:
    print(f"API error: {e}")
```

## Supported Image Formats

* **JPEG** - `.jpg`, `.jpeg`
* **PNG** - `.png`
* **WebP** - `.webp`
* **GIF** - `.gif` (non-animated)

<Note>
  Format support varies by provider. OpenAI and Anthropic support all common formats.
</Note>

## Image Size Limits

| Provider      | Max Size         | Max Resolution         |
| ------------- | ---------------- | ---------------------- |
| OpenAI        | 20MB             | Varies by detail level |
| Anthropic     | 5MB per image    | 8000x8000              |
| Google Gemini | Varies           | Varies by model        |
| Ollama        | Depends on model | Depends on model       |

## Best Practices

<AccordionGroup>
  <Accordion title="Image Quality">
    * Use high-quality images for better results
    * Ensure images are clear and well-lit
    * Crop images to focus on relevant content
    * Use appropriate resolution (not too low or unnecessarily high)
  </Accordion>

  <Accordion title="Cost Optimization">
    * Use `detail="low"` for simple image tasks
    * Resize large images before sending
    * Use URLs instead of base64 when possible
    * Consider using cheaper models for simple vision tasks
  </Accordion>

  <Accordion title="Prompt Design">
    * Be specific about what you want to extract
    * Ask direct questions about the image
    * Use examples when requesting specific formats
    * Break complex tasks into multiple queries
  </Accordion>

  <Accordion title="Performance">
    * Cache images when used multiple times
    * Use streaming for long descriptions
    * Process multiple images in parallel when independent
    * Consider batch processing for many images
  </Accordion>
</AccordionGroup>

## Limitations

<Warning>
  * Vision models may hallucinate details not present in images
  * Text recognition accuracy varies
  * Some models have restrictions on certain image types
  * Privacy: Be cautious with sensitive images
</Warning>