> ## Documentation Index
> Fetch the complete documentation index at: https://mintlify.com/BerriAI/litellm/llms.txt
> Use this file to discover all available pages before exploring further.

# HuggingFace

> Use HuggingFace models through LiteLLM's unified interface

## Overview

LiteLLM provides support for HuggingFace models through multiple deployment options: HuggingFace Inference API, dedicated endpoints, and provider-specific routing.

## Quick Start

<Steps>
  <Step title="Install LiteLLM">
    ```bash theme={null}
    pip install litellm
    ```
  </Step>

  <Step title="Set API Key">
    ```bash theme={null}
    export HUGGINGFACE_API_KEY="hf_..."
    ```
  </Step>

  <Step title="Make Your First Call">
    ```python theme={null}
    from litellm import completion

    response = completion(
        model="huggingface/meta-llama/Llama-3.3-70B-Instruct",
        messages=[{"role": "user", "content": "Hello!"}]
    )
    print(response.choices[0].message.content)
    ```
  </Step>
</Steps>

## Deployment Options

<Tabs>
  <Tab title="Inference API">
    Use HuggingFace's serverless Inference API.

    ```python theme={null}
    from litellm import completion

    response = completion(
        model="huggingface/meta-llama/Llama-3.3-70B-Instruct",
        messages=[{"role": "user", "content": "Explain AI"}]
    )
    ```
  </Tab>

  <Tab title="Dedicated Endpoint">
    Use your own HuggingFace endpoint URL.

    ```python theme={null}
    from litellm import completion

    response = completion(
        model="huggingface/https://your-endpoint.aws.endpoints.huggingface.cloud",
        messages=[{"role": "user", "content": "Hello!"}],
        api_key="hf_..."
    )
    ```
  </Tab>

  <Tab title="Provider Routing">
    Route through specific providers via HuggingFace Router.

    ```python theme={null}
    from litellm import completion

    # Route through Fireworks AI
    response = completion(
        model="huggingface/fireworks-ai/meta-llama/Llama-3.3-70B-Instruct",
        messages=[{"role": "user", "content": "Hello!"}]
    )

    # Route through Novita
    response = completion(
        model="huggingface/novita/meta-llama/Llama-3.3-70B-Instruct",
        messages=[{"role": "user", "content": "Hello!"}]
    )
    ```
  </Tab>
</Tabs>

## Authentication

<Tabs>
  <Tab title="Environment Variable">
    ```bash theme={null}
    export HUGGINGFACE_API_KEY="hf_..."
    # Or
    export HF_API_BASE="https://your-endpoint.huggingface.cloud"
    ```

    ```python theme={null}
    from litellm import completion

    response = completion(
        model="huggingface/meta-llama/Llama-3.3-70B-Instruct",
        messages=[{"role": "user", "content": "Hello!"}]
    )
    ```
  </Tab>

  <Tab title="Direct Parameter">
    ```python theme={null}
    from litellm import completion

    response = completion(
        model="huggingface/meta-llama/Llama-3.3-70B-Instruct",
        messages=[{"role": "user", "content": "Hello!"}],
        api_key="hf_...",
        api_base="https://api-inference.huggingface.co/models"
    )
    ```
  </Tab>
</Tabs>

## Chat Completions

```python theme={null}
from litellm import completion

response = completion(
    model="huggingface/meta-llama/Llama-3.3-70B-Instruct",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Explain quantum computing"}
    ],
    temperature=0.7,
    max_tokens=500
)

print(response.choices[0].message.content)
```

## Streaming

```python theme={null}
from litellm import completion

response = completion(
    model="huggingface/meta-llama/Llama-3.3-70B-Instruct",
    messages=[{"role": "user", "content": "Write a story"}],
    stream=True
)

for chunk in response:
    if chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="")
```

## Embeddings

HuggingFace supports various embedding models.

<Tabs>
  <Tab title="Sentence Transformers">
    ```python theme={null}
    from litellm import embedding

    response = embedding(
        model="huggingface/sentence-transformers/all-MiniLM-L6-v2",
        input=["Text to embed", "Another text"]
    )

    embeddings = [data.embedding for data in response.data]
    ```
  </Tab>

  <Tab title="BGE Models">
    ```python theme={null}
    from litellm import embedding

    response = embedding(
        model="huggingface/BAAI/bge-large-en-v1.5",
        input=["Query text"]
    )
    ```
  </Tab>

  <Tab title="Custom Endpoint">
    ```python theme={null}
    from litellm import embedding

    response = embedding(
        model="huggingface/https://your-embedding-endpoint.cloud",
        input=["Text to embed"],
        api_key="hf_..."
    )
    ```
  </Tab>
</Tabs>

## Reranking

Use HuggingFace reranking models for improved search.

```python theme={null}
from litellm import rerank

response = rerank(
    model="huggingface/BAAI/bge-reranker-v2-m3",
    query="What is machine learning?",
    documents=[
        "Machine learning is a subset of AI.",
        "Deep learning uses neural networks.",
        "Python is a programming language."
    ],
    top_n=2
)

for result in response.results:
    print(f"Score: {result.relevance_score}")
    print(f"Document: {result.document}")
```

## Provider-Specific Routing

Route requests through different inference providers.

```python theme={null}
from litellm import completion

# Fireworks AI provider
response = completion(
    model="huggingface/fireworks-ai/meta-llama/Llama-3.3-70B-Instruct",
    messages=[{"role": "user", "content": "Hello!"}]
)

# Novita provider
response = completion(
    model="huggingface/novita/meta-llama/Llama-3.3-70B-Instruct",
    messages=[{"role": "user", "content": "Hello!"}]
)

# HF Inference provider
response = completion(
    model="huggingface/hf-inference/meta-llama/Llama-3.3-70B-Instruct",
    messages=[{"role": "user", "content": "Hello!"}]
)
```

<Note>
  Provider availability varies by model. LiteLLM validates provider support automatically.
</Note>

## Configuration

```python theme={null}
from litellm import completion

response = completion(
    model="huggingface/meta-llama/Llama-3.3-70B-Instruct",
    messages=[{"role": "user", "content": "Hello!"}],
    temperature=0.8,
    max_tokens=1000,
    top_p=0.95,
    stop=["\n\n"]
)
```

## Supported Parameters

| Parameter           | Type  | Description         |
| ------------------- | ----- | ------------------- |
| `temperature`       | float | Randomness (0-1)    |
| `max_tokens`        | int   | Max output tokens   |
| `top_p`             | float | Nucleus sampling    |
| `frequency_penalty` | float | Reduce repetition   |
| `presence_penalty`  | float | Encourage diversity |
| `stop`              | list  | Stop sequences      |
| `stream`            | bool  | Enable streaming    |

<Note>
  Not all parameters are supported by all HuggingFace models. Check model documentation.
</Note>

## Error Handling

```python theme={null}
from litellm import completion
from litellm.exceptions import APIError, RateLimitError

try:
    response = completion(
        model="huggingface/meta-llama/Llama-3.3-70B-Instruct",
        messages=[{"role": "user", "content": "Hello!"}]
    )
except RateLimitError as e:
    print(f"Rate limit: {e}")
except APIError as e:
    print(f"API error: {e.status_code} - {e.message}")
```

## LiteLLM Proxy

```yaml theme={null}
model_list:
  - model_name: llama-3.3-70b
    litellm_params:
      model: huggingface/meta-llama/Llama-3.3-70B-Instruct
      api_key: os.environ/HUGGINGFACE_API_KEY
  
  - model_name: custom-endpoint
    litellm_params:
      model: huggingface/https://your-endpoint.cloud
      api_key: os.environ/HF_TOKEN
```

```python theme={null}
import openai

client = openai.OpenAI(
    api_key="sk-1234",
    base_url="http://0.0.0.0:4000"
)

response = client.chat.completions.create(
    model="llama-3.3-70b",
    messages=[{"role": "user", "content": "Hello!"}]
)
```

## Best Practices

<AccordionGroup>
  <Accordion title="Model Selection">
    * Use Inference API for testing and prototyping
    * Use dedicated endpoints for production workloads
    * Check model availability on HuggingFace Hub
  </Accordion>

  <Accordion title="Performance">
    * Dedicated endpoints provide better latency
    * Provider routing offers alternative inference options
    * Monitor staging vs production provider status
  </Accordion>

  <Accordion title="Cost Optimization">
    * Inference API is free tier available
    * Dedicated endpoints are billed separately
    * Compare provider pricing when routing
  </Accordion>
</AccordionGroup>

## Common Models

| Model                                    | Use Case           |
| ---------------------------------------- | ------------------ |
| `meta-llama/Llama-3.3-70B-Instruct`      | General chat       |
| `mistralai/Mixtral-8x7B-Instruct-v0.1`   | Advanced reasoning |
| `sentence-transformers/all-MiniLM-L6-v2` | Embeddings         |
| `BAAI/bge-large-en-v1.5`                 | Search embeddings  |
| `BAAI/bge-reranker-v2-m3`                | Reranking          |