> ## Documentation Index
> Fetch the complete documentation index at: https://docs.baseten.co/llms.txt
> Use this file to discover all available pages before exploring further.

# Deploy LLMs with Ollama

> Run LLMs on Ollama as a custom Docker server.

[Ollama](https://ollama.com/) is a popular lightweight LLM inference server, similar to vLLM or SGLang. This guide deploys an Ollama model as a custom Docker server on Baseten.

This configuration serves [TinyLlama](https://ollama.com/library/tinyllama) with Ollama on a CPU instance. The deployment process is the same for larger Ollama models. Adjust the `resources` and the model in `build_commands` to match your model's requirements.

## Set up your environment

This guide uses the [Baseten CLI](/reference/cli/baseten/overview#install). Sign in and install `requests` to call the deployed model from Python. Browser login opens a tab to approve this device, so there's no API key to copy and paste.

<Columns cols={2}>
  <Column>
    **Sign in to Baseten**

    ```sh theme={"system"}
    baseten auth login
    ```
  </Column>

  <Column>
    **Install requests**

    ```sh theme={"system"}
    uv pip install requests
    ```
  </Column>
</Columns>

## Configure the model

Create a directory with a `config.yaml` file:

```sh theme={"system"}
mkdir tinyllama-ollama
touch tinyllama-ollama/config.yaml
```

Copy the following configuration into `config.yaml`:

```yaml config.yaml theme={"system"}
model_name: ollama-tinyllama
base_image:
  image: python:3.11-slim
build_commands:
  - apt-get update && apt-get install -y curl ca-certificates zstd
  - curl -fsSL https://ollama.com/install.sh | sh
  # Bake the model into the image with a thread count that matches the instance.
  - >-
    sh -c "mkdir -p /home/app/.ollama/models && export OLLAMA_MODELS=/home/app/.ollama/models
    && ollama serve >/tmp/ollama-serve.log 2>&1 & sleep 8
    && ollama pull tinyllama
    && printf 'FROM tinyllama\nPARAMETER num_thread 4\n' > /tmp/Modelfile
    && ollama create tinyllama:4t -f /tmp/Modelfile"
docker_server:
  start_command: sh -c "ollama serve"
  readiness_endpoint: /api/tags
  liveness_endpoint: /api/tags
  predict_endpoint: /api/generate
  server_port: 11434
resources:
  cpu: "4"
  memory: 8Gi
```

The `base_image` is a lightweight Python image. The `build_commands` install the system packages that the Ollama install script requires (`curl`, `ca-certificates`, and `zstd`), install Ollama, and then pull TinyLlama while the image builds. The slim base image doesn't include these packages by default.

The build step also creates `tinyllama:4t`, a copy of the model with `num_thread` pinned to 4. Ollama sizes its thread pool from the host CPU count, not the instance's vCPU count, so the unmodified model spreads across 192 threads on the host and a short reply takes minutes instead of seconds. Baking the thread count into the image means every request and every replica runs with it, and no request has to pass it. Raise `num_thread` if you move the deployment to a larger instance.

The `start_command` runs the Ollama server. The `readiness_endpoint` and `liveness_endpoint` both point to `/api/tags`, which returns successfully when Ollama is running. The `predict_endpoint` maps Baseten's `/predict` route to Ollama's `/api/generate` endpoint.

This example only needs 4 CPUs and 8 GB of memory. For a complete list of resource options, see the [Resources](/deployment/resources) page.

## Deploy the model

Push the model to Baseten to start the deployment:

```sh theme={"system"}
baseten model push --dir tinyllama-ollama
```

You should see output like:

```output theme={"system"}
Pushing model "ollama-tinyllama"...
Uploading model...
Uploaded model in 0s
✨ Model ollama-tinyllama was successfully pushed ✨

  Model:       ollama-tinyllama (abc1d2ef)
  Deployment:  xyz123
  Environment: production

🪵 View logs:
   deployment:   baseten model deployment logs --model-id abc1d2ef --deployment-id xyz123
   environment:  baseten model environment logs --model-id abc1d2ef --environment production  (once deployed)
   app:          https://app.baseten.co/models/abc1d2ef/logs/xyz123

🚀 Invoke your model:
   URL:  https://model-abc1d2ef.api.baseten.co/deployment/xyz123/predict
   CLI:  baseten model predict --model-id abc1d2ef
```

Copy the model ID from the output for the next step.

The first deploy takes several minutes while Baseten builds the image, and Ollama downloads TinyLlama into it. Later pushes of the same configuration reuse the cached image, so scaling up starts a server that already has the model on disk.

## Call the model

Ollama's `/api/generate` is mapped to Baseten's `/predict` route, so you can call the deployed model with any HTTP client:

<Tabs>
  <Tab title="Baseten CLI">
    To run inference with the Baseten CLI, use the `predict` command:

    ```sh theme={"system"}
    baseten model predict --model-id MODEL_ID --data '{"model": "tinyllama:4t", "prompt": "Write a short story about a robot dreaming", "stream": false, "options": {"num_predict": 50}}'
    ```
  </Tab>

  <Tab title="cURL">
    To run inference with cURL, use the following command:

    ```sh theme={"system"}
    curl -s -X POST "https://model-MODEL_ID.api.baseten.co/production/predict" \
      -H "Authorization: Bearer $BASETEN_API_KEY" \
      -d '{"model": "tinyllama:4t", "prompt": "Write a short story about a robot dreaming", "stream": false, "options": {"num_predict": 50}}' \
      | jq -j '.response'
    ```
  </Tab>

  <Tab title="Python">
    To run inference with Python, use the following:

    ```python call_model.py theme={"system"}
    import os
    import requests

    model_id = "MODEL_ID"
    baseten_api_key = os.environ["BASETEN_API_KEY"]

    response = requests.post(
        f"https://model-{model_id}.api.baseten.co/production/predict",
        headers={"Authorization": f"Bearer {baseten_api_key}"},
        json={
            "model": "tinyllama:4t",
            "prompt": "Write a short story about a robot dreaming",
            "stream": False,
            "options": {"num_predict": 50},
        },
    )
    print(response.json()["response"])
    ```
  </Tab>
</Tabs>

Replace `MODEL_ID` with the model ID from your deployment output.

You should see:

```output theme={"system"}
It was a dreary, grey day when the robots started to dream.
They had been programmed to think like humans, but it wasn't until they began to dream that they realized just how far apart they actually were.
```

***

## Next steps

For higher-throughput serving on GPUs with OpenAI-compatible endpoints, see the vLLM and SGLang examples.

<CardGroup cols={2}>
  <Card title="Deploy LLMs with vLLM" icon="bolt" href="/examples/vllm">
    Serve open-source LLMs on vLLM with prefix caching and the OpenAI-compatible API.
  </Card>

  <Card title="Deploy LLMs with SGLang" icon="gauge-high" href="/examples/sglang">
    Serve open-source LLMs on SGLang's high-performance runtime with the OpenAI-compatible API.
  </Card>
</CardGroup>