Deploy LLMs with Ollama

Ollama is a popular lightweight LLM inference server, similar to vLLM or SGLang. This guide deploys an Ollama model as a custom Docker server on Baseten. This configuration serves TinyLlama with Ollama on a CPU instance. The deployment process is the same for larger Ollama models. Adjust the resources and the model in build_commands to match your model’s requirements.

Set up your environment

This guide uses the Baseten CLI. Sign in and install requests to call the deployed model from Python. Browser login opens a tab to approve this device, so there’s no API key to copy and paste.

Sign in to Baseten

baseten auth login

Install requests

uv pip install requests

Configure the model

Create a directory with a config.yaml file:

mkdir tinyllama-ollama
touch tinyllama-ollama/config.yaml

Copy the following configuration into config.yaml:

config.yaml

model_name: ollama-tinyllama
base_image:
  image: python:3.11-slim
build_commands:
  - apt-get update && apt-get install -y curl ca-certificates zstd
  - curl -fsSL https://ollama.com/install.sh | sh
  # Bake the model into the image with a thread count that matches the instance.
  - >-
    sh -c "mkdir -p /home/app/.ollama/models && export OLLAMA_MODELS=/home/app/.ollama/models
    && ollama serve >/tmp/ollama-serve.log 2>&1 & sleep 8
    && ollama pull tinyllama
    && printf 'FROM tinyllama\nPARAMETER num_thread 4\n' > /tmp/Modelfile
    && ollama create tinyllama:4t -f /tmp/Modelfile"
docker_server:
  start_command: sh -c "ollama serve"
  readiness_endpoint: /api/tags
  liveness_endpoint: /api/tags
  predict_endpoint: /api/generate
  server_port: 11434
resources:
  cpu: "4"
  memory: 8Gi

The base_image is a lightweight Python image. The build_commands install the system packages that the Ollama install script requires (curl, ca-certificates, and zstd), install Ollama, and then pull TinyLlama while the image builds. The slim base image doesn’t include these packages by default. The build step also creates tinyllama:4t, a copy of the model with num_thread pinned to 4. Ollama sizes its thread pool from the host CPU count, not the instance’s vCPU count, so the unmodified model spreads across 192 threads on the host and a short reply takes minutes instead of seconds. Baking the thread count into the image means every request and every replica runs with it, and no request has to pass it. Raise num_thread if you move the deployment to a larger instance. The start_command runs the Ollama server. The readiness_endpoint and liveness_endpoint both point to /api/tags, which returns successfully when Ollama is running. The predict_endpoint maps Baseten’s /predict route to Ollama’s /api/generate endpoint. This example only needs 4 CPUs and 8 GB of memory. For a complete list of resource options, see the Resources page.

Deploy the model

Push the model to Baseten to start the deployment:

baseten model push --dir tinyllama-ollama

You should see output like:

Pushing model "ollama-tinyllama"...
Uploading model...
Uploaded model in 0s
✨ Model ollama-tinyllama was successfully pushed ✨

  Model:       ollama-tinyllama (abc1d2ef)
  Deployment:  xyz123
  Environment: production

🪵 View logs:
   deployment:   baseten model deployment logs --model-id abc1d2ef --deployment-id xyz123
   environment:  baseten model environment logs --model-id abc1d2ef --environment production  (once deployed)
   app:          https://app.baseten.co/models/abc1d2ef/logs/xyz123

🚀 Invoke your model:
   URL:  https://model-abc1d2ef.api.baseten.co/deployment/xyz123/predict
   CLI:  baseten model predict --model-id abc1d2ef

Copy the model ID from the output for the next step. The first deploy takes several minutes while Baseten builds the image, and Ollama downloads TinyLlama into it. Later pushes of the same configuration reuse the cached image, so scaling up starts a server that already has the model on disk.

Call the model

Ollama’s /api/generate is mapped to Baseten’s /predict route, so you can call the deployed model with any HTTP client:

Baseten CLI
cURL
Python

To run inference with the Baseten CLI, use the predict command:

baseten model predict --model-id MODEL_ID --data '{"model": "tinyllama:4t", "prompt": "Write a short story about a robot dreaming", "stream": false, "options": {"num_predict": 50}}'

To run inference with cURL, use the following command:

curl -s -X POST "https://model-MODEL_ID.api.baseten.co/production/predict" \
  -H "Authorization: Bearer $BASETEN_API_KEY" \
  -d '{"model": "tinyllama:4t", "prompt": "Write a short story about a robot dreaming", "stream": false, "options": {"num_predict": 50}}' \
  | jq -j '.response'

To run inference with Python, use the following:

call_model.py

import os
import requests

model_id = "MODEL_ID"
baseten_api_key = os.environ["BASETEN_API_KEY"]

response = requests.post(
    f"https://model-{model_id}.api.baseten.co/production/predict",
    headers={"Authorization": f"Bearer {baseten_api_key}"},
    json={
        "model": "tinyllama:4t",
        "prompt": "Write a short story about a robot dreaming",
        "stream": False,
        "options": {"num_predict": 50},
    },
)
print(response.json()["response"])

Replace MODEL_ID with the model ID from your deployment output. You should see:

It was a dreary, grey day when the robots started to dream.
They had been programmed to think like humans, but it wasn't until they began to dream that they realized just how far apart they actually were.

Next steps

For higher-throughput serving on GPUs with OpenAI-compatible endpoints, see the vLLM and SGLang examples.

Deploy LLMs with vLLM

Serve open-source LLMs on vLLM with prefix caching and the OpenAI-compatible API.

Deploy LLMs with SGLang

Serve open-source LLMs on SGLang’s high-performance runtime with the OpenAI-compatible API.

Examples

Models

Engines

Custom Docker servers

Custom Python models

Chains

API workflows

Baseten for Model Labs

Deploy LLMs with Ollama

Set up your environment

Configure the model

Deploy the model

Call the model

Next steps

Deploy LLMs with vLLM

Deploy LLMs with SGLang

​Set up your environment

​Configure the model

​Deploy the model

​Call the model

​Next steps

Deploy LLMs with vLLM

Deploy LLMs with SGLang

Set up your environment

Configure the model

Deploy the model

Call the model

Next steps