Vision

Vision-language models in Loops train on examples that pair images with text. The trainer turns each image into visual tokens, places them before the text tokens, and computes loss over the combined sequence in the same forward_backward() and optim_step() round trip as text-only training. Checkpoints, sampling, and deployment work the same way; only the input construction changes. For a complete run that fine-tunes on geometry diagrams, deploys the checkpoint, and sends an image prompt, see Fine-tune a vision model with the Loops SDK.

Supported models

The Vision column of the model catalog marks the models that accept image inputs.

Build an image input

Each training example becomes a Datum:

model_input contains one ImageChunk with the image bytes and one text chunk with the problem and answer tokens.
loss_fn_inputs supplies the loss targets through a TensorData named target_tokens, with one target token per input position.

ImageChunk takes the image bytes inline, as JPEG or PNG. One chunk holds one image; add more chunks for more images. Pass the image already resized: the trainer doesn’t resize images and rejects a decoded image larger than 18 MB or 50 megapixels.

Loops doesn’t fetch external image URLs. Pass the bytes inline in ImageChunk. The ImageAssetPointerChunk type exists for Tinker compatibility, but its constructor raises ValueError.

ImageChunk.expected_tokens tells the trainer how many input tokens the image becomes. The model replaces the image with exactly that many image tokens, so the number must match what the model actually produces. For Qwen VL and Muse Spark models, compute it from the image processor as prod(image_grid_thw) // merge_size**2. The following functions build a Datum from an image, a prompt, and an answer. image_processor is the model’s Hugging Face image processor, loaded with AutoImageProcessor.from_pretrained(base_model, trust_remote_code=True), and tokenizer comes from training_client.get_tokenizer():

import io

from PIL import Image

from baseten.loops import Datum, ImageChunk, ModelInput, TensorData


def compute_expected_tokens(image: Image.Image) -> int:
    result = image_processor(images=[image])
    grid = result["image_grid_thw"][0]
    merge = image_processor.merge_size
    return int(grid[0] * grid[1] * grid[2]) // (merge * merge)


def to_image_chunk(image: Image.Image, expected_tokens: int) -> ImageChunk:
    buf = io.BytesIO()
    image.convert("RGB").save(buf, format="JPEG")
    return ImageChunk(
        data=buf.getvalue(),
        format="jpeg",
        expected_tokens=expected_tokens,
    )


def to_datum(image: Image.Image, prompt: str, answer: str, tokenizer) -> Datum:
    expected_tokens = compute_expected_tokens(image)
    img_chunk = to_image_chunk(image, expected_tokens)

    p = tokenizer.encode(prompt, add_special_tokens=False)
    a = tokenizer.encode(answer, add_special_tokens=False)

    # The trainer doesn't shift labels: drop the last input token and make
    # each target the token that follows its position.
    text_chunk = ModelInput.from_ints(p + a[:-1]).chunks[0]
    targets = [-100] * (expected_tokens + len(p) - 1) + list(a)

    return Datum(
        model_input=ModelInput(chunks=[img_chunk, text_chunk]),
        loss_fn_inputs={
            "target_tokens": TensorData(
                data=targets, dtype="int64", shape=[len(targets)]
            )
        },
    )

The trainer expands each ImageChunk into expected_tokens copies of the model’s image-placeholder token ID and appends the text chunk. Because to_datum omits the final answer token from the input, the input sequence is [image: expected_tokens] [prompt: len(p)] [answer: len(a) - 1].

Set the loss targets

The input and target_tokens have the same length, expected_tokens + len(p) + len(a) - 1. Mask the image and prompt positions with -100, which cross-entropy ignores, except for the last prompt position. That position targets the first answer token. The remaining positions target the subsequent answer tokens, including the final answer token.

Send images to a deployed checkpoint

Deploy a checkpoint from a vision run like any other. The deployment serves an OpenAI-compatible /v1/chat/completions endpoint that accepts images as base64-encoded data in the image_url format:

{
  "role": "user",
  "content": [
    {"type": "image_url", "image_url": {"url": "data:image/jpeg;base64,<base64_image>"}},
    {"type": "text", "text": "Find x."}
  ]
}

Text-only prompts work on the same deployment.

Next steps

Fine-tune a vision model with the Loops SDK: Train, deploy, and query a vision checkpoint end to end.
Migrate from Tinker: How image inputs differ from Tinker.
Loss functions: The loss inputs a Datum can carry.

Get started

Model APIs

Baseten Grid

Inference

Frontier Gateway

Training

Sandboxes

Development

Deployment

Engines

Model Labs

Organization

Hosting options

Observability

Troubleshooting

Supported models

Build an image input

Set the loss targets

Send images to a deployed checkpoint

Next steps

​Supported models

​Build an image input

​Set the loss targets

​Send images to a deployed checkpoint

​Next steps

Supported models

Build an image input

Set the loss targets

Send images to a deployed checkpoint

Next steps