Model I/O with files

Baseten supports file-based input and output during inference, whether the file is local or remote and whether you handle it in the Truss or in your client code.

Files as input

Send a file with JSON-serializable content

The Truss CLI has a -f flag to pass file input. If you’re using the API endpoint from Python, get file contents with the standard f.read() function.

truss predict -f input.json

import os
import json
import requests

model_id = ""
# Read secrets from environment variables
baseten_api_key = os.environ["BASETEN_API_KEY"]

# Read input as JSON
with open("input.json", "r") as f:
    data = json.load(f)

resp = requests.post(
    # Endpoint for production deployment, see API reference for more
    f"https://model-{model_id}.api.baseten.co/production/predict",
    headers={"Authorization": f"Bearer {baseten_api_key}"},
    json=data,
)

print(resp.json())

Send a file with non-serializable content

The -f flag for truss predict only applies to JSON-serializable content. For other files, like the audio files required by MusicGen Melody, base64-encode the file content before you send it:

call_model.py

import os
import base64
import requests

model_id = ""
# Read secrets from environment variables
baseten_api_key = os.environ["BASETEN_API_KEY"]

# Open a local file and base64-encode it (mono WAV, 48kHz sample rate)
with open("mymelody.wav", "rb") as f:
    encoded_str = base64.b64encode(f.read()).decode("utf-8")

# Build a JSON-serializable payload
data = {"prompts": ["happy rock", "energetic EDM", "sad jazz"], "melody": encoded_str, "duration": 8}

resp = requests.post(
    # Endpoint for production deployment, see API reference for more
    f"https://model-{model_id}.api.baseten.co/production/predict",
    headers={"Authorization": f"Bearer {baseten_api_key}"},
    json=data,
)

# Decode the base64 audio clips in the response and save them
for idx, clip in enumerate(resp.json()["data"]):
    with open(f"clip_{idx}.wav", "wb") as f:
        f.write(base64.b64decode(clip))

Send a URL to a public file

Rather than encoding and serializing a file to send in the HTTP request, write a Truss that takes a URL as input and loads the content in the preprocess() function. Here’s an example from Whisper in the model library:

model/model.py

from tempfile import NamedTemporaryFile
import requests

# Get file content without blocking GPU
def preprocess(self, request):
    resp = requests.get(request["url"])
    return {"content": resp.content}

# Use file content in model inference
def predict(self, model_input):
    with NamedTemporaryFile() as fp:
        fp.write(model_input["content"])
        result = whisper.transcribe(
            self._model,
            fp.name,
            temperature=0,
            best_of=5,
            beam_size=5,
        )
        segments = [
            {"start": r["start"], "end": r["end"], "text": r["text"]}
            for r in result["segments"]
        ]
    return {
        "language": whisper.tokenizer.LANGUAGES[result["language"]],
        "segments": segments,
        "text": result["text"],
    }

Files as output

Save model output to a local file

Saving model output to a local file needs no Baseten-specific code. Use the standard > operator in bash or the file.write() function in Python:

truss predict -d '"Model input!"' > output.json

import os
import json
import requests

model_id = ""
# Read secrets from environment variables
baseten_api_key = os.environ["BASETEN_API_KEY"]

# Call model
resp = requests.post(
    # Endpoint for production deployment, see API reference for more
    f"https://model-{model_id}.api.baseten.co/production/predict",
    headers={"Authorization": f"Bearer {baseten_api_key}"},
    json="Model input!",
)
# Serialize the JSON response and write it to file
with open("output.json", "w") as f:
    f.write(json.dumps(resp.json()))

Output for some models, like image and audio generation models, may need to be decoded before you save it. See our image generation example for how to parse base64 output.

Overview

Get started

Model APIs

Inference

Development

Deployment

Engines

Frontier Gateway

Training

Organization

Observability

Troubleshooting

Model I/O with files

Files as input

Send a file with JSON-serializable content

Send a file with non-serializable content

Send a URL to a public file

Files as output

Save model output to a local file

​Files as input

​Send a file with JSON-serializable content

​Send a file with non-serializable content

​Send a URL to a public file

​Files as output

​Save model output to a local file

Files as input

Send a file with JSON-serializable content

Send a file with non-serializable content

Send a URL to a public file

Files as output

Save model output to a local file